Standard Operating Procedures: Preparing and running an experiment with the Rule Game Server

Updated 2022-10-24 for Game Server ver. 4.003

Preparing the experiment control files

The control files

As discussed in the Rule Game server data guide, a number of files need to be prepared in the server's game data directory in order to control the players' experience during your experiment. (The game data directory on sapir at UWM is /opt/tomcat/game-data. An administrator setting up an instance of Game Server at a host in another institution can choose a different location, and indicate it in the GAME_DATA variable in the master config file).

To summarize, this set of files needed to run an experiment should include, at a minimum:

The experiment plan directory (a subdirectory under /opt/tomcat/game-data/trial-lists), containing one or several trial list files. The server will pseudo-randomly assign players, in a balanced manner, to different trial lists, so each of the trial list files will be the top-level file defining the experiences of a particulalry (randomly selected) subgroup of the participants of your experiment.
For each rule set mentioned in the trial list files, a rule set file must be created in the rules directory, /opt/tomcat/game-data/rules. If you are creating a group of rule set files to be used in just one experiment, or in some series of experiments, it would make sense to place them into a new, specially created subdirectory under the rules directory. This will make it easier to manage rule set files.

Since GS 4.*, an alternative way to create experiment plans, known as dynamic experiment plans, has been made available. If you want to create an R:-type dynamic plan, you don't need an experiment plan directory, but you need a modifier file in the modifier directory.

Depending on the design of your experiment, you may also need to create or modify other files:

If your trial lists refer to pre-created initial boards, you need to create the initial board directory, which will contains the initial border files, and, possibly, an ordering control file.
If your experiment intends to use custom shapes (as reflected in the trial list files or rule sets), you need to make sure you have the SVG files for all shapes at the appropriate locations. (See: Using custom shapes and colors in Rule Game Server 2.*)
If your experiment intends to use custom colors (as reflected in the trial list files or rule sets), make sure you have those colors defined in the color map file
If you want your experiment plan, or a dynamic plan based on your rule set, to appear in the MLC Launch Page (for the staff) or the APP Launch Page (for the students), you need either to follow certain rules for placing the trial list files or the rule set files into certain special directories, or to edit the "launch files". For details, see Setting up launch pages.

Using a GitHub repository

One should not try to edit the experiment control files directly in the server's game data directory. Instead, one should follow the procedures outlined in the Deployment SOP (in that document, you only need to look at the sections written on yellow background). In a nutshell:

The "master copy" of the experiment control files (the entire content of the /opt/tomcat/game-data directory tree) is kept in a GitHub repository;
You can "check out" these files to your working directory, either on your home PC or on sapir;
You can edit the files in your working directory as desired, adding new subdirectories and files as needed;
Once done, you can check in your work into the repository; that will result in the updated files being automatically deployed to the /opt/tomcat/game-data directory tree on sapir.

This procedure ensures that at any point in the future we will have a reliable to see, by examining the repository's history, what set of control files was deployed to the server during any particular experiment.

Checking your math

One of the most complicated parts of the syntax of rule set files is the field which specifies the bucket(s) into which a rule allows game pieces to be put. (See the Bucket expression arithmetic Guide for details). To help you test your arithmetic, we have the Arithmetic test form, which you can use to test your expression, for various values of the built-in variables, before actually using it in a rule.

Validating your plan

Once you have prepared your trial list files, rule set files, and any other data you need, you should validate your experiment plan. Go to the Validate your plan link, and enter the name of your experiment plan. The validation tool will read the trial list files for your experiment, and various other files (rule sets etc) they refer to, and will check them for syntax errors, references to non-existing resources, etc. Please review the validator's report; if any deficiencies are reported, try to correct them, clear the server tables, and validate your plan again.

Testing your plan

One can do this with the GUI client or with the HTML play interface.

If you realize that your rules need to be corrected, and edit your trial list files or rule set files, you should clear the server tables before playing again (with another player ID). This will ensure that the server will remove the earlier version of the rule sets from its memory.

Keeping track of what was done

For any later analysis to be valid, it is important that complete information were kept about the experimental environment, which includes both the control files and the software version.

Once you have developed and extensively tested your experiment plan, you open your experiment to real players (e.g. those coming from Amazon's Mechanical Turk). After this has been done, it would be a very good idea to never modify the experiment's trial list files and the files it refers to (rule set files etc). This will ensure that all participants have a comparable experience, and hat the collected data could be properly analyzed in the future. If you want to modify your rules, or the order in which they are presented to players, it is much better to create a new experiment plan.
Make a record of what set of control files was deployed. If you make sure to always have the current set of files in the GitHub repository, it is, in principle, sufficient to know when the experiment was run, and you would be able to find the appropriately timed snapshot of the control files in the repository. However, you may also want to make a "snapshot" of the server's game data directory before the experiment starts, e.g. by making a zip file:
```
	    cd /opt/tomcat
	    zip -r ~/game-data-`date -Idate`.zip game-data
	  
```
This will produce a file with a name such as game-data-2021-03-15.zip in your home directory.
Make a record of what version of the GUI client is used, and what version of the Game Server it uses. The GUI client has a special URL to check what it's version is; for the production version, it is http://sapir.psych.wisc.edu/rule-game/prod/?version=true, and for the development version, it is http://sapir.psych.wisc.edu/rule-game/dev/?version=true. The version reported generated by that URL may look like this:
```
Client Version [environment-commitHash]: dev-962254ba6e35966540d0f06a2470ec89beaa0474
Server URL: http://sapir.psych.wisc.edu:7150/w2020-dev
Server Version: 2.006  
	  
```
Make sure to have thus information recorded in your experiment log. (Note that the server version is not reported prior to 2.006).

Configuring the instructions pages

We refer to the sequence of screens shown by the GUI client to a new player as the "instruction booklet". These screens contain information about the rules of game, to the extent we want the player to know them in advance.

The sequence of screens presented to players in GS 1.* thru 3.* (and still available in GS 4.*) is referred to as the "instruction booklet No. 0". As of December 2021, it is expected that Kevin will add to the GUI client the functionality to display different instruction booklets to different players; the choice of the booklet will be controlled by the trial set files.

If the experiment designer decided to create a new instruction booklet to be presented to some players, here are the steps:

Check with Kevin how many booklets exist now. If, for examples, booklets Nos. 0 thru 3 are currently in existence, you may choose to designate your new booklet as No. 4.
In the experiment plan you are designing, indicate that this instruction booklet is to be presented to players assigned to some trial lists. To do that, put the appropriate integer value (in this example, 4) in the column named init of the first data row (= the first parameter set) of these trial lists.
Check in the trial list files to the GitHub repository, in order for them to be auto-deployed to the server (so that Kevin will be able to test your booklet).
Create the pages of your booklet, as JPEG or PNG files, and send them to Kevin, asking him to add them to the GUI application, to be presented as the instruction booklet with the desired number (e.g. 4, in this example).
Kevin will add these image files to the appropriate directory of the client, rebuild the client, deploy it to the server, and inform you of its availability.

Data processing

See the section on exporting data from the Game Srever's SQL tables, and the section on files saved by the Game Server in the Data Guide for guidance on what data are recorded during the experiment, and how they can be extracted from the server.

Appendix: Tips and tricks for rule set development

Editing existing rules is always somewhat problematic. The issue is, even if you have cleared the server tables, the particular player in whose "persona" you are playing already has its own compiled version of the rule set, and it is going to be used for the duration of that player's current series. This means that clearing the tables will affect the players created in the future, but not the players who are currently playing a series of episodes.

This is likely to inconvenience experiment designers who keep modifying the same rule set, testing it again and again. However, there are a number of techniques to go around it. They are described below.

Approach 1

Suppose you like doing your testing from a page such as http://sapir.psych.wisc.edu:7150/w2020-dev/launch/launch-rules-cgs.jsp . In this case, you can do the following:

Every time you edit your rule set file, you put the new version in a separate file with a new name (e.g. test-01.txt, test-02.txt, test-03.txt, etc), and manually copy it to the directory the server takes it from ( /opt/tomcat/game-data/rules/CGS ), instead of using the normal deployment process (checking in to GitHub).
Once a new file (e.g. test-03.txt) has been copied to the server directory, you reload the lunch page, and the new rule set now appears in the table as a new row. Click on the "PLAY" button in that row, and a new player will be created (under your usual username), which is guaranteed to use the brand-new rule.
Once you have finished your development of the new rule, delete all these temp files (test-*.txt) from your own directory and frome the /opt/tomcat/game-data tree, while renaming the final version so that it has its permanent name (e.g. my-good-rule.txt). Then you can do the proper deployment, by adding the new rule to the GitHub repository ("git add my-good-rule.txt", if it has not been done before) and committing it to the repository ("git commit -a; git push origin master"), which will deploy it under that permanent name.

Approach 2

If you like doing testing through the developers' launch page for the GUI client (or its version with the dev version of the GUI client) , or the HTML Play (or the corresponding URLs on whatever machine you're deploying to), then it's even simpler:

You have modified the rule test set file, and have deployed it to the /opt/tomcat/game-data tree, either through the official SOP process (Github checkin), or simply by copying it there manually for the time being.
You clear the server tables.
You go to one of the launch pages mentioned above. In the player name field you enter a unique name that has never been used used yet (e.g. john-doe-2022-10-24-a). In the experiment plan name you can enter a so-called "dynamic plan name of the R: type", which is composed of the prefix "R:", the name of the rule set, and the modifier name. For example, R:CGS/color-match:APP/APP-max-feedback will test the rule set that lives in rules/CGS/color-match.txt, with one of the modifiers from modifiers/APP.

[Main page] [Rule Game Server data guide]