Image-and-property-based description of objects

This is a more detailed development of an earlier proposal, Wildcard syntax for a wider object space (2021-04-01). It describes the proposed approach to defining and handling objects (game pieces) in Game Engine 3.*.

Notation

For conciseness, in this document I will use the term "object" to refer to a class of identically looking entities, e.g. "black square(s)". The term "game piece" will refer to an individual object at a particular location on the board.

Defining object types

In addition, a new way of defining object types will be introduced. In this approach, each object will be defined by an image file (SVG, PNG, JPEG...) which already contains the desired coloring of the objects, and a list of properties, which represent some concepts with which humans can reason about objects. We will refer to this new type of objects as image-and-properties-based objects, or IPB objects.

Since multiple experiments with different sets of objects can be carried out on the game server, it will be expected that the IPB object descriptions will be located in multiple directories under the main shape directory, /opt/tomcat/game-data/shapes. For example, if an experiment designer has named a particular experiment exp-20210501-a and decided to use two groups of objects, heraldic animals and arrows, as game pieces in that experiment, he may want to put them into new directories, /opt/tomcat/game-data/shapes/exp-20210501-a/animals and /opt/tomcat/game-data/shapes/exp-20210501-a/arrows. (Of course, the experimenters may decide to use the same group of images in multiple experiments, so maybe one of the directories will be simply named /opt/tomcat/game-data/shapes/arrows, or whatever.

Each directory containing image files (unless it only contains colorless SVG images for use in GS 2.*-style experiments) will need to contain a properties file, named properties.csv, which will contain the descriptions of all objects defined in that directory. This CSV file, therefore, will need to contain 1 line of text per each image file in the directory, in addition to the header line on top. For example, it may look something like this:

Property names should be written using lower case Latin letters; they may also contain the underscore character (_) and digits, but not in the first position.

If, for a particular object, the cell corresponding to property X is empty (contains an empty string), this means that property X is not defined for that object (e.g., all our tigers are striped, and we don't associate a color property with them). This means that any rule atom that only select objects with property X having a particular value won't select this object.

It is also possible for an object to contain an * in the column for property X. This object will match any X-based selector.

Reserved names

The property names shape and color are perfectly legal to use in a property file. The Game Server software and the GUI client will know that the objects involved are IPB objects, rather than as shape-color tuples, and will handle them accordingly.

Automatic generation of images and property files.

It is expected than in many cases, both a group of object images and the property file describing them will be generating programmatically, by means of a customized program or shell script. For example, if one wants to create a directory with 24 images of arrows, consisting of red, green, and yellow arrows pointing in 8 directions (N, NE, E, SE, etc), one can manually create a single image (in an image editing program such as Inkscape, Gimp, or Microsoft Paint), and then have a script produce multiple images by rotating the original image and changing its color, using a command-line image manipulation utility such as ImageMagick.

In a more sophisticated example, you can have a 3D model of solid structure, such as a statue (e.g. in VRML or X3D format), and a program that rotates it in various ways and saves various 2-D projections (equivalent to views from different directions) as separate images; for each image it will create an entry in the properties file describing the direction from which the statue is viewed (e.g. using spherical coordinates).

If you need some help with generating a series of images along with an accompanying property file, contact Vladimir.

Controlling the initial board generation

Both of these methods exist in GS 3.* as well, and support not only traditional SC objects but also new IPB objects.

When using predefined initial boards

If your parameter set uses predefined initial boards, there is not much difference from GS 2.*. The way the parameter set specifies the location and ordering of prefedined initial boards is exactly the same; any of the initial board files themselves may contain either SC objects, or IPB objects, or any combination of both types of objects.

When using a random generator

In GS 2.*, a parameter set could specify the parameters of the random board genrator, i.e. the min and max number of pieces on the board, the min and max number of colors and shapes, as well as the set of shapes and the set of colors from which the shapes and colors of all pieces are drawn. (If those two sets are not specified in the parameter set, the legacy 4-shape and 4-color sets are used as the defaults).

In GS 3.*, if you want to generate random boards using IPB objects, you need to first specify the set from which these objects are drawn, by using the images parameters. You also need to provide the min_objects and max_objects, same as for the traditional SC objects. You don't need the min_shapes, max_shapes, min_colors, max_colors any more, since they are not applicable in the IPB context.

Question for discussion: if using IPBs, do you still feel that there is a need to provide something analogous to min_shapes, max_shapes, min_colors, max_colors? That is, an ability to specify that the objects of your set have property X (e.g. posture), and you want every board to have objects with no fewer than n1 and no more than n2 different postures? If needed, I can work on a syntax for this feature, although it likely will be rather cumbersome.

Answer (decidesd at 2021-04-19 meeting): no, we don't need that. If the experiment need this kind of distribution (or any other specialized distribution), they can write their own script to create a large set of initial boards, and then, in the parameter set, specify random selection from that set.

The value in the images parameter, is essentially, a list of image files, with *-based and ?-based wildcard expressions allowed, and the [x,y,...] notation for lists. The file locations can be relative (interpreted as relative to the server's shape directory) or absolute. For example,

Note that, if you are using a random board generator in your parameter set, it cannot combine SC objects and IPB objects in the same parameter set. If you want your random boards to contain, for example, both black squares and rampant lions, you have to create a directory in which shape-and-color tuples are defined as IPB objects, i.e. with an individual image file for each object (black-suqare.svg etc)

Rule set files

An alternative proposal is for a format where every field in the atom has to be explicitly labeled, and no field is mandatory:

           (count:count, property1:valueList1 [, property2:valueList2]  [pos:positions,], bucket:buckets)

An absent field is equivalent to a present field with the value *, i.e. "anything is allowed, as far as this type of condition is concerned".

As it is the case with the "legacy" atoms, each new-style atom can be understood as a conjunction. That is, for an atom to allow moving a game piece to a bucket, each part of the atom, viewed as a condition, must yield true on this game piece. The possible conditions include:

an optional condition on the number of times the atom can be used until the rule line needs to be reset;

zero, one, or several conditions applied to various properties of the game piece the player wants to move. (No more than one condition per property);

an optional condition on the position of the game piece to be moved;

an optional condition on the choice of the destination bucket Thus, the simplest atom,

()

is the trivial conjunction of no conditions -- so it means, "take any number of pieces and put them into any buckets". If for example, the condition specifies count and the property species, e.g.

(count:3 species:[lion,mouse])

it means, "take 3 pieces that are lions or mice, and put them into any buckets". The atom

	  (species:tiger color[pink:blue] pos:T bucket:[0,1])

means, "take any number of pink or blue tigers from the top occupied row of the board, and put them to bucket 0 or 1".

or in the alternative format,

       (species:tiger, brightness:bright, bucket:0)

       (count:3, color:black, pos:T, bucket:1)

Just like one could do it with shapes and colors in GS 1.* and 2.*, one will also be able to use lists of values with IBP objects. E.g. the atom

       (count:2, species:[tiger,lion], direction:right, bucket:0)

Value ranges

Suppose the property angle is integer-valued, with values ranges from 0 to 360; it is used to describe the orientation of objects (the rotation angle from some initial position). For example, suppose the experiment desisgner uses this property to indicate the angle by which an arrow is rotated, counterclockwise, from the direction "east" (X axis):

When creating a rule with a range for some property, the experiment designer should ensure that all objects for which this property is defined contain eiher a numerical value for this propery, or the special value * (which matches all selectors).

Effect on the buckets statement

As of GS 2.*, the expression used as the last element of the atom makes use of the variables ps and pc, which refer to "the most recent bucket into which an object of this shape was put" and "the most recent bucket into which an object of this shape was put". In GS 3.*, we need to extend this syntax to apply to IPB objects, so that we could express concepts such as e.g. "the most recent bucket into which an object with this orientation was put". I propose to use the following syntax:

Describing a board as a JSON structure

The Game Engine exports the informastion about the current state of the board in JSON format when the Web-based Game Server transmits this information to the GUI client, or when the Captive Game Server sends this information to the ML program that has spawned the CGS.

At present, the plan is that when a JSON representation is former for sending to the GUI client, each IPB game piece will be identified just by the image attribut. The GUI client does not need to know about the properties associated with this object in the properties table, since all it needs is the image.

How will this work with ML?

Still, a human player is never explicitly told what properties of objects may be used in the rules, and he has to figure that on his own, likely making natural guesses of what features are salient. E.g. if the game pieces are alphabetic glyphs from various alphabets, such as A, Λ, V, М, Δ, E, Γ, L, Π Ш, O, U, Ո, Φ, P, З, C, a human player may need to make guesses as to which features may matter. Is it the geometrical shape of the glyph (with a sharp angle vs. with a right angle vs. with a rounded element)? Or the topology (having or not having a loop)? Is it the alphabet the character may belong too (Latin vs. Greek vs. ...)? Is it the sound expressed by the letter (vowel vs. stop vs. fricative vs. liquid)? On the other hand, if we explicitly give the ML program the list of properties (as given in the property file) for each game piece on the board, the ML program will be able to only look at this finite set of properties.

Recently (2021-04-13) on Slack Jerry apparently suggested that they can use :"deep net feature representation" to anlyze the images. If something like this can be tried, then perhaps a ML program could try to play against the the Captive Game Server without being explicitly supplied features, just like a human would...

Based on Jerry's and Shubham's input at the 2021-04-19 meeting, it was understood that the ML team indeed intends to use image recognition. So their ML application can get the SVG or JPEG file based on the image attribute of each object. They don't need to use the explciti property information, so the captive server may choose not to supply it. However, I may also provide an option for the captive server to include it, so that the ML team can have an easier time in some experiments.

Game transcripts and other output files

The introduction of the IPB objects will necessitate some changes to the CSV data files written by the Game Server for subsequent analysis.

The initial board

In GS 1.* and 2.*, the initial board file describes each game piece by two columns, shape and color. In GS 3.*, we will add one more column, objectType. The traditional SC objects will continue to be described primarily by the two old columns, while the objectType column will contain something like BLACK_SQUARE empty; the new IPB objects will leave the shape and color columns empty (or write null to them), while the objectType column will contain the path to the image file (either relative to the server's shape directory, or absolute, as appropriate).

Note that we don't explicitly write the properties of IPB objects to the saved initial board file. The researchers can combine each game piece's image name from this file with the data from the properties file in the directory where the image is located in order to find out the object's properties. If this is an issue for Aria or Ellise, please let me (Vladimir) know!

The transcript

The transcript files won't be affected, as they identify game pieces by their positions on the board.

The detailed transcript

When the detailed transcript format was first introduced in GS 1.*, Aria wisely requested that a field named objectType be provided. In GS 1.* and 2.*, the value of this field is created from the color and shape properties of the object, and then capitalized, e.g. BLACK_CIRCLE. In GS 3.*, we will write the same value in this field for SC objects, while for the new IPB objects the image path will be written into this field, e.g. /opt/tomcat/game-data/shapes/exp-20210501-a/animals/rampant-lion-03.jpg.

Compatibility with Game Engine 1.* and 2.*

All old (GS 1.* and 2.*) experiment control files (trial list files, rule sets files, initial board files) will continue to be usable in GS 3.*, with the same effect (behavior of the system) as in GS 2.*.

A JSON structure describing a board (such as an initial board file, or a JSON structure sent by the Game Server to the GUI client in response to a /display API call) may contain both GS 2.* legacy pieces (described by a shape and color and GS 3.* IPB objects.

A trial list file may also contain both parameter sets using the tradition SC objects and those making use of the new IPB objects. As mentioned above, however, one cannot combine both types of objects in the random board generator within a single parameter set.

A single rule set file may also have atoms with the legacy 5-tuple structure, and atoms in the new format. Internally, an SC object is handled the same way as an IPB object that has exactly 2 properties defined (shape and color), and a 5-tuple rule atom has the same effect as a new-format rule that explicitly refers to these two properties.

Proposal: image-and-property-based description of objects