Unicode mappings for Sierra games, keen4, ss2 and sf

[horsemankukka]

Some generators, namely:

• King’s Quest 4 and 5 (kq4, kq5);
• Space Quest 1, 3 and 4 (sq1vga, sq3, sq4);
• The Colonel’s Bequest (lb1);
• Police Quest 2 and 3 (pq2, pq3);
• System Shock 2 (ss2);
• Commander Keen 4 (keen4); and
• Star Fox (sf)

have (mostly) identity mappings to the character sheet (as in character at position 1 is mapped to U+0001) instead of proper Unicode mappings.

I have created this gist, which has

1. proposed mappings as easier-to-read YAML file; along with
2. a JSON file with resolved references and expanded ranges. It also has
3. a python script to convert the YAML to the same JSON or alternatively to keys¹.

Example of kq5 mapping with reusable parts as a separate referencable template:

x-kq5-base: &x-kq5-base
  0x01: ⛰ # Sierra logo
  # 0x01: ":sierra:"
  # 0x02: ":alt:"
  0x02: ⎇
  # 0x03: ":ctrl:"
  0x03: ⎈
  0x04: ©
  0x05: ™
  0x07: ␈
  0x08: ␉ # "TAB"

kq5: &kq5
  <<: *x-kq5-base
  0x00: null
  0x06: null
  0x09..0x1F: null

Processed to JSON

"kq5":{
  "0":null,"1":"⛰","2":"⎇","3":"⎈","4":"©","5":"™","6":null,
  "7":"␈","8":"␉","9":null,"10":null,"11":null,"12":null,"13":null,"14":null,"15":null,"16":null,"17":null,"18":null,"19":null,"20":null,"21":null,"22":null,"23":null,"24":null,"25":null,"26":null,"27":null,"28":null,"29":null,"30":null,"31":null
}

And a chart for visual comparison (with Times) with changes highlighted:

The example also highlights some problems: for BS and TAB I selected corresponding control pictures, but CTRL and ALT are mapped as keyboard key symbols. I’m not entirely sure what the semantic meaning is.

They could also be mapped to :insertables: as there seems to be support for those since 6f92437, but I’m not sure if they are basically identical to characters or not. (aside: have insertion links for [subfont][/] disappeared from the UI or were they ever there?)

For Star Fox I mapped the lowercase punctuation to U+FE50..U+FE6F Small Form Variants compatibility block and to Katakana-appropriate compatibility characters (such as fullwidth forms). The generator would optimally, maybe, have a hook that selected the proper punctuation based on preceding letter’s case/script, but then again it would prevent accessing the specific forms if needed and could be even more confusing than having the forms as separate characters:

I’ve tried to maintain common CP437 mappings where charset is clearly based on that, such as in the Sierra games.

Rest of the mappings as pictures, grouped by similarity for ease of comparison:

kq4+sq3+lb1+pq2

[Note: This game (sq3) has two fonts but the json file does not include the other. The subfont has different bounding boxes and a bit more characters than the base font, a bit difficult situation.]

kq5+sq4+pq3+sq1vga

ss2

keen4

[Note: 81..8A were mapped to fullwidth numbers as the glyphs differ from the ASCII numbers. Fullwidth numbers are usually monospaced, so I figured it would be most appropriate mapping.]

I can make a pull request out of these, but would maybe like some comments about the suitability of the mappings (or if :insertables: would be better for ambiguous ones) to make the merging easier (or possible). The gist data is for as-easy-as-possible application without pull request if desired.

Edit (2022-01-15): Changed some terms to match with the source code (like references to insertables) and added the reasoning for gist data. Added notes for sq3 and keen4. Formatting changes.

Footnotes

1. It was surprisingly difficult to get 1:1 output (normalized with jq) since the original JSON output was from my HTML+JS thingy for generating those comparisons that’s too messy to publish. However it does just that now. ↩