Add python otfautohint into afdko

[skef]

Description

This PR adds the Python port of psautohint back into afdko as otfautohint. Although the new name is more accurate, the primary motivation is to allow the older C-based psautohint to be installed and used simultaneously.

Some of the tests are missing from this PR because they depend on a git submodule, and we need to decide if we're going to retain that convention or do something else. For now these are represented in adobe-type-tools/psautohint-testdata#5 .

There is also some duplicate code, mostly between python/afdko/convertfonttocid.py and python/afdko/otfautohint/fdTools.py. We should decide what if anything to do about that.

There's a bit of documentation but no added messaging about the changes.

Most of the changes to the makeinstancesufo tests are just updated hinting, but I did need to change the blueFuzz for the bold instance because otherwise it had a pair of stems too close together. Not sure why the older hinter wasn't catching that.

As we discussed I did not include a command line script for splitpsdicts but it can be run as

python -m afdko.otfautohint.splitpsdicts [arguments]

Checklist:

• I have followed the Contribution Guidelines
• I have added test code and data to prove that my code functions correctly
• I have verified that new and existing tests pass locally with my changes
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

[skef]

I kept the _version.py from the previous repository in the otfautohint directory as an example. I'm not entirely happy with that solution (it's unfortunately easy to check the modified file back in) but I worry our current practices are worse.

Anyway, we can decide what to do about it.

[skef]

A few notes on tests, while they're at the front of my mind:

The tests in the psautohint repo were divided into "unittests" and "integration", but while that might have been the original intention it was really "tests that use stuff checked into the repo directly" vs "tests that use stuff in the submodule repo".

Accordingly, the most of the (top-level) contents of the "dummy" directory of the submodule are for unit tests, or at least fairly specific tests. The main exception is the "mm0" directory, which is also the largest (29 out of 33 megs). It would probably make sense to move that stuff, minus mm0, into afdko/tests/otfautohint_data.

As for the rest, it mostly seems to involve hinting whole open-source fonts, either individually or as part of multi-master or VF workflow. I think the main thing this buys is diversity of geometry to see if the code crashes. It also arguably involves a correctness test, in that we check to see if the output has changed, but absent actual evaluation of the output it's not clear what is "correct". If someone changes the tool somewhat, would we do anything other than copy the modified output over so that the tests pass again? I did some checking with hintdiff but I know that was broadly my plan with the port.

In any case, one thing we could do is make a single font or a modest number of small fonts with "interesting" glyphs pulled from that larger set, and check those into afdko for basic testing. Then we can either punt on the rest or maintain a separate repository with its own tests directory that we check against when there are significant changes to otfautohint code. (Maybe the requirement can be that if a code change requires any change to the otfautohint tests included in the repo, you are required to check the tests in the external repo as well.)

[skef]

I have now ported most of the tests over.

Right now there is a largish-subdirectory called "dummy/mm0" which is used for basic multi-master hint testing but the output is never evaluated. The glyphs are from some san-serif font although the UFO metadata says "source serif".

Here is one way to get this PR to where it can be integrated:

1. Pare down the list of glyphs in mm0 a bit so that it's more of a sanity check.
2. Sample interesting/relevant glyphs from among the various fonts we use for "bulk" testing (cantarell, libertinus-serif, source-code-pro, and source-serif-pro) and build a single font with multiple dictionaries (so that the blue values and other parameters of the respective fonts are preserved). Make a VF version of this font as well as separate OTFs. Add tests to hint these and match the output to equivalent files in expected_output.

Later, when we've added the UFO support for FDArray/FDSelect to otfautohint we can also make UFO versions of these sample fonts.

[skef]

There's one test that hints a glyph with a huge number of points that now takes an annoyingly long time with the python code. Since we normally skip these I'm not sure it's worth running this test routinely -- it will probably get annoying.

[skef]

Here is a list of tests that have been entirely ported over in terms of the files they're stored in in the psautohint repository:

• unittests/test_fdtools.py
• integration/test_cli.py
• integration/test_stemhist.py
• integration/test_ufo.py

Partial ports:

• integration/test_hint.py (missing tests: test_ufo, test_otf, test_cff, test_flex_otf, test_flex_ufo, test_too_long_glyph_name (not needed))
• integration/test_mmhint.py (missing tests: test_mmufo, test_mmotf, test_incompatible_masters)

I believe these files aren't relevant, mostly because we don't use bez anymore and/or because the C code had hard array limits that the python code doesn't:

• unittests/test_extension.py
• unittests/test_hint.py

We might want to disable test_big_glyph in test_hint.py given how long it takes to run now.

[skef]

I backed out the _version.py file to a hard-coded version like the other facilities, bumped to v3.0.0. I also modified the tests to skip big_glyph_test.

In addition, I added this branch and draft PR to psautohint-tests: adobe-type-tools/psautohint-testdata#5 .

[skef]

Topics for messaging:

• Python port of psautohint was (re)integrated into this repository as "otfautohint".
• The name-change better reflects how hinted input is now published, and lets the new version coexist with the old version when needed.
• Other tools in AFDKO have been updated to call oftautohint instead of psautohint, and the dependency on the latter's repository has been removed. We expect development of psautohint to stop as of this release.
• Because psautohint was partly written in (very old) C code, and otfautohint is written entirely in Python, the new code takes significantly longer to hint an individual glyph. However, we have also enhanced otfautohint to hint different glyphs on multiple CPU cores by default. As a result the tool will be 5-8 times slower when running on a single core but will typically be slightly faster when running on 8 cores.
• The new code fixes a number of bugs and in our judgement produces better results on average. It also uses a slightly different encoding in UFO glif files. Accordingly, users should expect that running the new code results in many differences compared with psautohint, but this should be a one-time change.
• The new code also supports hinting of variable CFF-based fonts, including directly hinting a built CFF2 font. Glyphs that include overlap will typically be hinted close to how they would have been hinted with overlap removed.

[skef]

We agreed in discussion that the minimal duplication of code can be cleaned up later.

And with that, I think this PR is ready to be evaluated for merging.

[kaydeearts]

FYI — GitHub is having a hard time loading the diffs even after I filter out the .glif files, so when the page is not responsive I'll directly comment here & include the file & lines I'm referring to.

[kaydeearts]

otfautohint_Notes.md, lines 18-19
glyphData.py, lines 44 - 60

Is there a definition of what a and o is in the documentation, and if not, could we add that somewhere? What are those values in relation to vertical & horizontal hinting?

[kaydeearts]

otfautohint_Notes.md lines 84 - 106

• Could you add a key for what each variable means in these examples? I feel like this is otherwise helpful for someone like me reading about the technicalities of hinting, but I'm trying to guess what c is, for example
• When you say stems i & j, are these stems of the same glyph but in different masters? I kind of feel like this gets clearer as I read on, but takes some assumption

[skef]

otfautohint_Notes.md, lines 18-19 glyphData.py, lines 44 - 60

Is there a definition of what a and o is in the documentation, and if not, could we add that somewhere? What are those values in relation to vertical & horizontal hinting?

Well, I would say that the definition is in the lines you mention in the comment to pt.setAlign.

In terms of interpretation "a" is meant to suggest "aligned" and "o" is meant to suggest "opposite", and I guess we could add that somewhere.

If you look at the C code of the current psautohint there are separate implementations for vertical and horizontal stem hinting, with many functions that (mostly) differ only in having "x" and "y" swapped. The cleanest way of combining the implementations is to have a property on the pt object that can refer to "x" when hinting stems horizontally and "y" when hinting them vertically, and another property that does the opposite.

[skef]

OK, I pushed some code-review related changes. This doesn't resolve all of the concerns in hinter.py, but I did add a "conventions" comment to the top of that. We can talk about that stuff.

[kaydeearts]

Skef and I talked about some things offline and I reviewed his latest commit — this looks great!