Measuring/estimating text in VexFlow quickly and accurately

[AaronDavidNewman]

I started this with a specific issue in mind, but this quickly became a larger discussion in some PRs, so we are moving the discussion here.

Introduction

In many cases we need to know metrics of text font files to place and render them with music glyphs. Drawing and measuring the text has some problems: 1) during the format part of the pipeline, the rendering context isn't available, and 2) rendering anything on the VF renderer canvas/svg can cause a reflow which will greatly slow down the rendering of large scores (especially in SVG).

Summary of the Proposals

There are a couple of solutions to this that have been introduced, each has its own drawbacks:

1. TextFormatter is a class that uses font metrics derived directly from the font, using the same tools we use to derive music font metrics. It was introduced with the ChordSymbol module. Drawbacks: 1) you need to have the metrics for the font a priori, if the metrics aren't available the accuracy won't be as good. And 2) representing information like kerning pairs could result in some large, static files.
2. @rvilarl had the idea to create a canvas context specifically for the purpose of rendering the bounding box. Since this isn't the canvas where the score is rendered, it shouldn't cause a reflow and will does not require the text metrics to be derived in advance. Drawbacks: 1) lazy-loaded fonts might not be available when rendering (this is the issue that initially started this thread) 2) we need to investigate if there are cases where canvas context would not be available and text estimation needs to be done. Right now there is only one DOM element required to use VexFlow, when the renderer is created.

[AaronDavidNewman]

This is the initial text from this issue:

I can't get the tempos in my music to look right, if I am using a custom web font for the text, because the font is lazy loaded and the wrong font is used for estimation.

[mscuthbert]

The only way I have found to do this properly is to append to an HTML element, measure it, and remove it, and make sure that the second text element falls after that. Or to use a custom SVGForeign element of the two HTML Elements.

(the particular solution in this case, though, is not to encourage people to play music arrogantly. there's enough of that in music as it is. :-D )

[AaronDavidNewman]

Ha! Not my tune, but yeah a bit overdone.

I suppose the html and canvas ideas are similar. Canvas might be faster since it's a pure raster operation and doesn't affect the flow.

I wonder why the browser doesn't just expose an API to get metrics without having to draw anything - it must know what they are. There must be some technical or security reason I don't understand, I'm sure other people have asked for it.

[rvilarl]

In #1466 I prototyped the idea from @ronyeh to use the font directly. I did it only for noteheadBlack, see latest commit 7be6d55 . The result is also very promising.

noteheadBlack rendered as a font

current svg path generation approach

The size of the SVGs produced by flow.html is reduced by 10MB as each print out of a notehead is now simply

<text stroke="none" x="317.11" y="110"></text>

rather than

<path stroke="none" d="M321.041 115.054C324.636 115.054,329.044 111.741,329.044 108.315C329.044
 106.237,327.416 104.946,325.113 104.946C320.676 104.946,317.11 108.231,317.11 111.685C317.11 
 113.791,318.851 115.054,321.041 115.054"></path>

[ronyeh]

Yes! I never had time to look into just using the Bravura / Petaluma / Leland OTF files directly, but it would be cool for sure. Not sure if we could make it a backwards compatible upgrade. Or it might just be for VexFlow 5. :-)

[AaronDavidNewman]

Does using the font for the note head speed rendering also? The size of the SVG is definitely a problem for me. That is probably worthy of its own issue.

Unfortunately we don't have any good large-scale renders for benchmarking in native vex format. We need a Mahler symphony or something. I have a few of my big-band charts and I'd like to get them, at least a few pages, into vex code. We probably don't run them as part of the unit tests but they could be seperate.

[rvilarl]

I can extend my prototype to use the font approach for Bravura. This is easy. I can also use VexMl to convert a MusicXML score to VexFlow factory API. This is also easy although VexMl is not complete and some elements will be missing. I would need that @ronyeh releases a new Alpha to get more completeness.

[rvilarl]

@AaronDavidNewman I have reopen #1466 as draft so that you can look at the details in the prototype:

Prototype flow.html

• 50,3MB
• 8259ms

Master flow.html

• 60,3MB
• 8872ms

10MB less and 600ms less 👍

[AaronDavidNewman]

I don't think removing TextFormatter is related to reading music font directly. Is the latter change in 466? I thought that was only about text fonts. Which file reads the music fonts directly?

[rvilarl]

I tried to make a minor change on master but TextFormatter was causing a runtime error. I did not investigate further. The important changes are glyph.ts, the list of WEB_FONT_FILES in font.ts and the change in textfonts.ts.