Tests are unstable

[remcohaszing]

The tests are currently unstable due to the unstable output of mermaid. As a result I had to allow test failures in CI, which is really bad. I’m open to suggestions to make the tests more stable.

[aloisklink]

You can try setting deterministicIds to true in the Mermaid config: https://github.com/mermaid-js/mermaid/blob/03c5bc112947eebb428e1d540a665392b5eb800c/packages/mermaid/src/defaultConfig.ts#L146, which should help for most IDs (if that's the only thing that is changing).

Unfortunately, that still doesn't help everywhere: I just had to fix a bug in the entity relationship diagrams that weren't deterministic, see mermaid-js/mermaid#3916.

There's also some discussion in mermaid-js/mermaid#3491 about test snapshots in Mermaid, but it's mainly centered around rendered .png snapshots, not .svg snapshots.

.png snapshots do seem to be more stable (but much much slower) and it's what I use for https://github.com/aloisklink/remark-mermaid-dataurl (although I have much less tests than your project!)

[remcohaszing]

Thanks for the suggestions and cross-referencing related issues! It helps to have this information together.

The problem isn’t really deterministic IDs though. It’s mostly pixel offsets. The tests are relatively stable on my machine. In #17 Bruce says they are for him too. I’m inclined to believe they are in CI too. The problem is that they are unstable across different environments.

I like the idea of adding support for different output options, such as SVG or PNG data URLs. They would be a additional options though, not a replacement.

[aloisklink]

The problem is that they are unstable across different environments.

It might help to make sure all environments have the same fonts installed. Mermaid defaults to using trebuchet ms as the default font, which isn't great, since ms stands for Microsoft. You can install it pretty easily on CI using sudo apt-get install ttf-mscorefonts-installer, but it's not open-source, so you need to accept a EULA first.

(my wishlist is for Mermaid to default to using an open-source font like Atkinson Hyberlegible)

I doubt you'll get it to be pixel perfect on all environments though. I've heard that ARM-based CPUs and x86_64 CPUs both render CSS shadows slightly differently. My gut feeling is that ClearType/FreeType might also change how text is rendered, depending on what sub-pixel arrangement your screen has, so plugging an external monitor into your laptop might change the output!

I like the idea of adding support for different output options, such as SVG or PNG data URLs. They would be a additional options though, not a replacement.

If you do add SVG data URLs to your project, then I'll be able to archive my https://github.com/aloisklink/remark-mermaid-dataurl! Or convert it to a wrapper that uses this plugin.

[remcohaszing]

This issue has been solved by 2312642 and 236c16c.

@aloisklink

If you do add SVG data URLs to your project, then I'll be able to archive my https://github.com/aloisklink/remark-mermaid-dataurl! Or convert it to a wrapper that uses this plugin.

rehype-mermaidjs supports multiple rendering strategies, including img-png for base64 PNG data URLs and img-svg for SVG data URLs. It also supports better accessibility.