Tracking: Turing Way

[rowanc1]

We had a wonderful conversation this morning with the TuringWay team on how best to support the TuringWay team in their adoption of JupyterBook v2, which will be using mystmd. There were a lot of exciting possibilities that we talked about in terms of creating an ecosystem of content that the TuringWay could expose (figure, equations, glossaries), and that be included, linked to and cross-referenced by other books, tutorials, and communities in the wider open-science ecosystem. There are also some current deficiencies in MyST MD at the moment, one of the core ones is around internationalization. We got a really good demo from @BatoolMM on how that process is working, and some of the sticking points on how to bring content together to have a unified experience between languages. The localization also must include right-to-left languages, which will require a number of style updates and improvements across the components.

Resources for translations:

• https://turingway-arabic.netlify.app/welcome.html
• https://github.com/TWTranslation/the-turing-way/tree/main/book/translation

Example of Turing Way content deployed in MyST

• https://rowanc1-turingway.curve.space/ (this is deployed using Curvenote)

Some cool things

Updates that need to happen

• Glossary Compatibility with RST Glossaries #618
• Improve the upgrade process 🆙 Upgrade Jupyter Books in myst init #1223 (in progress)
• Internationalization and localization (this is a big task that we need to break down, what is necessary is a lot clearer to me notw after @BatooIMM's demo) Add support for translations / internationalization / localization of page elements #166
• Host multiple languages in a single theme
• PR: fix admonitions with Note --> note (that can be upstreamed right away)
• Way to turn off @username references on a page (e.g. changelogs) or have those fallback to a configured list of IDs and improve, e.g. our GitHub person cards.
• Discuss and demo different ways to build out the infrastructure for the book (and tradeoffs between the different approaches)
• Idea (curious on your take): Add source attribution information to images/figures #1305
• Improved header / footer information

There are probably a number of other things that will come out as we work towards these goals, I encourage members of the TuringWay team to jump in here and let us know how we can support you. Please try the preview above and get a sense of where we are at. One of the cool wins is that the performance of navigating between pages should be improved quite a bit. There are also some abbreviations that will work and increase accessibility etc. for various pages.

I know that my team and @agoose77 do have some time to help with specific things, and some issues are in progress this week, so hopefully when we meet again in a few weeks there will be some progress to show!

@JimMadge @da5nsy @BatoolMM @agoose77 @sgibson91

We have had some previous conversations with the Environmental DataScience Book as well:

https://github.com/alan-turing-institute/environmental-ds-book

@acocac

[da5nsy]

Amazing, thank you so much for getting the ball rolling on this @rowanc1. Really excited to see where we can go with this. And happy to assist where we can!

Linking to the issue over on TTW where we've been tracking our own attempts/progress on enabling multi-language support: the-turing-way/the-turing-way#3255

[agoose77]

@rowanc1 was this meeting recorded in any fashion? I had some pre-conceived ideas around what TTW is doing regarding Crowdin, etc, and I know a bit about the space of tools. My read is that there are many ways to do this, so any guidance on the solution that TTW is converging on would be useful context.

[da5nsy]

Hi @agoose77!
Not recorded I'm afraid, but I'm happy to answer (or redirect) any questions you have.

@BatoolMM is the person to ask about translation, or have a look here at our existing documentation.

And then from an infrastructure perspective this is the latest. In summary, there's currently no obvious out-of-the-box tools that we could use to deploy a multi-language site. We might be able to get something working with RTD, but honestly we would prefer an internal process that didn't rely on an external build service (so that we would be able to host anywhere).
(@JimMadge, @sgibson91, @bsipocz - correct me if I've misunderstood!)

[rowanc1]

@fwkoch just made a change that takes turing way from >1min to build to 17 seconds. 🚀

[JimMadge]

Just leaving some of my thoughts/opinions,

Hosting isn't the same as the build. However, hosting that we have considered/used (Netlify, RTD) do place restrictions on the build.
Ideally, I would prefer that we control the build and just copy the output somewhere to host.

Multilingual support seems surprising difficult given how many languages are commonly used.
A lot of the solutions we have looked at seem to have inherited the idea of translations from software, where you have either

• A set of messages with .po files for each language
• A primary language with translations for each other language

Neither of these feel like they suit us. We don't want to enforce the idea of a primary or default language. We don't really see the task as reproducing the same information in each language.

Maybe we want something more like Wikipedia, where different languages are fairly independent but if there are pages on the same topic in two languages, you can switch between them.

This really isn't my area so I have no idea how difficult or possible that is 😆

[da5nsy]

Hi folks!
Just adding a note to say that I tried following the instructions at https://next.jupyterbook.org/upgrade to test upgrading TTW and totally failed.
the-turing-way/the-turing-way#3958
Might be a Windows thing, or just me goofing something up, but I don't know.

[agoose77]

@da5nsy thank you for trying out Jupyter Book and reporting back! I'll take a look at this within the next day.

[da5nsy]

We're going to work through the error logs from the-turing-way/the-turing-way#4019 tomorrow at a collab cafe. If any folks from mystmd would like to join they'd be super welcome!

[acocac]

I've experimented JB2 for EDS book since last month - Happy to join the collab session to have a look at the error logs too!

[rowanc1]

It would be awesome to have a some of the exploration of this documented here so that we can improve things! I think that @agoose77 will be there, however I can't make it. Good luck!! 🚀

[agoose77]

We had a look at some error logs. There were a lot of hard-failure errors in @sgibson91's local build that look suspiciously like resource-exhaustion errors during the HTML build specifically.

We also found a lot of errors pertaining to genuinely missing links.

I observed:

1. We should perhaps support caching link resolution (i.e. "it resolved") for a number of days
2. We should enable cache persistence of myst builds with the GHA generated by myst init.
3. Our DOI heuristic might be too aggressive: Examples of discrimination in AI include [racism in predictive policing](https://www.technologyreview.com/2020/07/17/1005396/predictive-policing-algorithms-racist-dismantled-machine-learning-bias-criminal-justice/) is treated as a DOI with 1005396
4. Our wiki heuristic doesn't understand permalinks to wiki content: https://en.wikipedia.org/w/index.php?title=Tidy_data&oldid=962241815 fails
5. We end up with a lot of errors RE Waiting for response from http://localhost:3000/remote-collab-guidelines.json that ultimately crash
6. We also see ECONNRESET errors
7. Perhaps we want to be able to ignore errors in a particular file e.g. automatic link fragment generation
8. Do we support {cite}`Corporate Governance and Ethics<Dessain2011governance>`

[rowanc1]

DOI fixed in continuous-foundation/doi-utils#15

The ECONNRESET is likely #1775 if you are building html. Especially for turing way we hit the web server with all requests at once; this should be looped in a promise limit. That should fix 5/6.

[rowanc1]

This should fix 5/6:

• 🔄 Fetch with Retry for HTML build #1793

[da5nsy]

As of today, the-turing-way/the-turing-way#4019 is merged and therefore The Turing Way is running JB2! 🎉

Thank you all for all your assistance in getting us to this point. Looking forward to being able to be a test case for new developments going forwards!

We're working through some residual points in the-turing-way/the-turing-way#4056

[JimMadge]

It would be great if anyone has insight on the-turing-way/the-turing-way#4055 and could comment.

The problem only seems to occur on MacOS. It happens with The Turing Way, but not with a minimal Jupyter Book. So I'm not entirely sure if it is a Turing Way problem or perhaps a Jupyter/Myst bug. If there are some other sizeable books I could test building I would be happy to try 🙏.

[JimMadge]

And to echo what @da5nsy has said. All of the collaboration has been amazing. It is so great to have the developers looking so closely at our problems. I hope it is also helpful for you to find bugs and get some user feedback 🙏.

[choldgraf]

Woohoo! Thanks y'all for being so helpful and collaborative. Could we write up a little case study for the jupyter blog?

[fperez]

This is awesome news, so grateful to both teams for the excellent collaboration!! Kudos to everyone involved :)

[JimMadge]

Could we write up a little case study for the jupyter blog?

I think that would be great 🚀. Do we want to find some time to draft something together? From our side, @da5nsy really helped push it over the line, I did a little fixing around the edges, and @sgibson91 did the lion's share of the migration work.

[da5nsy]

Could we write up a little case study for the jupyter blog?

Happy to be involved! Can you link us to a similar example (if such thing exists)? What would the audience be? (A list of the technical steps vs a broader and more accessible "why we upgraded").
We've got a collab cafe at 3pm - 5pm GMT today if it would be useful to videochat about it? (I personally can only do the first hour today)

[rowanc1]

A hearty congrats to the Turing Way folks. 🎉 I was going through the new site and was really having fun with the deep dive links in the glossary. I am very excited to share this case study of folks upgrading — seeing Turing Way switch over is a huge milestone and I am excited to ship some new features that improve a few of the things I saw as well. :)

@da5nsy @sgibson91 @JimMadge I would be very interested in both accounts. I am going through a similar upgrade process for @mmcky's content at QuantEcon for large, executable books and the technical upgrade requirements would be good to compare on, esp. regarding where we can smooth some things out.

Content wise, I think that the JupyterBlog should probably be a bit higher level — why upgrade, new capabilities, and a vision for reuse between resources. Again, I would be very interested in your perspectives there @da5nsy @sgibson91 @JimMadge.

[agoose77]

I've opened a PR to update some syntax to the way we do things in MyST: the-turing-way/the-turing-way#4061

This probably did parse differently under docutils, but I don't hate the change that we're imposing — the extra newline improves readability imo.

https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#block-quotes

[choldgraf]

Quick thoughts on a blog from me:

Could we write up a little case study for the jupyter blog? Can you link us to a similar example (if such thing exists)? What would the audience be? (A list of the technical steps vs a broader and more accessible "why we upgraded").

Personally, I think that having a regular heartbeat of blog content is more important than having "the right" content. I'd be a fan of communicating and celebrating wins wherever we can. In this case, I think a "cast study" is the right framing. The audience might be other communities that are interested in using Jupyter Book 2, and that have experience with Jupyter Book 1.

A lightweight structure could be:

• A summary of what you did ("We upgrade to JB2 to by doing X, Y, and Z")
• What this unlocks for you, or improvements it brings from your perspective ("This allows us to A, B,C")
• Any lessons you learned the hard way, anything you wish was different. ("Along the way, we hit unexpected challenge X, Y, Z")
• Anything you'd like to see different or improved ("We're particularly excited to see JB2 improve in areas A, B, C")

I'd be fine putting this either on the JupyterBook blog, or on the Jupyter Blog. Probably lean towards the Jupyter Book blog since this is more relevant to JB-specific users rather than Jupyter at large.

If you want inspiration, here's a blog post about a Project Pythia workshop we participated in - though this is way, way more information than we'd need - I'm just imagining like 300 words or so.

[fperez]

Huge 👍 on @choldgraf's suggestion above, with emphasis on making it lightweight enough that it's not a huge burden for anyone to produce, to keep the ❤ actually beating on a regular cadence :)

[JimMadge]

That structure sounds good to me. Maybe we could start a shared document where we answer those questions asynchronously. Would that work for the JB team and whoever will write/edit the blog post?

[choldgraf]

@JimMadge and @da5nsy I've put together a child issue to track a little blog post, and have a short draft template for writing. Let me know what you think! Happy to iterate on that with you:

#1871

[stefanv]

xref #1876

[stefanv]

xref #1877