Sync missing translation keys in app/locales/*.json files to a Google Sheet #481

michaelmcmillan · 2020-03-29T19:45:54Z

The script is heavily commented with what I've been thinking. There might be a logical error here, so please read through the script to see if it makes sense.

This script will produce false positives (since it looks at all changes to the file ever). Why did I make it like that? Because translations are added at different points in time – a new feature branch will contain new translations, while all the existing branches and locales will be missing those.

This will produce some false positives (translations which have been removed etc.), but it in turn include translations which are being worked on right now in different branches (which should allow us to start translating before the PR is done, and widen the bottleneck which is currently halting PRs). We could also just add the false positives to an ignore list to make them go away.

I'm hoping this should make it easier for us to quickly grab all missing translations for all locales and then handing them off to a non-technical person that don't know JSON or git, maybe via. a more friendly medium like Google Sheets.

This is what I get if I run the script right now for no, en-IN and se.

no missing translation for: Sweden
... basically means that the Norwegian locale (no.json) has never had a row in its JSON file where the key is Sweden and the value is Sverige.

Does this make sense? Haha, I'm actually not sure. Here's the full output:

no missing translation for: Sweden
no missing translation for: https://www.whatismyzip.com
no missing translation for: What is my Zip code?
---
se missing translation for: this
se missing translation for: https://who.org
se missing translation for:  article as an attempt to chart these numbers.
se missing translation for: In total <%= numberWithSpaces(totalPeopleInContactWithInfected) %> people have reported that they have been in close contact with a person who was tested positive for COVID-19.
se missing translation for: In total <%= numberWithSpaces(totalInfectedPeopleWithSymptoms) %> people have reported that they have tested positive for COVID-19 and experience symptoms.
se missing translation for: In total <%= numberWithSpaces(totalPeopleInContactWithInfected) %> people have reported that they have been tested for COVID-19.
se missing translation for: Join the most important crowdsource! Regardless if you're healthy or not, please submit the form below – that is also valuable information!
se missing translation for: Zip code information
se missing translation for: Netherlands
se missing translation for:  if you can't get it to work.
se missing translation for: I agree to my being data in accordance with the privacy statement
se missing translation for: To WHO.org
se missing translation for: /healthcondition/
se missing translation for: Using self-report, you can get a better picture of how many people have symptoms without exposing health workers to potential contamination hazards and without using up valuable infection control equipment that is already in short supply. We at Bustbyte, well helped by other volunteers, have created this tool in response to
se missing translation for: In total <%= numberWithSpaces(totalPeopleWithSymptoms) %> people have reported that they experience symptoms.
---
en-IN missing translation for: this
en-IN missing translation for: https://who.org
en-IN missing translation for:  article as an attempt to chart these numbers.
en-IN missing translation for: In total <%= numberWithSpaces(totalPeopleInContactWithInfected) %> people have reported that they have been in close contact with a person who was tested positive for COVID-19.
en-IN missing translation for: In total <%= numberWithSpaces(totalInfectedPeopleWithSymptoms) %> people have reported that they have tested positive for COVID-19 and experience symptoms.
en-IN missing translation for: In total <%= numberWithSpaces(totalPeopleInContactWithInfected) %> people have reported that they have been tested for COVID-19.
en-IN missing translation for: Join the most important crowdsource! Regardless if you're healthy or not, please submit the form below – that is also valuable information!
en-IN missing translation for: Zip code information
en-IN missing translation for: Netherlands
en-IN missing translation for:  if you can't get it to work.
en-IN missing translation for: I agree to my being data in accordance with the privacy statement
en-IN missing translation for: To WHO.org
en-IN missing translation for: /healthcondition/
en-IN missing translation for: Using self-report, you can get a better picture of how many people have symptoms without exposing health workers to potential contamination hazards and without using up valuable infection control equipment that is already in short supply. We at Bustbyte, well helped by other volunteers, have created this tool in response to
en-IN missing translation for: In total <%= numberWithSpaces(totalPeopleWithSymptoms) %> people have reported that they experience symptoms.

ops/lost-in-translation.js

fossecode · 2020-03-29T21:39:28Z

Nice! Couple of questions:

When and who runs the script? On PRs?
Does it make sense to run it on all branches and all commits, or should we only run it on the current state on the branch you are on?
Any reasons javascript is used instead of typescript?

fossecode · 2020-03-29T21:44:36Z

ops/lost-in-translation.js

+ *
+ * So what we do here is simply to find *all* the english translation keys across the entire
+ * project. That means looking at all the english translations keys in all the locale files
+ * since the start of the project across all branches and PRs. We throw them into a set that


This sounds unnecessary to me, as it would also include WIP's and stale branches. But it might be some cases where this makes sense that I have not thought about?

Yeah, that's the part that sucks, but having it would make sure that all translations are added and (hopefully) translated by the time we want to merge it.

Aha, I understand. So a possible solution would be to run this if this run as a cronjob or something, so that the sheet is updated as soon as there are new texts in the repo?

Yeah, so in the case with Stano's PR, those translations are now added to the sheet (even though his PR is WIP). Not sure if we want that or not, but we could try it out?

fossecode · 2020-03-29T21:47:06Z

ops/lost-in-translation.js

+    if (translationKeysByLocale[locale] === undefined) {
+      translationKeysByLocale[locale] = new Set([]);
+    }
+    translationKeysByLocale[locale] = new Set([...translationKeysByLocale[locale], ...translationKeys]);


I feel stuff like this is mentally much easier to visualize when translationKeysByLocale is typed :)

Yeah we can tweak that, wanted to get it out quickly before bed

michaelmcmillan · 2020-03-29T21:47:12Z

Nice! Couple of questions:

When and who runs the script? On PRs?

Not sure what's best. PRs would work

Does it make sense to run it on all branches and all commits, or should we only run it on the current state on the branch you are on?

See comment above.

Any reasons javascript is used instead of typescript?

Wanted to crank it out fast. Types can be added.

fossecode · 2020-03-29T21:54:30Z

ops/lost-in-translation.js

+  for (let sheetIndex = 0; sheetIndex < doc.sheetCount; sheetIndex++) {
+    const sheet = doc.sheetsByIndex[sheetIndex];
+    for (const locale of allLocales) {
+      if (sheet.title !== locale) {


If I understand it correctly, there is one sheet per locale?

Is it then necessary to loop through the sheets here, can't we just loop through the locales and try to add them (from the code it seems like it throws an error if it does not exist)?

try { await doc.addSheet({ title: locale, headerValues: ['key', 'translation'] }); } catch (error) { // We don't do anything if the sheet for this locale exists. }

That's correct, take a look at the sheet here: https://docs.google.com/spreadsheets/d/1ILFfc1DX4ujMnLnf9UqhwQGM9Ke3s1cAWciy8VqMHZw/edit#gid=462835362

The reason I'm looping over locales is because that way we will create sheets for new locales automatically. There is no way in the API to retrieve the sheet names in the spreadsheet. This way we don't have to worry about keeping locales and sheets in sync (it will sort it self out).

Didn't read your comment thoroughly enough. Yeah, maybe we can get away without that loop.

Haha getting late... The reason we have to loop over those are because there is no way of retrieving a sheet by its name. Does that make sense? The doc.addSheet doesn't return the sheet if it doesn't exist, it only throws an exception. So we have to, at some point, look at all the sheets to check if they match the current locale we are looping over.

I see, I thought the for-loop ended after adding a sheet. A little hard to wrap my head around the logic, but as long as it works it is more than good enough.

I agree. Just refactored it, better now?

fossecode · 2020-03-29T22:01:25Z

Great job, I think it will make it much easier to handle translations :)

michaelmcmillan · 2020-03-29T22:09:30Z

Just tested this on @sbocinec's PR #483 and it seems to have worked 👍

fossecode · 2020-03-29T22:19:31Z

Just tested this on @sbocinec's PR #483 and it seems to have worked 👍

Haha nice, also found this bug from looking at the picture: b2988f4

fossecode · 2020-03-29T22:26:26Z

Although a few of the keys are not part of the translations anymore. Maybe add a whitelist or something where we can declare those?

fossecode · 2020-03-29T22:34:26Z

ops/lost-in-translation.js

+  for (const locale of allLocales) {
+
+    // Create sheet for this locale if it doesn't already exist.
+    let matchingSheet = sheets.find(sheet => sheet.title === locale);


Nice, much better 👍

michaelmcmillan · 2020-03-29T23:11:36Z

Next step before it's ready to merge:

Find a way of git fetching all PRs to local.
Add sync in the other direction (from Sheet to locale.json).

trkoch · 2020-04-10T21:38:04Z

@michaelmcmillan hub (an extension to git) might be of use for fetching PRs.

michaelmcmillan added 3 commits March 29, 2020 17:09

add script for finding missing translations across all locales

b379cbc

remove unused var

06d1ef0

port lost-in-translation.py to javascript for consistency

5a50bc1

michaelmcmillan requested a review from fossecode as a code owner March 29, 2020 19:45

michaelmcmillan mentioned this pull request Mar 29, 2020

Add script for finding missing translations across all locales #469

Closed

michaelmcmillan requested a review from adriaandotcom March 29, 2020 19:48

michaelmcmillan added 5 commits March 29, 2020 21:49

ported over script usage

004d5e0

remove python script

721769d

delimit language with ---

432870d

determine locales from dir

f70d5a6

upload missing translations to a google sheet

ae00680

michaelmcmillan commented Mar 29, 2020

View reviewed changes

ops/lost-in-translation.js Show resolved Hide resolved

michaelmcmillan added 4 commits March 29, 2020 23:03

empty string as missing

490106c

sync all available sheets

6ecc6fa

add sheet if it doesnt already exist

34e600a

avoid rate limit

e2def1e

michaelmcmillan changed the title ~~Lost in translation script in JavaScript rather than Python~~ Sync missing translation keys in app/locales/*.json files to a Google Sheet Mar 29, 2020

rate limit circumvension

1fe146f

fossecode reviewed Mar 29, 2020

View reviewed changes

prettify comment

99f1c80

fossecode approved these changes Mar 29, 2020

View reviewed changes

refactored code and made it less complex after feedback from @fossecode

d355fbf

fossecode reviewed Mar 29, 2020

View reviewed changes

linted

be1a7e1

sbocinec mentioned this pull request Mar 30, 2020

Sort translation files by keys #493

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync missing translation keys in app/locales/*.json files to a Google Sheet #481

Sync missing translation keys in app/locales/*.json files to a Google Sheet #481

michaelmcmillan commented Mar 29, 2020 •

edited

Loading

fossecode commented Mar 29, 2020 •

edited

Loading

fossecode Mar 29, 2020

michaelmcmillan Mar 29, 2020

fossecode Mar 29, 2020

michaelmcmillan Mar 29, 2020

fossecode Mar 29, 2020

michaelmcmillan Mar 29, 2020

michaelmcmillan commented Mar 29, 2020 •

edited

Loading

fossecode Mar 29, 2020

michaelmcmillan Mar 29, 2020

michaelmcmillan Mar 29, 2020

michaelmcmillan Mar 29, 2020 •

edited

Loading

fossecode Mar 29, 2020

michaelmcmillan Mar 29, 2020

fossecode commented Mar 29, 2020

michaelmcmillan commented Mar 29, 2020

fossecode commented Mar 29, 2020

fossecode commented Mar 29, 2020 •

edited

Loading

fossecode Mar 29, 2020

michaelmcmillan commented Mar 29, 2020

trkoch commented Apr 10, 2020

Sync missing translation keys in app/locales/*.json files to a Google Sheet #481

Are you sure you want to change the base?

Sync missing translation keys in app/locales/*.json files to a Google Sheet #481

Conversation

michaelmcmillan commented Mar 29, 2020 • edited Loading

fossecode commented Mar 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelmcmillan commented Mar 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelmcmillan Mar 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fossecode commented Mar 29, 2020

michaelmcmillan commented Mar 29, 2020

fossecode commented Mar 29, 2020

fossecode commented Mar 29, 2020 • edited Loading

Choose a reason for hiding this comment

michaelmcmillan commented Mar 29, 2020

trkoch commented Apr 10, 2020

michaelmcmillan commented Mar 29, 2020 •

edited

Loading

fossecode commented Mar 29, 2020 •

edited

Loading

michaelmcmillan commented Mar 29, 2020 •

edited

Loading

michaelmcmillan Mar 29, 2020 •

edited

Loading

fossecode commented Mar 29, 2020 •

edited

Loading