What would be the recommended implementation for internationalization

[tiniscule]

Looking at implementing this, but we leverage Next's built in internationalization support - so when creating content for a blog, would need to be able to specify the locale en or fr for example, and then when pulling content paths/body's back out would need to be able to specify it again there.

End result is the ability to publish English articles to the US locale, French to the FR locale.

Any ideas on how that might be possible with the existing implementation?

[avitorio]

Hi @tiniscule,

For now there's no support for i18n, and I haven't given much thought about how to implement it. But here are a couple of quick suggestions:

• Creating two collections, one for English, and a second for French. Then adding the logic to pull from one or another in Next.js.

• Alternatively, you could also just create different slugs for each language ex: post-title-en, post-title-fr, and filter slugs on the frontend.

The project is still in its infancy and there are a lot of features on the roadmap. Internationalization should come at some point in the near future.

[tiniscule]

Great, thanks for getting back with me. I'm going to play around with it a bit. Are you open to a PR request if I come up with something useful?

[avitorio]

Yes, definitely. Before creating the PR, link the fork here with a description about the approach you are taking. Then I can take a look at it and make sure it aligns with our vision for the framework. 👍🏼

[DCzajkowski]

We used to use Netlify CMS. However, we have migrated to Vercel since and are looking for alternatives. Outstatic looks great, but we need i18n support.

Setup

1. Add a new environment variable:

OST_LOCALES=en,pl # Specify supported locales with a comma-separated list. If empty, localization is not supported.

2. Add a locale to the file name between the slug and extension (also known as structure: multiple_files from Netlify CMS):

outstatic/
  content/
    posts/
      post-1.en.md
      post-1.pl.md
      post-2.en.md
      post-2.pl.md

Changes in the UI

1. Add a language switcher in the collection list. This would allow to preview the list with titles in specific locale.

2. Add a side-by-side editor, or a locale switcher in the editor, to allow editing in multiple languages. Notify before saving if one of the locales has no content.

[avitorio]

Hi @DCzajkowski thanks for the suggestions.

I am not familiar with Netlify CMS, but should definitely check it out to try getting some ideas.

The concept you are proposing is cool, although I have a couple of concerns.

First, I think i18n is something a minority of users will require. And for this reason, the feature and UI changes, should only be available to those who decide to add the OST_LOCALE, environment variable.

Second, as of now, we don't have pagination/sorting inside the dashboard, which is something I definitely want to add in the future.
I am thinking one of the possible solutions for that would be having an id as the file name instead of the slug, or a combination of both. That's why I'm a bit hesitant to making changes to filenames or creating patterns for other usages.

As for the side-by-side editor. I think this one would be tricky to implement and would require some decent refactoring of how our editor currently works, but it could be done.

That being said, what I ideally would like to do with Outstatic, is to create an architecture that would allow extensions/plugins. So that these features can be added on a per install basis and not be part of the core system.

These are my 2 cents. Let me know what you think.

[tiniscule]

I think one of my concerns with the Netlify approach is that it assumes translation instead of localization a bit (one-to-one posts across languages). I'd also be a bit concerned with impact to the existing interface - though I don't necessarily agree that it's a small use case. It always seems that way until you hit it, which is why you'll notice Next advertises it first class on their features.

The approach i was thinking was something along the lines of:

• Load defined locales from existing Next installation next.config.js
• Add frontmatter support for locale on individual posts
• If locales enabled in next install, show locale selector in post interface with a default to the configured next locale in next.config.js
• Update the getDocuments and getDocumentsBySlug functions to take optional filters, one of which would be locale. Just loads documents with added frontmatter filters

The thinking here is that all of this fails gracefully when not in use, but is available when needed. Also shares the majority of the arch you'd expect for something like tags or other filters down the road.

[avitorio]

Thanks everyone for the suggestions. I've been following this discussion and played with the Next.js i18n demo. I think we are moving in the right direction.

I like @jakobo's approach. It feels like the content folder would get less cluttered in time with it.

However, I have a couple of remarks, and perhaps this is just how Next.js handles things, but what happens to localized slugs? For example, in english it makes sense to have mywebsite.com/about-me, but in portuguese it should probably be mywebsite.com/pt/sobre-mim. Any thoughts?

Of course, the first implementation of this feature doesn't need to be perfect and I'd be happy with pushing something that makes it work even if slugs remain the same as the default language.

[thecodedrift]

Localized slugs would be a whole other problem. To do this, you'd probably need IDs instead of slugs to identify individual outstatic posts. It's a system used by drupal and other CMSes with a node/leaf pattern, where everything after the ID is for SEO purposes.

example.com/posts/1/about-me slug is for SEO purposes only and 304 to the correct slug if wrong/ommitted
example.com/posts/en/1/about-me canonical URL maps back to the default locale
example.com/posts/pt/1/sobre-mim pt locale, "sobre-mim" slug is SEO and can 304 to the correct slug if it changes

/
  outsatatic/
    posts/
      1/ # <= the id
        metadata.json # <= common information about post 1
        en.md # <= from next locale
        pt.md

It's going to be a tradeoff, either with the slug about-me as the identifying marker for the data or some other ident such as an integer or timestamp to tell outstatic where something is.

[tiniscule]

yeah, localized slugs and the back and forth here is why I'm not a huge fan of being so prescriptive in the format (slug/[locale]).

I'd rather just be able to create different markdown files and set the locale per-post. I always end up needing more flexibility than the formulaic approaches allow, and as a customer I find content translated 1:1 isn't very good anyways.

[thecodedrift]

Since we're discussing / exploring, maybe the core technology is just sub-collection / subfolder routing. This doesn't depend on next.js' routing layer anymore, and the people can structure the content in a way that makes sense for their mental model. The structure would look something like:

outstatic/
  posts/ # <= collection
    about-me.md # example.com/blog/about-me
    pt/ # <= sub-collection
      sobre-mim.md # example.com/blog/pt/sobre-mim

In this model, you have no system-level way of knowing those two pieces of content are related, but that may not be an outstatic concern since we have access to frontmatter. We can suggest in the docs ways to map the locale into getDocumentBySlug for those who want a more seamless next.js/outstatic solution.

const fields = [
  'title',
  'publishedAt',
  'slug',
  'author',
  'content',
  'coverImage'
];

// try locale, fall back to non-locale. Completely userland implemented
const post = getDocumentBySlug(`posts/${locale}`, params.slug, fields)
  ?? getDocumentBySlug(`posts`, params.slug, fields)

The other places this becomes tricky is for things like indexes (posts by tags / categories, or chronological listings) but I think with good use of frontmatter we can sidestep those concerns for all but the largest of sites where the filesystem becomes a liability.

[avitorio]

I like the idea of an id based system. I've considered it before when thinking about other features, such as pagination. Although the simplicity of keeping everything in slug.md files is great, ids can give us more flexibility in the long term and allow for an efficient way to sort and organize documents. The permalink structure example.com/post-name, or example.com/fr/nom-du-post might still be maintained if the right logic is placed inside getStaticPaths.

I've even considered adding SQLite with something like sql.js for storing collection or post specific relation/data. Still have to think a bit more about what this would entail. But that's another discussion.

Meanwhile, I've thought about a weird approach for i18n... hear me out.

What if, for outstatic installs where locales are set, posts had additional lang and locale fields, ex:

slug: good-morning
lang: en
locales:
  fr: bonjour
  pt: bom-dia

Then on bonjour.md:

slug: bonjour
lang: fr
locales:
  en: good-morning
  pt: bom-dia

In the UX we'd have a language dropdown that lets users pick the language for the current document (for quick switching) and also set the corresponding slugs for other locales.

When a user defines a new locale and clicks save, the commit would look for the specific slugs and add the relation back to the current slug being edited. The folder structure would look like this:

/posts/good-morning.md # regular file for non-i18n setups
/posts/fr/bonjour.md
/posts/pt/bom-dia.md

This way we can still filter by language inside the documents list page or in the front end.
I know this is going in an entirely new direction and it may require more file changes when a locale slug is specified, but the upside is having localized slugs and also a relation between documents (admittedly not in the most elegant way).

[thecodedrift]

I've even considered adding SQLite with something like sql.js for storing collection or post specific relation/data. Still have to think a bit more about what this would entail. But that's another discussion.

I think you've got a solid idea w/ the filesystem as the source of truth and agree that's probably beyond the current scope. Plus, once the content is no longer readable / editable text files, the value of outstatic living in GitHub is diminished.

What if, for outstatic installs where locales are set, posts had additional lang and locale fields, ex:

lang: fr is good. locales probably not. Having filterable properties will definitely be a requirement for developers displaying lists, and the language property is easy to understand. I'd avoid adding relational data into the frontmatter at the outstatic level; that feels like an advanced use case that is a developer decision. (Specifically, it becomes much more error-prone to do file-system edits and break your install by forgetting to update a file in a local commit.)

To @tiniscule's comment, the least prescriptive solution is likely just subfolder support. We even have that proposed with /posts/fr/bonjour.md, just fr is a decision by the author, not the framework. Then, it doesn't matter to outstatic that you have fr/posts/slug or posts/fr/slug, you're free to arrange the content in a way that fits your use case. This also aligns with the original suggestion "Creating two collections, one for English, and a second for French. Then adding the logic to pull from one or another in Next.js" which is probably the easiest to explain to someone new to outstatic.

It also means developers are free to opt in to Next's locale support or control the experience manually.

[thecodedrift]

Hey all, chatted a bit with @avitorio about how we get to i18n, and wanted to share the current thinking with everyone. This will be a bit of a long reply since it includes what you can do today, some of the prerequisite technology & features we need for a seamless i18n experience, and some thoughts about how it will hypothetically work once some of the other infrastructure is in place.

The today case: 1 Collection Per Language

This is the original proposal by avitorio in this discussion. It's not an ideal solution, but it is forward compatible. Using this method, you will only be able to have a single content "type", but you can begin using it today for internationalization if your goal is an i18n-backed blog or other single-content-type.

• Create a collection for each language (en, fr, etc)
• Write outstatic content as you normally would
• Manually link between versions if you'd like to tell people about the post being available in another language
  • (developer alternative) Use the next.js locale to automatically select the "collection"

The remainder of this post is about the path forward, so for those who need a quick answer, you're set. 🎉

---

Looking Forward

Outstatic's strength is the limited primitives that exist. At it's core, Outstatic is a filesystem CMS backed by git and GitHub and with getStaticProps, it is closer on the spectrum of CMSes to 11ty or gatsby than it is to Strapi or Wordpress. This is useful in framing features, because we can talk about something as complex as i18n in terms of folders, files (frontmatter and markdown), and git. Adding something that's not a folder, a file, or a git operation is beyond this CMS; not because it isn't important, but because focusing on strengths make it easier for developers to select the best tool for the job at hand.

Using these terms, internationalization isn't a folder structure, it's metadata about a file. In a given collection, this post happens to be in french fr, and its parent is slug xyz-abc. The same infrastructure that will let us filter and sort posts should also filter and sort by metadata in these posts. Instead of getDocumentBySlug, developers should have a richer query language for getting a document's content. Getting there will require a few core features which benefit more than just i18n.

For each of these steps, there's also a bulleted list of the kinds of things you can do w/ each feature in addition to the eventual goal of i18n content.

Feature: Custom Fields (near future)

Custom fields allow you to add additional structured data to a post's frontmatter. The current thinking is that each collection has an optional schema.json that lets you define additional custom fields for your posts. With even a few custom fields such as date and string, you can expose a large number of customizations in Outstatic

• co-authored posts, credits (thanks to)
• last updated timestamps instead of just publishing times
• SEO information (open graph images or additional meta keywords)

avitorio's been working on this, and it also happens to be a good first step towards i18n. Using even these basic primitives, we can express what locale a post is written for by adding a locale field to the metadata. This helps, but we still can't query/filter by that data, which brings us to the next feature...

Feature: Metadata "DB" (future)

Metadata in individual posts is great, but it isn't useful at scale for querying. With even a few dozen files, reading each off the FS and doing a yaml parse on the frontmatter (per lambda) becomes very cumbersome. Categories, tags, and even i18n benefit from a queryable data store. Instead, it's useful on save to sync the post's frontmatter into some DB-like solution and provide a queryable interface that can be used in getStaticProps

• Sort (by published, updated, etc)
• Filter (by tag, category, cross-collection, locale, any frontmatter)
• Primitive search (by funneling raw metadata into something like fuzzy or fuse)

Because post files remain the source of truth, this db-like metadata system is altered, regenerated, or rebuilt on saves and commits. Ideally it's a single file or a single file per collection, following the custom fields design. Adding this when custom fields are done will make a "true" i18n design possible since custom metadata (ie locale) can now be filtered and sorted upon. Initially, this metadata sync will probably be just hardcoded in as part of saving a post with a UI button for regenerating the DB if things get out of sync. In an ideal world, the metadata sync is one of the final steps in...

Feature: Lifecycle Hooks & Plugins (far far future... like v2 future)

This isn't a hard requirement for i18n, as we can get a basic i18n implementation with the above two features. However, avitorio would love to eventually support plugins, and I agree; the ability to quickly add 3rd party comment systems such as disqus or webmentions, trackbacks, contact forms, etc is incredibly powerful. However, if we'd like to maintain relations between a primary post and its secondary translated posts, we'd want a more modular way to do this

• Keyword extractor - automatically pull the best keywords out for SEO, add to post metadata
• New post mail - Send an email via 3rd party service and notify your readers of new content
• Agolia push - Sync posts to 3rd party service for even better search tools

Existing outstatic features that are currently hardcoded can be added to these lifecycle hooks or even swapped out if someone has a custom install that needs really elaborate things. It would also allow us to split i18n off into a dedicated plugin which folks could enable (adding a required locale field, building relationships between translations of a post, etc). This also reduces the maintenance surface area of Outstatic; by the time we're building this as a community, we'll probably also have a lot of things we'd like to move out of the core.

Getting There

Of course, in addition to these 2-3 features, we also need to improve our testing infrastructure, add some CI/CD work, and have confidence that these features won't break or corrupt existing installs. That's where everyone can truly help. avitorio will have a pinned issue soon with some roadmap work, and I know that testing is top of mind for him. If you can identify areas where we can add tests and build robustness into outstatic, those PRs would be massively appreciated. ❤️

[ahaasler]

Hello everyone,

The answer from @jakobo is really promising, but I feel we are forgetting about i18n for Outstatic itself, not just the content. This is also related to #54 because we need to identify and properly separate logic from what is presented in the UI.

Some sites may not be multilingual, but may be in another language different than English, so Outstatic should be able to be configured to display all the functionality in that language. We should also consider that the UI may be in a multilingual site and the how that affects the interface. Should an English content mantainer only see English content in the English UI, i think not. That should be the default, but someone may need to update some external link in all language versions for the same collection entry. And not force that user to press Spanish buttons for the Spanish UI.

Has someone experience in that kind of i18n?

[thecodedrift]

How to internationalize the UI is a valid concern. Tools such as https://react.i18next.com or React-Intl can simplify the process greatly; the work can be done in parallel, and hopefully someone can put together a research discussion like we've done with testing and the metadata db. That first step helps clarify thoughts around the problem space.

Regarding how the ui impacts content authoring, I agree that by default, any editor can see all content. Post i18n, we can add support for searching / filtering content. This would make it easier for an editor to change something in all posts with a specific set of metadata.

[ahaasler]

Nice, I can start researching those tools, look a bit more at the current state of the code and when ready start the discussion.