How should I ... do ... robots.txt? Sitemap?

[kylehodgson]

I spent some time on a quick node script that pulls in paths entries and my dynamicPaths generator function from observable.config.jsand hands them to npm'ssitemap` library. Now I have a nice little sitemap.xml, and I'd like to reference it in my robots.txt. Which is easy! Except, how do I deploy either of these?

I saw documentation somewhere about adding "external" files not managed by observable, and I did add this to head in observable.config.js:

<a href="/robots.txt" rel="external">robots.txt</a>
<a href="/robots.txt" rel="external">sitemap.xml</a>

npm run build doesn't copy robots.txt to the dist folder. I can do this as part of the CI pipeline of course - but I wanted to see if I was just missing something?

[Fil]

One way to do this is to use the dynamicPaths option to add references to files that must be built at a given URL. For example, see https://github.com/Fil/pangea/blob/c7b2ee2fec2aead7b2741e7256556c16b4c04ebc/observablehq.config.ts#L16C17-L16C29

See #1199 for a related discussion.