We now have the response header that injects the following into every response:

x-robots-tag: noindex, nofollow

found in this PR:

#74

However, obviously some publishers will want their sites indexed and crawled so we just need a simple setting to turn this on (for testing, blocking LLMs, etc) or off (for prod, open web traffic)