fix(supply-chain): derive the mirror path instead of describing a rule for it

The mirror instruction was wrong for the five first-party images. The inventory
lists what you pull — `ghcr.io/simstudioai/app` — but `ghcr.io` is the chart's
default registry, so a redirected install pulls `<registry>/simstudioai/app`.
Following "mirror to <registry>/<reference as listed>" put them at
`<registry>/ghcr.io/simstudioai/...` and every Sim image failed to pull. The
device plugin is the opposite case: its `nvcr.io/` really is part of the
repository and survives the rewrite.

Rather than write that rule out and hope it stays true, the generator now
renders the chart a second time against a sentinel registry and strips it back
off, so each entry carries both the source reference and the exact path the
chart resolves to. Verified by rendering with a real mirror and diffing: all 13
mirror paths match.

Also retries the moving-alias read. A mismatch meant "the guard held this tag
back", but it can equally mean GHCR is still serving the previous digest a
moment after this run wrote it — and that reading dropped three subjects from
the matrix without a word.

Docs, each verified against source:

- Chat drops a disallowed tool while generating a workflow and records the
  omission; a run that reaches one fails. Both statements were needed, and the
  page previously asserted one or the other in different places.
- A quickly-failing CronJob pod is retried six times by Kubernetes; the
  five-minute deadline only preempts one that hangs.
- Selecting a sandbox without a provider is a 503; the 422 is the file path.
- Chat's public code tool is run_function, not the internal function_execute.
- The parity step needs SANDBOX_PROVIDER=e2b, or an exported daytona provider
  makes step 1 exit before writing the manifest.
- PII_URL is auto-wired by the chart and needs a service URL on Compose.
- NO_PROXY should use the cluster suffix; the Service names depend on the
  release name.
- The bucket loops now fail loudly instead of silently iterating nothing when
  SIM_BUCKETS is unset in a fresh shell.
- Generating CRON_SECRET beside the curl suggested it was the token to use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt

Author: waleedlatif1

.github/workflows/ci.yml

@@ -746,7 +746,18 @@ jobs:
             fi
 
             for alias in $alias_groups; do
-              alias_index="$(digest_of "${image}:${alias}")"
+              # A mismatch has two very different causes: the guard deliberately
+              # held the tag back, or GHCR is still serving the previous digest
+              # moments after this run wrote it. Re-read before concluding the
+              # former, or a read landing a second early silently drops three
+              # subjects from the matrix.
+              alias_index=""
+              for alias_attempt in 1 2 3; do
+                alias_index="$(digest_of "${image}:${alias}")"
+                [ "$alias_index" = "$sha_index" ] && break
+                [ "$alias_attempt" -lt 3 ] && sleep 5 || true
+              done
+
               if [ "$alias_index" != "$sha_index" ]; then
                 # `latest` is legitimately allowed to lag: the monotonic guard in
                 # create-ghcr-manifests holds it back when the branch has moved

apps/docs/content/docs/platform/enterprise/access-control.mdx

@@ -27,7 +27,7 @@ Assignment-time checks keep this unambiguous: a workspace has at most one all-me
 When a user runs a workflow or uses Chat, Sim reads the resolved group's configuration and applies it:
 
 - **In the executor:** If a workflow uses a disallowed block type or model provider, execution halts immediately with an error. This applies to both manual runs and scheduled or API-triggered deployments.
-- **In Chat:** Disallowed blocks are filtered out of the block list so they cannot be added to a workflow. Disallowed tool types (MCP, custom tools, skills) make the block fail with an error if Sim attempts to use them — they are refused, not silently dropped.
+- **In Chat:** Disallowed blocks are filtered out of the block list so they cannot be added to a workflow. Disallowed tool types (MCP, custom tools, skills) are dropped while Sim generates a workflow, with the omission recorded. If a run reaches one anyway, the block fails with an error naming the restriction rather than skipping it.
 
 ---
 
@@ -209,7 +209,7 @@ This applies regardless of how the workflow is triggered — manually, via API,
 When a user opens Chat, their permission group is read before any block or tool suggestions are made:
 
 - Blocks not in the allowed list are filtered out of the block picker entirely — they do not appear as options.
-- If Sim generates a workflow step that would use a disallowed tool (MCP, custom, or skills), the run fails with an error naming the restriction.
+- If Sim generates a workflow step that would use a disallowed tool (MCP, custom, or skills), the tool is left out of the generated workflow. A run that reaches a disallowed tool fails with an error naming the restriction.
 
 ---
 

apps/docs/content/docs/platform/enterprise/data-retention.mdx

@@ -199,8 +199,8 @@ Neither shipped deployment schedules these three endpoints — not the Helm char
 </Callout>
 
 ```bash
-CRON_SECRET=$(openssl rand -hex 32)
-
+# Use the value already configured as CRON_SECRET on the app — a token
+# generated here and not installed there returns 401.
 curl -H "Authorization: Bearer $CRON_SECRET" \
   https://sim.example.com/api/logs/cleanup
 ```

apps/docs/content/docs/platform/self-hosting/background-jobs.mdx

@@ -71,7 +71,7 @@ cronjobs:
   enabled: true
 ```
 
-Each job runs a small `curlimages/curl` pod that calls the app's **in-cluster Service** (not the ingress), with `concurrencyPolicy: Forbid` so a slow run never overlaps itself. The Service load-balances that single call to one replica, so a job never fans out across pods. The chart keeps three completed and one failed Job per CronJob, but `ttlSecondsAfterFinished` defaults to 600, so a finished Job is deleted ten minutes after it ends — inspect a failure promptly or collect the logs. It sets no `backoffLimit`, but `cronjobs.activeDeadlineSeconds` (default `300`) terminates the whole Job after five minutes, and the pod's own script already retries three times inside that window — so a failing job gets roughly one pod attempt and is then marked `DeadlineExceeded`, not six retries.
+Each job runs a small `curlimages/curl` pod that calls the app's **in-cluster Service** (not the ingress), with `concurrencyPolicy: Forbid` so a slow run never overlaps itself. The Service load-balances that single call to one replica, so a job never fans out across pods. The chart keeps three completed and one failed Job per CronJob, but `ttlSecondsAfterFinished` defaults to 600, so a finished Job is deleted ten minutes after it ends — inspect a failure promptly or collect the logs. It sets no `backoffLimit`, so a pod that exits quickly — an immediate `401`, say — is retried up to the Kubernetes default of six times. A pod that hangs instead is cut off by `cronjobs.activeDeadlineSeconds` (default `300`), which caps the whole Job at five minutes and marks it `DeadlineExceeded`. Set `backoffLimit: 0` if you want exactly one attempt.
 
 Disable individual jobs you do not need — billing reconciliation is the obvious one on a self-hosted install:
 

apps/docs/content/docs/platform/self-hosting/environment-variables.mdx

@@ -199,7 +199,7 @@ See [Observability](/platform/self-hosting/observability).
 |----------|-------------|
 | `COPILOT_API_KEY` | API key for Chat. Without it the Sim Chat block, scheduled prompt jobs, and Inbox cannot run |
 | `NEXT_PUBLIC_CHAT_DISABLED` | Set to `true` to hide the Chat module: the workspace lands on your first workflow, with no chats list, scheduled tasks, or editor Chat panel. Chat is shown when unset; `npx sim-setup` sets it for you if you skip the chat key |
-| `PII_URL` | Base URL of the Presidio service backing PII detection and redaction. Defaults to `http://localhost:5001`, which exists only in local development — set it to your own Presidio deployment |
+| `PII_URL` | Base URL of the Presidio service backing PII detection and redaction. The Helm chart wires it to its own `pii` Service when `pii.enabled`; on Compose point it at the PII service on your network. The default `http://localhost:5001` exists only in local development, and leaving it makes redaction fail |
 | `DURABLE_SECRET_PROVENANCE_ENFORCED_SURFACES` | Durable stores where a value whose secret provenance was never recorded fails the run instead of logging a warning. `all`, or a comma-separated subset of `memory`, `table-row`, `knowledge`, `workspace-file`. Unset (nothing enforced) by default |
 | `ADMIN_API_KEY` | Admin API key for GitOps operations and organization provisioning |
 

apps/docs/content/docs/platform/self-hosting/object-storage.mdx

@@ -50,7 +50,7 @@ export SIM_BUCKETS="workspace-files knowledge-base execution-files chat-files
                     copilot-files profile-pictures og-images workspace-logos"
 
 # Create buckets (names must be globally unique — prefix with your org)
-for name in $SIM_BUCKETS; do
+for name in ${SIM_BUCKETS:?run the export above first}; do
   aws s3api create-bucket \
     --bucket "myorg-sim-$name" \
     --region "$AWS_REGION" \
@@ -86,7 +86,7 @@ cat > /tmp/cors.json <<'EOF'
 }
 EOF
 
-for name in $SIM_BUCKETS; do
+for name in ${SIM_BUCKETS:?run the export above first}; do
   aws s3api put-bucket-cors --bucket "myorg-sim-$name" --cors-configuration file:///tmp/cors.json
 done
 ```
@@ -280,7 +280,7 @@ export SIM_BUCKETS="workspace-files knowledge-base execution-files chat-files
                     copilot-files profile-pictures og-images workspace-logos"
 
 # Create buckets (names must be globally unique — prefix with your org)
-for name in $SIM_BUCKETS; do
+for name in ${SIM_BUCKETS:?run the export above first}; do
   gcloud storage buckets create "gs://myorg-sim-$name" \
     --project "$PROJECT_ID" \
     --location "$LOCATION" \
@@ -318,7 +318,7 @@ cat > /tmp/cors.json <<'EOF'
 ]
 EOF
 
-for name in $SIM_BUCKETS; do
+for name in ${SIM_BUCKETS:?run the export above first}; do
   gcloud storage buckets update "gs://myorg-sim-$name" --cors-file=/tmp/cors.json
 done
 ```
@@ -338,7 +338,7 @@ Create a service account (or reuse the one your workload runs as) and grant it o
 ```bash
 gcloud iam service-accounts create sim-storage --project "$PROJECT_ID"
 
-for name in $SIM_BUCKETS; do
+for name in ${SIM_BUCKETS:?run the export above first}; do
   gcloud storage buckets add-iam-policy-binding "gs://myorg-sim-$name" \
     --member "serviceAccount:sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
     --role roles/storage.objectAdmin

helm/sim/images.yaml

@@ -10,24 +10,38 @@
 # their public registries. Pin each image's `digest` to what your mirror
 # resolved.
 #
-# Mirror to `<your-registry>/<reference exactly as listed below>`. The chart
-# prepends your registry to the repository verbatim, so the device plugin lands
-# at `<your-registry>/nvcr.io/nvidia/k8s-device-plugin` — the `nvcr.io/`
-# segment is part of the path, not a source host to strip. If your registry
-# cannot nest that path, override the one image directly instead:
-# `ollama.gpu.devicePlugin.image.repository=<your-registry>/nvidia/k8s-device-plugin`.
+# Each entry pairs the reference to pull FROM with the path the chart resolves
+# to once redirected. Copy `source` to `<your-registry>/<mirror>` — they differ:
+# `ghcr.io/` is the default registry and is replaced, while the device plugin's
+# `nvcr.io/` is part of its repository and is kept. If your registry cannot nest
+# that path, override that one image:
+# `ollama.gpu.devicePlugin.image.repository=<your-registry>/nvidia/k8s-device-plugin`
+# with global rewriting left on, since the override is used verbatim.
 appVersion: v0.8.18
 images:
-  - busybox:1.36
-  - curlimages/curl:8.5.0
-  - ghcr.io/simstudioai/copilot:v0.8.18
-  - ghcr.io/simstudioai/migrations:v0.8.18
-  - ghcr.io/simstudioai/pii:v0.8.18
-  - ghcr.io/simstudioai/realtime:v0.8.18
-  - ghcr.io/simstudioai/simstudio:v0.8.18
-  - nvcr.io/nvidia/k8s-device-plugin:v0.18.2
-  - ollama/ollama:0.23.2
-  - otel/opentelemetry-collector-contrib:0.91.0
-  - pgvector/pgvector:pg17
-  - postgres:17-alpine
-  - redis:7-alpine
+  - source: busybox:1.36
+    mirror: busybox:1.36
+  - source: curlimages/curl:8.5.0
+    mirror: curlimages/curl:8.5.0
+  - source: ghcr.io/simstudioai/copilot:v0.8.18
+    mirror: simstudioai/copilot:v0.8.18
+  - source: ghcr.io/simstudioai/migrations:v0.8.18
+    mirror: simstudioai/migrations:v0.8.18
+  - source: ghcr.io/simstudioai/pii:v0.8.18
+    mirror: simstudioai/pii:v0.8.18
+  - source: ghcr.io/simstudioai/realtime:v0.8.18
+    mirror: simstudioai/realtime:v0.8.18
+  - source: ghcr.io/simstudioai/simstudio:v0.8.18
+    mirror: simstudioai/simstudio:v0.8.18
+  - source: nvcr.io/nvidia/k8s-device-plugin:v0.18.2
+    mirror: nvcr.io/nvidia/k8s-device-plugin:v0.18.2
+  - source: ollama/ollama:0.23.2
+    mirror: ollama/ollama:0.23.2
+  - source: otel/opentelemetry-collector-contrib:0.91.0
+    mirror: otel/opentelemetry-collector-contrib:0.91.0
+  - source: pgvector/pgvector:pg17
+    mirror: pgvector/pgvector:pg17
+  - source: postgres:17-alpine
+    mirror: postgres:17-alpine
+  - source: redis:7-alpine
+    mirror: redis:7-alpine

scripts/generate-image-manifest.test.ts

@@ -1,5 +1,5 @@
 import { describe, expect, it } from 'vitest'
-import { collectImages, renderManifest } from './generate-image-manifest'
+import { collectImages, pairSources, renderManifest } from './generate-image-manifest'
 
 describe('collectImages', () => {
   it('finds images across every container key a pod spec can use', () => {
@@ -71,25 +71,61 @@ describe('collectImages', () => {
   })
 })
 
+describe('pairSources', () => {
+  it('strips the sentinel registry so the mirror path is what the chart resolves to', () => {
+    const paired = pairSources(
+      ['ghcr.io/simstudioai/simstudio:v1', 'redis:7-alpine'],
+      ['mirror.invalid/simstudioai/simstudio:v1', 'mirror.invalid/redis:7-alpine']
+    )
+
+    // ghcr.io is the default registry and is replaced, so it must not survive.
+    expect(paired).toEqual([
+      { source: 'ghcr.io/simstudioai/simstudio:v1', mirror: 'simstudioai/simstudio:v1' },
+      { source: 'redis:7-alpine', mirror: 'redis:7-alpine' },
+    ])
+  })
+
+  it('keeps a registry host that is part of the repository', () => {
+    const paired = pairSources(
+      ['nvcr.io/nvidia/k8s-device-plugin:v0.18.2'],
+      ['mirror.invalid/nvcr.io/nvidia/k8s-device-plugin:v0.18.2']
+    )
+
+    expect(paired[0].mirror).toBe('nvcr.io/nvidia/k8s-device-plugin:v0.18.2')
+  })
+
+  it('fails rather than emitting a half-paired inventory when the renders disagree', () => {
+    expect(() => pairSources(['redis:7-alpine'], [])).toThrow(/two renders disagree/)
+  })
+})
+
 describe('renderManifest', () => {
-  it('renders the app version and a sorted image list', () => {
+  it('renders the app version and each image as a source/mirror pair', () => {
     const manifest = renderManifest({
       appVersion: 'v0.8.18',
-      images: ['busybox:1.36', 'redis:7-alpine'],
+      images: [{ source: 'ghcr.io/simstudioai/simstudio:v1', mirror: 'simstudioai/simstudio:v1' }],
     })
 
     expect(manifest).toContain('appVersion: v0.8.18')
-    expect(manifest).toContain('  - busybox:1.36\n  - redis:7-alpine\n')
+    expect(manifest).toContain(
+      '  - source: ghcr.io/simstudioai/simstudio:v1\n    mirror: simstudioai/simstudio:v1'
+    )
   })
 
   it('omits the chart version so a chart-only bump does not fail the check', () => {
-    const manifest = renderManifest({ appVersion: 'v1', images: ['a:1'] })
+    const manifest = renderManifest({
+      appVersion: 'v1',
+      images: [{ source: 'a:1', mirror: 'a:1' }],
+    })
 
     expect(manifest).not.toContain('chartVersion')
   })
 
   it('ends with a trailing newline so the checked-in file is POSIX-clean', () => {
-    const manifest = renderManifest({ appVersion: 'v1', images: ['a:1'] })
+    const manifest = renderManifest({
+      appVersion: 'v1',
+      images: [{ source: 'a:1', mirror: 'a:1' }],
+    })
 
     expect(manifest.endsWith('\n')).toBe(true)
     expect(manifest.endsWith('\n\n')).toBe(false)

scripts/generate-image-manifest.ts

@@ -50,6 +50,28 @@ const RENDER_ARGS = [
   'ollama.gpu.enabled=true',
 ] as const
 
+/**
+ * Sentinel registry used for the second render.
+ *
+ * The path an operator must mirror TO is not derivable from the pullable
+ * reference: `global.imageRegistry` defaults to `ghcr.io`, so a Sim image reads
+ * `ghcr.io/simstudioai/app` but redirects to `<registry>/simstudioai/app` — the
+ * `ghcr.io/` is the default registry, not part of the repository. The NVIDIA
+ * plugin is the reverse: `nvcr.io/` IS in its repository and survives the
+ * rewrite. Rendering a second time with a known registry and stripping it back
+ * off yields the real destination for every image without encoding that rule
+ * twice.
+ */
+const MIRROR_SENTINEL = 'mirror.invalid'
+
+const MIRROR_RENDER_ARGS = [
+  ...RENDER_ARGS,
+  '--set',
+  `global.imageRegistry=${MIRROR_SENTINEL}`,
+  '--set',
+  'global.useRegistryForAllImages=true',
+] as const
+
 /** Pod-spec keys whose entries carry an image reference. */
 const CONTAINER_KEYS = new Set(['containers', 'initContainers', 'ephemeralContainers'])
 
@@ -87,8 +109,43 @@ export function collectImages(documents: readonly unknown[]): string[] {
 }
 
 /** Renders the checked-in manifest. */
-export function renderManifest(input: { appVersion: string; images: readonly string[] }): string {
-  const entries = input.images.map((image) => `  - ${image}`).join('\n')
+export interface ImageEntry {
+  /** The reference to pull from today. */
+  source: string
+  /** The path the chart pulls once redirected, relative to your registry. */
+  mirror: string
+}
+
+/**
+ * Pairs each pullable reference with the repository path the chart resolves to
+ * under a mirror, by stripping the sentinel registry off the second render.
+ */
+export function pairSources(sources: readonly string[], mirrored: readonly string[]): ImageEntry[] {
+  const byTag = new Map<string, string>()
+  for (const ref of mirrored) {
+    const stripped = ref.startsWith(`${MIRROR_SENTINEL}/`)
+      ? ref.slice(MIRROR_SENTINEL.length + 1)
+      : ref
+    byTag.set(stripped.slice(stripped.lastIndexOf('/') + 1), stripped)
+  }
+
+  return sources.map((source) => {
+    const key = source.slice(source.lastIndexOf('/') + 1)
+    const mirror = byTag.get(key)
+    if (!mirror) {
+      throw new Error(`No mirrored path rendered for ${source}; the two renders disagree`)
+    }
+    return { source, mirror }
+  })
+}
+
+export function renderManifest(input: {
+  appVersion: string
+  images: readonly ImageEntry[]
+}): string {
+  const entries = input.images
+    .map((image) => `  - source: ${image.source}\n    mirror: ${image.mirror}`)
+    .join('\n')
 
   return `# Generated by \`bun run images:generate\`. Do not edit this file directly.
 #
@@ -102,12 +159,13 @@ export function renderManifest(input: { appVersion: string; images: readonly str
 # their public registries. Pin each image's \`digest\` to what your mirror
 # resolved.
 #
-# Mirror to \`<your-registry>/<reference exactly as listed below>\`. The chart
-# prepends your registry to the repository verbatim, so the device plugin lands
-# at \`<your-registry>/nvcr.io/nvidia/k8s-device-plugin\` — the \`nvcr.io/\`
-# segment is part of the path, not a source host to strip. If your registry
-# cannot nest that path, override the one image directly instead:
-# \`ollama.gpu.devicePlugin.image.repository=<your-registry>/nvidia/k8s-device-plugin\`.
+# Each entry pairs the reference to pull FROM with the path the chart resolves
+# to once redirected. Copy \`source\` to \`<your-registry>/<mirror>\` — they differ:
+# \`ghcr.io/\` is the default registry and is replaced, while the device plugin's
+# \`nvcr.io/\` is part of its repository and is kept. If your registry cannot nest
+# that path, override that one image:
+# \`ollama.gpu.devicePlugin.image.repository=<your-registry>/nvidia/k8s-device-plugin\`
+# with global rewriting left on, since the override is used verbatim.
 appVersion: ${input.appVersion}
 images:
 ${entries}
@@ -121,10 +179,10 @@ function readChartField(chart: string, field: string): string {
   return match[1].trim()
 }
 
-function renderChart(): unknown[] {
+function renderChart(args: readonly string[] = RENDER_ARGS): unknown[] {
   let rendered: ReturnType<typeof Bun.spawnSync>
   try {
-    rendered = Bun.spawnSync(['helm', ...RENDER_ARGS], { cwd: ROOT })
+    rendered = Bun.spawnSync(['helm', ...args], { cwd: ROOT })
   } catch {
     throw new Error('Could not run `helm`. Install the Helm CLI to regenerate the image inventory.')
   }
@@ -141,7 +199,10 @@ async function main(): Promise<void> {
   const chart = await readFile(resolve(CHART_DIR, 'Chart.yaml'), 'utf8')
   const manifest = renderManifest({
     appVersion: readChartField(chart, 'appVersion'),
-    images: collectImages(renderChart()),
+    images: pairSources(
+      collectImages(renderChart()),
+      collectImages(renderChart(MIRROR_RENDER_ARGS))
+    ),
   })
 
   if (!CHECK_MODE) {