flyte-deploy-kind: local kind + R2 deploy needs task-pod storage creds, runs.storagePrefix override, and console-origin fix

[cosmicBboy]

Summary

While deploying a local kind-based Flyte v2 (flyte-binary v2.0.27) cluster on macOS using the flyte-deploy-kind skill, the control plane came up healthy and the API (ListProjects) worked, but task execution and the web console did not work out of the box. Each issue was diagnosable but not covered by the skill runbook. Filing to suggest concrete additions.

Deployment shape: kind on the local machine + Supabase Postgres (session pooler) + Cloudflare R2 object store.

---

Issues faced (in the order they surfaced)

1. Web console can't reach the API — console and API are separate origins

Port-forwarding flyte-console directly (:8080) serves only the Next.js SPA. The console frontend resolves its API base URL as NEXT_PUBLIC_ADMIN_API_URL || "/", and that env var is unset, so the browser calls the API at the same origin it was served from (localhost:8080/flyteidl2.*) — which 404s because :8080 doesn't serve the API. Result: run pages load blank / "connection refused" on the URL the SDK prints (http://localhost:8080/v2/...).

What actually works: put console + API behind one origin. Traefik (installed by the auth flow) with two IngressRoutes — PathPrefix('/flyteidl2.') → flyte-http:8090 (scheme h2c) and / → flyte-console:80 — then kubectl -n traefik port-forward service/traefik 8080:80 and open http://localhost:8080/v2.

2. Task pods have no object-store credentials → fall back to AWS IMDS

Tasks fail with:

OSError: Generic S3 error: Error performing PUT http://169.254.169.254/latest/api/token ... HTTP error: error sending request

The task-side SDK (obstore) reads static creds from FLYTE_AWS_ENDPOINT / FLYTE_AWS_ACCESS_KEY_ID / FLYTE_AWS_SECRET_ACCESS_KEY (flyte/storage/_config.py, S3._KEY_ENV_VAR_MAPPING). With none set it uses the default AWS credential chain and hits the EC2 metadata endpoint 169.254.169.254. The chart's storage.* values configure the control plane but do not propagate credentials to task pods.

Fix: inject them via configuration.inline.plugins.k8s.default-env-vars. Caveat: this list replaces the chart default, so the control-plane vars must be repeated:

configuration:
  inline:
    plugins:
      k8s:
        default-env-vars:
          - _U_EP_OVERRIDE: "flyte-http.flyte:8090"
          - _U_INSECURE: "true"
          - _U_USE_ACTIONS: "1"
          - FLYTE_AWS_ENDPOINT: "https://<account>.r2.cloudflarestorage.com"
          - FLYTE_AWS_ACCESS_KEY_ID: "<key>"
          - FLYTE_AWS_SECRET_ACCESS_KEY: "<secret>"

3. Task I/O written to the wrong bucket → 403 AccessDenied

After creds were fixed, tasks then failed with:

403 Forbidden AccessDenied ... PUT https://<account>.r2.cloudflarestorage.com/flyte-data/.../error.pb

runs.storagePrefix defaults to s3://flyte-data (a bucket that doesn't exist), so all task input/output/error writes go to the wrong bucket. storage.metadataContainer / userDataContainer only set the dataproxy container — not this prefix.

Fix:

configuration:
  inline:
    runs:
      storagePrefix: s3://<real-bucket>

4. Minor / papercuts

• helm upgrade kills the flyte-http port-forward. Every upgrade rolls the flyte pod, dropping kubectl port-forward service/flyte-http 8090:8090; the SDK then reports "Flyte system is currently unavailable." Worth calling out to restart it after each upgrade.
• Console vs SDK use different ports. Browser → :8080 (Traefik unified origin), SDK → :8090 (flyte-http). Easy to conflate; the run URL advertises :8080 while the SDK config points at :8090.
• Default SDK config port. A generated project-local .flyte/config.yaml pointed at localhost:8080 for admin.endpoint; the API is on 8090.

---

Suggested skill improvements

1. Add a "Storage for task pods" step for any S3-compatible store (R2/MinIO), documenting the three FLYTE_AWS_* env vars and the default-env-vars list-replacement caveat. This is required for tasks to run at all — not optional.
2. Add runs.storagePrefix override to the storage step (or derive it from metadataContainer), and explain it's distinct from metadataContainer/userDataContainer.
3. Document console access without auth. The base (no-auth) path currently leaves the console unreachable for run pages; add the Traefik unified-origin recipe (or set NEXT_PUBLIC_ADMIN_API_URL) so /v2 run pages load. Today Traefik only appears in the optional auth section.
4. Note the port-forward lifecycle — that helm upgrade drops the flyte-http forward, and that browser (:8080) and SDK (:8090) use different forwards.
5. Consider a values snippet that bundles items 1–2 into the R2/S3 example so the base deployment is task-ready, not just API-ready.

Happy to open a PR against the skill with these additions if useful.

Environment: macOS, Docker Desktop, kind, flyte-binary v2.0.27, flyte SDK 2.5.8, Cloudflare R2 + Supabase (session pooler).