Hyperloom Queue Worker

Go workers for the HF 15K optimization pipeline.

What It Does

• download-worker: consumes Redis list items from hio:download_queue, SSHes into the core42 bash workspace, and prewarms models into /wekafs/models.
• optimize-worker: consumes hio:optimize_queue, builds the Hyperloom prompt, and submits directly to Claw using the one-step create session with message API.
• direct-claw-submit: single-model CLI for smoke tests and debugging.

Redis items are intentionally minimal:

{"task_id":"hio-example-001","model":"HuggingFaceH4/zephyr-7b-beta"}

Build

go test ./...
GOOS=linux GOARCH=amd64 go build -o dist/download-worker ./cmd/download-worker
GOOS=linux GOARCH=amd64 go build -o dist/optimize-worker ./cmd/optimize-worker
GOOS=linux GOARCH=amd64 go build -o dist/direct-claw-submit ./cmd/direct-claw-submit

Runtime Environment

Common:

export REDIS_ADDR=127.0.0.1:6379
export DOWNLOAD_QUEUE=hio:download_queue
export OPTIMIZE_QUEUE=hio:optimize_queue
export REPORT_ENDPOINT=https://global.primus-safe.amd.com/pulse/api/v1/ci-funnel/l0l1

Download worker:

export CORE42_SSH_TARGET='697f4f8340e397169e72840c6ea51cc4.chenyi-rzcg2-master-0.pytorch.bash.core42-hyperloom@core42.primus-safe.amd.com'
export CORE42_SSH_PORT=2222
export REMOTE_PREWARM_SCRIPT=/wekafs/HyperloomV2/ci/prewarm_models.py
export REMOTE_MODEL_ROOT=/wekafs/models
export MAX_DOWNLOAD_ATTEMPTS=5

Optimize worker:

export CLAW_BASE_URL=https://core42.primus-safe.amd.com/claw-api/v1
export CLAW_API_KEY=...
export CLAW_AGENT_ID=agent_default
export CLAW_PLUGIN_ID=4
export OPT_WORKSPACE=core42-hyperloom
export MAX_DISPATCH_ATTEMPTS=5

Do not commit secrets or private SSH keys. Put runtime secrets in deployment-specific environment files or Kubernetes Secrets.

Queue Flow

hio:download_queue
  -> download-worker
  -> /wekafs/models/<org>-<repo>
  -> hio:optimize_queue
  -> optimize-worker
  -> POST /claw-api/v1/sessions with message
  -> Pulse L0/L1 report

Both download and dispatch retry up to 5 times. After repeated failure, the worker reports submit_failed to Pulse and pushes the same two-field task back to the download queue tail.