exp232: run contacts-v1 decontam sweep

[eric-czech]

Summary

• add a CoreWeave tokenization driver that mirrors only issue exp: retrain our best model on decontaminated training data #232's AFDB and ESM Atlas training corpora from the Hugging Face bucket into the requested S3 data/ prefix, verifies the complete mirror, and tokenizes only from S3
• add the dated AFDB/ESM cache paths (2026.08.14) and retain exp199's validation cache and tokenizer
• add the exp199-derived CoreWeave 1.5B sweep with the unaugmented arm removed, leaving ten always-augmented trials
• preserve exp199's permanent checkpoint cadence and set temporary checkpoints to 15 minutes
• add a pinned CPU/GPU environment and concise launch documentation

Codex Prompt

---

Execute #232.

Details

• Base
  • Work from exp/232-sweep-cv1-decontam branch in fresh repos/MarinFold-br latest main clone; use experiments/exp232_sweep_cv1_decontam dir
  • Review PR #205 thoroughly first
• Grid: Remove no augmentation option from trials (always augment) to cut grid in half
• Arch: No changes; same 1.5B
• Code
  • Start from exp199_sweep_cw.py in exp199: optimize 1.5B contacts-v1 AFDB + ESM models #205 as exp232_sweep.py
  • Assume only CoreWeave/GPU
  • Rewrite, simplify and consolidate implementation/docs to carry only the most essential details from prior experiments
• Data
  • Use ONLY training data from #232
    • Write a separate script for tokenization
    • Download from HF to s3://marin-us-east-02a/MarinFold/exp232_sweep_cv1_decontam/data/ first
      • Tokenize from CWS3, not from HF
    • Create ESM_CACHE and AFDB_CACHE s3://marin-us-east-02a/MarinFold/exp232_sweep_cv1_decontam/tokenized/contacts_v1/{esm,afdb}/$CACHE_VERSION
      • Set CACHE_VERSION to today
  • Keep contacts-v1 validation data from exp: optimize 1.5B contacts-v1 AFDB + ESM models #199 (same VALIDATION_CACHE)
  • Use same tokenizer as exp199 (eczech/contacts-v1-tokenizer-5d68a24a899f)
• Checkpoints
  • Save permanent checkpoints on same schedule as exp199
  • Save temp checkpoints every 15 mins

Steps

• Write tokenization and sweep scripts; open draft PR; stop and review
• Run tokenization on CW; stop and review
• Execute sweep with /run-training-sweep; set /goal to completion

Begin with step 1.

Data

Full CoreWeave tokenization (/eczech/exp232-tokenize, exit 0) produced and validated cache version 2026.08.14:

Corpus	exp232 documents	exp232 tokens	exp199 tokens	Change
AFDB	3,963,003	4,432,940,838	4,676,753,425	-5.21%
ESM Atlas	65,553,178	70,042,923,165	71,450,105,324	-1.97%
Total	69,516,181	74,475,864,003	76,126,858,749	-2.17%

The sweep pins and verifies these counts. m2 is now 5.9522% AFDB / 94.0478% ESM; the fixed 152,253,235,200-token budget covers 2.0443 exp232 corpora, and half-AFDB evaluation cadence is 2,114 steps (2,230 in exp199).

Updates

• 2026-08-14:The s02 production sweep is underway: all 10 trials have active CoreWeave dispatches; 8 were reporting progress at the latest heartbeat and 2 were still starting. W&B Report: MarinFold #232 / Phase 1.

• 2026-08-15 — m1-p01-aug diverged at 43.7% and was dropped; all associated jobs are stopped. The remaining nine trials continue: m1 trials are 13.8–23.9% complete and m2 trials are 6.1–10.6%. Eight are actively reporting to the W&B group, while m1-p06-aug recovers from a routine preemption.

• 2026-08-24 — Three 91.35B-token cooldowns are underway from full Levanter state, each linearly decaying its checkpoint LR to zero. Source (latest preceding eval/contacts-v1/loss, peak LR): m2-p06 a03 3.0715, 1e-3; m1-p02 3.0976, 3.1623e-4; TRC m2-p06 lr005 2.9681, 5e-5. W&B group.