Evaluation sweep: DeepSeek V4 Flash, further Together models, Marin 32B

[timodonnell]

Tracking issue for the first real evaluation sweep over the expanded benchmark. I will post results here as each run lands.

Where the benchmark stands

The candidate set is 117 semantic instances over 49 protein groups, 247 rendered variants, at PROMPT_VERSION = v2. Recent changes worth knowing about before reading any number below:

• 69 of 117 instances (59%) sit on post-2023 structures drawn from the FoldBench low-homology target lists. The other half is the classical pool (crambin, BPTI, lysozyme, calmodulin …), which is in every training corpus. Release date is recorded per instance, so scores can be split by it.
• Seven automatic families now carry a context-only control — P01, P02, S03, S04, S05, S08, S09 — plus the six mechanistic episodes. Same question, same gold answer, no coordinates. See below for why this changes how the scores should be read.
• Gold answers are not in the repository. build regenerates them byte-identically and validate checks them against committed hashes. See docs/contamination.md.

What is being run

model	provider	status
DeepSeek V4 Flash 0731	Together (batch)	queued
further Together models	Together (batch)	queued
Marin 32B (Stanford)	Modal	queued

Together's Batch API is roughly half the price of synchronous calls, so every Together run goes through it.

Two things that will be reported alongside every score

Truncation, separately from wrongness. These score identically — zero — and conflating them turns a benchmark of reasoning models into a benchmark of output budgets. In the earlier smoke run, DeepSeek V4 Flash moved 0.475 to 0.638 to 0.710 purely by raising the output budget from 8k to 32k to 64k tokens. Nothing about the model changed. Every table here reports the truncation count.

The context-only floor. For a mechanistic episode the prompt still describes the experiment, so a model that recognises the system can answer from the literature, and the gain over context-only separates structure use from recall. For the automatic families the sanitised question names nothing — no entry ID, no organism, no ligand code — so what the control measures is the guessing floor, and that floor is neither uniform nor small:

family	answer space	why it matters
S03	buried / exposed	a coin flip before any prior
S04	helix / strand / coil	three-way, coil is the plurality class
S05	three fold classes	three-way
S09	g+ / t / g-	three-way, g- is commonest
P01	chain identifiers	answering A is right much of the time
P02, S08	integer / residue label	unconstrained, should sit near zero

A model scoring 0.55 on S03 has demonstrated nothing at all. Reports print this as What the coordinates are worth.

Response cache

All completions now go into a content-addressed cache keyed on model, sampling parameters, output budget and the exact prompt text — deliberately not on the render or instance identifier, both of which move when the dataset is rebuilt. Adding a question costs only the calls for that question; removing one costs nothing. Each entry keeps the provider's full response body, so reasoning traces survive for inspection rather than being reduced to the answer text at call time.

Caveats that apply to everything below

• The candidate set is proposed, not curator-accepted. Numbers may move once curation lands.
• S02 realises 2 instances against a target of 4 — phosphorylation is rare in the pool and absent from all 49 FoldBench entries.

[timodonnell]

Update: infrastructure in place, runs submitted

Response cache and batch inference — landed (726b6bb)

Every completion now goes into a content-addressed cache keyed on the model, revision, sampling parameters, output budget, completion index and the exact prompt text — and deliberately not on render_id or the instance identifier, both of which move when the dataset is rebuilt. Today's FoldBench expansion would have cost 24 calls instead of 247.

The output budget is part of the key on purpose. Truncation and inability score identically, so an 8k-budget answer must never be silently reused for a 64k run.

Entries keep the provider's full response body, so reasoning traces survive for inspection instead of being reduced to the answer text at call time — which is what was happening before. Prompts are stored by hash only; they are already in the dataset's prompts/ directory.

Batch inference is wired in as a cache filler and nothing else: batch submits the uncached prompts, evaluate afterwards finds them already there. Scoring and reporting never learn that batching happened, and a batch that fails costs only time. 176 tests pass.

DeepSeek V4 Flash — submitted

All 247 renders are in a single Together batch (1cd93d41), 64k output budget. Batch queues are slow by design — that is what the ~50% discount buys — so this is expected to take hours rather than minutes. Results posted here when it lands.

Marin 32B — two problems worth stating before any number appears

Marin 32B is not hosted by Together (checked all 280 models), so it is being served on Modal with vLLM behind an OpenAI-compatible endpoint. Two things about the model materially limit what a score from it will mean, and both are properties of the model rather than choices made here.

1. It is a base model. marin-community/marin-32b-instruct does not exist — only marin-32b-base. A base model has never been trained to obey "place the machine-readable answer after FINAL:". A low score may therefore be format non-compliance rather than an inability to read coordinates. The scorer already counts format errors as their own category; that number should be read before the score.

2. Its context is short, and its config contradicts itself. config.json declares max_position_embeddings: 4096, while the llama3 rope_scaling block in the same file claims an original length of 8192 with factor 8. Those cannot both be right. At face value, 4096 tokens covers very little of this benchmark:

context limit	renders covered	families covered
4,096	48/247 (19%)	8/20
8,192	49/247 (20%)	8/20
16,384	73/247 (30%)	15/20
32,768	133/247 (54%)	19/20
131,072	247/247 (100%)	20/20

Median prompt is ~30k tokens; the largest is ~87k. At 4096 the only families that fit are the ones with small structures or no coordinates at all — P01, P02, S03, S04, S05, S08, S09 and the context-only mechanistic variants. In other words, at its declared context length Marin 32B can only attempt the questions whose context-only controls exist, which is an unusually clean accident: for those families we will be able to compare its coordinate score directly against its guessing floor.

It is being served at --max-model-len 32768 to exercise the rope scaling. Prompts that do not fit are skipped and counted, never truncated — the run configuration records how many, and the report shows which families lost coverage. Any Marin number posted here will carry its coverage fraction next to it.

[timodonnell]

Result: Marin 32B cannot be evaluated on this benchmark

This is a null result and worth stating plainly rather than burying under a low score.

The smallest prompt in the candidate set that contains any coordinates is 7,132 tokens. Marin 32B's context is 4,096. Not one structural question fits — the shortest is 1.74× the entire window.

smallest coordinate-bearing prompts	tokens	atoms
S09-fb_7v3o normalized_coordinates	7,132	307
G03-fb_7t9r normalized_coordinates	8,197	352
G02-fb_7t9r normalized_coordinates	8,216	352

Of the 247 renders, 48 fit in 4,096 tokens — and all 48 are context_only controls, which by construction show no structure at all. So a Marin run on this dataset measures its guessing floor and strictly nothing else.

Why not just serve it with a longer context

config.json carries a llama3 rope_scaling block claiming factor 8 over an original length of 8,192, which would imply 65,536. It contradicts max_position_embeddings: 4096 in the same file, and the model card settles it: "Sequence length: 4096".

vLLM refuses --max-model-len 32768 outright and will only accept it behind VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, whose warning is the relevant part:

If the model uses relative position encoding (RoPE), positions exceeding derived_max_model_len lead to nan.

Forcing it would produce garbage that scores zero, and a zero from NaN activations is indistinguishable from a zero from an inability to reason. That is exactly the failure mode this benchmark reports truncation separately to avoid, so it was not done.

What is being run instead

The deployment is live at its real 4,096-token limit and will run the 48 context-only renders. That yields the guessing floor of a base model — a genuinely useful reference point next to the instruct models, and the only honest number available here. It will be reported as a floor, never as a benchmark score.

What would make Marin 32B evaluable

A dataset built to a ~3,000-token prompt budget: single small chains, aggressive cropping, normalized_coordinates throughout (~22 tokens/atom against minimal PDB's ~41). That caps structures at roughly 130 atoms, which supports the parsing and geometry families and rules out anything needing a fold, an interface or a binding site. It is a real piece of work and a defensible one — 4k-context open models are otherwise entirely excluded from structural benchmarks — but it is a separate dataset, not a filter on this one, and it would need its own targets and curation.

Say the word if that is worth doing.

Meanwhile

Five Together batches are queued over all 247 renders: DeepSeek V4 Flash (1cd93d41), gpt-oss-120b (889bdd9d), DeepSeek V3.1 (7c3e3a84), Qwen3-235B-A22B (dee47d0c) and GLM-4.7 (cbae2a86). The set is chosen to isolate two axes — reasoning versus not, within one family (V4 Flash against V3.1), and a size ladder across three unrelated model lineages.

[timodonnell]

Marin 32B: ran, and the failure mode is instructive

Served on Modal (1×H100, vLLM, max_model_len 4096) and run over the 48 renders that fit. 199 of 247 renders were skipped as too long — recorded in the run config, not silently dropped.

macro score	0.053
micro score	0.055
format errors	26 / 48
truncated	34 / 48
coverage	48/247 renders, 0 of them containing coordinates

family	n	score	format errors	truncated
S05 fold class	6	0.333	0	0
P01 chain identifiers	7	0.094	0	0
P02, S03, S04, S08, S09, MECH	35	0.000	26	34

S05 scoring exactly 0.333 on a three-way classification is the guessing floor, arrived at honestly.

What it actually produced

The two non-zero families are non-zero for revealing reasons.

P01 — it answered every letter of the alphabet:

FINAL: A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z

Set F1 gives that 0.094: perfect recall, hopeless precision. A degenerate strategy, and exactly the kind of thing a set-valued metric should expose rather than reward.

Everywhere else — it continued the document instead of answering it:

Question: What is the secondary structure at residue B:G128? Answer helix, strand or coil.

Answer with exactly one of the listed categories.
Example: FINAL: helix


Question: What is the secondary structure at residue B:G128? Answer helix, strand or coil.

Answer with exactly one of the listed categories.
Example: FINAL: helix
...

It repeats the prompt until it hits the token cap. That is textbook base-model behaviour — it was never trained to treat a prompt as an instruction — and it accounts for all 34 truncations. This is the one case where truncation and inability genuinely coincide, and the benchmark reporting both separately is what makes that visible rather than a mysterious 0.05.

Conclusion: 0.053 is not a structural-reasoning score. It is a base model failing to follow an answer format, on 48 prompts none of which contain a structure. It belongs in the record as a floor and a warning, not in a leaderboard.

---

A methodological problem the response cache uncovered

Filling the cache reported 10 hits on 48 calls against an empty cache. That should be impossible — and the cause is a real issue with the context-only control, not with the cache.

For P01, P02 and S05 the context-only prompts are byte-identical across different proteins. Those three questions carry no distinguishing parameter — "List all chain identifiers", "How many protein residues are in chain A", "Classify chain A" — and with the coordinates removed there is nothing left to tell the instances apart:

family	context-only renders	distinct prompts	distinct gold answers
P01	7	1	2
P02	5	1	4
S05	6	1	3

Seven identical prompts, two different correct answers. The families that name a residue (S03, S04, S08, S09) are unaffected, since the residue label differentiates them.

This does not invalidate the floor — measuring "how often does the model's one guess match the distribution of golds" is exactly what a guessing floor is. But it has a consequence worth stating: at temperature 0 the effective sample size for those three families is one prompt, not five to seven. The clustered bootstrap will report an interval as if there were independent samples per protein, and there are not; the correlation is total. Any confidence interval on the P01, P02 or S05 context-only floor is therefore too narrow and should be read as a point estimate.

The cache deduplicating them is the right behaviour — it saved 10 of 48 calls and will save proportionally more on the larger models — but the statistics need the caveat. I will fix the interval reporting for these three families before publishing floors.

---

Batch progress

All seven Together batches still in_progress after ~1h. Three earlier submissions had to be replaced: DeepSeek V3.1, GLM-4.7 and Qwen3-235B are listed in the catalogue but not servable on this account. Qwen3-235B was the expensive lesson — it accepted a 247-request batch, reported COMPLETED at 100%, and put every request in the error file with "Unable to access non-serverless model", discovered only after the completion window. batch now spends one token on a synchronous probe before submitting (be0c7e7).

[timodonnell]

Results

Five runs complete enough to compare, over all 247 renders at a 32,768-token output budget. Macro average across the 20 question families, clustered bootstrap by protein.

model	macro	95% CI	truncated	format errors
Kimi K3	0.742	[0.694, 0.784]	64/247	71
MiniMax M3	0.501	[0.432, 0.562]	98/247	109
Qwen3.5 9B	0.398	[0.334, 0.443]	155/247	157
gpt-oss-120b	0.288	[0.210, 0.360]	10/214	68
gpt-oss-20b	0.200	[0.132, 0.255]	97/241	166
Marin 32B (base, 4k ctx)	0.053	—	34/48	26

DeepSeek V4 Flash (80 renders, 8 families) and Gemma 4 31B (60 renders, 8 families) are not in the table: their batches were cut off by the credit limit below, and batches process in render-id order, so each covers a different alphabetically-early slice. Their macro scores are over different question sets and are not comparable to anything.

The finding: these scores are output budgets, not capabilities

Kimi K3's per-family results split cleanly in two. Twelve families at 1.000 — G01, G02, G03, N01, P01, P02, P03, S01, S02, S08, S09, T01 — and S07 at 0.967. Then a cliff: I01 0.000, G04 0.143, S03 0.143, S04 0.200, MECH 0.431, S06 0.451.

In every one of the low families, the format-error count equals the truncation count. I01 is 0.000 with 10 of 10 truncated. That is not an inability to reason about interfaces; it is a model that never finished writing.

Restricting to responses that actually terminated:

family	context-only floor	with coordinates	truncated	coordinates, untruncated only
S03	0.167	0.167	83%	1.000
S04	0.667	0.167	83%	1.000
S05	0.333	0.500	50%	1.000
MECH	0.522	0.368	58%	0.883
P01, P02, S08, S09	0.000	1.000	0%	1.000

Kimi K3 scores 1.000 on every automatic family it is given room to finish, and 0.883 on the mechanistic episodes. Its headline 0.742 is a measurement of a 32k output cap.

A confound in the context-only control

The middle columns above show something I had not anticipated and that anyone using this control needs to know.

Read naively, S04 says coordinates make the model worse — floor 0.667, with coordinates 0.167, a gain of −0.500. MECH reads −0.154 and S03 reads 0.000.

All three are artefacts. Context-only prompts are ~200 tokens and never truncate; coordinate prompts are up to 87,500 tokens and truncate at 50–83% in exactly these families. The contrast is therefore not like-for-like whenever truncation is present: it silently compares a completed cheap answer against a cut-off expensive one, and can invert the sign of the effect it exists to measure.

Corrected against untruncated responses only, the same three families give +0.833, +0.333 and +0.361. The control is sound; the naive difference is not. Any published context-only gain has to be conditioned on completion, and I will make report do that rather than leaving a −0.500 in a table.

Why three runs are incomplete

The Together account hit its credit limit mid-sweep: insufficient_balance — this job would exceed your maximum account credit limit. That is what killed DeepSeek V4 Flash at 31% and Gemma 4 31B at 24%, and it blocks resubmission of the ~230 missing prompts. Completing those two plus the gpt-oss gaps needs credits added to the account.

Cache recovery

Upstream landed a parallel cache implementation (#5 and the commits before it) that bumped the format to 3 and added endpoint to the cache key. Both changes made every one of the 1,284 responses bought overnight unreadable — right key, wrong digest, wrong format.

They were re-keyed rather than re-bought, by walking the same (model, render, completion) space the runner walks and asking it for the key, instead of reconstructing one by hand. All 1,384 entries now resolve; the three complete models verify at 0 misses. Nothing was paid for twice.

[timodonnell]

Deck and writeup published

Deck: docs/slides/pdbthink-results-2026-08-15.pptx
Writeup: docs/results.md

Both are generated from a collected results file rather than hard-coded, because these numbers moved four times while the sweep ran. collect_results.py computes three things a plain score does not: every score restricted to responses that terminated, the context-only gain conditioned on completion, and per-run family coverage so a run cut short by a credit limit is never averaged against a complete one.

Final numbers

model	renders	macro	completed responses only	truncated
Kimi K3	247	0.668	0.839	64
MiniMax M3	247	0.447	0.755	98
Qwen3.5 9B	247	0.310	0.579	155
gpt-oss-120b	214	0.260	0.272	10
gpt-oss-20b	241	0.174	0.247	97
Marin 32B (base, 4k ctx)	48	0.053	0.142	34
Gemma 4 31B	60	0.808	0.808	0
DeepSeek V4 Flash	80	0.627	0.761	19

Italic rows cover 8 of 20 families each — different subsets, not comparable to anything, and marked as such on the slide rather than quietly averaged.

Still running

The high-budget re-runs of the truncated subsets are in flight but slow: at 65,536–131,072 output tokens per response, throughput is about one completion a minute regardless of concurrency, and there are roughly 600 calls left across the remaining models. They are confirmatory — the completion-conditioned analysis already answers the same question without them, which is why the deck did not wait.

The two incomplete Together runs are also back in flight now that credits are available. Both will be folded in when they land; the deck regenerates from the same command.

One note on the batch path

With the credit limit lifted, large batch submissions still fail at reservation time — Together appears to reserve the full max_output_tokens × requests up front, and DeepSeek at 65,536 × 158 exceeds the account cap even with a positive balance. A 40-request batch submits fine. The remainder was completed synchronously instead: roughly twice the price on a small remainder, against a sequential chunked queue that would have run for the better part of a day.