Skip to content

feat: batched tick-level inference (llm:chat-all) #53

Description

@JNK234

Roadmap priority: B4 — described in the progress report as "the single biggest simulation-speed win for population-scale models."

Summary

Add a batched, tick-level inference primitive — an llm:chat-all-style call that fires every agent's request concurrently with one barrier per tick, instead of N sequential per-agent calls.

Why it matters

The dominant cost in a population-scale LLM model is wall-clock, not tokens. ask turtles [ let r llm:chat ... ] serializes: 100 turtles at 2s per call is over 3 minutes per tick. Firing them together and joining once per tick collapses that to roughly the slowest single call.

llm:chat-async already exists but retrieval is a blocking runresult, so the modeler has to hand-roll the two-pass pattern (fire all, then collect all) and gets no concurrency control.

What needs to be done

  1. Decide the API shape. Options: a command that takes an agentset and a prompt reporter, a two-phase fire/collect pair, or a batched reporter returning a list aligned to the agentset.
  2. Respect NetLogo's execution model — results must land back on the right agents, and the barrier must complete before the primitive returns.
  3. Per-agent history must stay correct under concurrent commit (the historyLock / commitExchange path from 14f7c35 already handles this; verify it holds at batch scale).
  4. Per-agent failure isolation — one failed call must not abort the batch.
  5. Must compose with throttling (feat: request throttling — bounded concurrency, jitter, and per-provider pacing #48): "fire all" needs to mean "queue all, run at most N concurrently."

Open questions

  • What is the natural NetLogo idiom here? A primitive taking an agentset and a prompt-producing reporter is powerful but unusual for this codebase.
  • Should the result be assigned to a turtle variable automatically, or returned as a list for the modeler to distribute?
  • Interaction with llm:chat-async — does this replace it, or sit alongside?

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions