Skip to content

memory-pattern.md

Latest commit

 

History

History
88 lines (64 loc) · 4.08 KB

memory-pattern.md

File metadata and controls

88 lines (64 loc) · 4.08 KB
Raw

Decision: Memory Outside the Agent Loop

The Question

The quality agent learns from past reviews via pgvector. Should the agent manage its own memory (tools inside the loop) or should the service handle retrieval and storage around the agent (pre/post processing)?

Rejected: Tools Inside the Agent Loop

The argument: Give the agent find_similar_reviews and store_review tools. The agent decides when to search for past context and when to store its results. Maximum autonomy -- the agent manages its own memory.

Problem 1: Retrieval Is Deterministic

Finding similar past reviews is a fixed sequence:

  1. Embed the diff summary (Voyage API call)
  2. Query pgvector with the embedding + repo filter + date range
  3. Return the top 5 results

There's no reasoning involved. The agent will always want past reviews before starting its review. Making it "decide" to call a tool for this wastes a turn (and tokens) on a decision that has exactly one correct answer every time.

Concrete cost: one tool call turn is ~500-1,000 tokens of input (the tool result) plus whatever the agent says about it. At $3/M input tokens for Sonnet, that's ~$0.003 per review wasted on a no-op decision. Multiply by 500 reviews/day, that's $1.50/day -- not catastrophic, but pointless.

Problem 2: Storage Is Not a Decision

After producing a verdict, the review must be stored. Always. Unconditionally. There's no scenario where the agent should decide "I won't store this one." Making storage a tool the agent calls means:

  • The agent might forget to call it (LLMs are not reliable at remembering procedural steps)
  • The agent might call it at the wrong time (mid-review instead of after)
  • You need error handling for "what if the agent didn't store the review?"

When the answer is always "yes, store it," it shouldn't be a question.

Problem 3: Agent Turns Are Expensive

The agent has 20 turns and a $1.00 budget. Every turn spent on memory management is a turn not spent reviewing code. The retrieval tool call costs one turn. The storage tool call costs one turn. That's 10% of the turn budget consumed by bookkeeping.

Problem 4: Embedding Is a Side Effect

Embedding the diff summary and the review text requires Voyage API calls. These are I/O-bound operations with variable latency (200-800ms each). Inside the agent loop, they add latency to turns and can trigger the per-tool timeout. Outside the loop, they can be parallelized with other post-processing steps.

Accepted: Pre/Post Processing Outside the Loop

BEFORE the agent loop:
  1. Summarize the diff (fast Claude call)
  2. Embed the summary (Voyage)
  3. Retrieve similar past reviews (pgvector query)
  4. Inject past reviews into the agent's prompt

AGENT LOOP:
  Agent reviews code with full context.
  Uses its tool budget for codebase investigation, not memory management.

AFTER the agent loop:
  5. Embed the review text (Voyage)
  6. Store everything in the database
  7. Post GitHub status check

The agent gets past reviews as prompt context -- it doesn't need to know they came from pgvector. It just sees "here are similar reviews from the last 90 days" and uses that information in its review. Clean separation.

Steps 5-7 can be parallelized and even backgrounded (return the verdict to the developer immediately, do storage and status posting async).

Approach Rejected Because
Tools inside agent loop Retrieval and storage are deterministic, not decisions. Wastes turns and budget on bookkeeping. Embedding adds latency inside the loop. Agent might forget to call storage.
Pre/post processing Accepted. Deterministic operations happen outside the loop. Agent uses its full turn budget for actual code review. Post-processing can be parallelized.