Skip to content

Return error chunk in low memory situations#1626

Draft
rltakashige wants to merge 7 commits intomainfrom
leo/handle-low-memory-situations
Draft

Return error chunk in low memory situations#1626
rltakashige wants to merge 7 commits intomainfrom
leo/handle-low-memory-situations

Conversation

@rltakashige
Copy link
Collaborator

Motivation

We currently allow runners (and their instance) to OOM. Let's try to prevent this as much as possible.

Changes

  • Check a prompt will fit into memory
  • Check on each cancellation check step that we are not close to running out of memory
  • If either happen, return an error chunk

Test Plan

Manual Testing

Seems to work on Mac Minis.

Automated Testing

Wrote a test script to test this, but it's very flaky so won't be in the final PR.

@rltakashige rltakashige force-pushed the leo/handle-low-memory-situations branch 2 times, most recently from 1b346d4 to fef3703 Compare February 26, 2026 13:16
@rltakashige rltakashige force-pushed the leo/handle-low-memory-situations branch from fef3703 to 20ea13f Compare February 26, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant