feat: add circuit breaker for upstream provider overload protection #75

kacpersaw · 2025-12-11T13:51:13Z

Summary

Implement per-provider circuit breakers that detect upstream rate limiting (429/503/529 status codes) and temporarily stop sending requests when providers are overloaded.

This completes the overload protection story by adding the aibridge-specific component that couldn't be implemented as generic HTTP middleware in coderd (since it requires understanding upstream provider responses).

Key Features

Per-provider circuit breakers - Separate circuit breakers for Anthropic and OpenAI since upstream rate limits are provider-specific
Configurable parameters - Failure threshold, time window, cooldown period, half-open max requests
Half-open state - Allows gradual recovery testing after cooldown
Prometheus metrics - State gauge, trips counter, rejects counter for monitoring
Thread-safe - Proper mutex protection for concurrent access
Disabled by default - For backward compatibility, must be explicitly enabled

Circuit Breaker States

┌─────────┐  failures >= threshold  ┌─────────┐
│ CLOSED  │ ──────────────────────► │  OPEN   │
│(normal) │                         │(reject) │
└─────────┘                         └────┬────┘
     ▲                                   │
     │                          cooldown expires
     │                                   │
     │  successes >= threshold     ┌─────▼─────┐
     └──────────────────────────── │ HALF-OPEN │
                                   │ (testing) │
     ┌─────────────────────────────┤           │
     │         any failure         └───────────┘
     │
     ▼
┌─────────┐
│  OPEN   │
│(reject) │
└─────────┘

Status Codes That Trigger Circuit Breaker

Code	Description
429	Too Many Requests
503	Service Unavailable
529	Anthropic Overloaded

Other error codes (400, 401, 500, 502, etc.) do not trigger the circuit breaker since they indicate different issues that circuit breaking wouldn't help with.

Default Configuration

Parameter	Default	Description
`Enabled`	`false`	Must be explicitly enabled
`FailureThreshold`	`5`	Failures needed to trip circuit
`Window`	`10s`	Time window for counting failures
`Cooldown`	`30s`	Time to wait before testing recovery
`HalfOpenMaxRequests`	`3`	Requests allowed in half-open state

New Prometheus Metrics

aibridge_circuit_breaker_state{provider} - Current state (0=closed, 1=open, 2=half-open)
aibridge_circuit_breaker_trips_total{provider} - Total times circuit opened
aibridge_circuit_breaker_rejects_total{provider} - Requests rejected due to open circuit

Files Changed

circuit_breaker.go - Core circuit breaker implementation
circuit_breaker_test.go - Comprehensive test suite (13 tests)
bridge.go - Integration into RequestBridge
interception.go - Apply circuit breaker to intercepted requests
metrics.go - Add Prometheus metrics

Testing

All tests pass:

go test -count=1 -short ./...
go vet ./...
go build ./...

Implement per-provider circuit breakers that detect upstream rate limiting (429/503/529 status codes) and temporarily stop sending requests when providers are overloaded. Key features: - Per-provider circuit breakers (Anthropic, OpenAI) - Configurable failure threshold, time window, and cooldown period - Half-open state allows gradual recovery testing - Prometheus metrics for monitoring (state gauge, trips counter, rejects counter) - Thread-safe implementation with proper state machine transitions - Disabled by default for backward compatibility Circuit breaker states: - Closed: normal operation, tracking failures within sliding window - Open: all requests rejected with 503, waiting for cooldown - Half-Open: limited requests allowed to test if upstream recovered Status codes that trigger circuit breaker: - 429 Too Many Requests - 503 Service Unavailable - 529 Anthropic Overloaded Relates to: coder/internal#1153

circuit_breaker.go

interception.go

circuit_breaker.go

…solation - Replace custom circuit breaker implementation with sony/gobreaker - Change from per-provider to per-endpoint circuit breakers (e.g., OpenAI chat completions failing won't block responses API) - Simplify API: CircuitBreakers manages all breakers internally - Update metrics to include endpoint label - Simplify tests to focus on key behaviors Based on PR review feedback suggesting use of established library and per-endpoint granularity for better fault isolation.

Rename fields to match gobreaker naming convention: - Window -> Interval - Cooldown -> Timeout - HalfOpenMaxRequests -> MaxRequests - FailureThreshold type int64 -> uint32

circuit_breaker.go

bridge.go

interception.go

go.mod

interception.go

metrics.go

circuit_breaker_test.go

…onfigs Address PR review feedback: 1. Middleware pattern - Circuit breaker is now HTTP middleware that wraps handlers, capturing response status codes directly instead of extracting from provider-specific error types. 2. Per-provider configs - NewCircuitBreakers takes map[string]CircuitBreakerConfig keyed by provider name. Providers not in the map have no circuit breaker. 3. Remove provider overfitting - Deleted extractStatusCodeFromError() which hardcoded AnthropicErrorResponse and OpenAIErrorResponse types. Middleware now uses statusCapturingWriter to inspect actual HTTP response codes. 4. Configurable failure detection - IsFailure func in config allows providers to define custom status codes as failures. Defaults to 429/503/529. 5. Fix gauge values - State gauge now uses 0 (closed), 0.5 (half-open), 1 (open) 6. Integration tests - Replaced unit tests with httptest-based integration tests that verify actual behavior: upstream errors trip circuit, requests get blocked, recovery after timeout, per-endpoint isolation. 7. Error message - Changed from 'upstream rate limiting' to 'circuit breaker is open'

- Add CircuitBreaker interface with Allow(), RecordSuccess(), RecordFailure() - Add NoopCircuitBreaker struct for providers without circuit breaker config - Add gobreakerCircuitBreaker wrapping sony/gobreaker implementation - CircuitBreakers.Get() returns NoopCircuitBreaker when provider not configured - Add http.Flusher support to statusCapturingWriter for SSE streaming - Add Unwrap() for ResponseWriter interface detection

bridge.go

circuit_breaker.go

- Changed CircuitBreaker interface to Execute(fn func() int) (statusCode, rejected) - Use gobreaker.Execute() to properly handle both ErrOpenState and ErrTooManyRequests - NoopCircuitBreaker.Execute simply runs the function and returns not rejected - Simplified middleware by removing separate Allow/Record pattern

…Bridge

bridge.go

circuit_breaker.go

…e gobreakerCircuitBraker along with the interface and noop struct

bridge.go

Co-authored-by: Paweł Banaszewski <[email protected]>

kacpersaw marked this pull request as draft December 11, 2025 13:52

chore: apply make fmt

aad288c

kacpersaw marked this pull request as ready for review December 15, 2025 11:18

kacpersaw requested review from dannykopping and pawbana December 15, 2025 11:18

dannykopping requested changes Dec 16, 2025

View reviewed changes

circuit_breaker.go Show resolved Hide resolved

pawbana reviewed Dec 16, 2025

View reviewed changes

interception.go Outdated Show resolved Hide resolved

circuit_breaker.go Show resolved Hide resolved

kacpersaw added 3 commits December 16, 2025 10:27

refactor: align CircuitBreakerConfig fields with gobreaker.Settings

8cf2d18

Rename fields to match gobreaker naming convention: - Window -> Interval - Cooldown -> Timeout - HalfOpenMaxRequests -> MaxRequests - FailureThreshold type int64 -> uint32

refactor: remove CircuitState, use gobreaker.State directly

8e44145

kacpersaw requested review from dannykopping and pawbana December 16, 2025 11:01

dannykopping requested changes Dec 17, 2025

View reviewed changes

kacpersaw added 5 commits December 17, 2025 08:51

docs: clarify noop behavior when provider not configured

521df9b

Update go.mod

c85b836

fix: update metrics help text to reflect 0/0.5/1 gauge values

e446954

pawbana reviewed Dec 17, 2025

View reviewed changes

kacpersaw added 7 commits December 17, 2025 10:27

refactor: remove unused circuitBreakers field and getter from Request…

6a7d578

…Bridge

use per-provider maps for endpoints

b0ff0eb

make fmt

bee7a4d

use mux.Handle for cb middleware

98c7b7a

Move CircuitBreakerConfig to the Provider struct

7733266

Update tests

7c7c85b

kacpersaw requested a review from pawbana December 17, 2025 14:38

pawbana reviewed Dec 17, 2025

View reviewed changes

bridge.go Outdated Show resolved Hide resolved

bridge.go Outdated Show resolved Hide resolved

circuit_breaker.go Outdated Show resolved Hide resolved

default noop func for onChange

8943ef0

create CircuitBreakers per Provider instead of a global one and remov…

7d2dcb1

…e gobreakerCircuitBraker along with the interface and noop struct

pawbana reviewed Dec 17, 2025

View reviewed changes

bridge.go Show resolved Hide resolved

kacpersaw and others added 2 commits December 17, 2025 17:27

Update bridge.go

e3438f4

Co-authored-by: Paweł Banaszewski <[email protected]>

fix format

a32f246

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add circuit breaker for upstream provider overload protection #75

feat: add circuit breaker for upstream provider overload protection #75

kacpersaw commented Dec 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: add circuit breaker for upstream provider overload protection #75

Are you sure you want to change the base?

feat: add circuit breaker for upstream provider overload protection #75

Conversation

kacpersaw commented Dec 11, 2025

Summary

Key Features

Circuit Breaker States

Status Codes That Trigger Circuit Breaker

Default Configuration

New Prometheus Metrics

Files Changed

Testing

Related

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants