feat: add ENABLE_LITELLM deployment-wide flag (default true) by saurya · Pull Request #474 · OpenHands/enterprise

saurya · 2026-09-23T01:15:21Z

Summary

Introduces ENABLE_LITELLM as a deployment-wide setting (database-managed feature flag, env-var fallback, default true for backward compatibility). With the flag off, OHE never contacts the external LiteLLM gateway.

This addresses each of the requested behaviors when the flag is off:

New user provisioning: LiteLlmManager.create_entries now succeeds without creating a LiteLLM user/team or generating a gateway key — the user is left to configure a direct provider (BYOK).
Organization management: org creation/membership add/remove already route exclusively through LiteLlmManager, so the manager-level gate covers them — OHE records update, LiteLLM sync is skipped.
OpenHands Models: LiteLLMProxyModelService returns an empty managed-model list (BYOK catalogue only) instead of querying the proxy.
Budgets: the Budgets and "Your budget" screens render a FeatureDisabledScreen ("Please enable LiteLLM to use this feature") instead of fetching; the backend budget routes, maintenance job, and preflight job reject/skip outright.
Usage Monitoring: per-user usage rows omit stale budget_monthly_limit/budget_is_disabled; conversation/usage stats themselves (OHE's own tracking) are unaffected.
Billing and credits: unaffected by this change — already gated by the existing ENABLE_BILLING flag; LiteLLM-backed reads (e.g. get_credits) degrade gracefully to $0.
Managed LLM keys: the managed/BYOR refresh and generate endpoints are rejected (403) when disabled; regular OpenHands application API keys (create/list/delete, current-key info) are unaffected.
Re-enabling LiteLLM / backfill: intentionally out of scope for this PR — flagged as follow-up work (an async backfill job would be needed to provision existing users/orgs/memberships into the gateway before flipping the flag back on for an already-running deployment).

How it works

storage/lite_llm_manager.py gets a new is_litellm_enabled() runtime resolver, mirroring the existing _is_billing_enabled() pattern (DB-flag-row wins, env var is the fallback, ENABLE_LITELLM defaults to true).
Every one of the ~28 existing "LiteLLM not configured" guards in LiteLlmManager, plus the shared with_http_client transport wrapper and verify_key, now also check this flag — so a disabled deployment never constructs an httpx.AsyncClient targeting LiteLLM, let alone sends a request.
WebClientFeatureFlags.enable_litellm is threaded through to the frontend (both the new and legacy config paths), DB-resolved on every request like enable_billing.
A new FeatureDisabledScreen shared component renders "Please enable LiteLLM to use this feature."; Budgets/Your-Budget wire it in and skip their data fetches; their nav entries are hidden.

Testing

Backend: tests/unit/test_lite_llm_manager.py — new tests assert the disabled path returns the existing safe fallback (None/False/{}/[]) and never constructs an httpx.AsyncClient, across create_entries, verify_key, get_user_team_info, get_team_members_financial_data, remove_user_from_team, delete_team, ensure_free_team_models, sync_free_model_allowlists.
Frontend: new Budgets test asserts the placeholder renders and getBudgetSettings is never called when the flag is off.
Full backend unit suite (pytest tests/unit): 6045 passed (2 pre-existing, unrelated failures — S3 testcontainers region config and a macOS-specific tmp-dir path assertion — reproduce identically on main).
Full frontend suite (vitest run): same 151 pre-existing failures on main and on this branch (verified via git stash); no new failures introduced.
pre-commit run --config ./dev_config/python/.pre-commit-config.yaml (ruff, mypy) and npm run lint:fix + npm run build both pass clean.

Manual/exploratory test plan (flag OFF)

Provisioning: create a brand-new user; confirm no LITE_LLM_API_URL requests are made (check LiteLLM proxy access logs / a network capture) and the user lands with no LLM configured (must add a direct provider key in Settings).
Org lifecycle: create an org, add/remove members; confirm org/org_member rows update in Postgres but no LiteLLM team/user calls occur.
Model selector: confirm the OpenHands/managed model group is empty and only direct-provider (BYOK) options are offered.
Budgets: navigate to Settings → Budgets and Your Budget; confirm the placeholder renders, the nav items are hidden, and the budget maintenance/preflight cron jobs report skipped: litellm_disabled in logs.
Managed keys: attempt POST /api/keys/llm/managed/refresh and /api/keys/llm/byor/refresh; confirm 403; confirm regular API key create/list/delete still works.
Usage Monitoring: confirm conversation counts/spend still show, but budget columns are blank/omitted.
Billing: with ENABLE_BILLING=true and ENABLE_LITELLM=false, confirm /api/billing/credits returns $0.00 rather than erroring.
Flag flip back on: re-enable and confirm existing behavior is fully restored (no persisted "disabled" state anywhere in the DB blocks it).

Deployment verification (Replicated fleet)

Plan (not yet executed — pending confirmation before touching shared sandbox infra):

make release on this branch in OpenHands/OpenHands-Cloud publishes to a per-branch Replicated channel named after the branch.
Use OpenHands/infra's replicated_fleet.yaml workflow (clean-install op) against two fleet VMs (e.g. shared-1 with ENABLE_LITELLM=true and shared-2 with ENABLE_LITELLM=false in that VM's config-values), pointed at the enable-litellm-flag channel, to compare behavior side-by-side.

This PR description was drafted by an AI agent (OpenHands) on behalf of the user.

Enterprise server image for this PR:

ghcr.io/openhands/enterprise-server:sha-2c6b768

github-actions · 2026-09-23T01:47:28Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
openhands/app_server/server_config
server_config.py
openhands/app_server/web_client
default_web_client_config_injector.py
web_client_models.py
server
config.py
constants.py
server/maintenance_task_processor
managed_llm_key_ownership_processor.py					67-71
server/routes
api_keys.py					221, 234, 634-637
orgs.py					1233, 1344-1345
server/services
feature_flag_service.py
org_budget_service.py					535-541
org_conversation_service.py					1406-1436
server/verified_models
litellm_proxy_model_router.py					230-231, 243
storage
lite_llm_manager.py					1201, 1379-1380, 2246
org_member.py
org_store.py
saas_settings_store.py
Project Total

_{This report was generated by python-coverage-comment-action}

Fixes a regression found while fleet-testing ENABLE_LITELLM=false (#474): saving personal LLM settings for a member provisioned while the flag is off returned 500 "Something went wrong storing settings" for any managed (openhands/*) model. Root cause: with ENABLE_LITELLM off, new-user provisioning (LiteLlmManager.create_entries) correctly skips minting a LiteLLM key, leaving agent_settings.llm.api_key as None. That None flowed into OrgMember.llm_api_key = None, whose setter (unlike its llm_api_key_for_byor sibling) unconditionally encrypted the raw value -- "successfully" JWE-encrypting a null payload into the NOT NULL _llm_api_key column. On the next settings save, decrypting that row produces a Python None wrapped in SecretStr, and pydantic's SecretStr.__len__/__bool__ call len(None) unconditionally, raising TypeError instead of reporting "empty". That TypeError isn't a ValueError, so store_settings' generic `except Exception` handler swallowed it into an opaque 500. - storage/org_member.py: the llm_api_key setter now normalizes None to '' before encrypting, matching the "empty means no key" convention has_real_api_key/llm_api_key_is_set already use. - storage/saas_settings_store.py: _ensure_api_key now checks has_real_api_key() instead of a bare bool()/truthiness check, as defense-in-depth against any already-poisoned rows (e.g. rows written before this fix, or via any other future code path that passes None through the same setter). - tests/unit/test_models.py: new regression test constructing an OrgMember with llm_api_key=None and asserting the round-tripped key is falsy/empty rather than raising. - tests/unit/test_saas_settings_store.py: new regression test calling _ensure_api_key with a SecretStr(None) fallback key and asserting a fresh key is generated instead of raising. Reproduced directly against the ENABLE_LITELLM=false fleet VM (shared-2) via a live Python script inside the running pod, confirmed identical to the reported traceback, then verified all four tests (2 new + existing suite) pass with the fix. Co-authored-by: openhands <openhands@all-hands.dev>

Picks up OpenHands/enterprise#474's settings-save 500 fix (sha-9dc7e70), found while fleet-testing ENABLE_LITELLM=true/false. Co-authored-by: openhands <openhands@all-hands.dev>

Picks up OpenHands/enterprise#474's managed-key mint/rotate guard fix (sha-a8024c3). Co-authored-by: openhands <openhands@all-hands.dev>

Introduces ENABLE_LITELLM as a database-managed feature flag (env-var fallback default true, mirroring the ENABLE_BILLING pattern) that gates every network-touching path in the app on whether this deployment may contact the LiteLLM gateway. Backend: - storage/lite_llm_manager.py: new is_litellm_enabled() runtime resolver; every existing "LiteLLM not configured" guard (28 call sites, plus the with_http_client transport wrapper and verify_key) now also checks the flag, so a disabled deployment never opens a connection to LiteLLM. create_entries (new-user provisioning) now succeeds without ever creating a LiteLLM user/team/key when disabled, instead of failing. - Budgets (server/routes/orgs.py, org_budget_service.py, run_budget_preflight.py): reads/writes/maintenance/preflight/alerts are rejected or skipped outright with a clear "enable ENABLE_LITELLM" error, rather than silently degrading. - Managed/BYOR LLM keys (server/routes/api_keys.py): refresh/generate endpoints are rejected when disabled; regular OpenHands application API keys (create/list/delete, current-key info) are untouched. - OpenHands/managed model discovery (server/verified_models/litellm_proxy_model_router.py): returns an empty managed-model list (BYOK catalogue only) instead of fetching from the proxy. - Usage monitoring (org_conversation_service.py): omits stale budget_monthly_limit/budget_is_disabled fields when LiteLLM is disabled so per-user usage stats don't imply enforcement that isn't happening; conversation/usage stats themselves are untouched (OHE's own data). - Org creation/membership (storage/org_service.py, org_store.py, org_member_service.py) already routed exclusively through LiteLlmManager, so they're covered by the manager-level gate with no further route changes needed. - New enable_litellm field on WebClientFeatureFlags, threaded through both config paths (DefaultWebClientConfigInjector + legacy ServerConfig.get_config), DB-resolved on every request like enable_billing. Frontend: - New FeatureDisabledScreen shared component ("Please enable LiteLLM to use this feature"). - Budgets and "Your budget" pages render that screen (skipping their fetches) instead of the full feature UI; their nav items are hidden. Tests: - New unit tests asserting the disabled path never constructs an httpx.AsyncClient across create_entries, verify_key, get_user_team_info, get_team_members_financial_data, remove_user_from_team, delete_team, ensure_free_team_models, and sync_free_model_allowlists. - New frontend test asserting the Budgets screen shows the placeholder and skips its data fetch when the flag is off. Co-authored-by: openhands <openhands@all-hands.dev>

Fixes a regression found while fleet-testing ENABLE_LITELLM=false (#474): saving personal LLM settings for a member provisioned while the flag is off returned 500 "Something went wrong storing settings" for any managed (openhands/*) model. Root cause: with ENABLE_LITELLM off, new-user provisioning (LiteLlmManager.create_entries) correctly skips minting a LiteLLM key, leaving agent_settings.llm.api_key as None. That None flowed into OrgMember.llm_api_key = None, whose setter (unlike its llm_api_key_for_byor sibling) unconditionally encrypted the raw value -- "successfully" JWE-encrypting a null payload into the NOT NULL _llm_api_key column. On the next settings save, decrypting that row produces a Python None wrapped in SecretStr, and pydantic's SecretStr.__len__/__bool__ call len(None) unconditionally, raising TypeError instead of reporting "empty". That TypeError isn't a ValueError, so store_settings' generic `except Exception` handler swallowed it into an opaque 500. - storage/org_member.py: the llm_api_key setter now normalizes None to '' before encrypting, matching the "empty means no key" convention has_real_api_key/llm_api_key_is_set already use. - storage/saas_settings_store.py: _ensure_api_key now checks has_real_api_key() instead of a bare bool()/truthiness check, as defense-in-depth against any already-poisoned rows (e.g. rows written before this fix, or via any other future code path that passes None through the same setter). - tests/unit/test_models.py: new regression test constructing an OrgMember with llm_api_key=None and asserting the round-tripped key is falsy/empty rather than raising. - tests/unit/test_saas_settings_store.py: new regression test calling _ensure_api_key with a SecretStr(None) fallback key and asserting a fresh key is generated instead of raising. Reproduced directly against the ENABLE_LITELLM=false fleet VM (shared-2) via a live Python script inside the running pod, confirmed identical to the reported traceback, then verified all four tests (2 new + existing suite) pass with the fix. Co-authored-by: openhands <openhands@all-hands.dev>

Found via a second live repro against the fleet VM after the first fix (9dc7e70): saving settings for an *existing* user whose default model is openhands/* (virtually every existing user/org) still 500'd with ENABLE_LITELLM off, because ValueError('LiteLLM API configuration not found') propagated straight out of LiteLlmManager.generate_key -- the managed-key ensure/rotate helpers had no flag check of their own; they relied on their LiteLLM-not-configured guard, which only trips when the env vars are unset, not when the flag disables an otherwise fully configured gateway. - storage/saas_settings_store.py (_ensure_api_key): returns immediately when disabled, leaving the existing key field untouched, instead of unconditionally trying to verify/rotate/generate a managed key. - storage/org_store.py (_ensure_managed_llm_key_for_user): same guard for the org-defaults save path. - server/maintenance_task_processor/managed_llm_key_ownership_processor.py: skips the whole batch (reporting all targets as skipped) instead of letting every target fail individually and get logged as a spurious repair error. - tests/unit/test_saas_settings_store.py, tests/unit/test_org_store.py: new regression tests asserting no LiteLlmManager call is made and the personal/org-defaults save paths return successfully when the flag is off. Found by reproducing the settings-save flow end-to-end inside the running shared-2 (ENABLE_LITELLM=false) fleet pod, on the image built from the prior None-key fix (9dc7e70): a synthetic org/user/member with no LiteLLM key on record and an openhands/* model still raised ValueError out of LiteLlmManager.generate_key. Will redeploy the fleet VMs on the image built from this commit to re-verify live. Co-authored-by: openhands <openhands@all-hands.dev>

_set_team_blocked and _block_team landed on main after this branch was cut. with_http_client passes client=None when LiteLLM is disabled and relies on each wrapped function re-checking the flag; these two did not, so they raised AttributeError instead of no-oping. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dylan-openhands · 2026-09-25T00:15:20Z

Superseded by the stack #519 → #520 → #521 (same changes, rebased on main and split into backend / frontend / model-discovery layers).

github-actions Bot added the type: feat A new feature label Sep 23, 2026

saurya and others added 4 commits September 24, 2026 17:09

dylan-openhands force-pushed the enable-litellm-flag branch from a8024c3 to 2c6b768 Compare September 24, 2026 23:45

dylan-openhands mentioned this pull request Sep 25, 2026

feat: add ENABLE_LITELLM deployment-wide flag (backend) #519

Open

2 of 6 tasks

dylan-openhands closed this Sep 25, 2026

Navigation Menu

feat: add ENABLE_LITELLM deployment-wide flag (default true) - #474

feat: add ENABLE_LITELLM deployment-wide flag (default true)#474
saurya wants to merge 4 commits into
mainfrom
enable-litellm-flag

saurya commented Sep 23, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

dylan-openhands commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

saurya commented Sep 23, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Testing

Manual/exploratory test plan (flag OFF)

Deployment verification (Replicated fleet)

Uh oh!

github-actions Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

dylan-openhands commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

saurya commented Sep 23, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 23, 2026 •

edited

Loading