- Added cost tracking for
gpt-4o-mini
- Removed in-memory cache of provider settings
- Removed in-memory cache of keys
- Added support for OpenAI vector store APIs
- Added support for OpenAI vector store files APIs
- Added support for OpenAI vector store batches files APIs
- Added a new env variable for removing
User-Agentfrom http headers.
- Added Prometheus integration thanks to @GingerMoon
- Added latest models to Azure OpenAI thanks to @minhoryang
- Fixed OpenAI transcription endpoint not working error
- Added support for storing metadata per request via the
X-METADATAheader
- Added deletion endpoint for routes
- Added support for new models from OpenAI and Azure OpenAI to routes
- Added support for Anthropic sonnet 3.5
- Added retry strategy to routes
- Added support for Azure OpenAI completions endpoint
- Changed to using sha256 hashing algorithm for computing cache key within routes
- Added filtering by status code for events v2 API
- Fixed issues with updating provider settings
- Removed unused logs
- Changed from using response model to using request model as the source of truth for customized pricing
- Removed debug logs
- Added support for customized model pricing for Azure OpenAI, OpenAI, vLLM and Deep Infra
- Added support for OpenAI batch APIs
- Fixed event not being recorded when request body is empty and request logging is turned on
- Added support for
gpt-4oinside Azure OpenAI
- Fixed Azure OpenAI token counting when streaming
- Fixed configuration files priority issue
- Removed unused env variables
- Fixed
.envfile cannot be openned error
- Added
.envandjsonfile configurations for env variables
- Removed unused funcition variables such as
cid
- Updated
goversion to1.22
- Fixed token counting for
gpt-4o
- Reconfigured logger to always log
correlationId
- Reconfigured logger to always log
correlationId
- Added
requestFormatandrequestParamsto routes
- Added policy integration with assistant APIs
- Fixed issue with policy running even if policy input is empty
- Added support for gpt-4o
- Fixed policy bug where regex redaction rules run even when there isn't a match
- Added
redactedas an option for filters
- Updated v2 events API to allow more array params such as
userIdsinstead ofuserId
- Fixed an issue with forwarding requests to OpenAI's Assistant APIs
- Added v2 get events API
- Added a new filter for retrieving the total number of keys for the get keys v2 API
- Changed get keys V2 API response format
- Fixed PII detection bug
- Added new env variable for setting AWS region
AMAZON_REGION - Added new
nameandorderfilters to get keys V2 API - Added new
nameandrevokedfilters to top keys API
- Fixed top keys API for not returning keys with no spend
- Added a new API
/api/v2/key-management/keysfor retrieving keys - Added a new API
/api/reporting/top-keysfor retrieving key IDs sorted by spend - Added a new API
/api/usersfor cerating a user - Added a new API
/api/users/:idfor updating a user via user ID - Added a new API
/api/usersfor updating a user via tags and user ID - Added a new API
/api/usersfor getting users
- Added new admin API endpoints for retriving
customIdsanduserIds
- Added Deepinfra integration
- Changed update provider setting behavior to only do partial updates for
settingfield
- Provider settings APIs start returning
settingfield without containingapikey
- Fixed an issue with
apikeybeing required for making vLLM requests
- Added support for
apikeyin vLLM integration
- Added vLLM integration
- Fixed revoking key without hitting cost limit error
- Added fetch from DB if a provider setting associated with a key is not found in memory
- Fixed user Id not being captured with Anthropic's new messages endpoint
- Added integration with Anthropic's new messages endpoint
- Fixed a db query issue with the new reporting endpoint
- Added new reporting endpoint for retrieving metric data points by day
- Added a new key field called
isKeyNotHashed
- Fixed http response exceeding
1GBwhen receiving a streaming error from OpenAI
- Updated error messages for authentication errors
- Changed error message for authentication errors
- Added more detailed logs for authentication errors
- Added stats when key is not found in cache
- Added fallback when key is not found in cache
- Fixed
ttlbehavior not to disable keys
- Updated default postgresql read operation timeout from
15sto2m
- Fixed
nilpointer error s
- Added
policyAPIs
- Started supporting storing streaming responses in bytes as part of
event
- Fixed issue with event not being recorded if
shouldLogResponseis set totruefor streaming responses
- Updated update key API to support setting
costLimitInUsdOverTime,costLimitInUsdandrateLimitOverTimeto 0
- Added querying keys by
keyIds - Increased default postgres DB read timeout to
15sand write timeout to5s
- Added setting rotation feature to key
- Added cost tracking for OpenAI audio endpoints
- Added inference cost tracking for OpenAI finetune models
- Added
userIdas a new filter option for get events API endpoint - Added option to store request and response using keys
- Added support for updating key cost limit and rate limit
- Removed validation to updating revoked key field
- Added new model "gpt-4-turbo-preview" and "gpt-4-vision-preview" to the cost map
- Added support for calculating cost for the cheaper 3.5 turbo model
- Added validation to updating revoked key field
- Added CORS support in the proxy
- Fixed custom route tokens recording issue incurred by the new architecture
- Fixed OpenAI chat completion endpoint being slow
- Drastically improved performance through event driven architecture
- Fixed API calls that exceeds cost limit not being blocked bug
- Added support for new chat completion models
- Added new querying options for metrics and events API
- Extended default proxy request timeout to 10m
- Fixed streaming response stuck at context deadline exceeded error
- Added key authentication for admin endpoints
- Changed code to string in OpenAI error response
- Fixed inability to parse OpenAI chat completion result
- Fixed problem with keys containing duplicated API credentials
- Fixed key exceeding spend limit error code issue
- Fixed a bug with route failover not working as expected
- Fixed route caching issue
- Added default values to route step timeout and cache TTL
- Started supporting caching in routes
- Added the ability to make API calls with fallbacks
- Added the support for multiple sets of provider settings per key
- Added support for Azure OpenAI embeddings and chat completions endpoints
- Added support for minute spend limits
- Added support for OpenAI audio endpoints
- Added support for OpenAI image endpoints
- Added support for monthly spend limits
- Removed deperacated fields when updating keys
- Fixed issues with inconsistent rate limit cache expiration date
- Added path access control at the API key level
- Added model access control at the provider level
- Inability to update provider settings with only
settingfield
- Added support for Anthropic completion endpoint
- Fixed streaming latency with custom provider and openai chat completion endpoints
- Fixed bug with updating a custom provider
- Added new API endpoints for custom providers
- Added validation for changing setting when updating provider setting
- Removed support for changing provider when updating provider setting
- Fixed issues with not being able to update provider setting name
- Added new filters tags and provider to getting keys API
- Added a new API endpoint for fetching an event
- Added a new API endpoint for fetching all provider settings
- Added more integration tests that cover critical paths
- Fixed issues with API key creation
- Introduced new field called
nameto provider settings
- Fixed issues with PostgreSQL schema inconsistencies
- Added support for OpenAI's embeddings API
- Fixed configuration not found inconsistency with key and provider settings
- Updated admin server to pull configurations every 5 sec by default
- Updated prod Dockerfile to have privacy mode turned on
- Added a default OpenAI proxy timeout of 180s
- Added model and keyId filters to the metrics API
- Added health check endpoints for both admin and proxy servers
- Started recording path and method for each proxy request
- Fixed Datadog tags being in the wrong format
- Added Datadog integration
- Fixed bug with proxy not fetching existing provider settings
- Added support for gpt-4-turbo models in cost estimation
- Fixed proxy logic to prevent internal errors from forwarding requests to OpenAI
- Fixed proxy logic to prevent internal errors from receiving responses from OpenAI
- Added a new API endpoint for creating and updating provider settings (e.g. OpenAI API key)
- Removed having OpenAI API key as an required env vairable for running the proxy
- Forced key creation to include a new field called settingId
- Added a new API endpoint for retrieving metrics
- Fixed open ai error response parsing
- Added new reporting endpoint for retrieving the cumulative cost of an API key
- Added support of cost estimation for fined tuned models
- Added logging for openAI error response
- Fixed incorrect logging format in production environment
- Added logging for admin and proxy web servers
- Fix PostgreSQL connection URI format issue
- Updated PostgreSQL connection URI to be more versatile
- Changed env var POSTGRESQL_SSL_ENABLED to POSTGRESQL_SSL_MODE
- Updated proxy error response to exclude empty fields
- Added a new environment varible for configurting PostgreSQL DB Name
- Fixed issue with rate limit
- Fixed issue with cost limit
- Added a web server for managing key configurations
- Added a proxy server for openai request
- Added rate limit, cost limit and ttl to API keys