How to use rate limit in add_routes

[kientv]

@eyurtsev
I have 2 problems:

1. Handle rate limit of OpenAI and other llm.
2. Rate limit on langserve (add_routes).

How to do it by langserve?

Thanks in advances

[eyurtsev]

Easiest way is to prepend a runnable lambda to your runnable. That runnable lambda should be a passthrough that implements a rate limiting algorithm. The easiest way to do this is using a token bucket algorithm. We haven't built in support into the framework for this yet. In the meanwhile you can adapt the code here: https://github.com/langchain-ai/langchain-benchmarks/blob/main/langchain_benchmarks/rate_limiting.py#L90