Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cool down when hitting rate limit with online processors #256

Merged
merged 4 commits into from
Dec 14, 2024
Merged

Cool down when hitting rate limit with online processors #256

merged 4 commits into from
Dec 14, 2024

Conversation

RyanMarten
Copy link
Contributor

Fixes #252

We had cooldown before for openai:

curator/src/bespokelabs/curator/request_processor/openai_online_request_processor.py

Lines 456 to 473 in 40aa7df

# if a rate limit error was hit recently, pause to cool down
seconds_since_rate_limit_error = (
time.time()
- status_tracker.time_of_last_rate_limit_error
)
if (
seconds_since_rate_limit_error
< seconds_to_pause_after_rate_limit_error
):
remaining_seconds_to_pause = (
seconds_to_pause_after_rate_limit_error
- seconds_since_rate_limit_error
)
await asyncio.sleep(remaining_seconds_to_pause)
# ^e.g., if pause is 15 seconds and final limit was hit 5 seconds ago
logger.warn(
f"Pausing to cool down until {time.ctime(status_tracker.time_of_last_rate_limit_error + seconds_to_pause_after_rate_limit_error)}"
)

However, the way the main loop is running has changed so we need to change it a bit

@RyanMarten
Copy link
Contributor Author

Test

python tests/simple_online.py --log-level DEBUG --model deepinfra/meta-llama/Llama-3.3-70B-Instruct  --n-requests 5000 --max-requests-per-minute 3000 --max-tokens-per-minute 10000000000000 --max-retries 0

Observations:
This cool down only helps you if you are close to the correct rate limit and just need to pause sometimes.
If you are well over the correct rate limit, then as soon as you are un-paused you will send as many requests as possible and then once again be hit with tons of errors.
Because there is a delay between sending a request and getting a response, you can sends tons of requests even though you are already over the rate limit without knowing it.

Having something that is more responsive, as discussed in #233, like exponential retries that intelligently adapts and finds the correct rate limit would be better.

Right now we are relying on the local rate limit being correctly set. This PR helps mitigate failures when we hit rate limits randomly in patches (e.g. connection limit isn't actually a RPM rate limit so longer requests cause our estimation of RPM to be too large and we hit rate limits. This is also potentially better solved by #253)

@CharlieJCJ
Copy link
Contributor

Tested on claude 3.5 haiku, with a manual high rate limit, expected behaviors and logs
Command

(bespokelabs-curator-py3.12) (base) ➜  bella git:(ryanm/rate-limit-cool-down) ✗ python tests/simple_online.py --log-level DEBUG --model claude-3-5-haiku-20241022  --n-requests 5000 --max-retries 0;;
image

Some of my local logs

Pausing to cool down for 9 seconds
Seconds since rate limit error: 9.77731466293335
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: True
Pausing to cool down for 0 seconds
2024-12-14 21:42:28,402 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Pausing to cool down for 0 seconds
Pausing to cool down for 0 seconds
Seconds since rate limit error: 10.002909183502197
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.0047025680542
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.007076501846313
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.010509729385376
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.012789964675903
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.014800310134888
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.016777753829956
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.019784450531006
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.022103071212769
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.024208068847656
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.031663656234741
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.034281969070435
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.036867618560791
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.039520263671875
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.042527198791504
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.045552492141724
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.05044412612915
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.05648422241211
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.06947636604309
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.079070329666138
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.084420442581177
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.088730573654175
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.094618320465088
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.098593950271606
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.104597091674805
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.117860555648804
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.127331495285034
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.130967140197754
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.1381094455719
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.14184308052063
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.147015571594238
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.152969598770142
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.162304878234863
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.168402194976807
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.175575494766235
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.17983102798462
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.185556888580322
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.191916465759277
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.200887680053711
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.207088947296143
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.213093519210815
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.21965503692627
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.226195573806763
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.232726573944092
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.239461421966553
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Status tracker time of last rate limit error: 1734212548.655001
2024-12-14 21:42:28,655 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 228 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,655 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 228 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Seconds since rate limit error: 0.0009074211120605469
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: True
Pausing to cool down for 9 seconds
Status tracker time of last rate limit error: 1734212548.65985
2024-12-14 21:42:28,659 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 220 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,660 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 220 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.6647618
2024-12-14 21:42:28,665 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 232 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,665 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 232 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.6858182
2024-12-14 21:42:28,685 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 236 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,686 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 236 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.6901386
2024-12-14 21:42:28,690 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 234 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,690 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 234 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.694756
2024-12-14 21:42:28,694 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 233 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,695 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 233 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7018754
2024-12-14 21:42:28,702 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 235 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,702 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 235 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7118537
2024-12-14 21:42:28,711 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 237 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,712 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 237 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7199416
2024-12-14 21:42:28,720 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 238 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,720 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 238 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.724836
2024-12-14 21:42:28,724 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 239 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,725 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 239 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7375472
2024-12-14 21:42:28,737 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 241 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,737 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 241 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.744753
2024-12-14 21:42:28,744 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 242 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,745 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 242 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7602687
2024-12-14 21:42:28,760 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 240 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,760 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 240 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7695148
2024-12-14 21:42:28,769 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 243 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,769 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 243 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7756224
2024-12-14 21:42:28,775 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 244 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,775 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 244 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.781321
2024-12-14 21:42:28,781 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 255 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,781 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 255 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.7926455
2024-12-14 21:42:28,792 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 246 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,792 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 246 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.799815
2024-12-14 21:42:28,799 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 245 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,799 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 245 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.8107188
2024-12-14 21:42:28,810 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 247 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,811 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 247 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.8162231
2024-12-14 21:42:28,816 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 248 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,816 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 248 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.8222675
2024-12-14 21:42:28,822 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 249 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,822 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 249 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.8436055
2024-12-14 21:42:28,843 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 252 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,843 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 252 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.850122
2024-12-14 21:42:28,850 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 251 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,850 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 251 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.864091
2024-12-14 21:42:28,864 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 250 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,864 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 250 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.8703935
2024-12-14 21:42:28,870 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 253 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,870 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 253 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212548.9305582
2024-12-14 21:42:28,930 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 254 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:28,930 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 254 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Processing LiteLLMOnlineRequestProcessor requests:   2%|▋                            | 120/5000 [00:59<21:33,  3.77it/s]2024-12-14 21:42:38,662 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Pausing to cool down for 9 seconds
Pausing to cool down for 9 seconds
Seconds since rate limit error: 9.733609676361084
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: True
Pausing to cool down for 0 seconds
2024-12-14 21:42:38,932 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Pausing to cool down for 0 seconds
Pausing to cool down for 0 seconds
Seconds since rate limit error: 10.003473043441772
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.00525712966919
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.00659465789795
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.00950026512146
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.012519359588623
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.01540207862854
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.01784610748291
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.020509719848633
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.022376298904419
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.025001049041748
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.02773380279541
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.030853986740112
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.033184289932251
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.0364248752594
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.039759874343872
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.042680740356445
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.04600477218628
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.051254987716675
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.059996843338013
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.071706771850586
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.081772327423096
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.090312004089355
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.09453797340393
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.10104250907898
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.109780550003052
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.120887517929077
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.128454685211182
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.135640859603882
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.141072034835815
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.147302865982056
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.154208183288574
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.1580810546875
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.167264223098755
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.179534673690796
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.183794736862183
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.191077709197998
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.19514799118042
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.200405359268188
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.206177473068237
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.213131666183472
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.225002527236938
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.23212718963623
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.24032735824585
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Seconds since rate limit error: 10.246054649353027
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: False
Status tracker time of last rate limit error: 1734212559.1813638
2024-12-14 21:42:39,181 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 278 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,181 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 278 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Seconds since rate limit error: 0.006224393844604492
if seconds_since_rate_limit_error < SECONDS_TO_PAUSE_ON_RATE_LIMIT: True
Pausing to cool down for 9 seconds
Status tracker time of last rate limit error: 1734212559.1921756
2024-12-14 21:42:39,192 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 276 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,192 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 276 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2025673
2024-12-14 21:42:39,202 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 279 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,202 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 279 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.219472
2024-12-14 21:42:39,219 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 280 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,219 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 280 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2276735
2024-12-14 21:42:39,227 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 282 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,227 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 282 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2321038
2024-12-14 21:42:39,232 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 281 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,232 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 281 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2428505
2024-12-14 21:42:39,242 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 284 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,243 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 284 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2473881
2024-12-14 21:42:39,247 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 283 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,247 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 283 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2573447
2024-12-14 21:42:39,257 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 286 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,257 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 286 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2642558
2024-12-14 21:42:39,264 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 285 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,264 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 285 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.27671
2024-12-14 21:42:39,276 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 287 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,276 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 287 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2813115
2024-12-14 21:42:39,281 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 288 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,281 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 288 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.2903912
2024-12-14 21:42:39,290 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 289 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,290 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 289 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.295375
2024-12-14 21:42:39,295 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 300 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,295 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 300 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3054955
2024-12-14 21:42:39,305 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 277 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,305 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 277 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3112357
2024-12-14 21:42:39,311 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 291 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,311 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 291 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3181143
2024-12-14 21:42:39,318 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 290 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,318 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 290 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3278427
2024-12-14 21:42:39,327 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 292 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,328 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 292 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3329508
2024-12-14 21:42:39,333 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 301 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,333 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 301 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3407898
2024-12-14 21:42:39,341 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 293 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,341 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 293 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3492148
2024-12-14 21:42:39,349 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 294 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,349 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 294 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3543978
2024-12-14 21:42:39,354 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 295 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,354 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 295 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3633752
2024-12-14 21:42:39,363 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 296 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,363 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 296 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.368515
2024-12-14 21:42:39,368 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 297 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,368 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 297 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.3914592
2024-12-14 21:42:39,391 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 298 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,391 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 298 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']
Status tracker time of last rate limit error: 1734212559.4104059
2024-12-14 21:42:39,410 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 299 failed with Exception litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}, attempts left 0
2024-12-14 21:42:39,410 - bespokelabs.curator.request_processor.base_online_request_processor - ERROR - Request 299 failed permanently after exhausting all 0 retry attempts. Errors: ['litellm.RateLimitError: AnthropicException - {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your organization’s rate limit of 80,000 output tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}']

CharlieJCJ
CharlieJCJ approved these changes Dec 14, 2024
View reviewed changes
Copy link
Contributor

@CharlieJCJ CharlieJCJ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@RyanMarten RyanMarten merged commit 879016e into dev Dec 14, 2024
2 checks passed
@RyanMarten RyanMarten deleted the ryanm/rate-limit-cool-down branch December 14, 2024 22:08
@RyanMarten RyanMarten mentioned this pull request Dec 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CharlieJCJ CharlieJCJ CharlieJCJ approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@RyanMarten @CharlieJCJ