OnlineRequestProcessor only retry once

[CharlieJCJ]

Related code:
https://github.com/bespokelabsai/curator/blob/dev/src/bespokelabs/curator/request_processor/base_online_request_processor.py#L331-L364

Minimal reproduction, run examples/poem.py without passing into a key, at the end, it will retry once, and put back to the queue, but never retry again.

Relevant log

(bespokelabs-curator-py3.12) (base) ➜  bella git:(CURATOR-28-add-a-lite-llm-backend-for-curator) ✗ python examples/poem.py
2024-12-04 18:21:19,188 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Running OpenAIOnlineRequestProcessor completions with model: gpt-4o-mini
2024-12-04 18:21:19,369 - bespokelabs.curator.request_processor.openai_online_request_processor - WARNING - Failed to get rate limits from OpenAI API, using default values
Processing OpenAIOnlineRequestProcessor requests:   0%|                                           | 0/1 [00:00<?, ?it/s]2024-12-04 18:21:19,481 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 0 failed with Exception API error: {'message': 'Incorrect API key provided: sk-proj-********************************************************************************************************************************************************EHsA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}, attempts left 5
2024-12-04 18:21:19,481 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Adding request 0 to retry queue. Will retry in next available slot. Attempts remaining: 4
2024-12-04 18:21:19,482 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Processing final retry for request 0 (attempt #2 of 5). Previous errors: [Exception("API error: {'message': 'Incorrect API key provided: sk-proj-********************************************************************************************************************************************************EHsA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}")]
2024-12-04 18:21:19,550 - bespokelabs.curator.request_processor.base_online_request_processor - WARNING - Request 0 failed with Exception API error: {'message': 'Incorrect API key provided: sk-proj-********************************************************************************************************************************************************EHsA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}, attempts left 4
2024-12-04 18:21:19,550 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Adding request 0 to retry queue. Will retry in next available slot. Attempts remaining: 3
Processing OpenAIOnlineRequestProcessor requests:   0%|                                           | 0/1 [00:00<?, ?it/s]
2024-12-04 18:21:19,551 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Processing complete. Results saved to /home/charlieji/.cache/curator/50f9cdfb94fa2acd/responses_0.jsonl
2024-12-04 18:21:19,551 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Status tracker: Tasks - Started: 1, In Progress: 1, Succeeded: 0, Failed: 0, Already Completed: 0
Errors - API: 2, Rate Limit: 0, Other: 2, Total: 4
Traceback (most recent call last):
  File "/home/charlieji/workspace/bella/examples/poem.py", line 29, in <module>
    topics: Dataset = topic_generator()
                      ^^^^^^^^^^^^^^^^^
  File "/home/charlieji/workspace/bella/src/bespokelabs/curator/prompter/prompter.py", line 187, in __call__
    return self._completions(self._request_processor, dataset, working_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/workspace/bella/src/bespokelabs/curator/prompter/prompter.py", line 275, in _completions
    dataset = request_processor.run(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/workspace/bella/src/bespokelabs/curator/request_processor/base_online_request_processor.py", line 177, in run
    return self.create_dataset_files(working_dir, parse_func_hash, prompt_formatter)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/workspace/bella/src/bespokelabs/curator/request_processor/base_request_processor.py", line 200, in create_dataset_files.0.1.1-604800000', 'Content-Length': '148', 'Content-Type': 'application/json')>, real_url=URL('https://api.openai.com/v1/chat/completions')), (), status=502, message='Attempt to decode JSON with unexpected mimetype: text/html', heade
(bespokelabs-curator-py3.12) (base) ➜  bella git:(CURATOR-28-add-a-lite-llm-ba

Previous testings in #141 all succeeded after one retry.