Improve error handling for API access issues

[devin-ai-integration]

Improve Error Handling for API Access Issues

This PR improves error handling in the curator library by making error messages more informative when API requests fail, particularly for API access and model permission issues.

Changes

• Enhanced error messages when all API requests fail to include specific error details
• Added collection and display of individual error messages (up to 5) in failure summary
• Improved user feedback for API key and model access issues
• Added comprehensive tests for error handling scenarios
• Maintained backward compatibility with existing functionality (verified with examples/poem.py)

Testing

✅ Changes have been tested with:

• Unit tests for model access errors
• Function-based error handling tests
• Backward compatibility tests with examples/poem.py
• Tests with both valid and invalid API keys
• Tests for error message clarity and helpfulness

Implementation Details

• Added error message collection in BaseRequestProcessor
• Enhanced error aggregation to show specific error details
• Improved error messages to guide users in fixing API access issues
• Maintained existing functionality while adding better error reporting

Link to Devin run: https://preview.devin.ai/devin/56e42bcae3954c0aa5e6316239b5f717

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR
• Look at CI failures and help fix them

⚙️ Control Options:

• Disable automatic comment and CI monitoring

Additional Controls

You can control how I interact with specific messages:

• Add "(aside)" to your comment to have me ignore it
• Use "@devin" to explicitly get my attention on a comment

[RyanMarten]

Fixes #213

[RyanMarten]

There actually looks like there is another error - looking closer at #213

d with Exception API error: {'message': 'Project `proj_849B36OelpiRtrG7hEPPgVSD` does not have access to model `gpt-4o`', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}, attempts left 4
2024-12-04 18:44:30,733 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Adding request 0 to retry queue. Will retry in next available slot. Attempts remaining: 3
Processing OpenAIOnlineRequestProcessor requests:   0%|                                           | 0/10 [00:00<?, ?it/s]
2024-12-04 18:44:30,736 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Processing complete. Results saved to /Users/ryan/.cache/curator/f467067fea2470e9/responses_0.jsonl
2024-12-04 18:44:30,737 - bespokelabs.curator.request_processor.base_online_request_processor - INFO - Status tracker: Tasks - Started: 10, In Progress: 10, Succeeded: 0, Failed: 0, Already Completed: 0
Errors - API: 20, Rate Limit: 0, Other: 20, Total: 40
Traceback (most recent call last):
  File "/Users/ryan/curator/examples/persona-hub/synthesize.py", line 75, in <module>
    main(args)
  File "/Users/ryan/curator/examples/persona-hub/synthesize.py", line 46, in main
    output = generator(persona_dataset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ryan/curator/src/bespokelabs/curator/prompter/prompter.py", line 191, in __call__
    return self._completions(self._request_processor, dataset, working_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ryan/curator/src/bespokelabs/curator/prompter/prompter.py", line 279, in _completions
    dataset = request_processor.run(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ryan/curator/src/bespokelabs/curator/request_processor/base_online_request_processor.py", line 177, in run
    return self.create_dataset_files(working_dir, parse_func_hash, prompt_formatter)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ryan/curator/src/bespokelabs/curator/request_processor/base_request_processor.py", line 200, in create_dataset_files
    raise ValueError(f"No responses files found in {working_dir}")
ValueError: No responses files found in /Users/ryan/.cache/curator/f467067fea2470e9

Even though it said it wrote to response file, no response file was written and this leads to that value error (which is raised before getting to your new code)