Delete input and output files for successful batches

[RyanMarten]

Fixes #189

Test

from bespokelabs.curator import Prompter
from datasets import Dataset
import logging

# To see more detail about how batches are being processed
logger = logging.getLogger("bespokelabs.curator")
logger.setLevel(logging.INFO)
dataset = Dataset.from_dict({"prompt": ["write me a poem"] * 3})

prompter = Prompter(
    prompt_func=lambda row: row["prompt"],
    model_name="gpt-4o-mini",
    response_format=None,
    batch=True,
    batch_size=1,
)

dataset = prompter(dataset)
print(dataset.to_pandas())

Should print something like

2024-12-03 18:33:02,712 - bespokelabs.curator.request_processor.openai_batch_request_processor - INFO - Batch batch_674fbea486d881919d3eed7675b6ee4f returned with status: completed
Completed OpenAI requests in batches:  67%|████████████████████████████▋              | 2/3 [03:05<01:32, 92.92s/request]2024-12-03 18:33:03,337 - bespokelabs.curator.request_processor.openai_batch_request_processor - INFO - Batch batch_674fbea486d881919d3eed7675b6ee4f completed and downloaded
2024-12-03 18:33:03,341 - bespokelabs.curator.request_processor.openai_batch_request_processor - INFO - Batch batch_674fbea486d881919d3eed7675b6ee4f written to /Users/ryan/.cache/curator/d62efd02c09bf424/responses_1.jsonl
2024-12-03 18:33:03,848 - bespokelabs.curator.request_processor.openai_batch_request_processor - INFO - Deleted file file-RBs1sVu52BGJRXYP29Le4B
2024-12-03 18:33:04,257 - bespokelabs.curator.request_processor.openai_batch_request_processor - INFO - Deleted file file-MKZdvmugFwc46RLHf6q17H

You can also see files appear and disappear in the dashboard:
https://platform.openai.com/storage/files

[RyanMarten]

The default in prompter is

        delete_successful_batch_files: bool = True,
        delete_failed_batch_files: bool = False,  # To allow users to debug failed batches

Test with this toggled off

from bespokelabs.curator import Prompter
from datasets import Dataset
import logging

# To see more detail about how batches are being processed
logger = logging.getLogger("bespokelabs.curator")
logger.setLevel(logging.INFO)
dataset = Dataset.from_dict({"prompt": ["write me a poem"] * 3})

prompter = Prompter(
    prompt_func=lambda row: row["prompt"],
    model_name="gpt-4o-mini",
    response_format=None,
    batch=True,
    batch_size=1,
    delete_successful_batch_files=False
)

dataset = prompter(dataset)
print(dataset.to_pandas())

You should expect to see the files still in the dashboard and no delete logs.

[vutrung96]

I cannot test given my unstable internet but the logic makes sense to me. LGTM but please make sure the tests work :D