Gemini Flash runs very slow (1 it/s)

[RyanMarten]

• api key and paid tier https://aistudio.google.com/app/apikey
• quota (rate limits) monitoring https://console.cloud.google.com/apis/api/generativelanguage.googleapis.com/quotas?inv=1&invt=AbjX7w&project=bespokelabs
• usage (server response and latency) montoring https://console.cloud.google.com/apis/api/generativelanguage.googleapis.com/metrics?project=bespokelabs

Only doing one request at a time

[CharlieJCJ]

What happened here was that 1) we've set a slow default tpm, which is 150k, and 2) we did a conservative output token estimation, which doesn't exhaust the token capacity.
Related issues: #206

The fix for this is manually set the tpm to a higher value. I'll followup with a higher default tpm.

[CharlieJCJ]

Example script

from bespokelabs.curator import Prompter
from datasets import Dataset
import logging

logger = logging.getLogger("bespokelabs.curator")
logger.setLevel(logging.DEBUG)

dataset = Dataset.from_dict({"prompt": ["write me a poem"] * 100_000})

prompter = Prompter(
    prompt_func=lambda row: row["prompt"],
    model_name="gemini/gemini-1.5-flash-002",
    response_format=None,
    max_requests_per_minute=2000,
    max_tokens_per_minute=4000000,
)

dataset = prompter(dataset)
print(dataset.to_pandas())

[CharlieJCJ]

Also works for gemini-1.5-pro-002

[CharlieJCJ]

gemini-1.5-pro-002

gemini-1.5-flash-002