Feature: tlm batch api #149
Conversation
| @@ -46,21 +63,147 @@ def __init__(self, api_key: str, quality_preset: QualityPreset) -> None: | |||
|
|
|||
| self._quality_preset = quality_preset | |||
|
|
|||
| self._event_loop = asyncio.get_event_loop() | |||
| self._query_semaphore = asyncio.Semaphore(max_concurrent_requests) | |||
There was a problem hiding this comment.
do we want to limit concurrency with the asyncio semaphore? if i set it to a high value, i.e. 1000, i get the generic API error. note: it doesn't hit the RateLimitError, which seems to be the expected one based on the error message suggesting to lower max_concurrent_requests . not sure how likely it is that the typical user will be setting this value or just using the default
|
When trying to run scenario 1 linked for testing from the staging environment, I get below error. I am not sure if it is my environment or code issue. My env: staging, cleanlab-studio is pointed to editable install of feature/tlm-batch-api (this branch) api_key = <api_key>
tlm = Studio(api_key).TLM(quality_preset="low", max_concurrent_requests=64)
tlm.batch_prompt([f"what's 2 + {i}?" for i in range(128)], retries=1)
|
Running async code in a Jupyter notebook will require the following: https://github.com/erdewit/nest_asyncio |
Adds
batch_prompt,batch_get_confidence_scoreAPIs to enable scale use of TLMIncludes refactor to use of async requests (enabling request concurrency)
Both batch methods are intended to gracefully handle query exceptions w/ retries. For rate limit errors, the retry occurs after a wait specified by the backend. For other errors, an exponential backoff is applied.
See testing instructions: https://github.com/cleanlab/cleanlab-studio-backend/pull/988