Batched inferencing using HuggingfaceLocalGenerator

[srsingh24]

Is there a way to perform batched inferencing using HuggingfaceLocalGenerator? I could not find any information about this on the docs.

[anakin87]

Hello!

While it should be possible to configure the batch_size of the Hugging Face pipeline/model under the hood,
this component only accepts a prompt (single str) as input, therefore this practically does not allow batching.

Could you tell me more about your use case?

[srsingh24]

@anakin87 Since the HuggingFaceLocalGenerator only accepts a single str input, I am having to sequentially loop through each prompt. This makes running my code very slow. I would like to perform batched inference so I can make better use of my GPUs and run the inference faster

[anakin87]

Clear... I wanted to better understand what you are building...
Are you running evaluation? Performing RAG? ...

[srsingh24]

I am using the HuggingfaceLocalGenerator for all sorts of use cases from basic inferencing on a list of prompts to evaluation to RAG