You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the LlavaStreamGenerator function at tinychat/stream_generators/llava_stream_gen.py processes inputs one at a time. To improve performance and throughput, we should implement batch processing capabilities. This will allow the function to handle multiple inputs simultaneously, potentially leading to significant speed improvements.
Quick action plan is to adjust the main generation loop to work with multiple sequences. But before starting wanted to confirm, Are there any model-specific considerations for batch processing with VILA models?
The text was updated successfully, but these errors were encountered:
Currently, the
LlavaStreamGenerator
function attinychat/stream_generators/llava_stream_gen.py
processes inputs one at a time. To improve performance and throughput, we should implement batch processing capabilities. This will allow the function to handle multiple inputs simultaneously, potentially leading to significant speed improvements.Quick action plan is to adjust the main generation loop to work with multiple sequences. But before starting wanted to confirm, Are there any model-specific considerations for batch processing with VILA models?
The text was updated successfully, but these errors were encountered: