Investigate / improve streaming speed

[abetlen]

Streaming speed seems to be noticeably slower than regular completion.

[m-from-space]

Just my two cents here: I prefer streaming over regular inference with llama-cpp-python, because somehow it works "smoother". When I ask the model for a reply using normal inference, my graphics card's fans immediately go into overdrive (while they are at rest when idle), and my PC sounds like it is going to fly away. When using streaming on the other hand, the fans only smoothly start, when it takes a bit longer, but the performance is nearly the same.