[CB] add min batch size of 2 in decode

[nikolaospapandreou]

This PR adds support for minimum batch size of 2 in decode steps.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[sducouedic]

LGTM (just a minor comment)
nice changes 👍

[sducouedic]

I think appendleft won't do exactly what you want. It will append the values from dummy_req_ids2blocks to the left, but in reverse order. e.g. [6, 7] --> appendleft --> [7, 6, 8, 9, 10, ...]. I would suggest to use append simply, just as you did in line 620. Or iterate the dummy_req_ids2blocks in reverse order (and do the same in 620)

[joerunde]

Is this popping free blocks for every decode iteration of a batch with a single request? It looks like len(input_tokens) == 1 will always be true when there's only a single request running, since cached_requests comes from the scheduler and won't have our dummy request in it

[joerunde]

ahhh I see the num_blocks - len(self.dummy_req_ids2blocks) is only popping ones as the tkv increases 👍

[yannicks1]

Couldn't we simply use self.req_ids2blocks for this?
Just add an entry self.req_ids2blocks['dummy_req'] to it when needed and clean it up again afterwards. This would not only eliminate this additional variable, but potentially you could also reuse some of the code of freeing up the blocks again.

[yannicks1]

Can you explain why we need this cleanup here as well? self._update_states() is always called before self.prepare_model_input() (which invokes _prepare_prompt()). Would only doing the cleanup in one place be possible?

[sducouedic]

I guess this would do the cleanup when the last requests finishes?

[yannicks1]

Not sure if these have to make sense (align in some way) for fms/AIU Spyre compiler, but to be on the safe side, we could simply re-use the values from the real request here.

[sducouedic]

I am not completely sure if this won't automatically be handled correctly in the subsequent _prepare_decode, but probably in reduce_left_padding we should also free the dummy blocks

[sducouedic]

Suggested change

	for i in range(num_blocks - len(self.dummy_req_ids2blocks)):
	for _ in range(num_blocks - len(self.dummy_req_ids2blocks)):

[yannicks1]

I am not completely sure if this won't automatically be handled correctly in the subsequent _prepare_decode, but probably in reduce_left_padding we should also free the dummy blocks

good point! this will be solved when doing the removal of the left padding in every step (decode and prefill). I am taking care of this on this branch: https://github.com/vllm-project/vllm-spyre/tree/ysc-refactor-left-padding-stripping