KeyError: 'raw_prompt'

[YixinSong-e]

Very nice work!
I'm runing PPO using the hhrlhf datasets in verl repo. And the error is here.

  File "/home/syx/rlhf/verl/single_controller/ray/base.py", line 395, in func
    return getattr(self.worker_dict[key], name)(*args, **kwargs)
  File "/home/syx/rlhf/verl/single_controller/base/decorator.py", line 404, in inner
    return func(*args, **kwargs)
  File "/home/syx/rlhf/verl/verl/trainer/ppo/workers/fsdp_workers.py", line 795, in compute_rm_score
    rm_data = self._switch_chat_template(data)
  File "/home/syx/rlhf/verl/verl/trainer/ppo/workers/fsdp_workers.py", line 745, in _switch_chat_template
    chat: list = data.non_tensor_batch['raw_prompt'][i].tolist()
KeyError: 'raw_prompt

[YixinSong-e]

And there is another error.

def _pre_process_inputs(pad_token_id, prompt_token_ids: torch.Tensor) -> List[int]:
    # remove the left padding in the prompt token_id
    # pad_token_id = self.llm_engine.tokenizer.pad_token_id if self.llm_engine.tokenizer.pad_token_id is not None else self.llm_engine.tokenizer.eos_token_id
    non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0]
    token_ids = prompt_token_ids[non_pad_index:].tolist()
    return token_ids

non_pad_index = torch.nonzero(prompt_token_ids != pad_token_id, as_tuple=False)[0][0]
TypeError: nonzero(): argument 'input' (position 1) must be Tensor, not bool

It seems that the first paramter should not be prompt_token_ids != pad_token_id, but a Tensor.

[PeterSH6]

Hi @YixinSong-e , for the first issue, it's because you have set the input_tokenizer in reward model. This field in config will be set only when the reward model utilize different tokenizer from the actor model.

So, you only need to set the data.return_raw_chat=True as shown in our example script to fix it. You can refer to our config explaination in the document for more information.

For the second issue, it seems that the input_ids in your case is a scalar not tensor. However, this input_ids should be the output of the tokenizer from the DataProto. Could you give us more information about the LoC that make the input_ids a scalar?