random_timesteps does not work for PPO_RNN

[khanhphan1311]

Hi @Toni-SM ,

I trained PPO with LSTM model and tried to add random_timesteps in cfg but it returned error.

To be specific, in ppo_rnn.py in act method:

if timestep < self._random_timesteps:
    return self.policy.random_act({"states": self._state_preprocessor(states), **rnn}, role="policy")

Because self.policy.random_act returns log_prob as None, it will cause error when we add samples to memory here:

self.memory.add_samples(states=states, actions=actions, rewards=rewards, next_states=next_states,
                                    terminated=terminated, truncated=truncated, log_prob=self._current_log_prob, values=values, **rnn_states)

Here is error log:

Traceback (most recent call last):
  File "D:\2. COMPANY\9. SIMPLIZE\DATABASE\8_simplize_rl\nebula\custom_trainer.py", line 203, in train_single_agent
    self.agents.record_transition(states=states,
  File "D:\1. APPS\Anaconda\envs\8_simplize_finrl\lib\site-packages\skrl\agents\torch\ppo\ppo_rnn.py", line 303, in record_transition
    self.memory.add_samples(states=states, actions=actions, rewards=rewards, next_states=next_states,
  File "D:\1. APPS\Anaconda\envs\8_simplize_finrl\lib\site-packages\skrl\memories\torch\base.py", line 266, in add_samples
    self.tensors[name][self.memory_index].copy_(tensor)
TypeError: copy_(): argument 'other' (position 1) must be Tensor, not NoneType

Are there any ways to random timesteps for PPO_RNN?

[Toni-SM]

Hi @khanhphan1311

The execution of random actions for stochastic policies is not yet implemented 😅🙈