Relationship between the skrl models and Isaac Gym models

[famora2]

Hello,

according to the documentation (https://skrl.readthedocs.io/en/latest/intro/getting_started.html#models), this library creates another model that functions as the agent's brain. How are these models connected to the model that is for example defined in the Isaac Gym settings (see below). As you can in the below example of Cartpole from Isaac Gym, it uses the a2c_continuous algorithm and utilizes a MLP with two layers as its brain, right?

params:
  seed: ${...seed}

  algo:
    name: a2c_continuous

  model:
    name: continuous_a2c_logstd

  network:
    name: actor_critic
    separate: False
    space:
      continuous:
        mu_activation: None
        sigma_activation: None

        mu_init:
          name: default
        sigma_init:
          name: const_initializer
          val: 0
        fixed_sigma: True
    mlp:
      units: [32, 32]
      activation: elu
      
      initializer:
        name: default
      regularizer:
        name: None

[Toni-SM]

Hi @famora2

The mapping between Isaac Gym's models (rl_games) and skrl's models is as follows:

rl_games	skrl
a2c_continuous (the implementation is PPO)	PPO
continuous_a2c_logstd	Gaussian or mutlivariate Gaussian Policy with deterministic Value
separate	whether to create a shared or separate model
mu_activation	final activation before returning the mean actions (in the policy) in .compute() method
sigma_activation	final activation before returning the log_std (in the policy) in .compute() method
mu_init	-
sigma_init	initial value for the log_std
fixed_sigma	whether to use nn.Parameter for returning the log_std instead of the output of an ANN layer
mlp: units	nn.Linear layers specification
mlp: activation	layers activation function
initializer	-
regularizer	-

You can inspect the skrl's Isaac Gym examples and compare the model definitions with the Isaac Gym configuration .yaml files to see how the mapping was done for each task :)

[famora2]

Thank you for the quick answer. By default, Isaac Gym uses train.py (https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/blob/main/isaacgymenvs/train.py) to start the train the agent, which again uses the model_builder.py from rl_games (https://github.com/Denys88/rl_games/blob/master/rl_games/algos_torch/model_builder.py).

The reason why skrl has its own scripts for models (gaussian.py, deterministic.py, etc) is to have universal scripts for Isaac Gym, OpenAi and Deepmind?

[Toni-SM]

The existence of different helpers (Python mixins) for constructing models does not depend on the environment interface (Isaac Gym, Gym/Gymnasium, or DeepMind), but on the nature of the observations/actions (Discrete, Box, etc...) supported by the environment and by the agent (deterministic or stochastic)...

Citing skrl docs:

In contrast to other libraries, skrl does not provide predefined models or fixed templates (this practice tends to hide and reduce the flexibility of the system, forcing developers to deeply inspect the code to make even small changes). Nevertheless, helper classes/mixins are provided to create discrete and continuous (stochastic or deterministic) models with the library. In this way, the user/researcher should only be concerned with the definition of the approximation functions (tables or artificial neural networks), having all the control in his hands.

For example:

• If you need a deterministic policy for the continuous domain (DDPG case, for example)...
  skrl offers the Deterministic mixin to help you to construct this model
• If you need a stochastic policy for the continuous domain (PPO case, for example)...
  skrl offers the Gaussian and MultivariateGaussian mixins to help you to construct the model
• If you need a stochastic policy for the discrete domain (CEM case, for example)...
  skrl offers the Categorical mixin to help you to construct the model

Refer to each agent docs (under the Spaces and models section) to know the supported model specification and observation and action space :)

[Toni-SM]

In the case of Isaac Gym, all the environments have continuous observation and action spaces (gym.spaces.Box)...