Agent Types

Basic Motor Joints and Free Joints

MuJoCo (Multi-Joint dynamics with Contact) is a physics engine commonly used in the field of robotics and reinforcement learning (RL). Using our environment wrapper, two agent types are can be employed for various RL-tasks: agents utilizing basic motor joints and agents with free joints. These agent types differ in their movement and orientation mechanisms, as well as the observations they receive. Users of the environment wrapper can switch between both modus operandi by setting the hyperparameter freeJoint to True or False.

from MuJoCo_Gym.mujoco_rl import MuJoCo_RL

configDict = {
   "agents": ["Agent1", "Agent2"], 
   "xmlPath": "/path/to/xml/file.xml",
   "freeJoint": True
}

environment = MuJoCo_RL(config_dict)

Basic Motor Joint Agent

The basic motor joint agent in MuJoCo employs motorized joints to control its movements and orientation in the environment. Each joint is actuated by a motor, enabling precise control over joint angles and forces. This agent receives the states of all its joints as part of its observations. These joint states typically include information such as joint angles, joint velocities, joint torques, and joint limits.

The basic motor joint agent's ability to manipulate its joints directly offers a high degree of control over its movements. By exerting forces and torques on the joints, it can perform complex actions such as reaching, grasping, and locomotion. The joint states provided as observations allow the agent to have detailed knowledge of its body configuration and dynamics, enabling it to make informed decisions in its interactions with the environment.

Free Joint Agent

In contrast to the basic motor joint agent, the free joint agent in MuJoCo utilizes free joints as its primary means of movement and orientation. Free joints allow unrestricted motion, enabling the agent to move freely in all directions and rotate around any axis. Instead of receiving joint states as observations, the free joint agent relies on the qvel (joint velocity) information to control its forward/backward movement and turning capabilities.

The free joint agent's simplicity lies in its reduced control complexity. By relying on joint velocities, it can achieve locomotion and rotational maneuvers without explicitly manipulating individual joint angles or torques. This simplicity can be advantageous in certain scenarios where fine-grained control over individual joints is unnecessary or impractical.

RL Considerations

When designing RL algorithms for agents in MuJoCo, certain considerations specific to each agent type should be taken into account:

1. Action Space: For basic motor joint agents, the action space consists of joint torques or desired joint angles. In contrast, free joint agents use only two actions to move. One is moving forwards/backwards (-1/1) and one is turning left and right (-1/1).

2. Observation Space: Basic motor joint agents receive detailed joint state information, which can be used to reconstruct the agent's full body configuration. Free joint agents, on the other hand, rely on qvel values and may require additional information, such as global positions or orientations, to reconstruct their complete state.

3. Reward Design: The design of rewards in RL tasks should consider the capabilities and limitations of each agent type. For basic motor joint agents, rewards can be defined based on achieving specific joint configurations or exerting forces/torques. I.e. an additional reward might be needed to prevent the agent from turning on its head. In the case of free joint agents, rewards can be tied to accomplishing tasks that rely on locomotion or turning capabilities.