Create a tracking body and a simulated body

[Hardwarize]

Intro

Hi!

I am a master student at UNED, I use MuJoCo for my research on generative human motion.

My setup

MuJoCo python

My question

I’m trying to replicate a publication on human motion generation—specifically ControlVAE, in case anyone needs more references. In this publication, there’s an initial part of the model where a neural network must learn a world model based on a BVH file. It works as follows:

• The world model receives:
  1. The current state of the character (the rotation of each joint relative to its parent joint, e.g., the rotation of the knee joint relative to the hip joint’s axes).
  2. The action (torques) applied to each joint.
• The model’s output is the character’s state at the next simulation step.

For visualization, I need a character that is not affected by physics, onto which I can apply only the desired rotations. I achieved this by modifying qpos and calling mj_forward. The problem is that I now also need a new character that is subject to physics (i.e., one to which I can apply actions/torques and that responds accordingly). For this second character, I need to call mj_step.

I tried calling mj_forward and then mj_step in the loop, assigning zero mass to all the geoms of the “visualization” character so that it wouldn’t be affected by gravity. However, I’m getting the error “body mass is too small, cannot compute center of mass,” which suggests this approach is incorrect.

What would be the proper way to do this?

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[Balint-H]

Hello,

mj_step already calls mj_forward, so you do not need to call it separately; just manually set qpos of the visualization kinematic tree. It is also a good idea to set its qvel to 0 as well (otherwise, depending on your integrator it could lead to your character drifting away due to accumulating qvel). Don't forget to disable collisions to visualization character, and I also recommend setting its geoms transparent for clarity.

If you do these steps, you can leave the masses as they are for the visu character. In fact, you get the benefit of being able to calculate measure such as CoM, which you can compare with your simulated character.

Note that this approach does waste some computation, the engine will try to simulate the visu character, you just overwrite its results every time step. You particularly should take care to only compare kinematics between sim and visu characters, and only after updating the visu state, but before mj_step (otherwise the visu state would be affected).

Another approach is using mocap bodies for each separate body segment, and setting their positions and rotations to the global one from the reference animation. Mocap bodies are not updated by the engine hence are cheaper, but there is more book keeping involved when you need to convert back and forth from local and global configurations.

Related but not necessary is the gravcomp field for bodies, using which you can disable gravity selectively. However, if you do set the qpos and qvel each frame, it won't be necessary.

[Hardwarize]

Thanks a lot, I've tried with just mj_step and setting qvel to 0 and seems to work. The problem now is that when I apply torque to the controller in the simulated character (orange) it seems to restart to the initial position to the body, as you can see in the video

Screen.Recording.2025-01-20.at.14.30.49.mov

Here is the fragment of the code where I set the qpos and qvel for visu character and apply torque to simu character:

data.qpos[58] = rElbow_angles[i][0]
data.qpos[59] = rElbow_angles[i][1]
data.qpos[60] = rElbow_angles[i][2]
    
data.qpos[61] = rWrist_angles[i][0]
data.qpos[62] = rWrist_angles[i][1]
data.qpos[63] = rWrist_angles[i][2]
    
data.qvel[:63] = np.zeros(63)

data.ctrl = np.ones(57) * 0.02

mj.mj_step(model, data)
    
i += 1
while (((tm.time() * 1000) - start_frame_time_ms) < dt):
    tm.sleep(0.001)

note that the visu character do not have any attached controller. Additionally the logs says:
WARNING: Nan, Inf or huge value in QPOS at DOF 64. The simulation is unstable. Time = 0.1560.

Thanks for your help

[Balint-H]

If you do not animate the visu character do you still get the instability?

If yes, then you probably just have to go down the usual stability checklist that includes:

• Timestep
• Swap to implicit integrator
• Adjust actuator gains
• Passive joint damping
• Small increase to armature

[Hardwarize]

Yes, increasing the armature and limiting the joints range fix the issue. Just out of curiosity, do you know if there is a way to detect which joint or element is creating the issue? some kind of exception in Mujoco or something like that?

[Balint-H]

You could keep track of the qacc/qvel vectors, and identify problematic DoFs that way. In some cases instability arises from a singular inertia matrix, you can check that as well for suspect DoFs. From the video the self-rotating torso element is a bit suspicious.