Plotting and projecting skeleton onto original Image

[Aiman-Jabaren]

@akanazawa First of all, thank you for the code and the interesting paper.
I have extracted the [x,y,z] coordinates from joints3d and plotted them (skeleton).
Then I have tried to project the skeleton onto the the original image through the z axis. It seems that the skeleton is a little bit rotated relatively to the projection (image). How can I align the skeleton projection onto the image plane?

Thanks

[justinmacp]

@ayjabia I am also stuck on this problem. I assume, that after some rotation, translation and scaling it would be possible to project them onto the original image. However it seems like the 3D coordinates are given at some random transformation (there is no one transformation matrix that fits all). I.e. if the torso is facing the camera in two different photos, that does not necessarily mean that the torso will face the same direction in the extracted 3D coordinates.

My question to @akanazawa is: Is there a pattern in the joints3d? Does one particular joint always have to point in a specific direction, is the origin fixed? I can't seem to find a pattern in the joints3d.

[wallflower95]

@ayjabia @justinmacp I also have this trouble. Have you solved this problem? If it is solved, can you share it, I am very grateful.

[akanazawa]

I need to see the images, but is the problem that when you project the skeleton it does not align with the rendered mesh and the 2d joints?
Note that the model already returns 2D joints, which is a projection of the 3D joints.
So take a look at how that projection is being done (dig from here:

hmr/src/RunModel.py

Line 119 in 5534c9b

pred_kp = self.proj_fn(Js, cams, name='proj_2d_stage%d' % i)

) , and you should be able to do the same thing.

Re: patten in the 3D joints, 3D joints are obtained from the vertices of the SMPL mesh, and SMPL comes with it's own coordinate system where the root is at the origin. The first 3-D of the pose parameter defines the rotation of the roots.

Best,

Angjoo

[justinmacp]

@ayjabia @justinmacp I also have this trouble. Have you solved this problem? If it is solved, can you share it, I am very grateful.

Hi @zhangkai95, I hope this is still useful:

I created a fork that implements very minor changes. It will write the coordinates as an array to a .npy file if I remember correctly. I mainly made changes to the demo.py and documented changes in line 144 of the demo.py script. Check them out! I think you will find them useful if this is something you're still working on.

Best,

Justin

[EtiReznikov]

Hello, does someone solved this problem?
Thank you.

[jszgz]

Hello, do you know how to use mpi_inf_3dhp_to_tfrecords.py to convert mpi_inf_3dhp dataset? I failed because the code use jpg as input but the dataset I downloaded is consisting of videos. Do I need to use ffmpeg and write code to convert avi to jpg?