WHAC: World-grounded Humans and Cameras

This is the official implementation of WHAC: World-grounded Humans and Cameras (ECCV 2024), featuring the latest foundation model for human pose and shape estimation with SMPLest-X.

[Homepage]      [Paper]      [arXiv]      [SMPLest-X]      [SMPLer-X]     

---

---

Installation

Prepare the environment

git clone https://github.com/wqyin/WHAC.git --recursive
cd WHAC

bash scripts/installation.sh

Download the pretrained model for WHAC

• Download the whac_motion_velocimeter.pth.tar from here and place it under ./pretrained_models.

Setup SMPLest-X

• Prepare the pretrained models and parametric human models for SMPLest-X following the official instructions here.
• Make sure the file structure under ./third_party/SMPLest-X is correct.

Setup DPVO

• Setup steps for DPVO are included in ./scripts/installation.sh.
• Refer to the Setup and Installation section if there is any issue during the installation.

File structure

.
├── assets
├── configs
├── demo
├── lib
├── outputs
├── pretrained_models
│   └── whac_motion_velocimeter.pth.tar 
├── scripts
├── third_party
│   ├── DPVO
│   │   └── pretrained_models
│   │       └── dpvo.pth
│   └── SMPLest-X
│       ├── pretrained_models
│       │   └── smplest_x_h40
│       │       ├── smplest_x_h40.pth.tar
│       │       └── config_base.py
│       └── human_models
│           └── human_model_files
├── whac
├── README.md
└── requirements.txt

Inference

• Place the video under ./demo folder.

bash scripts/inference.sh {SEQ_NAME}

• You may quick run the demo with our test videos by:

bash scripts/prepare_demo.sh

bash scripts/inference.sh dance_demo.mp4
bash scripts/inference.sh skateboard_demo.mp4

WHAC-A-Mole

Check out our homepage for dataset download links.

whac_a_mole_video_small.mp4

Citation

@inproceedings{yin2024whac,
  title={Whac: World-grounded humans and cameras},
  author={Yin, Wanqi and Cai, Zhongang and Wang, Ruisi and Wang, Fanzhou and Wei, Chen and Mei, Haiyi and Xiao, Weiye and Yang, Zhitao and Sun, Qingping and Yamashita, Atsushi and Yang, Lei and Liu, Ziwei},
  booktitle={European Conference on Computer Vision},
  pages={20--37},
  year={2024},
  organization={Springer}
}

@article{yin2025smplest,
  title={SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation},
  author={Yin, Wanqi and Cai, Zhongang and Wang, Ruisi and Zeng, Ailing and Wei, Chen and Sun, Qingping and Mei, Haiyi and Wang, Yanjun and Pang, Hui En and Zhang, Mingyuan and Zhang, Lei and Loy, Chen Change and Yamashita, Atsushi and Yang, Lei and Liu, Ziwei},
  journal={arXiv preprint arXiv:2501.09782},
  year={2025}
}