Sports2D

Sports2D automatically computes 2D joint positions, as well as joint and segment angles from a video or a webcam.

Announcement:
Complete rewriting of the code! Run pip install sports2d -U to get the latest version.

• Faster, more accurate
• Works from a webcam
• Results in meters rather than pixels. New in v0.5!
• Better visualization output
• More flexible, easier to run
• Batch process multiple videos at once

Note: Colab version broken for now. I'll fix it in the next few weeks.

Demo_sports2d.mp4

Warning: Angle estimation is only as good as the pose estimation algorithm, i.e., it is not perfect.
Warning: Results are acceptable only if the persons move in the 2D plane (sagittal or frontal plane). The persons need to be filmed as parallel as possible to the motion plane.
If you need 3D research-grade markerless joint kinematics, consider using several cameras, and constraining angles to a biomechanically accurate model. See Pose2Sim for example.

Contents

1. Installation and Demonstration
   1. Installation
   2. Demonstration
   3. Play with the parameters
2. Go further
   1. Too slow for you?
   2. What you need is what you get
   3. How it works
3. How to cite and how to contribute

Installation and Demonstration

Installation

• OPTION 1: Quick install
  Open a terminal. Type python -V to make sure python >=3.8 <=3.11 is installed. If not, install it from there. Run:

  pip install sports2d

• OPTION 2: Safer install with Anaconda
  Install Miniconda:
  Open an Anaconda prompt and create a virtual environment by typing:

  conda create -n Sports2D python=3.9 -y
  conda activate Sports2D
  pip install sports2d

• OPTION 3: Build from source and test the last changes
  Open a terminal in the directory of your choice and clone the Sports2D repository.

  git clone https://github.com/davidpagnon/sports2d.git
  cd sports2d
  pip install .

Demonstration

Just open a command line and run:

sports2d

You should see the joint positions and angles being displayed in real time.

Check the folder where you run that command line to find the resulting video, images, TRC pose and MOT angle files (which can be opened with any spreadsheet software), and logs.

Important: If you ran the conda install, you first need to activate the environment: run conda activate sports2d in the Anaconda prompt.

Note:
The Demo video is voluntarily challenging to demonstrate the robustness of the process after sorting, interpolation and filtering. It contains:

• One person walking in the sagittal plane
• One person doing jumping jacks in the frontal plane. This person then performs a flip while being backlit, both of which are challenging for the pose detection algorithm
• One tiny person flickering in the background who needs to be ignored

Play with the parameters

For a full list of the available parameters, check the Config_Demo.toml file or type:

sports2d --help

Run on custom video with default parameters:

sports2d --video_input path_to_video.mp4

Run on webcam with default parameters:

sports2d --video_input webcam

Get coordinates in meters rather than in pixels:

Just provide the height of the analyzed person (and their ID in case of multiple person detection).
The floor angle and the origin of the xy axis are computed automatically from gait. If you analyze another type of motion, you can manually specify them.
Note that it does not take distortions into account, and that it will be less accurate for motions in the frontal plane.

sports2d --to_meters True --calib_file calib_demo.toml

sports2d --to_meters True --person_height 1.65 --calib_on_person_id 2

sports2d --to_meters True --person_height 1.65 --calib_on_person_id 2 --floor_angle 0 --xy_origin 0 940

Run with custom parameters (all non specified are set to default):

sports2d --video_input demo.mp4 other_video.mp4

sports2d --show_graphs False --time_range 1.2 2.7 --result_dir path_to_result_dir --slowmo_factor 4

sports2d --multiperson false --mode lightweight --det_frequency 50 

Run with a toml configuration file:

sports2d --config Config_demo.toml

Run within a Python script:

from Sports2D import Sports2D; Sports2D.process('Config_demo.toml')

from Sports2D import Sports2D; Sports2D.process(config_dict)

Go further

Too slow for you?

Quick fixes:

• Use --save_vid false --save_img false --show_realtime_results false: Will not save images or videos, and will not display the results in real time.
• Use --mode lightweight: Will use a lighter version of RTMPose, which is faster but less accurate.
• Use --det_frequency 50: Will detect poses only every 50 frames, and track keypoints in between, which is faster.
• Use --multiperson false: Can be used if one single person is present in the video. Otherwise, persons' IDs may be mixed up.
• Use --load_trc <path_to_file_px.trc>: Will use pose estimation results from a file. Useful if you want to use different parameters for pixel to meter conversion or angle calculation without running detection and pose estimation all over.

Use your GPU:
Will be much faster, with no impact on accuracy. However, the installation takes about 6 GB of additional storage space.

1. Run nvidia-smi in a terminal. If this results in an error, your GPU is probably not compatible with CUDA. If not, note the "CUDA version": it is the latest version your driver is compatible with (more information on this post).

   Then go to the ONNXruntime requirement page, note the latest compatible CUDA and cuDNN requirements. Next, go to the pyTorch website and install the latest version that satisfies these requirements (beware that torch 2.4 ships with cuDNN 9, while torch 2.3 installs cuDNN 8). For example:

   pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

2. Finally, install ONNX Runtime with GPU support:

   pip install onnxruntime-gpu
   

3. Check that everything went well within Python with these commands:

   python -c 'import torch; print(torch.cuda.is_available())'
   python -c 'import onnxruntime as ort; print(ort.get_available_providers())'
   # Should print "True ['CUDAExecutionProvider', ...]"

What you need is what you get

Analyze a fraction of your video:

sports2d --time_range 1.2 2.7

Customize your output:

• Choose whether you want video, images, trc pose file, angle mot file, and real-time display:

  sports2d --save_vid false --save_img true --save_pose false --save_angles true --show_realtime_results false

• Choose which angles you need:

  sports2d --joint_angles 'right knee' 'left knee' --segment_angles None

• Choose where to display the angles: either as a list on the upper-left of the image, or near the joint/segment, or both:

  sports2d --display_angle_values_on body

• You can also decide not to calculate and display angles at all:

  sports2d --calculate_angles false

Run on several videos at once:

You can individualize (or not) the parameters.

sports2d --video_input demo.mp4 other_video.mp4 --time_range 1.2 2.7

sports2d --video_input demo.mp4 other_video.mp4 --time_range 1.2 2.7 0 3.5

How it works

Sports2D:

• Detects 2D joint centers from a video or a webcam with RTMLib.
• Converts pixel coordinates to meters.
• Optionally computes selected joint and segment angles.
• Optionally saves processed image files and video file. Optionally saves processed poses as a TRC file, and angles as a MOT file (OpenSim compatible).

Okay but how does it work, really?
Sports2D:

1. Reads stream from a webcam, from one video, or from a list of videos. Selects the specified time range to process.

2. Sets up pose estimation with RTMLib. It can be run in lightweight, balanced, or performance mode, and for faster inference, keypoints can be tracked instead of detected for a certain number of frames. Any RTMPose model can be used.

3. Tracks people so that their IDs are consistent across frames. A person is associated to another in the next frame when they are at a small distance. IDs remain consistent even if the person disappears from a few frames. This carefully crafted sports2d tracker runs at a comparable speed as the RTMlib one but is much more robust. The user can still choose the RTMLib method if they need it by specifying it in the Config.toml file.

4. Chooses the right persons to keep. In single-person mode, only keeps the person with the highest average scores over the sequence. In multi-person mode, only retrieves the keypoints with high enough confidence, and only keeps the persons with high enough average confidence over each frame.

5. Converts the pixel coordinates to meters. The user can provide a calibration file, or simply the size of a specified person. The floor angle and the coordinate origin can either be detected automatically from the gait sequence, or be manually specified.

6. Computes the selected joint and segment angles, and flips them on the left/right side if the respective foot is pointing to the left/right.

7. Draws the results on the image:
   Draws bounding boxes around each person and writes their IDs
   Draws the skeleton and the keypoints, with a green to red color scale to account for their confidence
   Draws joint and segment angles on the body, and writes the values either near the joint/segment, or on the upper-left of the image with a progress bar

8. Interpolates and filters results: Missing pose and angle sequences are interpolated unless gaps are too large. Results are filtered according to the selected filter (among Butterworth, Gaussian, LOESS, or Median) and their parameters

9. Optionally show processed images, saves them, or saves them as a video
   Optionally plots pose and angle data before and after processing for comparison
   Optionally saves poses for each person as a TRC file in pixels and meters, angles as a MOT file, and calibration data as a Pose2Sim TOML file

Joint angle conventions:

• Ankle dorsiflexion: Between heel and big toe, and ankle and knee.
  -90° when the foot is aligned with the shank.
• Knee flexion: Between hip, knee, and ankle.
  0° when the shank is aligned with the thigh.
• Hip flexion: Between knee, hip, and shoulder.
  0° when the trunk is aligned with the thigh.
• Shoulder flexion: Between hip, shoulder, and elbow.
  180° when the arm is aligned with the trunk.
• Elbow flexion: Between wrist, elbow, and shoulder.
  0° when the forearm is aligned with the arm.

Segment angle conventions:
Angles are measured anticlockwise between the horizontal and the segment.

• Foot: Between heel and big toe
• Shank: Between ankle and knee
• Thigh: Between hip and knee
• Pelvis: Between left and right hip
• Trunk: Between hip midpoint and shoulder midpoint
• Shoulders: Between left and right shoulder
• Head: Between neck and top of the head
• Arm: Between shoulder and elbow
• Forearm: Between elbow and wrist

How to cite and how to contribute

How to cite

If you use Sports2D, please cite Pagnon, 2024.

 @article{Pagnon_Sports2D_Compute_2D_2024,
   author = {Pagnon, David and Kim, HunMin},
   doi = {10.21105/joss.06849},
   journal = {Journal of Open Source Software},
   month = sep,
   number = {101},
   pages = {6849},
   title = {{Sports2D: Compute 2D human pose and angles from a video or a webcam}},
   url = {https://joss.theoj.org/papers/10.21105/joss.06849},
   volume = {9},
   year = {2024}
 }

How to contribute

I would happily welcome any proposal for new features, code improvement, and more!
If you want to contribute to Sports2D, please follow this guide on how to fork, modify and push code, and submit a pull request. I would appreciate it if you provided as much useful information as possible about how you modified the code, and a rationale for why you're making this pull request. Please also specify on which operating system and on which python version you have tested the code.

Here is a to-do list: feel free to complete it:

• Compute segment angles.
• Multi-person detection, consistent over time.
• Only interpolate small gaps.
• Filtering and plotting tools.
• Handle sudden changes of direction.
• Batch processing for the analysis of multiple videos at once.
• Option to only save one person (with the highest average score, or with the most frames and fastest speed)
• Run again without pose estimation with the option --load_trc for px .trc file.
• Convert positions to meters by providing the person height, a calibration file, or 3D points to click on the image
• Perform Inverse kinematics and dynamics with OpenSim (cf. Pose2Sim, but in 2D). Update this model (add arms, markers, remove muscles and contact spheres). Add pipeline example.
• Run with the option --compare_to to visually compare motion with a trc file. If run with a webcam input, the user can follow the motion of the trc file. Further calculation can then be done to compare specific variables.
• Colab version: more user-friendly, usable on a smartphone.
• GUI applications for Windows, Mac, and Linux, as well as for Android and iOS.

• Track other points and angles with classic tracking methods (cf. Kinovea), or by training a model (cf. DeepLabCut).
• Pose refinement. Click and move badly estimated 2D points. See DeepLabCut for inspiration.
• Add tools for annotating images, undistort them, take perspective into account, etc. (cf. Kinovea).