Real-Time AI Posture Correction for Powerlifting Exercises Using YOLOv5 and MediaPipe

• Paper: Paper Download
• PPT: PPT Download
• Demo Video: https://youtu.be/u4f_sdjk1Ig

Description

A Study on the big three exercises AI posture correction service Using YOLOv5 and MediaPipe
study duration: 2023.09.01 ~ 2023.11.20

Yeong-Min Ko

Development Environment

• OS: MAC m1 & Windows 11(NVIDIA GeForce RTX 4080 Ti)
  
• Frameworks & Libraries: YOLOv5, MediaPipe, OpenCV, Streamlit
• Device: iPhone 12 Pro(WebCam using iVCam)

Data

• YOLOv5: Detect only one Person
  • Scraping Images from Google & Roboflow
    • Bench Press
      • Google Search Keyword: Bench Press
      • Faller Computer Vision Project with lying down
    • Squat
      • Squat-Depth Image Dataset
      • HumonBody1 Computer Vision Project with Standing
    • Deadlift
      • SDT Image Dataset
    • More(bending, lying, sitting, standing)
      • Silhouettes of human posture
  • Inference

    Test 1	Test 2
    Applied 1	Applied 2

• Exercise Posture Correction
  • Shooting Stand Position

    Bench Press	Squat and Deadlift

  • Example of Shooting stand position for Bench Press

    Picture(Left, Center, Right)

  • Bench Press: read more
    • 
  • Squat: read more
    • 
  • Deadlift: read more
    • 

Train & Evaluate

YOLOv5

• Detect only a Person exercising something
  • Hyperparameters to train
    • epochs 200(but early stopping: 167)
    • batch 16
    • weights yolov5s.pt
    • etc are set by 'default'
• Performance Evaluation

  Precision	Recall	mAP_0.5	mAP_0.5:0.95
  0.987	0.990	0.99	0.686

Exercise Classfication

• Bench Press (Algorithm: Random Forest)

  Accuracy	Precision	Recall	F1-Score
  0.961	0.963	0.961	0.961

• Squat (Algorithm: Random Forest)

  Accuracy	Precision	Recall	F1-Score
  0.989	0.989	0.989	0.989

• Deadlift (Algorithm: Random Forest)

  Accuracy	Precision	Recall	F1-Score
  0.947	0.949	0.947	0.948

Feedback

Bench Press	Squat	Deadlift
허리가 과도한 아치 자세 허리를 너무 아치 모양으로 만들지 말고 가슴을 피려고 노력하세요.	척추가 중립이 아닌 자세 척추가 과도하게 굽지 않도록 노력하세요	척추 중립이 아닌 자세 척추가 과도하게 굽지 않도록 노력하세요
허리가 과도한 아치 자세 골반을 조금 더 들어올리고 복부를 긴장시켜 허리를 평평하게 유지하세요.	척추가 중립이 아닌 자세 가슴을 들어올리고 어깨를 뒤로 넣으세요.	척추가 중립이 아닌 자세 가슴을 들어올리고 어깨를 뒤로 넣으세요.
바를 너무 넓게 잡은 자세 바를 너무 넓게 잡았습니다. 조금 더 좁게 잡으세요.	무릎이 움푹 들어간 자세 무릎이 움푹 들어가지 않도록 주의하세요.	바를 너무 넓게 잡은 자세 바를 너무 넓게 잡았습니다. 조금 더 좁게 잡으세요.
바를 너무 넓게 잡은 자세 바를 잡을 때 어깨 너비보다 약간만 넓게 잡는 것이 좋습니다.	무릎이 움푹 들어간 자세 엉덩이를 뒤로 빼서 무릎과 발끝을 일직선으로 유지하세요.	바를 너무 넓게 잡은 자세 바를 잡을 때 어깨 너비보다 약간만 넓게 잡는 것이 좋습니다.
	발을 너무 넓게 벌린 자세 발을 어깨 너비 정도로만 벌리도록 좁히세요.	바를 너무 좁게 잡은 자세 바를 어깨 너비보다 조금 넓게 잡는 것이 좋습니다.

How to Use

• Open your terminal in mac, linux or your command prompt in Windows. Then, type "Streamlit run Streamlit.py".
  

  This Service

Major project records

• 2023/09/10: 2023/09/10: Successfully concluded a project utilizing YOLOv5 to detect a singular individual.
• 2023/09/11: Integration with Mediapipe yielded lower accuracy than anticipated. Consequently, we decided to enhance labeling by introducing additional spatial dimensions around individuals.
• 2023/09/16: Significantly refined bounding boxes for model training, resulting in a triumphant pose estimation with remarkable accuracy when employing YOLOv5 and Mediapipe in tandem. Implemented a Streamlit file for holistic pose estimation after detecting the nearest person using YOLOv5. And the streamlit file was impleted to estimate holistic pose after detecting only person closest to the camera using yolov5.
• 2023/09/30 ~ 2023/10/02: Gathered datasets for training an exercise posture classification model.
• 2023/10/03 ~ 2023/10/08: Commenced with class labeling of the dataset, followed by model training and conclusive evaluations.
• 2023/10/18: Established a connection between the bench press model and the server, implementing an algorithm to count bench press repetitions. Additionally, in the process of linking two additional models: deadlift and squat.
• 2023/10/24: Successfully integrated all models and the server, culminating in the completion of the paper.
• 2023/11/05: Implemented feedback mechanisms for each specific posture.
• 2023/11/20: Submitted the finalized paper along with experimental results.

Project Progress

• Week 1: Requirement Analysis
  • Read More
• Week 2: Prototype Development & Mini Test
  • Read More
• Week 3: Retrain the model detecting only person and Estimate holistic pose after detecting only person closest to the camera using yolov5
  • Read More
• Week 4: Write the paper
• Week 5: Write the paper and Develop machine learning pipelines
  • Read More
• Week 6: Presentation of project mid-progress
  • Read More
• Week 7: Link the bench press model and the streamlit server / Implement an algorithm to count the number of bench press
  • Read More
• Week 8: Write the paper and Link all models(bench press, squat, deadlift) and the streamlit server
  • Read More
• Week 9: Implement feedback for each posture
  • Read More
• Week 10: Paper Feedback
• Week 11: Paper Feedback
• Week 12: Finish the project

Award