Skip to content

Plug and Play Real-Time Object Detection App with Tensorflow and OpenCV

License

Notifications You must be signed in to change notification settings

gustavz/realtime_object_detection

Repository files navigation

realtime_object_detection

Realtime Object Detection based on Tensorflow's Object Detection API and DeepLab Project

Version 1: use branch r1.0 for the original repo that was focused on high performance inference of ssd_mobilenet
(x10 Performance Increase on Nvidia Jetson TX2)

Version 2: use branch Master or to be additionally able to run and test Mask-Detection Models, KCF-Tracking and DeepLab Models (merge of the repo realtime_segmenation)

ROS Support: To use this Repo as ROS-Package including detection and segmentation ROS-Nodes use branch ros. Alternativley use the repo objectdetection_ros

About the Project

The Idea was to create a scaleable realtime-capable object detection pipeline that runs on various systems.
Plug and play, ready to use without deep previous knowledge.

The project includes following work:

  • optional download of tensorflow pretrained models
  • do Inference with OpenCV, either through video input or on selected test_images.
    supported Models are all research/object_detection as well as research/deeplab models
  • enjoy this project's own ssd_mobilenet speed hack, which splits the model in a mutlithreaded cpu and gpu session.
    Results in up to x10 performance increase depending on the running system
    ⇒ which makes it (one of) the fastest inference piplines out there
  • run statistic tests on sets of images and get statistical information like mean and median fps, std dev and much more
  • create timeline files measuring the exact time consumption of each operation in your model
  • inspect, summarize, quantize, transform and benchmark models with the provided scripts/
  • Use this Repo as ROS Package. the detection subscirbes a ROS Image topic and publishes the detection as ROS Node.

Inference:

  • create a copy of config.sample.yml named config.yml and only change configurations inside this file
    For example: If you are not interested in visualization: set VISUALIZE to False,
    or if you want to switch off the speed hack set SPLIT_MODEL to False,
  • to be able to use KCF_Tracking inside scripts/ run bash build_kcf.sh to build it and set USE_TRACKER to True to use it
    (currently only works for pure object detection models without SPLIT_MODEL)
  • new class (Model,Config,Visualizer) structure. Simply create your own test file with:
    from rod.model import ObjectDetectionModel, DeepLabModel
    from rod.config import Config
    
    model_type = 'od'                                              #or 'dl'
    input_type = 'video'                                           #or 'image'
    config = Config(model_type)
    model = ObjectDetectionModel(config).prepare_model(input_type) #or DeepLabModel
    model.run()
    
  • Alternativley run python + objectdetection_video.py or objectdetection_image.py or deeplab_video.py or deeplab_image.py or allmodels_image.py

Scripts:

To make use of the tools provided inside scripts/ follow this guide:

  • first change all paths and variables inside config_tools.sh to your needs / according to your system
  • When using the first time run: source config_tools.sh and in the same terminal run only once source build_tools.sh to build the tools. this will take a while.
  • For all following uses first run: source config_tools.sh(due to the exported variables) and after that you are able to run the wanted scripts always from the same terminal with source script.sh.
  • All scripts log the terminal output to test_results/

Setup:

Use the following setup for best and verified performance

  • Ubuntu 16.04
  • Python 2.7
  • Tensorflow 1.4 (this repo provides pre-build tf wheel files for jetson tx2)
  • OpenCV 3.3.1

Note: tensorflow v1.7.0 seems to have massive performance issues (try to use other versions)

Current max Performance on ssd_mobilenet:

  • Dell XPS 15 with i7 @ 2.80GHZ x8 and GeForce GTX 1050 4GB: 100 FPS
  • Nvidia Jetson Tx2 with Tegra 8GB: 30 FPS

Related Work:

  • objectdetection_ros: This Repository as ROS Package ready to use
  • test_models: A repo for models i am currently working on for benchmark tests
  • deeptraining_hands: A repo for setting up the ego- and oxford hands-datasets.
    It also contains several scripts to convert various annotation formats to be able to train Networks on different deep learning frameworks
    currently supports .xml, .mat, .csv, .record, .txt annotations
  • yolo_for_tf_od_api: A repo to be able to include Yolo V2 in tf's object detection api
  • realtime_segmenation: This repo was merged into v2.0
  • Mobile_Mask_RCNN: a Keras Model for training Mask R-CNN for mobile deployment
  • tf_training: Train Mobile Mask R-CNN Models on AWS Cloud
  • tf_models: My tensorflow/models fork which includes yolov2 and mask_rcnn_mobilenet_v1_coco
  • eetfm_automation: Export and Evaluation of TensorFlow Models Automation based on the Object Detection API