Vehicle Detection and Tracking with Yolov4 and DeepStream SDK

This application runs on Jetson Nano and detects and tracks vehicles in a roundabout from a fixed camera stream using Nvidia DeepStream SDK and sends information about the entries and the exits along with the vehicle ids to a Kafka Message Bus in order for a client application to process the data.

For detection the application uses a custom trained Yolov4-Tiny network based on RoundaboutTraffic dataset. DeepStream-Yolo was used to improve inference performance.

For tracking the Discriminative Correlation Filter tracker as well as the DeepSORT tracker were tested and used.

A Kafka producer based on librdkafka library was implemented to send information about the tracked vehicles to the message bus.

Table of contents

• Requirements
• Installation
• Configuration
• Usage
• Discussion

Requirements

• Nvidia Jetson Nano (not mandatory)
• JetPack 4.6
• NVIDIA DeepStream SDK 6.0

Installation

Once we have all the requirements installed we download the repo and the third parties:

$ git clone https://github.com/alxandru/vehicle_tracking_deepstream.git
$ cd vehicle_tracking_deepstream
$ git submodule update --init --recursive

DeepStream-Yolo and librdkafka are used by the application as described in the introduction. First we need to configure and compile them.

Configure librdkafka:

$ cd 3pp/librdkafka
$ ./configure
$ cd ../../

Compile both third parties:

$ CUDA_VER=10.2 make subsystem

librdkafka needs installation:

$ sudo CUDA_VER=10.2 make install

Finally we compile the application:

$ CUDA_VER=10.2 make

Configuration

1. Models

The application uses a yolov4-tiny custom model for vehicle detection and mars-small128 model for DeepSORT tracking algorithm. Both models will be downloaded by running the following script:

$ bash models/get_models.sh

2. Kafka

Open the cfg/kafka_config.txt file and change the endpoint to reflect where your Kafka message bus is installed. The topic field may be left as it is.

If you don't already have a kafka message bus running you can check this simple deployment: zk-single-kafka-single.yml. You need to have docker and docker-compose installed on your machine.

Usage

The application takes as input a video in h264 format and outputs a mp4 video with the annotations:

$ ./bin/vehicle-tracking-deepstream test002.h264 output.mp4

For testing purposes you can download this video and use it as input for the app.

For tracking, the DeepStream discriminative correlation filter (DCF) is used but it can be changed to DeepSORT tracker by modifying the cfg/tracker_config.txt file. Just uncomment the ll-config-file line for DeepSORT and comment it for NvDCF tracker:

#ll-config-file=config_tracker_NvDCF_perf.yml
ll-config-file=config_tracker_DeepSORT.yml

Here is a video snippet with the NvDCF tracker (click on image to open the Youtube video):

And the same video but with DeepSORT tracker activated:

Discussion

The main issues for the trackers in this fixed camera scenario are the occlusions and the changes in angle of the vehicles. The NvDCF tracker suffers more from the occlusion problem. On the other hand the DeepSORT tracker handles the changes in vehicle angles worse. Nevertheless, in general, the two tracking algorithms perform similarly. At the end of the day almost the same number of vehicles are lost due to re-identification.

The DeepSORT algorithm uses mars-small128 model which originally was trained for recognizing humans. As a next step, it would be interesting to train a model for recognizing cars for DeepSORT tracker and check how much it would improve.

In terms of number of frames processed per second (FPS) there is no big difference between the two trackers. With DeepSORT the application processes around 14 FPS on average, whereas with NvDCF processes around 13 FPS.