[SoRo Project] Human touch-gesture recognition using MiRo Social Robot

Project Contributers:

1. Prajval Kumar Murali ([email protected])
2. Nicolò Bastianell ([email protected])
3. Luca Macchiusi ([email protected])

Objective:

Detecting and classifying touch patterns with MiRo. Doing deeplearning using TensorFlow (keras).

Accomplishments

Collected (recorded from MiRo tactile sensors and saved to a text file) 6 types of gesture data from 9 persons. Each person repeating the gesture 10 times

Gestures:

1. Caress body top-bottom
2. Caress body bottom-top
3. Pat body
4. Fixed body
5. Pat Head
6. Fixed Head

Actions

1. Go straight forward
2. Go straight backward
3. Go in circle
4. Stop motion
5. Move head left to right
6. Stop head and reorient to center

For all gesture recognitions we have very good accuracy (94%) and for some high precision and for some low precision.

Gestures with low precision:

1. Pat Body (high false positive rate: we predict pat-body when it is actually not pat-body)

Gestures with High precision (because the sensors are decopled):

1. Pat Head
2. Fixed Head

Modules or Nodes in the system

Offline part: 2 files (collecting and storing data, training the RNN) Online part: 2 ROS-nodes (data input and machinelearning, miro actions)

Limitations of the system

Offline part:

Improve the precision and recall for all the gestures

Optimize the number of Hidden Neurons

Online part:

After classification(recognizing) of gesture, we must have a voting system. As there is a sliding window, there are jumps in classified outputs, hence we must take the average (do the voting system)

Analyse the temporal performace (latency), ie, the time taken for action to happen after doing the gesture.

How to run:

Configure MiRo with your workstation

To configure MiRo with your workstation follow the setup guide on official website or https://github.com/EmaroLab/MIRO

To Run our application

Offline part (Collecting data and training RNN)

#Collecting data after connecting to MiRo ROS node
python miro_touch_sub.py robot=rob01 name_file="name_of_gesture"
#Training the network
source ~/tensorflow/bin/activate
python DataSet.py #remember to change the path to dataset
python testset.py #to evaluate your model

Online part (Classifying gestures in real-time)

#Data input and miro model node
source ~/tensorflow/bin/activate
python Data_input.py #remember to change the path to model :my_model_new.h5
#miro actions 
python miro_action.py #to move miro according to the gestures

Project Video

Can be found here: https://www.youtube.com/watch?v=N7AkDnkxILg