Diagnostics of neural networks characteristic using chaotic initializations.
We will treat the weight updates of a neural network as a dynamical system and use concepts from chaos theory to study the properties of a neural network.
The project is for 2 semesters, most of the work would be done in the first semester.
This is a research project, and can continue as long as you're willing to work.
Domain - Deep Learning, Chaos theory
Prerequisite - Deep learning, basic machine learning concepts
Understanding of Chaos theory is not required but you will have to learn basic concepts while working.
Skills - Python, Matlab
For Matlab, you just need to run some code and some basic debugging at best.
Goal - Find if there is an element of chaos in the learning process of a neural network and find a way to test if that element is necessary for learning.
Current Status - Basic exploration of methods to test the presence of chaos has been done, and a pipeline has been created for this. We are currently working with the vowel dataset and will then perform similar experiments for other datasets.
Expectations - Perform computations and store the weights iterates and examine them using NLD methods.
For the initial phase, you have to make a neural network from scratch for the vowel dataset, then create visualization methods for monitoring the performance of the network. At the same time you'll get familiar with basic chaos theory, and then start working on the main pipeline.
This is a theoretical project, the faculty will provide us with the basic background and tell us what to do and why are we doing it. As you get more comfortable with the theory, you can come up with your own ideas and discuss them with us.
Other than that you have to write the code and carry out the necessary computations.
- Introduction to Dynamical Systems and Chaos (Recommended)
A course on dynamical systems and chaos.
This is a very simple course and familiarises you with basic concepts without going into too much detail. - Nonlinear Dynamics and Chaos - Steven Strogatz, Cornell University
A very thorough and mathematical course on NLD.
It is not at all required, you can just use it as a reference. The professor of this course, Steven Strogatz, has also made a book with the same name. You can also use that as a reference. - A Chaos Theory Approach to Understand Neural Network Optimization
Here's a paper that has done some work similar to what we are doing. I don't recommend going through it completely, but do give it a cursory reading. It has provided the theoretical background for our work, even I don't understand a lot of stuff in it.
The official paper is not available but you can go through their original submission here. - ChaosNet
If you are interested in uses of chaos theory in deep learning or just want to know how theoretical stuff can help in building models, you can check this paper.
This is not related to our work but is a new method that the faculty in our team have worked on. It is not that theory intensive, so you can give it a read if it piques your interest.
- Shivam Chadha
- Vaibhav Ganatra
- Vishal Bharambe
Dr. Snehanshu Saha is our faculty supervisor. You can find more about him here