Introduction

This project is a fusion of a chrome plugin and a Deep Learning model that can be used to annotate accurate captions for scientific lectures.

Goal

• The goal of this project is to create a tool that can be used to annotate captions for scientific lectures.
• The tool should be able to recognize the captions and annotate them correctly, based on the scientific terms unlike current captioning tools which are prone to errors with certain terms.

Description

The tool must be able to tackle the following problems:

• Recognize the captions and annotate them correctly.
• Take in the scientific terms in the form of a pdf or a text document.
• Output the annotated captions in a text file.

Technologies

• Javascript for the chrome extension.
• Python for the deep learning model. [More details about the actual libraries and tools in python can be decided based on the inputs of the students]

Tentative Timeline

We plan to work for around 12-15 weeks.

We will be working on the following milestones:

• Week 1: Research and analysis of the problem.
• Weeks 2-3: Scouting datasets and skeleton development of the chrome extension.
• Weeks 4-5: Data preprocessing, and learning how to integrate the deep learning model with the extension using Chrome APIs.
• Weeks 6-8: Developing the model and fine-tuning it, and continuing work on the extension.
• Weeks 9-11: Individual development of modules and unit testing.
• Weeks 12-14: Integration testing and bug-fixing.
• Week 15: Demo readiness.

Prerequisites

• Basic knowledge in coding.
• Knowledge of one of the following technologies for the extension:
  • Javascript
  • HTML
  • CSS
• Knowledge in one of the following technologies for the model:
  • Tensorflow
  • Keras
  • Pytorch
• Appreciated optional stuff:
  • UI/UX
  • XML parsing
  • Prior experience in the field of machine learning
  • Strong research aptitude

Project Mentors

• Rajath V
• Akhilesh Adithya
• Taarush Bhatia
• Manthan Asher