Slurm self supervised

[rrichajalota]

Optimised the code to use FAISS indexing for nearest neighbour search

• also debugged the previous implementation
• added support for Mac M1 (to be tested)
• current implementation has been tested on 1 GPU
• implemented other FAISS indexes for approximate nearest neighbour search
• logging the time it takes to finish indexing
• fixed the fairseq-generate pipeline.

Differences:

• Unlike the previous implementation that required breaking down a large dataset into smaller files to be able to compute the dot product btw matrices (MxN multiplication), the code now works on the entire dataset of 50k+ sentences.
• Reduced the number of nested for-loops, saving O(mn) time with each.

Observations:

• In terms of speed, IndexFlatL2 < IndexIVFPQ < IndexFlatIVF ; making IndexFlatIVF the best choice
• the speed up is almost 50% going from IndexFlatL2 (~7.2 mins per epoch) to IndexFlatIVF (~3.6 mins per epoch)

---

TODOS:

• Optimize further:
  • test on multiple GPUs/ distributed training
  • test on MAC M1
• Log the time it takes to complete one iteration