Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faiss xb update and other small changes #5

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

Gordon-BP
Copy link

The latest version of faiss depreciated the .xb attribute, causing errors when training & converting JPQ models. This PR replaces instances of:

coarse_embeds = faiss.vector_to_array(coarse_quantizer.xb)
centroid_embeds += coarse_embeds.reshape(ivf_index.pq.M, -1, ivf_index.pq.dsub)

with

coarse_embeds = faiss.rev_swig_ptr(coarse_quantizer.get_xb(), coarse_quantizer.ntotal * coarse_quantizer.d)
coarse_embeds = coarse_embeds.reshape(coarse_quantizer.ntotal, coarse_quantizer.d)

Additionally, I added a few small changes for quality of life and for running on systems without GPU:

  • Added missing newline '' chars in the readme bash scripts
  • Added boto3 to the requirements in setup.py
  • Added behavior for a default tokenizer
  • Removed the depreciating transformers AdamW optimizer and replaced it with pytorch AdamW

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant