Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tsk 372 add jina embeddings v3 #387

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from
Draft

Tsk 372 add jina embeddings v3 #387

wants to merge 19 commits into from

Conversation

hh-space-invader
Copy link
Contributor

@hh-space-invader hh-space-invader commented Nov 4, 2024 •
edited
Loading

  • Added jina embeddings v3 with variable dimensions [32, 64, 128, 256, 512, 768, 1024] and 5 tasks:
    • retrieval.query: Used for query embeddings in asymmetric retrieval tasks
    • retrieval.passage: Used for passage embeddings in asymmetric retrieval tasks
    • separation: Used for embeddings in clustering and re-ranking applications
    • classification: Used for embeddings in classification tasks
    • text-matching: Used for embeddings in tasks that quantify similarity between two texts
  • Added support for multi task models that can be called by task_embed
  • Added support for Matryoshka Embeddings where the user can define variable output dimension

#372

@hh-space-invader
feat: Added multi task text embedding base
3dc1df4
@hh-space-invader
feat: Defined the multi task text embeddings
9c2b806
@hh-space-invader
chore: Updated example supported models in multi task models
f0454e9
@hh-space-invader
chore: Added jina embedding v3 to multitask model registery
7f32604
@hh-space-invader
feat: First version of jina embedding multitask
4211dc6
@hh-space-invader
feat: First implementation of matryoshka embedding
f829bf5
@hh-space-invader
chore: Updated model example description
4a6ce30 
chore: Updated task_embed docstring
@hh-space-invader
new: Added implementation of task_embed
32b4f0f 
new: Added get_task_type_dict
chore: Modified model description to have the task types with its corresponding onnx ids
chore: Replaced process for multi processing to support task id
chore: Modified onnx_embed to support task id
@hh-space-invader
feat: Added dataclass for multi task embedding
e87d9d8
@hh-space-invader
chore: Changed return type to multi task embedding
75a4135
@hh-space-invader
fix: Fixed type hint of embed documents by changing return type to mu…
2ecd109 
…lti task embeddding
@hh-space-invader
nit
7515594
@hh-space-invader
new: Added model downloader
b1fedc2 
chore: Updated doc strings
@hh-space-invader
fix: Fix matryoshka
4ff34c4
@hh-space-invader
new: Added matryoshka support for jina embedding v3
60d2432 
new: Added normalized mean pooling
@hh-space-invader
test: Added tests for multi task models
2c4d320
@hh-space-invader hh-space-invader force-pushed the TSK-372-add-jina-embeddings-v3 branch from 831012c to 2c4d320 Compare November 4, 2024 21:09
@joein joein marked this pull request as draft December 17, 2024 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joein joein Awaiting requested review from joein

@I8dNLo I8dNLo Awaiting requested review from I8dNLo

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@hh-space-invader