[FEATURE] Sparse Models Pretraining Code & Data

[contrebande-labs]

Is your feature request related to a problem?
I could not find any information on the sparse neural search models hosted on HuggingFace. What's their archictecture? Are they multilingual? And most importantly how were they pre-trained? We would like to have our own models pretrained on our own data with the architectures that best suit our needs.

What solution would you like?
I would like to have access to the pretraining code and data.

[mingshl]

@xinyual can you please help take a look at this issue?

[dhrubo-os]

@zhichao-aws take a look?

[contrebande-labs]

@dhrubo-os if you trained the models, can you publish the training code and data? thanks!

[zhichao-aws]

Hi @contrebande-labs , the paper for the sparse model is public now, https://arxiv.org/abs/2411.04403

The training code and data is still under the review process

[zhichao-aws]

@dhrubo-os if you trained the models, can you publish the training code and data? thanks!

Hi @contrebande-labs , we have public the code of fine-tuning the model(repo link). It can also be used to train a sparse model from scratch. You can reproduce the results if following the process of generating training data described in the paper.

We also aim to release the training data generated by us, but not sure whether this comply with the licenses of all used datasets and it's still under review.

[contrebande-labs]

Hi @zhichao-aws ! Thanks so much. We are evaluating and benchmarking many sparse and late interaction models now on our own data and we will look into it in the next couple of days. Please leave this issue opened until the data is published or there are new models trained on open data.