Evaluating word embeddings

[osotsia]

I have a newly-trained set of word embeddings (like Glove) and I want to evaluate them on some benchmark datasets (with a basic classifier like random forests or SVM). Ideally the embeddings would remain fixed and only the classifier would be trained.

I just wanted to verify if this is this the right project for this task. I wasn't clear where to begin since the documentation seems more complicated than this. Any chance there is a tutorial for this? Thanks.

[zphang]

Hi osotsia,

My sense is that jiant might not be the appropriate library for your use-case:

• If you have a set of newly-trained word embeddings, they likely have a specific tokenization that differs from the models we currently support.
• We also do not currently support models such as SVMs and random forests jiant primarily supports transformer-based models from the transformers library.

Instead, I would recommend you look into the following:

• Use Hugging Face's datasets library to obtain task data as well as metrics
• Use scikit-learn's suite of machine learning models for operating on the embeddings

[osotsia]

Many thanks.

[jpmcd]

Hi,

I'm considering a similar question regarding evaluating word embeddings, and there seems to be a way to do this with the older GLUE library (utilizing the args.word_embs_file and then trying out the baseline models). However the dependencies are pretty old. Does jiant implement the baseline models from GLUE anywhere? Thanks.