Using float16 for word vectors

[ashirviskas]

Hi there! I'd like to try and use float16 in spaCy model. It works if you do:

nlp = spacy.load('en_core_web_md')
nlp.tokenizer.vocab.vectors.data = nlp.tokenizer.vocab.vectors.data.astype('float16')

Then you can call nlp('word') and it works well with fp16.

However, if you do nlp.to_disk('en_core_web_md_fp16') and then nlp = nlp.from_disk('en_core_web_md_fp16'), it doesn't seem to load model properly and gives an error, as it seems that spacy is not able to load fp16 vectors properly.

Would it be possible to either implement this in spaCy, or somehow work around this limitation?

Your Environment

• Operating System: Arch Linux
• Python Version Used: 3.8
• spaCy Version Used: Latest

Last error lines:

  File "ops.pyx", line 512, in thinc.neural.ops.NumpyOps.gemm
ValueError: Does not understand character buffer dtype format string ('e')

[adrianeboyd]

I suspect this would be difficult for the similar reasons to those mentioned in #6378. See this explanation for further problems related to Cython: https://stackoverflow.com/a/47441442

The way the vectors are loaded in the background is masking the fact that the second line of code above doesn't affect the vectors data that the currently loaded statistical thinc models are accessing. You see what's really happening if you save and reload the model, and then the statistical components actually see the float16 vectors, and break.

I'm afraid I don't really think this going to be a workable idea. It's not theoretically impossible, but I don't think you'll see any gains beyond memory usage and the other drawbacks are pretty severe.

[ashirviskas]

My main goal is to lower the filesize of en_core_web_md, so using fp16 in memory is not necessary, it was just to check how feasible it is.

I think I may need to save in fp16 and then convert to fp32 just before loading, which may be sufficient for storage/bandwidth savings and my use case. Thank you for answering.

[adrianeboyd]

Yes, that would probably be easier. You would just need to modify the to/from_bytes/disk methods to handle the numpy type conversion.