Added device parameter to allow usage with MPS (Apple Silicon) and Vu…

…lkan (#32)

* Bumps dependencies for compatibility with libtorch 2.2

* Adds device parameter to enable usage with MPS (Apple Silicon) and Vulkan

* Disables default features for rust-bert

v0.4.1: Add Apple Silicon support + update deps + fix Docker image build

CI Integration (#14)

* Add: rust.yml for CI

* Distilroberta draft (#13)

* Add: sentence_piece tokenizer implementation for RoBERTa

* Add: temporarly using a fix branch on my fork of rust-bert while we wait
for the PR to be merged

* Add: distilroberta for sequence classification implementation

* Add: conversion script for distilroberta from PyTorch to Rust

* Add: test for distilroberta and sentence piece tokenizer

* Fix: padding token id is now extracted from the vocabulary +
Fix: attention mask after padding +
Add: new example test

* Fix: formatting

* Add: using rust-bert master branch with my fix PR instead of my fork until a new
release comes up

* Add: using the PR merge commit of rust-bert

* Clean: encode is now forward +
Add: all models in models/ directory +
Update: tests with new naming

* Clean: pad_sort is not in models/mod.rs

* Clean: encode_with_attention is now forward_with_attention

* Add: 0.4.0 version in Cargo.toml

* Add: download model before cargo test in CI

* Fix: directory in .gitignore

* Add: ls and tree to see what's going on

* Add: tree and ls as main script

* Add: apt tree

* Fix: before_script

* Add: cargo test

* Fix: formatting

* Add: exclude tests from cargo test

* Fix: skip filter in cargo test --

Update: rust_bert dep to 0.10.0 +

Fix: adjust error handling to the new dep version +
Add: const DIM in pooling

Fix: remove multithread error +

Add: expect at end of crossbeam scope

Attention highlight (#11)

* Add: sbert now handles important word highlighting +
Add: sbert can return all attention heads +
Add: tokenizer can return text tokens +
Add: unit test for attention +
Update: crate deps versions

* Some minor improvements

* Update src/sbert.rs

Co-authored-by: cpcdoy <[email protected]>

* More minor improvements

Co-authored-by: Romain Leroux <[email protected]>
Co-authored-by: Romain Leroux <[email protected]>

Add: tag v0.1.2

Add: convert-tensors into src/bin

Add: keywords to toml