Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for latest version of transformers repo #47

Open
prajdabre opened this issue Sep 27, 2022 · 0 comments
Open

Add support for latest version of transformers repo #47

prajdabre opened this issue Sep 27, 2022 · 0 comments
Labels
help wanted Extra attention is needed

Comments

@prajdabre
Copy link
Owner

prajdabre commented Sep 27, 2022

Currently I have provided my own modded fork of transformers but if someone doesnt care about the features I have added and only wants to work with the code mbart code then this should be enabled.

What this would mean is that all those other arguments I pass to the mbart config class to instantiate the object will be sent to kwargs. The main change will be minimal and most likely related to the tokenizer. In the batch creation logic, I pass some extra arguments to the tokenizer to support stochastic tokenization. The way I see it is we have a flag called --is_official_repo which if passed means that the official transformers repo is passed. This argument will then be passed to the batching function which wont pass the flags relevant for stochastic tokenization.

@prajdabre prajdabre added the help wanted Extra attention is needed label Sep 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant