-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Would you please release the hyper-parameters for FreeLB based on ALBERT(hugging-face) #9
Comments
Do you have any comments for the scale of norm( |
@PantherYan
|
@YasinQiu #1. Leave this question to @zhuchen03 I will read more literature to answer our confusion question. It confused me a lot. I will figure out why and post it out. |
@YasinQiu
To the explicit value. Should be around 1e-1? |
I have added the hyperparameters for 8 of the GLUE tasks in the bash script. For epsilon, in the current setting, you can set it to 0 first, which will put no restriction on the maximum norm, and tune other hyperparameters. In this way, the maximum norm will be restricted by the ascent step size, number of ascent steps and the initialization. In the context of security, epsilon restricts the strength of the adversary for better comparisons. However, in our case, you should first observe the norm of the embeddings to choose a strength/epsilon that is not ignorable but also won't outweigh the embeddings. |
@zhuchen03 @PantherYan thx ~!!! |
There are only 4 tasks' hyper parameters in this file, would you please release others?
The text was updated successfully, but these errors were encountered: