This is a repo to locate induction heads in a 4-bit quantized version of Mistral 7B (ih_sweep.ipynb) and then study the impact of ablating them on a question-answering task (ih_babi.ipynb). The dictionaries ih_evaluators.pkl and ih_scores.pkl contain the output of ih_sweep.ipynb, and ih_scores.pkl is necessary to run ih_babi.pynb. See each notebook for more detailed documentation.
-
Notifications
You must be signed in to change notification settings - Fork 0
hollymandel/Mistral7B_Induction_Heads
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Colab notebooks to locate and ablate induction heads in mistral 7b
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published