Skip to content

Commit

Permalink
Update @lenml/tokenizers for tokenizer fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
zcbenz committed Oct 21, 2024
1 parent 13ba668 commit c07c192
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 3 deletions.
3 changes: 1 addition & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,13 @@ jobs:
huggingface download --silent mlx-community/Llama-3.2-1B-Instruct-8bit
echo 'Who are you?' | yarn tsx src/chat.ts Llama-3.2-1B-Instruct-8bit
huggingface download --silent mlx-community/llava-1.5-7b-4bit --revision 82b38f2
huggingface download --silent mlx-community/llava-1.5-7b-4bit
echo 'What is in the photo? <image:https://www.techno-edge.net/imgs/zoom/19711.jpg>' | yarn tsx src/chat.ts llava-1.5-7b-4bit --temperature=0
huggingface download --silent Qwen/Qwen2-0.5B
yarn tsx src/generate.ts --max-tokens=128 Qwen2-0.5B
huggingface download --silent google-t5/t5-small --hf
huggingface download --silent Xenova/t5-small --hf # fix tokenizer
yarn tsx src/generate.ts --temperature=0 t5-small 'translate English to German: As much as six inches of rain could fall in the New York City region through Monday morning, and officials warned of flooding along the coast.'
huggingface download --silent mlx-community/codegemma-1.1-2b-8bit
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
},
"dependencies": {
"@frost-beta/mlx": "0.0.23",
"@lenml/tokenizers": "3.0.1",
"@lenml/tokenizers": "3.0.3",
"sharp": "0.33.5"
}
}

0 comments on commit c07c192

Please sign in to comment.