Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Allow loading .gguf and .bin files #1114

Open
8 tasks done
mhussaincov94 opened this issue Feb 5, 2024 · 3 comments
Open
8 tasks done

bug: Allow loading .gguf and .bin files #1114

mhussaincov94 opened this issue Feb 5, 2024 · 3 comments

Comments

@mhussaincov94
Copy link

mhussaincov94 commented Feb 5, 2024

would it be possible to add support to run gguf moddles like this?
https://huggingface.co/TheBloke/Kunoichi-7B-GGUF
I have been testing, and simply changeing the file extention from .gguf to .bin loads it ok.
but having support for the .gguf extention would help.
image I am using:
ghcr.io/serge-chat/serge:main
debian unstable with all updates installed.
docker docker compose installed and up-to-date.
using chromium from the debian repo.
thank you for considdering my request.

Checklist
  • Modify docker-compose.ymlf408cac Edit
  • Running GitHub Actions for docker-compose.ymlEdit
  • Modify charts/serge/values.yaml105f0fd Edit
  • Running GitHub Actions for charts/serge/values.yamlEdit
  • Create scripts/model_loader.pyeadd93c Edit
  • Running GitHub Actions for scripts/model_loader.pyEdit
  • Modify README.md761e0dc Edit
  • Running GitHub Actions for README.mdEdit
@gaby
Copy link
Member

gaby commented Feb 5, 2024

That works already the .bin is just how we rename the files. Serge supports GGUF.

@gaby gaby closed this as completed Feb 5, 2024
@mhussaincov94
Copy link
Author

hi,
they don't work.
if added with the file extention .gguf
I have just tested by adding the moddle to the default moddles directory, restarting the container and it is not showing up.

@gaby
Copy link
Member

gaby commented Feb 6, 2024

You have to add them with .bin, else they wont get loaded into the UI.

@gaby gaby reopened this Feb 6, 2024
@gaby gaby changed the title the ability to run gguf moddles? bug: Allow loading .gguf and .bin files Feb 6, 2024
@gaby gaby self-assigned this Feb 6, 2024
@gaby gaby added the sweep 🤖 SweepAI label Feb 26, 2024
@serge-chat serge-chat deleted a comment from sweep-ai bot Apr 13, 2024
@gaby gaby removed the sweep 🤖 SweepAI label Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants