Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for safetensors (non-GGUF) model format #642

Open
sallyom opened this issue Jan 28, 2025 · 3 comments
Open

Add support for safetensors (non-GGUF) model format #642

sallyom opened this issue Jan 28, 2025 · 3 comments

Comments

@sallyom
Copy link
Collaborator

sallyom commented Jan 28, 2025

Instead of a single model file, ramalama will need to mount and pull a directory. It would be really nice to have ramalama pull the/model --format safetensors or something like that

@bmahabirbu
Copy link
Collaborator

bmahabirbu commented Feb 6, 2025

Great idea, if we're using vllm (or others that support safe tensor) as the runtime we can mount the folder as is. If we are using llama.cpp we can convert the safetensors to a gguf then mount the file as usual!

@ericcurtin
Copy link
Collaborator

I agree we should implement this. I would say even drop the "--format safetensors", I think it would not be too hard to automatically detect a certain model is safetensors.

@rhatdan
Copy link
Member

rhatdan commented Feb 7, 2025

SGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants