[Question] using llama models in local for development

[axel7083]

I am able to run without any issue the inference worker with a model like OA_SFT_Pythia_12Bq_4, but I can't use llama models.

On the OpenAssistant/oasst-sft-6-llama-30b-xor repository on huggingface we can read the following:

Due to the license attached to LLaMA models by Meta AI it is not possible to directly distribute LLaMA-based models. Instead we provide XOR weights for the OA models.

Is there somewhere a nice documentation on how to setup the inference worker with a llama models ? Downloading the orginal weights etc..

[andreaskoepf]

Due to the license of LLaMA OA does not directly share the model weights. But searching for "oasst", "SFT-6" or "SFT-7" on the HuggingFace hub you might find models which are ready to use. You could for example try SinanAkkoyun/oasst-sft-7-llama-30b.

[axel7083]

I tried to use the following one elinas/llama-30b-hf-transformers-4.29 since the one you provided have the tokenizer_config.json file corrupted.
However the transformer version is not compatible. And lead to a lot of errors

[andreaskoepf]

You might try: https://huggingface.co/timdettmers/guanaco-33b-merged … it was not directly trained by us but also on the OA-Dataset and is a recent model.