Llama-v2-7B-Chat: State-of-the-art large language model useful on a variety of language understanding and generation tasks
Llama 2 is a family of LLMs. The "Chat" at the end indicates that the model is optimized for chatbot-like dialogue. The model is quantized to w4a16(4-bit weights and 16-bit activations) and part of the model is quantized to w8a16(8-bit weights and 16-bit activations) making it suitable for on-device deployment. For Prompt and output length specified below, the time to first token is Llama-PromptProcessor-Quantized's latency and average time per addition token is Llama-TokenGenerator-KVCache-Quantized's latency.
This is based on the implementation of Llama-v2-7B-Chat found here. This repository contains scripts for optimized on-device export suitable to run on Qualcomm® devices. More details on model performance accross various devices, can be found here.
Sign up for early access to run these models on a hosted Qualcomm® device.
Large Language Model (LLM) such as Llama 2 has the following complexities to deploy on-device:
- Model size is too large to fit in device memory for inference
- Multi-Head Attention (MHA) has large activations leading to fallback from accelerators
- High model load and inference time
We can tackle the above constraints with the following steps:
- Quantize weights to reduce on-disk model size, e.g., int8 or int4 weights
- Quantize activations to reduce inference time memory pressure
- Graph transformations to reduce inference time memory pressure, e.g., Multi-Head to Split-Head Attention (MHA -> SHA)
- Graph transformations to convert or decompose operations into more accelerator friendly operations e.g. Linear to Conv
- For LLM with 7B or more parameters, above steps are still not good enough on mobile, hence we go one step further and split model into sub-parts.
Here, we divide the model into 4 parts in order to
- Make model exportable with low memory usage
- Avoid inference time out-of-memory errors
In order to export Llama 2, please ensure
- Host machine has >40GB memory (RAM+swap-space)
- If you don't have enough memory, export.py will dump instructions to increase swap space accordingly.
- --prompt "what is gravity?" --max-output-tokens 30
-------- Response Summary --------
Prompt: what is gravity?
Response: Hello! I'm here to help you answer your question. Gravity is a fundamental force of nature that affects the behavior of objects with mass
- --prompt "what is 2+3?" --max-output-tokens 30
-------- Response Summary --------
Prompt: what is 2+3?
Response: Of course! I'm happy to help! The answer to 2+3 is 5.
- --prompt "could you please write code for fibonacci series in python?" --max-output-tokens 100
-------- Response Summary --------
Prompt: could you please write code for fibonacci series in python?
Response: Of course! Here is an example of how you could implement the Fibonacci sequence in Python:
```
def fibonacci(n):
if n <= 1:
return n
else:
return fibonacci(n-1) + fibonacci(n-2)
```
You can test the function by calling it with different values of `n`, like this:
```
print(fibonacci(5))
Install the package via pip:
pip install "qai_hub_models[llama_v2_7b_chat_quantized]"
Once installed, run the following simple CLI demo:
python -m qai_hub_models.models.llama_v2_7b_chat_quantized.demo
More details on the CLI tool can be found with the --help
option. See
demo.py for sample usage of the model including pre/post processing
scripts. Please refer to our general instructions on using
models for more usage instructions.
This repository contains export scripts that produce a model optimized for on-device deployment. This can be run as follows:
python -m qai_hub_models.models.llama_v2_7b_chat_quantized.export
Additional options are documented with the --help
option. Note that the above
script requires access to Deployment instructions for Qualcomm® AI Hub.
- The license for the original implementation of Llama-v2-7B-Chat can be found here.
- The license for the compiled assets for on-device deployment can be found here
- Join our AI Hub Slack community to collaborate, post questions and learn more about on-device AI.
- For questions or feedback please reach out to us.
This model may not be used for or in connection with any of the following applications:
- Accessing essential private and public services and benefits;
- Administration of justice and democratic processes;
- Assessing or recognizing the emotional state of a person;
- Biometric and biometrics-based systems, including categorization of persons based on sensitive characteristics;
- Education and vocational training;
- Employment and workers management;
- Exploitation of the vulnerabilities of persons resulting in harmful behavior;
- General purpose social scoring;
- Law enforcement;
- Management and operation of critical infrastructure;
- Migration, asylum and border control management;
- Predictive policing;
- Real-time remote biometric identification in public spaces;
- Recommender systems of social media platforms;
- Scraping of facial images (from the internet or otherwise); and/or
- Subliminal manipulation