Hugging Face's logo

qualcomm
/
Llama-v3-8B-Instruct

Text Generation
PyTorch
llm
generative_ai
android
Model card Files
xet
Community
Llama-v3-8B-Instruct / README.md
qaihm-bot's picture
qaihm-bot
v0.37.0
7eb738e verified
preview code
|
raw
history blame contribute delete
6.38 kB
metadata
library_name: pytorch
license: other
tags:
  - llm
  - generative_ai
  - android
pipeline_tag: text-generation

Llama-v3-8B-Instruct: Optimized for Mobile Deployment

State-of-the-art large language model useful on a variety of language understanding and generation tasks

Llama 3 is a family of LLMs. The model is quantized to w4a16 (4-bit weights and 16-bit activations) and part of the model is quantized to w8a16 (8-bit weights and 16-bit activations) making it suitable for on-device deployment. For Prompt and output length specified below, the time to first token is Llama-PromptProcessor-Quantized's latency and average time per addition token is Llama-TokenGenerator-Quantized's latency.

This model is an implementation of Llama-v3-8B-Instruct found here.

This repository provides scripts to run Llama-v3-8B-Instruct on Qualcomm® devices. More details on model performance across various devices, can be found here.

WARNING: The model assets are not readily available for download due to licensing restrictions.

Model Details

  • Model Type: Model_use_case.text_generation
  • Model Stats:
    • Input sequence length for Prompt Processor: 128
    • Context length: 4096
    • Precision: w4a16 + w8a16 (few layers)
    • Num of key-value heads: 8
    • Model-1 (Prompt Processor): Llama-PromptProcessor-Quantized
    • Prompt processor input: 128 tokens + position embeddings + attention mask + KV cache inputs
    • Prompt processor output: 128 output tokens + KV cache outputs
    • Model-2 (Token Generator): Llama-TokenGenerator-Quantized
    • Token generator input: 1 input token + position embeddings + attention mask + KV cache inputs
    • Token generator output: 1 output token + KV cache outputs
    • Use: Initiate conversation with prompt-processor and then token generator for subsequent iterations.
    • Minimum QNN SDK version required: 2.27.7
    • Supported languages: English.
    • TTFT: Time To First Token is the time it takes to generate the first response token. This is expressed as a range because it varies based on the length of the prompt. The lower bound is for a short prompt (up to 128 tokens, i.e., one iteration of the prompt processor) and the upper bound is for a prompt using the full context length (4096 tokens).
    • Response Rate: Rate of response generation after the first response token.

| Model | Precision | Device | Chipset | Target Runtime | Response Rate (tokens per second) | Time To First Token (range, seconds) |---|---|---|---|---|---| | Llama-v3-8B-Instruct | w4a16 | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite Mobile | GENIE | 12.9262 | 0.159383 - 5.100256 | -- | -- | | Llama-v3-8B-Instruct | w4a16 | Snapdragon X Elite CRD | Snapdragon® X Elite | GENIE | 10.0367 | 0.211644 - 6.772608 | -- | -- | | Llama-v3-8B-Instruct | w4a16 | SA8255P ADP | Qualcomm® SA8255P | GENIE | 7.55837 | 0.3644368 - 11.6619776 | -- | -- |

Deploying Llama 3 on-device

Please follow the LLM on-device deployment tutorial.

Installation

Install the package via pip:

pip install "qai-hub-models[llama-v3-8b-instruct]"

Configure Qualcomm® AI Hub to run this model on a cloud-hosted device

Sign-in to Qualcomm® AI Hub with your Qualcomm® ID. Once signed in navigate to Account -> Settings -> API Token.

With this API token, you can configure your client to run models on the cloud hosted devices.

qai-hub configure --api_token API_TOKEN

Navigate to docs for more information.

Demo off target

The package contains a simple end-to-end demo that downloads pre-trained weights and runs this model on a sample input.

python -m qai_hub_models.models.llama_v3_8b_instruct.demo

The above demo runs a reference implementation of pre-processing, model inference, and post processing.

NOTE: If you want running in a Jupyter Notebook or Google Colab like environment, please add the following to your cell (instead of the above).

%run -m qai_hub_models.models.llama_v3_8b_instruct.demo

Run model on a cloud-hosted device

In addition to the demo, you can also run the model on a cloud-hosted Qualcomm® device. This script does the following:

  • Performance check on-device on a cloud-hosted device
  • Downloads compiled assets that can be deployed on-device for Android.
  • Accuracy check between PyTorch and on-device outputs.
python -m qai_hub_models.models.llama_v3_8b_instruct.export

Deploying compiled model to Android

The models can be deployed using multiple runtimes:

  • TensorFlow Lite (.tflite export): This tutorial provides a guide to deploy the .tflite model in an Android application.

  • QNN (.so export ): This sample app provides instructions on how to use the .so shared library in an Android application.

View on Qualcomm® AI Hub

Get more details on Llama-v3-8B-Instruct's performance across various devices here. Explore all available models on Qualcomm® AI Hub

License

  • The license for the original implementation of Llama-v3-8B-Instruct can be found here.
  • The license for the compiled assets for on-device deployment can be found here

References

Community