# Running LLM on Linux

This page showcases an example of running LLM on RyzenAI NPU

- Open a Linux terminal and create a new folder

```bash
mkdir run_llm
cd run_llm
```

- Here is an Overview of all supported Models and their available variants [List of Supported Models](https://ryzenai.docs.amd.com/en/latest/llm_list.html)
- Note - Linux does not support Hybrid flow
- Choose any prequantized and postprocessed ready-to-run Model from Hugging Face collection of NPU models

```none
[Full-optimized Models with 4K Context length](https://huggingface.co/collections/amd/ryzen-ai-180-npu-4k)

[Long-context Models with 16K Context length](https://huggingface.co/collections/amd/ryzen-ai-180-npu-16k)
```

- For this flow, “Phi-3.5-mini-instruct_rai_1.7.1_npu_4K” is chosen for reference

```none
# Make sure git-lfs is installed (https://git-lfs.com)
sudo apt install git-lfs
git lfs install
git clone https://huggingface.co/amd/Phi-3.5-mini-instruct_rai_1.7.1_npu_4K
```

- Search for RYZEN_AI_INSTALLATION_PATH

```bash
# Activate the virtual environment created in Linux Installation step
source <TARGET-PATH>/venv/bin/activate

echo $RYZEN_AI_INSTALLATION_PATH
```

- Collecting the necessary files to get in current working directory

```bash
- Deployment folder - This has necessary libraries to run LLM Model
    # Navigate to <TARGET-PATH>/venv and copy the "deployment" folder
    cp -r <TARGET-PATH>/venv/deployment .

- Model Benchmark Script
    # Navigate to <TARGET-PATH>/venv/LLM/examples/ and copy "model_benchmark" file.
    cp <TARGET-PATH>/venv/LLM/examples/model_benchmark .

- Prompt file - Input to your LLM Model
    # Navigate to <TARGET-PATH>/venv/LLM/examples/ and copy "amd_genai_prompt.txt" file.
    cp <TARGET-PATH>/venv/LLM/examples/amd_genai_prompt.txt .
```

- Current working directory should have below files

```none
amd_genai_prompt.txt   deployment   model_benchmark   Phi-3.5-mini-instruct_rai_1.7.1_npu_4K
```

- Lastly, set required library path

```bash
export LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:deployment/lib:$LD_LIBRARY_PATH
export RYZENAI_EP_PATH=$PWD/deployment/lib/libonnxruntime_providers_ryzenai.so
source /opt/xilinx/xrt/setup.sh
```

- We can now run our Model with command below:

```bash
./model_benchmark -i Phi-3.5-mini-instruct_rai_1.7.1_npu_4K/ -l 128

 -i - Path to the ONNX model directory to benchmark
 -l - Number of tokens in the generated prompt (Default: 16)

# Use "./model_benchmark --help" to enable more options
```

## Expected output

```bash
-----------------------------
Prompt Number of Tokens: 128

Batch size: 1, prompt tokens: 128, tokens to generate: 128
Prompt processing (time to first token):
       avg (us):       169860
       avg (tokens/s): 753.562
       p50 (us):       169022
       stddev (us):    6108.17
       n:              5 * 128 token(s)
Token generation:
       avg (us):       20354.1
       avg (tokens/s): 49.1301
       p50 (us):       19964.9
       stddev (us):    4411.67
       n:              635 * 1 token(s)
Token sampling:
       avg (us):       192.274
       avg (tokens/s): 5200.91
       p50 (us):       202.417
       stddev (us):    76.6932
       n:              5 * 1 token(s)
E2E generation (entire generation loop):
       avg (ms):       2755.09
       p50 (ms):       2747.84
       stddev (ms):    14.0296
       n:              5
Peak working set size (bytes): 3543330816
```

## Preparing OGA Model

- Model Generate is not supported in current release. Choose any prequantized and postprocessed ready-to-run Model from the provided list.
- Read more on Windows specific Model Generation by visiting [Preparing OGA Models](https://ryzenai.docs.amd.com/en/latest/oga_model_prepare.html)

## Limitations

For some Models, the memory requirement is more than system default. Use the command below to change it to “unlimited”

```bash
sudo tee /etc/security/limits.d/99-memlock.conf >/dev/null <<'EOF'
*    soft    memlock    unlimited
*    hard    memlock    unlimited
EOF

# execute the command to verify
ulimit -l
```