Overview

OGA-based Flow

Ryzen AI Software supports deploying quantized 4-bit LLMs on Ryzen AI 300-series PCs using the OnnxRuntime GenAI (OGA) framework. OGA is a multi-vendor generative AI framework from Microsoft that provides a convenient LLM interface for execution backends such as Ryzen AI.

The flow supports two execution modes:

• NPU-only execution mode: the compute-intensive operations are exclusively offloaded to the NPU. The iGPU is not solicited and can be used for other tasks.

• Hybrid execution mode: the model is optimally partitioned such that different operations are scheduled on NPU or on the iGPU. This minimizes time-to-first-token (TTFT) in the prefill-phase and maximizes token generation (tokens per second, TPS) in the decode phase.

Supported Configurations

• Only Ryzen AI 300-series Strix Point (STX) and Krackan Point (KRK) processors support OGA-based hybrid execution.

• Developers with Ryzen AI 7000- and 8000-series processors can get started using the CPU-based examples linked in the Featured LLMs table.

• Windows 11 is the required operating system.

Development Interfaces

The Ryzen AI LLM software stack is available through three development interfaces, each suited for specific use cases as outlined in the sections below. All three interfaces are built on top of native OnnxRuntime GenAI (OGA) libraries, as shown in the Ryzen AI Software Stack diagram below.

The high-level Python APIs, as well as the Server Interface, also leverage the Lemonade SDK, which is multi-vendor open-source software that provides everything necessary for quickly getting started with LLMs on OGA.

A key benefit of both OGA and Lemonade is that software developed against their interfaces is portable to many other execution backends.

Ryzen AI Software Stack

Your Python Application	Your LLM Stack	Your Native Application
Lemonade Python API*	Lemonade Server Interface*	OGA C++ Headers
OGA Python API*
Custom AMD OnnxRuntime GenAI (OGA)
AMD Ryzen AI Driver and Hardware

* indicates open-source software (OSS).

High-Level Python SDK

The high-level Python SDK, Lemonade, allows you to get started using PyPI installation in approximately 5 minutes.

This SDK allows you to:

• Experiment with models in hybrid execution mode on Ryzen AI hardware.

• Validate inference speed and task performance.

• Integrate with Python apps using a high-level API.

To get started in Python, follow these instructions: High-Level Python SDK.

Server Interface (REST API)

The Server Interface provides a convenient means to integrate with applications that:

• Already support an LLM server interface, such as the Ollama server or OpenAI API.

• Are written in any language (C++, C#, Javascript, etc.) that supports REST APIs.

• Benefits from process isolation for the LLM backend.

To get started with the server interface, follow these instructions: Server Interface (REST API).

For example applications that have been tested with Lemonade Server, see the Lemonade Server Examples.

OGA APIs for C++ Libraries and Python

Native C++ libraries for OGA are available to give full customizability for deployment into native applications.

The Python bindings for OGA also provide a customizable interface for Python development.

To get started with the OGA APIs, follow these instructions: OnnxRuntime GenAI (OGA) Flow.

Featured LLMs

The following tables contain a curated list of LLMs that have been validated on Ryzen AI hybrid execution mode. The hybrid examples are built on top of OnnxRuntime GenAI (OGA).

The comprehensive set of pre-optimized models for hybrid execution used in these examples are available in the AMD hybrid collection on Hugging Face. It is also possible to run fine-tuned versions of the models listed (for example, fine-tuned versions of Llama2 or Llama3). For instructions on how to prepare a fine-tuned OGA model for hybrid execution, refer to Preparing OGA Models.

Ryzen AI OGA Featured LLMs

	CPU Baseline (HF bfloat16)		Ryzen AI Hybrid (OGA int4)
Model	Example	Validation	Example	TTFT Speedup	Tokens/S Speedup	Validation
DeepSeek-R1-Distill-Qwen-7B	Link	🟢	Link	3.4x	8.4x	🟢
DeepSeek-R1-Distill-Llama-8B	Link	🟢	Link	4.2x	7.6x	🟢
Llama-3.2-1B-Instruct	Link	🟢	Link	1.9x	5.1x	🟢
Llama-3.2-3B-Instruct	Link	🟢	Link	2.8x	8.1x	🟢
Phi-3-mini-4k-instruct	Link	🟢	Link	3.6x	7.8x	🟢
Qwen1.5-7B-Chat	Link	🟢	Link	4.0x	7.3x	🟢
Mistral-7B-Instruct-v0.3	Link	🟢	Link	5.0x	8.1x	🟢
Llama-3.1-8B-Instruct	Link	🟢	Link	3.9x	8.9x	🟢

The Ryzen AI OGA Featured LLMs table was compiled using validation, benchmarking, and accuracy metrics as measured by the ONNX TurnkeyML v6.1.0 lemonade commands in each example link. After this table was created, the Lemonade SDK moved to the new location found here.

Data collection details:

• All validation, performance, and accuracy metrics are collected on the same system configuration:

  • System: HP OmniBook Ultra Laptop 14z

  • Processor: AMD Ryzen AI 9 HX 375 W/ Radeon 890M

  • Memory: 32GB of RAM

• The Hugging Face transformers framework is used as the baseline implementation for speedup and accuracy comparisons.

  • The baseline checkpoint is the original safetensors Hugging Face checkpoint linked in each table row, in the bfloat16 data type.

• All speedup numbers are the measured performance of the model with input sequence length (ISL) of 1024 and output sequence length (OSL) of 64, on the specified backend, divided by the measured performance of the baseline.

• We assign the 🟢 validation score based on this criteria: all commands in the example guide ran successfully.