# Execution Providers Windows ML provides a system-level execution provider (EP) management layer for ONNX Runtime on Windows PCs. It automatically discovers, downloads, and registers the best-available EPs for your hardware, whether that is a CPU, GPU, or NPU, so your application always runs on the optimal hardware accelerator. Windows ML ships a shared, Windows-wide ONNX Runtime and exposes EP management APIs for C#, C++, and Python. Through these APIs you can: - **Auto-register all compatible EPs** with a single API call, Windows ML handles version resolution and updates. - **Set an execution policy** (e.g., `PREFER_NPU`) to steer workloads to a preferred device class with automatic fallback. - **Target a specific EP and device** by enumerating available EP devices and appending the one you need — for example, `VitisAIExecutionProvider` on an AMD NPU. - **Compile models for a specific EP** as a one-time step that optimizes the model for the target hardware, and the compiled artifact can be cached for all subsequent runs. Models from PyTorch, TensorFlow/Keras, TensorFlow Lite (TFLite), scikit-learn, and other frameworks can be converted to ONNX and executed through this managed EP infrastructure. For more details, see the [Windows ML official documentation](https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/overview). ## Automatic Execution Providers (EPs) Registration Windows ML will automatically discover, download, and register the latest version of all compatible execution providers. ### C++ Example ```cpp #include #include // First we need to create an ORT environment Ort::Env env(ORT_LOGGING_LEVEL_ERROR, "WinMLDemo"); // Use an ID of your own choice // Get the default ExecutionProviderCatalog winrt::Microsoft::Windows::AI::MachineLearning::ExecutionProviderCatalog catalog = winrt::Microsoft::Windows::AI::MachineLearning::ExecutionProviderCatalog::GetDefault(); // Ensure and register all compatible execution providers with ONNX Runtime catalog.EnsureAndRegisterAllAsync().get(); ``` ### Python Example ```python # Known issue: import winrt.runtime will cause the TensorRTRTX execution provider to fail registration. # As a workaround, run pywinrt related code in a separate thread. # winml.py import json def _get_ep_paths() -> dict[str, str]: from winui3.microsoft.windows.applicationmodel.dynamicdependency.bootstrap import ( InitializeOptions, initialize ) import winui3.microsoft.windows.ai.machinelearning as winml eps = {} with initialize(options = InitializeOptions.ON_NO_MATCH_SHOW_UI): catalog = winml.ExecutionProviderCatalog.get_default() providers = catalog.find_all_providers() for provider in providers: provider.ensure_ready_async().get() eps[provider.name] = provider.library_path # DO NOT call provider.try_register in Python. That will register to the native env. return eps if __name__ == "__main__": eps = _get_ep_paths() print(json.dumps(eps)) # In your application code import subprocess import json import sys from pathlib import Path import onnxruntime as ort def register_execution_providers(): worker_script = str(Path(__file__).parent / 'winml.py') result = subprocess.check_output([sys.executable, worker_script], text=True) paths = json.loads(result) for item in paths.items(): ort.register_execution_provider_library(item[0], item[1]) _ep_registered = True register_execution_providers() ``` The `register_execution_providers` function is used to download and register the latest version of all compatible execution providers. ## Execution Policy The EP selection policy can be configured to use specific execution provider or through general execution policy. For more details, refer to the Windows ML documentation on [Execution Providers](https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/select-execution-providers?tabs=python). For example, setting the execution policy to PREFER_NPU will prioritize the NPU execution provider if available, with a fallback to CPU execution if an NPU is not present. ### C++ Example ```cpp // Configure the session to select an EP and device for PREFER_NPU which typically // will choose an NPU if available with a CPU fallback. Ort::SessionOptions sessionOptions; sessionOptions.SetEpSelectionPolicy(OrtExecutionProviderDevicePolicy_PREFER_NPU); ``` ### Python Example ```python # Configure the session to select an EP and device for PREFER_NPU which typically # will choose an NPU if available with a CPU fallback. options = ort.SessionOptions() options.set_provider_selection_policy(ort.OrtExecutionProviderDevicePolicy.PREFER_NPU) assert options.has_providers() ``` Specifying the **specific execution provider** can be done through the set_providers API. For example, setting the execution provider to VitisAIExecutionProvider will only use the VitisAI EP for model execution. ### C++ Example ```cpp #include #include #include #include #include // Assuming you have an Ort::Env named 'env' // 1. Enumerate EP devices std::vector ep_devices = env.GetEpDevices(); // 2. Collect only ReplaceWithExecutionProvider NPU devices std::vector selected_ep_devices; for (const auto& d : ep_devices) { if (std::string(d.EpName()) == "VitisAIExecutionProvider" && d.HardwareDevice().Type() == OrtHardwareDeviceType_NPU) { selected_ep_devices.push_back(d); } } if (selected_ep_devices.empty()) { throw std::runtime_error("VitisAIExecutionProvider is not available on this system."); } // 3. Configure provider-specific options (varies based on EP) // and append the EP with the correct devices (varies based on EP) Ort::SessionOptions session_options; Ort::KeyValuePairs ep_options; ep_options.Add("optimize_level", "1"); session_options.AppendExecutionProvider_V2(env, { selected_ep_devices.front() }, ep_options); ``` ### Python Example ```python # 1. Enumerate and filter EP devices ep_devices = ort.get_ep_devices() selected_ep_devices = [ d for d in ep_devices if d.ep_name == "VitisAIExecutionProvider" and d.device.type == ort.OrtHardwareDeviceType.NPU] if not selected_ep_devices: raise RuntimeError("VitisAIExecutionProvider is not available on this system.") # 2. Configure provider-specific options in "vaiml_config.json" file session_options = ort.SessionOptions() provider_options = {'config_file':'vaiml_config.json'} session_options.add_provider_for_devices([selected_ep_devices[0]], provider_options) ``` For more details on the VitisAIExecutionProvider-specific provider_options, see [Model compilation and deployment](https://ryzenai.docs.amd.com/en/latest/modelrun.html) ## Model Compilation Models need to be compiled for specific EPs. This is a one-time process that stores the compiled model for subsequent runs: ### C++ Example ```cpp bool isCompiledModelAvailable = std::filesystem::exists(compiledModelPath); if (!isCompiledModelAvailable) { Ort::ModelCompilationOptions compile_options(env, sessionOptions); compile_options.SetInputModelPath(modelPath.c_str()); compile_options.SetOutputModelPath(compiledModelPath.c_str()); std::cout << "Starting compile, this may take a few moments..." << std::endl; Ort::Status compileStatus = Ort::CompileModel(env, compile_options); if (compileStatus.IsOK()) { std::cout << "Model compiled successfully!" << std::endl; isCompiledModelAvailable = true; } } ``` ### Python Example ```python input_model_path = "path_to_your_model.onnx" output_model_path = "path_to_your_compiled_model.onnx" model_compiler = ort.ModelCompiler( options, input_model_path, embed_compiled_data_into_model=True, external_initializers_file_path=None, ) model_compiler.compile_to_file(output_model_path) if not os.path.exists(output_model_path): # For some EPs, there might not be a compilation output. # In that case, use the original model directly. output_model_path = input_model_path ``` For more details on VitisAIExecutionProvider specific provider_options as shown in the reference documentation [Model compilation and deployment](https://ryzenai.docs.amd.com/en/latest/modelrun.html)