dachack/ollama-intel-docker repository overview

⁠Ollama with Intel GPUs

This repository demonstrates running Ollama⁠ with ipex-llm⁠ as an accelerated backend, compatible with both Intel iGPUs and dedicated GPUs (such as Arc, Flex, and Max). The provided docker-compose.yml file includes a patched version of Ollama for Intel acceleration with the required parameters and settings, along with the Open WebUI⁠ interface for convenience.

Read more about the project and benchmark results in blog post: https://nikolasent.github.io/hardware/deeplearning/2025/02/09/iGPU-Benchmark-VLM.html⁠. The results were obtained with intelanalytics/ipex-llm-inference-cpp-xpu:2.2.0-SNAPSHOT base Docker image running on Debian 12 with Kernel 6.11.5.

⁠Quick Start

Using Intel GPUs requires that you have Intel firmware installed. For example, on Debian-like systems:

sudo apt-get install firmware-misc-nonfree firmware-intel-graphics
sudo update-initramfs -u -k all  # Required after kernel updates as well.

Docker⁠ and docker compose⁠ are also required.

docker compose build
docker compose up -d

Open http://127.0.0.1:18080⁠ to access Open WebUI with accelerated ollama backend.

Tip: For performance monitoring, including GPU utilization and power usage, intel-gpu-top is a useful tool, which is provided as part of intel-gpu-tools package:

sudo apt install intel-gpu-tools
sudo intel_gpu_top

Alternatively, one may compile btop++⁠ with Intel GPU support.

⁠Parameters

The Docker environment is pre-configured to run on Intel iGPUs. Here are some parameters that may need adjustment:

In the docker-compose.yml⁠ file:

• Configure the volumes of services to set up where data and models will be stored. Prefer using disks with fast I/O.
• Use memory limit features, such as mem_limit: "32G", to limit RAM used by ipex_ollama service.
• Configure DEVICE variable if another hardware, such as a dedicated GPU, is used.
• Customize OLLAMA_NUM_GPU if required to manage GPU offload.
• Other Ollama variables can be specified in the environment section of the ipex_ollama service. For example, set OLLAMA_NUM_CTX to change the default model context length (it is set to be 8192 by default). Please note that some variable names are different from those used for similar purposes in the original Ollama.

⁠Advice on performance

1. If using CPU inference, tuning the num_thread model parameter in ollama for specific tasks (given the model and context length) may improve performance.
2. Use the cpuset option in docker-compose.yml to pin the ipex_ollama service to specific CPU cores. For example, use cpuset: "0-3" to utilize the first four CPU cores (e.g., to use performance cores only). Select the most performant value empirically.
3. It can be a good idea to use optimised models, for example, the models optimised by Unsloth⁠ to achieve better performance (for example, Qwen3:4b with Unsloth optimisations runs 3.8% faster on an iGPU).

⁠Benchmarks

The script scripts/benchmark.py⁠ contains a benchmarking tool that evaluates tokens/s generated by any OpenAI-compatible API, including benchmarks for both Language Models (LLMs) and Vision-Language Models (VLMs). The benchmarks are reported on an Intel Ultra 5 125H Meteor Lake SoC with 64GB RAM.

With sufficient RAM, this SoC can handle relatively large models locally, making it a power-efficient solution for low-cost experiments with local models.

Feel free to explore the benchmark code and adjust it as needed for your specific experimentation and setup. The provided code is configured to produce the results below, so ensure that the required models are pulled before running the benchmark script.

The benchmark script is designed to be a standalone script that can be executed from the host machine (not from inside the Docker environment). One can use this benchmark code to test any OpenAI-compatible APIs by adjusting the API_URI and specifying the required model names.

⁠Language models

⁠VLMs

⁠Links

1. Intel docs on ipex-llm: Run Ollama with IPEX-LLM on Intel GPU⁠.
2. ipex-llm repo⁠.