vllm with xpu kernel extensions for Intel Arc GPUs
1.7K
pre-built version of vllm with xpu kernel extensions from: https://github.com/vllm-project/
Built from vllm-project's Dockerfile.xpu
This includes optimized custom kernels for Intel GPUs (XPU) from: https://github.com/vllm-project/vllm-xpu-kernels
Example compose with a model that works well on Intel Arc B50 GPU
services:
vllm:
image: telnetdoogie/vllm-xpu:latest
ports:
- 8000:8000
container_name: vllm
environment:
HUGGING_FACE_HUB_TOKEN: <your_HF_token_here>
VLLM_WORKER_MULTIPROC_METHOD: spawn
privileged: true
restart: always
volumes:
- /dev/dri/by-path:/dev/dri/by-path
- /volume1/docker/ollama/vllm_cache:/root/.cache/huggingface
devices:
- /dev/dri:/dev/dri
group_add:
- "44" # video group
- "992" # render group
command: >
Qwen/Qwen2.5-7B-Instruct
--dtype=float16
--enforce-eager
--port 8000
--block-size 16
--gpu-memory-util 0.9
--no-enable-prefix-caching
--trust-remote-code
--disable-sliding-window
--max-num-batched-tokens 4096
--max-model-len 16384
--quantization fp8
--tool-call-parser hermes
--enable-auto-tool-choice
--safetensors-load-strategy=prefetch
Content type
Image
Digest
sha256:a3e3be8e8…
Size
8.3 GB
Last updated
3 months ago
docker pull telnetdoogie/vllm-xpu