mekayelanik/vllm-cpu repository overview

CPU-Optimized vLLM: Easy, Fast LLM Inference Without a GPU

Unified CPU wheel with automatic ISA detection at runtime (AVX2, AVX-512, VNNI, BF16, AMX, NEON, FP16, DOTPROD)

⁠ ⁠ ⁠

This is an independent, community-maintained package — not affiliated with or funded by the vLLM project, its sister concerns, or any hardware vendors. The first successful unification of different CPU ISAs (AVX2, AVX-512, VNNI, BF16, AMX) into a single wheel was done by Mekayel Anik⁠, for the benefit of the community.

⁠ ⁠ ⁠

⁠ ⁠ ⁠ ⁠

⁠ ⁠

⁠

⁠Why vllm-cpu?

The upstream vLLM project publishes CPU wheels only on GitHub Releases with a +cpu local version suffix, which cannot be uploaded to PyPI. Users must manually copy long URLs to install. This project solves that:

⁠Quick Start

⁠Install from PyPI

pip3 install vllm-cpu

⁠Start an OpenAI-compatible API server

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-0.6B", device="cpu")
output = llm.generate("The future of AI is", SamplingParams(temperature=0.8, max_tokens=128))
print(output[0].outputs[0].text)

⁠Or use the CLI

vllm serve Qwen/Qwen3-0.6B --device cpu --dtype auto

Then query it:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The future of AI is", "max_tokens": 128}'

⁠Requirements

• Python: 3.10+ (stable ABI, one wheel for all versions)
• OS: Linux (glibc 2.28+) — Debian 10+, Ubuntu 18.04+, RHEL 8+, Amazon Linux 2023+
• CPU: x86_64 with AVX2 (minimum) or AVX-512 (optimal), or aarch64 with NEON (BF16 recommended)

⁠Supported CPU Instructions

The unified wheel automatically detects and uses the best available instruction set:

No configuration needed — the correct .so is loaded automatically at import vllm.

⁠Install

⁠PyPI

# Latest
pip3 install vllm-cpu

# Specific version
pip3 install vllm-cpu==0.29.0

⁠Docker

# Docker Hub
docker pull mekayelanik/vllm-cpu:latest

# GHCR
docker pull ghcr.io/mekayelanik/vllm-cpu:latest

# Specific version
docker pull mekayelanik/vllm-cpu:0.29.0

# ARM64 without BF16 (for Graviton 2, Pi 5, older Altra)
docker pull mekayelanik/vllm-cpu:arm64-no-bf16-latest

⁠Docker Usage

⁠Quick start

docker run -d \
  --name vllm-cpu \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  mekayelanik/vllm-cpu:latest \
  --model Qwen/Qwen3-0.6B \
  --dtype auto

⁠Docker Compose

services:
  vllm:
    image: mekayelanik/vllm-cpu:latest
    ports:
      - "8000:8000"
    volumes:
      - huggingface-cache:/root/.cache/huggingface
    command: ["--model", "Qwen/Qwen3-0.6B", "--dtype", "auto"]
    deploy:
      resources:
        limits:
          memory: 16g
    restart: unless-stopped

volumes:
  huggingface-cache:

⁠Available Tags

⁠Supported Platforms

⁠How It Works

Starting with v0.17.0, vLLM ships a unified CPU wheel containing both AVX2 and AVX512 code paths:

1. The wheel includes _C.so (AVX512+BF16+VNNI+AMX) and _C_AVX2.so (AVX2 fallback)
2. At import time, vllm/platforms/cpu.py checks torch._C._cpu._is_avx512_supported()
3. The correct .so is loaded once — zero runtime dispatch overhead

⁠Stable ABI (cp38-abi3)

The wheels use Python's stable ABI⁠, meaning one wheel works with Python 3.10+. No per-Python-version builds needed.

⁠Build Process

Wheels are built from source inside manylinux_2_28 containers with GCC 14, ensuring broad glibc compatibility while using modern compiler optimizations.

⁠Registries

⁠Version Support

⁠Deprecated Variant Packages

The following variant packages have been deprecated as of v0.16.0 (last release). Starting with v0.17.0, the unified vllm-cpu package replaces all of them with automatic ISA detection at runtime.

These packages remain available on PyPI for older vLLM versions but will not receive further updates.

⁠Pipeline

Upstream vLLM release (v0.17.0+)
  --> Build unified CPU wheels in manylinux_2_28 (x86_64 + aarch64 + aarch64-no-bf16)
  --> Publish to PyPI + GitHub Releases
  --> Build multi-arch Docker images (linux/amd64 + linux/arm64)
  --> Build ARM64 no-BF16 Docker images (for CPUs without BF16 ISA)
  --> Push to GHCR + Docker Hub
  --> Promote :latest and :arm64-no-bf16-latest

⁠Links

• vLLM Documentation⁠
• vLLM GitHub⁠
• Report Issues⁠
• Changelog⁠

⁠License

This project is licensed under the GNU General Public License v3.0⁠ (GPL-3.0).

Your support encourages me to keep creating/supporting my open-source projects.

⁠