Sign inSign up

eilandert/rspamd-dcc-razor-pyzor

By eilandert

Updated about 21 hours ago

Image
1

5.0K

eilandert/rspamd-dcc-razor-pyzor repository overview

rspamd-dcc-razor-pyzor

A small, standalone Docker backend that brings the three classic collaborative-filtering networks — DCC, Razor and Pyzor — to rspamd through one HTTP endpoint, plus the rspamd plugin that talks to it.

The image runs no rspamd of its own. Your rspamd stays in its own container (or on the host); you drop the plugin shipped here into it and point the plugin at this backend.

Why a separate backend? rspamd has a built-in DCC module but nothing for Razor or Pyzor, and shelling out to those CLIs from inside the rspamd worker would block its event loop. This service is a single static Go binary that speaks all three networks in-process — Razor, Pyzor and DCC via the gazor, gyzor and gdcc libraries — answering over HTTP, so the plugin stays fully asynchronous, one request covers all three networks at once, and there are no per-message subprocess forks at all.

How it works

  ┌─────────────────────┐    HTTP :8077 + token    ┌──────────────────────────────┐
  │ rspamd (your image) │  ──── POST /check ───►   │  gozer  (distroless, nonroot)│
  │ dcc_razor_pyzor.lua │                          │   ├─ gdcc   (DCC,   in-proc) │
  │                     │  ◄─── JSON verdict ───   │   ├─ gazor  (Razor, in-proc) │
  │                     │                          │   └─ gyzor  (Pyzor, in-proc) │
  └─────────────────────┘                          └──────────────────────────────┘

The image is a single ~6 MB static gozer binary on a distroless/static base — no Debian, no s6 supervisor, no shell, no per-message fork. gozer is the container entrypoint and runs as nonroot. Its source lives in its own repo, eilandert/gozer, pulled in here as the docker/gozer submodule and compiled by this repo's image build. It queries the three networks concurrently, all in-process, and caches verdicts (see Configuration). All three talk to their servers directly (DCC needs no dccifd daemon). Every backend is best-effort: if one network is unreachable it simply doesn't score, and the container stays healthy — the healthcheck only depends on gozer's /health (probed by gozer health, since the image ships no shell or curl).

Hardening: gozer runs non-root with bounded concurrency, every POST is token-authenticated, and the bundled compose runs the container read-only with cap_drop: ALL, no-new-privileges, and no published host port.

Privacy: the message never touches disk

Gozer keeps the message in memory: all three checksums (Razor, Pyzor and DCC) are computed in-process — nothing is ever written to a temp file. The cache stores only sha256(body) → verdict (never the body itself), and the same goes for the optional Redis backend, so no message content is ever persisted locally.

(This is also why a tmpfs overlay would do nothing for speed: there is no per-message disk write to accelerate. The latency is network round-trips to the DCC/Razor/Pyzor servers.)

The only thing that leaves the container is what collaborative filtering needs: content fingerprints — DCC checksums, Razor signatures, Pyzor digests — sent to those networks (and, on /report, a spam submission). The raw message is never uploaded.

Quick start

1. Run the backend

Gozer rejects every POST until a token is configured, and it isn't published to the host — so run it with compose:

cd docker
mkdir -p secrets && openssl rand -hex 32 > secrets/drp_token.txt
docker compose up -d        # docker/docker-compose.yml

Containers on the same Docker network now reach it at http://rspamd-drp:8077. Give your rspamd (and Dovecot) the same token.

2. Install the plugin into rspamd

The plugin lives in rspamd/ at the repo root (it is not baked into the backend image):

cp rspamd/plugins/dcc_razor_pyzor.lua  /etc/rspamd/plugins/
cp rspamd/local.d/dcc_razor_pyzor.conf /etc/rspamd/local.d/
cp rspamd/local.d/groups.conf          /etc/rspamd/local.d/   # symbol scores
echo 'dofile("/etc/rspamd/plugins/dcc_razor_pyzor.lua")' >> /etc/rspamd/rspamd.local.lua

Then set the backend URL and the same token in local.d/dcc_razor_pyzor.conf:

url   = "http://rspamd-drp:8077/check";   # backend host:port
token = "the-shared-secret";              # must equal gozer's GOZER_TOKEN

Heads-up on DNS: rspamd resolves URLs through its own configured resolver. If that resolver can't see Docker service names (for example an RBL-only unbound), use the backend's IP address in url instead of rspamd-drp.

Restart rspamd. The plugin adds three symbols, scored in groups.conf (tune to taste):

SymbolMeaning
DRP_DCC_BULKDCC reports the body as bulk
DRP_RAZORRazor signature match
DRP_PYZORPyzor sightings above threshold

Identities

Anonymous works for every network and is the default. The image carries no writable state, so nothing persists between restarts unless you mount it. To use a known or shared identity, supply it as below — Razor and DCC take credentials from the environment (every var also accepts a <VAR>_FILE form for Docker secrets), Pyzor reads a standard accounts file:

NetworkHow to authenticateAnonymous default
RazorRAZOR_USER + RAZOR_PASS (or _FILE); obtain one with gozer razor-registeryes
DCCDCC_CLIENT_ID + DCC_CLIENT_PASSWD (or _FILE), or DCC_IDS; persist with gozer dcc-registeryes (id 1)
Pyzormount a pyzor accounts file at $PYZOR_HOME/accounts — see below; generate with gozer pyzor-registeryes
environment:
  DCC_CLIENT_ID: "1234567"
  DCC_CLIENT_PASSWD: "…"          # or DCC_IDS: <path to a DCC ids file>
  RAZOR_USER: "[email protected]"   # obtain one with `gozer razor-register`
  RAZOR_PASS: "…"

The gozer *-register subcommands persist a credential to the file the matching network loads it from and print it as bare KEY=value env lines, so you can feed it back through the environment instead of mounting a file. The image entrypoint is gozer serve, so override it to run a register subcommand:

docker compose run --rm --entrypoint /usr/local/bin/gozer rspamd-drp \
  pyzor-register --user [email protected] | grep '^GYZOR_' > pyzor.env
# likewise: razor-register --user … --pass …   |   dcc-register --client-id … --passwd …

See the gozer README for all the flags.

Pyzor authentication. Unlike Razor and DCC, Pyzor has no credential env var — the in-process gyzor client loads accounts the reference-pyzor way, from an accounts file in its home dir (PYZOR_HOME, default /var/lib/pyzor). Mount one read-only; this works even with the container's read_only: true rootfs:

environment:
  PYZOR_HOME: /var/lib/pyzor       # default; shown for clarity
volumes:
  - ./pyzor-accounts:/var/lib/pyzor/accounts:ro
# ./pyzor-accounts — one line per server: host : port : username : salt,key
public.pyzor.org : 24441 : [email protected] : 0123abcd,4567ef89…

Generate the salt,key with gozer pyzor-register (or the stock pyzor tool); gyzor consumes it byte-for-byte. With no accounts file the Pyzor client is anonymous to the public server, which suits most setups.

DNS-bypass server overrides

When the container's DNS is flaky or the public discovery servers are unreachable, pin each network's server list explicitly, bypassing DNS discovery entirely. Gozer forwards these to the matching in-process client (gdcc, gyzor, gazor); each accepts a comma list of host[:port] (hostname, IPv4, or bracketed IPv6 [::1]:port):

environment:
  DCC_SERVERS:     "dcc1.dcc-servers.net,dcc2.dcc-servers.net"  # → gdcc
  GYZOR_SERVERS:   "public.pyzor.org:24441"                     # → gyzor (Pyzor)
  GAZOR_DISCOVERY: "discovery.razor.cloudmark.com"              # → gazor (Razor), tried in order

Resolution order per network: explicit env → persisted identity file → anonymous. For Razor that means RAZOR_USER+RAZOR_PASS win, else the gazor-identity file under RAZORHOME (if you mounted one); for Pyzor, the mounted accounts file, else anonymous. Credential variables also accept a <VAR>_FILE form for Docker secrets — e.g. RAZOR_PASS_FILE=/run/secrets/razor_pass — so a secret never has to sit in the compose file.

Configuration

Every setting is a backend-container environment variable and also a gozer serve CLI flag (flag > env > default), so the same option works in compose or on the command line. The flag name is the env name lower-cased, de-prefixed and hyphenated — e.g. GOZER_MAX_CONCURRENT--max-concurrent, GYZOR_SERVERS--pyzor-servers. The full env/flag table and HTTP API are in the gozer README; the compose-relevant subset is below.

VariableDefaultPurpose
GOZER_TOKEN / GOZER_TOKEN_FILEShared secret for POST auth. Required — without it every POST returns 503.
GOZER_HOST / GOZER_PORT0.0.0.0 / 8077Bind address.
GOZER_CACHE_TTL300Verdict cache lifetime in seconds (0 disables). Bulk mail repeats, so cache hits are the main speed-up.
GOZER_CACHE_SIZE4096In-memory cache entries (LRU).
GOZER_REDIS_URLUse Redis for the cache so multiple scanners share it, e.g. redis://valkey:6379/5. Otherwise the cache is in-process.
GOZER_REDIS_PREFIXdrp:check:Key prefix in Redis.
GOZER_MAX_CONCURRENT8Max in-flight requests (bounds backend fan-out).
GOZER_BACKEND_TIMEOUT6Per-backend timeout in seconds.
GOZER_VERBOSE0 (off)Per-request logging — access line plus verdict, timing and cache hit/miss; also dumps the resolved config at startup. Off by default (only startup and errors are logged).
GOZER_LOG_STDOUT0 (off)Send info/access logs to stdout instead of stderr. Errors and warnings always stay on stderr so a log shipper can separate and alert on them. Both streams are captured by Docker regardless.
RAZOR_MIN_CFacRazor minimum confidence: ac, ac+N, ac-N, or a number.
DCC_SERVERS / GYZOR_SERVERS / GAZOR_DISCOVERYDNS-bypass server overrides forwarded to gdcc / gyzor / gazor (see above).
TZContainer timezone.

Benchmarks

From go test -bench (Go 1.26, 32-core host); reproduce with the commands shown.

Request path / throughputgo test -bench BenchmarkServe ./internal/gozer drives the full server (auth, concurrency gate, cache, single-flight, dispatch) against a 2 ms synthetic backend, over a mixed cache-hit ratio:

cache hit ratiothroughputbackend calls / request
90 %~29,600 msg/s0.11
50 %~11,500 msg/s0.53
0 % (all miss)~11,800 msg/s1.01

The verdict cache is the lever: at a 90 % hit ratio only ~1 request in 9 reaches a backend, and gozer clears 1000 msg/s comfortably even all-miss. The in-process cache hit itself is ~55 ns, zero allocations; an at-capacity LRU insert is ~344 ns.

Fingerprint compute (offline, one 256 KiB message) — the in-process CPU cost per network, all dwarfed by network RTT:

clientper messageallocations
gdcc — DCC checksums~3.2 ms (81 MB/s)24
gyzor — Pyzor digest~4.5 ms (58 MB/s)50
gazor — Razor signatures~8.4 ms (31 MB/s)833

Network round-trips dominate the cold path (anonymous, public servers, varies with distance): Pyzor ~50 ms, DCC ~170 ms, Razor ~1 s (multi-step discovery handshake). gozer queries all three concurrently, in-process, so a cold /check ≈ the slowest backend (Razor), not the sum — and a cached /check is a sub-millisecond local lookup.

HTTP API

The request body is always the raw RFC-822 message. POST endpoints require the token, sent as Authorization: Bearer <token> or X-DRP-Token: <token> (401 if it's wrong, 503 if gozer has no token). /health needs no auth.

  • POST /check — query only, never reports. Used by the rspamd plugin.

    { "dcc":   { "action": "reject", "bulk": 2147483647 },
      "razor": { "hit": true },
      "pyzor": { "count": 42, "wl": 0 } }
    
  • POST /report — report the message as spam to all three networks.

    { "dcc": true, "razor": true, "pyzor": true }
    
  • POST /revoke — report as ham. Razor and Pyzor support this; DCC has no network un-report, so its value is null.

  • GET /health200 ok, used by the container healthcheck.

  • GET /metrics — Prometheus exposition (no auth): per-endpoint request counters (gozer_check_total, gozer_report_total, gozer_revoke_total), gozer_error_total, gozer_busy_total, cache hit/miss/coalesced, Redis health (gozer_redis_error_total, gozer_redis_circuit_open_total), per-backend errors (gozer_backend_error_total{backend="dcc|razor|pyzor"}) and a gozer_latency_seconds histogram. gozer stats fetches and prints it locally (the image ships no curl).

Example

POST the raw message as the body — --data-binary keeps the bytes intact (the fingerprints are computed over them). From a container on the same network (or the host if you published the port):

TOKEN=$(cat docker/secrets/drp_token.txt)

# scan
curl -s --data-binary @message.eml \
  -H "Authorization: Bearer $TOKEN" http://rspamd-drp:8077/check
# {"dcc":{"action":"unknown","bulk":null},"razor":{"hit":false},"pyzor":{"count":42,"wl":0}}

# user feedback (X-DRP-Token works in place of the Bearer header)
curl -s --data-binary @spam.eml -H "X-DRP-Token: $TOKEN" http://rspamd-drp:8077/report
curl -s --data-binary @ham.eml  -H "X-DRP-Token: $TOKEN" http://rspamd-drp:8077/revoke
curl -s http://rspamd-drp:8077/metrics      # no auth

Reporting from Dovecot (sieve)

/check is for scanning. /report and /revoke are for user feedback — when someone moves a message into Junk (spam) or rescues it back out (ham). Sieve can't speak HTTP, so dovecot/drp-report bridges the message to gozer, triggered by imapsieve.

The eilandert/dovecot image already bakes this in. To wire it into any other Dovecot host (needs curl and dovecot-sieve):

cp dovecot/drp-report              /usr/lib/dovecot/sieve-pipe/drp-report   # chmod 0755
cp dovecot/sieve/report-spam.sieve /usr/lib/dovecot/sieve/
cp dovecot/sieve/report-ham.sieve  /usr/lib/dovecot/sieve/
sievec /usr/lib/dovecot/sieve/report-spam.sieve
sievec /usr/lib/dovecot/sieve/report-ham.sieve
cp dovecot/90-drp-sieve.conf       /etc/dovecot/conf.d/

# sieve_extprograms scrubs the environment, so pass the URL + token via a file:
printf 'DRP_URL=http://rspamd-drp:8077\nDRP_TOKEN=the-shared-secret\n' \
  > /etc/dovecot/drp.env

doveadm reload

What it does (90-drp-sieve.conf):

User action (IMAP)Sieve scriptGozer call
move/copy into Junkreport-spam.sievePOST /report (spam)
move out of Junkreport-ham.sievePOST /revoke (ham)

drp-report always exits 0, so a reporting hiccup never bounces mail or blocks the IMAP move.

Build

docker build -f docker/Dockerfile-deb -t eilandert/rspamd-dcc-razor-pyzor:latest docker/

In the dockerized monorepo this repo is a submodule at src/rspamd-dcc-razor-pyzor; build it with docker buildx bake debian-rspamd-drp.

Packages: none. The runtime is distroless/static plus the single static gozer binary — no Debian, no apt, no perl/python, no dcc package (dccproc/cdcc), no shell. All three clients are linked in.

The Go rewrite: gazor, gyzor, gdcc, gozer

Earlier versions of this backend were a thin Python HTTP shim that, for every message, forked the perl razor-check / razor-report, the python pyzor and the dccproc CLIs. That meant an interpreter (and a set-UID dccproc) start per check and a perl + python + dcc toolchain baked into the image. All three clients were rewritten from scratch in Go and are now linked into the backend in-process:

Was (per-message fork)Now (in-process Go)What it is
perl razor-agents (Razor2)gazorGo razor client
python pyzorgyzorGo pyzor client
set-UID dccproc (dcc package)gdccGo DCC client
python spamcheck_shim.pygozerthis backend — the binary in the image

gazor, gyzor and gdcc speak their wire protocols byte-for-byte compatibly with the reference perl/python/C clients (each is gated by parity tests against real razor, pyzor and dccproc in its own CI), so the servers see identical fingerprints and the switch is invisible on the wire. With no per-message fork left, the image dropped from ~268 MB (perl/python/dcc + s6 on Debian) to a ~6 MB distroless static binary.

Upgrading from an older build: the backend's environment variables were renamed SHIM_* to GOZER_* (for example SHIM_TOKEN becomes GOZER_TOKEN). The HTTP contract, the X-DRP-* headers, and drp-report's DRP_URL / DRP_TOKEN are unchanged.

See also

License

MIT — see LICENSE.

Tag summary

Content type

Image

Digest

sha256:e9573c9c7

Size

4.4 MB

Last updated

about 21 hours ago

docker pull eilandert/rspamd-dcc-razor-pyzor