Skip to content

EN

Local AI · Recipe · 8× RTX 3090

GLM-4.7 218B AutoRound INT4 with vLLM on 8× RTX 3090

GLM-4.7 218B AutoRound INT4 with vLLM on 8× RTX 3090: 38.5 tok/s according to huggingface.co (dataset as of Nov 3, 2026).

38.5tok/sPeak

1 request, mixed prompt set

Source: huggingface.co

Intelligence (original model) · Artificial Analysisno independent value

Instructions in the README Repo

Engine: vLLM
Quantization: W4A16 AutoRound (group 128) + FP8 KV
Model family: GLM-4.7
Context: 165,000
Parameters: 218B total / 32B active (REAP)
Creator: 0xSero
GitHub stars: 25
Repo updated: May 30, 2026

Measurements

Every sourced value of this recipe, each with its condition and source. Bars relative to the largest value in the group.

Decode · 1 request

Peak (decode)38.5 tok/s„Generation 38.5“ ohne Einheit/Concurrency/Promptlänge in der Card Source

Prefill by context

Prompt not stated375 tok/s„Prefill 375 tps“, Bedingungen nicht angegeben Source

Time to first token

Shorter is better.

Prompt not stated3,820 msTime to First Token 3.82 s, Bedingungen nicht angegeben Source

What you need

Hardware: 8 × NVIDIA RTX 3090
Weights: Main weights0xSero/GLM-4.7-218B-W4A16 Apache-2.0
Engine: vLLM
Context: 165,000 tokens

Notes

What matters before you rebuild it.

REAP-pruned

Sources

Related recipes

8× RTX 3090Qwen3.5-397B-A17B (REAP 264B)

Qwen3.5-397B-A17B FP8 with vLLM on 8× RTX 3090

62tok/sMixed

1 request, mixed prompt set, no speculative decoding Source

Intelligenceno independent value

W4A16 + FP8 KVvLLM 0.19.0Weights Repo Updated May 30, 2026

8× RTX 3090

Qwen3.5 35B-A3B with vLLM on 8× RTX 3090

13.9tok/sPeak

1 request, best value reported by source Source

Intelligence19with thinking no thinking 15

Experimental

vLLMWeights Repo Updated Sep 10, 2026

← Back to overview

Local AI in your company?

In a workshop we work out which models and which hardware fit your tasks, and build the first agent on your infrastructure.

See the AI workshop Back to overview