EN
Book a call

Local AI · Recipe · 2× DGX Spark

GLM-5.3-Flash FP4-Local (NVFP4/MXFP8) with vLLM on 2× DGX Spark

GLM-5.3-Flash FP4-Local (NVFP4/MXFP8) with vLLM on 2× DGX Spark: 34.4 tok/s according to github.com (dataset as of Nov 3, 2026).

by myllmbox

34.4tok/sEveryday

1 request, prose prompt (7,000-word story), with MTP (per recipe)

Source: github.com
Intelligence (original model) · Artificial Analysis42with thinking
Instructions in the README Repo

Engine

Engine
vLLM v0.32.0 (Image myllmbox/glm53-flash-cluster-vllm:v3: mbx-Loader, NVFP4-Experten auf Marlin, dynamische MTP-Draft-Tiefe 1–9, One-Shot-RoCE-All-Reduce via b12x), TP=2 über RDMA
Quantization
NVIDIA FP4-Local-Checkpoint wie ausgeliefert (NVFP4-Experten, MXFP8-Dense-Layer); KV-Pool 1.145.238 Token (12G/Box); Marlin MoE
Model family
GLM-5.3-Flash
Context
400,000
Parameters
320B MoE / 18B aktiv (lt. README)
Creator
myllmbox
GitHub stars
18
Repo updated
Oct 29, 2026

Measurements

Every sourced value of this recipe, each with its condition and source. Bars relative to the largest value in the group.

Decode · 1 request

  • Everyday (decode)34.4 tok/sLong prose (7.000-Wörter-Geschichte), c=1, Mittel aus 3 Läufen à 300 s; Image v3, ausgelieferte recipe.yaml Source 
  • Peak (decode)86.7 tok/sSpitzenwert eines Thinking-on-Requests (pasture), Ø desselben Requests 44,9; c=1: JSON-Schema 45,6, TypeScript 41,1, bestes 10-s-Fenster 60,4 Source 

Prefill by context

  • 128K1,836 tok/sKalt (ohne Prefix-Cache), 128k-Token-Prompt; 32k: 1.879 Source 

Time to first token

Shorter is better.

  • Prompt not stated1,130 mscontext 1K · Erster Token, 1k-Prompt; mit Prefix-Cache 0,26 s Source 

What you need

Hardware
2 × NVIDIA DGX Spark (GB10)
Engine
vLLM v0.32.0 (Image myllmbox/glm53-flash-cluster-vllm:v3: mbx-Loader, NVFP4-Experten auf Marlin, dynamische MTP-Draft-Tiefe 1–9, One-Shot-RoCE-All-Reduce via b12x), TP=2 über RDMA
Context
400,000 tokens

Notes

What matters before you rebuild it.

  • Custom kernel required
  • vision
  • tool-calling
  • License unclear

Sources

Related recipes

4× DGX Spark

GLM-5.3-Flash NVFP4/MXFP8 CSF with SparkRing on 4× DGX Spark

by fujitsupolycom

68tok/sEveryday

1 request, generation at 32K context (speed table; measured only as a path of four Sparks inside a ring of eight, not as a ring of four), thinking always on (measured before the profile default high, i.e. template default max), with MTP3 (per recipe) Source 

Intelligence42with thinking
  • Custom kernel required
  • License unclear

NVFP4/MXFP8 CSF QAD (Local Inference Lab)vLLM (local-inference-lab Fork) via SparkRing (SIRCL-Collectives, switchless RDMA ueber ConnectX-7), Image Release 2026.10.2Weights Repo Updated Nov 2, 2026

Details
2× DGX Spark

GLM-5.3-Flash EXL3 with TensorFold on 2× DGX Spark (1M context, 4 streams)

by MiaAI-Lab

60.4tok/sEveryday

Prose decode, 1 request; quality (FP8 KV, 4-bit dense): GSM8K 98.8% / HumanEval 95.7% / HumanEval+ & MBPP+ 86.5% Source 

Intelligence42with thinking
  • ≤ 3 bit: quantization may cost quality
  • Custom kernel required
  • Non-commercial
  • License unclear

EXL3/TR3 4bpw routed experts + 4-bit dense, BF16 elsewhere, FP8 KVTensorFold v0.6.0 + Recipe v1.10 (README: 96 Patches; gepinntes Image v0.6.0-e66c657a433d mit 107 Patches: DFlash2+Copy-Drafts, 4-bit dense, FP8-KV, RoCE-One-Shot-All-Gather, Shared KV-Pool, optionaler Spill-Tier, Vision, Tool-Calling, Anthropic /v1/messages; PARALLEL=8 opt-in)Weights Repo Updated Oct 25, 2026

Details

← Back to overview

Local AI in your company?

In a workshop we work out which models and which hardware fit your tasks, and build the first agent on your infrastructure.

See the AI workshopBack to overview
Cookies

With your consent we use Google Analytics (Google, USA) and our own audience measurement to understand how our website is used. Strictly necessary storage, such as your cookie choice, happens without consent. You can change your choice at any time via “Cookie settings” in the footer. Privacy Policy · Cookie Policy