FR
Réserver un appel

Local AI · Recipe · 4× RTX PRO 6000

Qwen3.5 397B-A17B NVFP4 with vLLM on 4× RTX PRO 6000

Qwen3.5 397B-A17B NVFP4 with vLLM on 4× RTX PRO 6000: 139 tok/s according to github.com (dataset as of Nov 3, 2026).

by brandonmmusic-max

139tok/sEveryday

1 request, prose prompt, with MTP (per recipe)

Source: github.com
Intelligence (original model) · Artificial Analysis18with thinkingno thinking 21
Instructions in the README Repo

Engine

Engine
vLLM 0.17.1rc1
Quantization
NVFP4
Model family
Qwen3.5
Context
262,144
Parameters
397B MoE / 17B active
Creator
brandonmmusic-max
GitHub stars
6
Repo updated
Sep 6, 2026

Measurements

Every sourced value of this recipe, each with its condition and source. Bars relative to the largest value in the group.

Decode · 1 request

  • Everyday (decode)139 tok/scontext 0 · Engine-Throughput (/metrics), 30 s/Zelle, "real generation prompts" Source 
  • Peak (decode)139 tok/scontext 0 · Engine-Throughput (/metrics), 30 s/Zelle, "real generation prompts" Source 

Prefill by context

Time to first token

Shorter is better.

What you need

Hardware
4 × NVIDIA RTX PRO 6000 Blackwell
Weights
Engine
vLLM 0.17.1rc1
Context
262,144 tokens

Notes

What matters before you rebuild it.

  • Custom kernel required

Sources

Related recipes

← Back to overview

Local AI in your company?

In a workshop we work out which models and which hardware fit your tasks, and build the first agent on your infrastructure.

See the AI workshopBack to overview
Cookies

Avec votre consentement, nous utilisons Google Analytics (Google, États-Unis) et notre propre mesure d’audience pour comprendre l’utilisation de notre site. Les stockages strictement nécessaires, comme votre choix de cookies, se font sans consentement. Vous pouvez modifier votre choix à tout moment via « Paramètres des cookies » en pied de page. Politique de confidentialité · Politique relative aux cookies