EN
Book a call

Local AI · Recipe · 2× RTX 3090

Nanbeige4.3-33B W4A16 AutoRound with vLLM on 2× RTX 3090

Nanbeige4.3-33B W4A16 AutoRound with vLLM on 2× RTX 3090: 62.2 tok/s according to github.com (dataset as of Nov 3, 2026).

by tonyd2wild

62.2tok/sEveryday

1 request, prose prompt, no speculative decoding

Source: github.com
Intelligence (original model) · Artificial Analysisno independent value
Instructions in the README Repo

Engine

Engine
vLLM 0.27.1
Quantization
W4A16 AutoRound (GPTQ INT4, g128) + FP8 e5m2 KV
Model family
Nanbeige4.3-33B
Context
131,072
Parameters
33B dense
Creator
tonyd2wild
GitHub stars
3
Repo updated
Oct 26, 2026

Measurements

Every sourced value of this recipe, each with its condition and source. Bars relative to the largest value in the group.

Decode · 1 request

  • Everyday (decode)62.2 tok/sProse, C1, thinking off, temp 0, 400-Token-Antworten, ohne Draft Source 
  • Peak (decode)66.2 tok/sCode, C1, thinking off, ohne Draft Source 

Prefill by context

  • 17.0731,765 tok/sCold (Prefix-Cache aus), C1, TTFT 9.67 s Source 

Time to first token

Shorter is better.

What you need

Hardware
2 × NVIDIA RTX 3090
Engine
vLLM 0.27.1
Context
131,072 tokens

Notes

What matters before you rebuild it.

  • tool-calling

Sources

Related recipes

← Back to overview

Local AI in your company?

In a workshop we work out which models and which hardware fit your tasks, and build the first agent on your infrastructure.

See the AI workshopBack to overview
Cookies

With your consent we use Google Analytics (Google, USA) and our own audience measurement to understand how our website is used. Strictly necessary storage, such as your cookie choice, happens without consent. You can change your choice at any time via “Cookie settings” in the footer. Privacy Policy · Cookie Policy