EN
Book a call

Local AI · Recipe · 1× RTX 5090

Qwen3.8 27B EXL3 4 bpw with ExLlamaV3 on 1× RTX 5090 (one-click kit)

Qwen3.8 27B EXL3 4 bpw with ExLlamaV3 on 1× RTX 5090 (one-click kit): 122 tok/s according to github.com (dataset as of Sep 29, 2026).

by MiaAI-Lab

122tok/sMixed

1 request, llama-benchy 2048-token prompt, 128 tokens out, depth 0, with MTP

Source: github.com
Intelligence (original model) · Artificial Analysis34with thinkingno thinking 20
Instructions in the README Repo

Engine

Engine
ExLlamaV3 v1.6.0 (tools/serve_openai.py), MTP; opt-in DRAFT=dflash2
Quantization
EXL3 4.0 bpw (turboderp) + int4 KV
Model family
Qwen3.8
Context
262,144
Parameters
27B dense
Creator
MiaAI-Lab
GitHub stars
546
Repo updated
Oct 9, 2026

Measurements

Every sourced value of this recipe, each with its condition and source. Bars relative to the largest value in the group.

Decode · 1 request

  • Mixed (decode)122 tok/scontext 2048 tok prompt, depth 0 · 1.6.0 + MTP (aktueller Default), 3 Läufe; Tiefe 8k 115,9 / 32k 112,5 / 64k 122,6 / 128k 102,9 / 248k 79,3; VRAM 22.366 MiB Source 
  • Peak (decode)366 tok/sStructured, opt-in DRAFT=dflash2 (RigMark 1.3.0, Median 5 Läufe, thinking off); dflash2 Prosa 113,2 / Code 274,6 Source 

What you need

Hardware
1 × NVIDIA RTX 5090
Engine
ExLlamaV3 v1.6.0 (tools/serve_openai.py), MTP; opt-in DRAFT=dflash2
Context
262,144 tokens

Notes

What matters before you rebuild it.

  • vision

Sources

Related recipes

← Back to overview

Local AI in your company?

In a workshop we work out which models and which hardware fit your tasks, and build the first agent on your infrastructure.

See the AI workshopBack to overview