Context Studios
Back to Blog
Lokale KI-ModelleLLM Self-HostingMia AI Lab
By Michael Kerkhoff, Founder & CEO

Which Local AI Model Fits Your Hardware? The Mia-Lab Recipe Guide (8 GB to 512 GB VRAM)

From 8 GB to 512 GB VRAM: which local AI models run on your hardware — with the production recipes and benchmarks from Mia's AI Lab.

Which Local AI Model Fits Your Hardware? The Mia-Lab Recipe Guide (8 GB to 512 GB VRAM)

Which Local AI Model Fits Your Hardware? The Mia-Lab Recipe Guide (8 GB to 512 GB VRAM)

The first question anyone asks when self-hosting LLMs is: Which model can I actually run on my hardware — and how close does it get me to frontier performance? Answers are hard to find objectively, because performance only makes sense in the context of your own machine. This guide takes exactly that angle and draws exclusively on the recipes and repositories of Mia's AI Lab (MiaAI-Lab) — a source that doesn't just review models but ships and deploys them for concrete hardware configurations (from a single GPU up to 2× DGX Spark, including NVFP4, EXL3 and vLLM stacks).

Important up front: The model recommendations come from a battle-tested setup; the Mia-Lab recipes linked here are the production recipes these models actually run on that hardware. Where there isn't (yet) a dedicated Mia recipe, I say so honestly and point you to the nearest reference — no recipe is invented for anyone.

8 GB VRAM — small, but surprisingly capable

On 8 GB every quantization step counts. The rule of thumb: slim models with plenty of context and an agentic focus instead of raw parameter count.

Spark-X2.5-4B

A model that benchmarks near the 9B class yet is tuned for coding and agentic performance — while offering a 1M-token context window. That combination (lots of context, very little VRAM) is exactly what makes the difference on small boxes.

Mia-Lab angle: No dedicated deployment recipe here (yet). The measurable reference is Mia's tool-eval-bench — a tool-calling benchmark for serving stacks that lets you compare agentic suitability and quality across small models before you commit.

Bonsai-27B (Prism ML, Ternary)

The next step up: this is the closest you get to a frontier model on 8 GB. The technique behind it: a Qwen3.6-27B model is quantized to Q2, which massively shrinks the weights — and in return you get a 262K-token context window plus vision.

Mia-Lab angle: Mia has measured this model on its own benchmark: Ternary-Bonsai-27B-tool-eval-bench-results — tool-eval-bench results for the Prism-ML Ternary-Bonsai-27B (Q2_0) with 8 runs and a score. That lets the quality ranking be demonstrated rather than merely asserted.

16 GB VRAM

Gemma-4-12B

If you need vision capabilities, this is the top choice here. Users report the model has enormous world knowledge, is excellent at finding things in images and videos, and holds coherence across long contexts. Fair caveat: it is not the best agent — but it holds up remarkably well at this level.

Mia-Lab angle: Mia runs Gemma models in NVFP4 recipes. The nearest reference is Gemma-4-31B-IT-NVFP4-Recipe (vLLM with MTP speculative decoding, tool-calling) — the pattern by which the Gemma family runs in the Mia environment.

24 GB VRAM — top pick

Qwen3.8-27B (Mia-EXL3-3.5bpw variant)

My clear favorite in this category. If you want precisely the model with the relevant quantization, reach for the EXL3-3.5bpw variant from Mia's AI Lab. The use case at the core: Qwen3.8-27B was used for coding, game dev, media editing, vLLM/SGLang configuration management, and fleet management — and the performance may surprise you. Worth knowing: it reasons less and has MTP, which lets you run it at sensible speeds. Realistically it lands at Sonnet/Luna level.

Mia-Lab angle: The matching deployment recipe is Qwen3.8-27B-DFlash2-EXL3-5.0bpw — a kit that serves exactly the EXL3-3.5bpw target (plus a DFlash2-EXL3-5.0bpw speculative draft). Alternatively, if you prefer SGLang: Qwen3.8-27B-SGLang-DGX-Spark.

32–64 GB VRAM — top pick

Nex-N2.5-Mini

Nex post-trains strong models (such as Qwen3.6-35B) on agentic, coding and media tasks — with the goal of shortening reasoning lengths while still squeezing out a few IQ points. This one is especially interesting for Macs and slower GPUs, because it is a MoE and only a small part of the model is active for each generated token. Result is again Sonnet/Luna level.

Mia-Lab angle: For the MoE/low-activation class, Mia offers the related recipe Qwen3.6-35B-A3B-NVFP4-vLLM as well as the benchmark reference Best-Local-Model_Agentic-Workflows_2026 (head-to-head comparison of local LLMs for agentic workflows with Hermes Agent + tool-eval-bench).

96–128 GB VRAM — this is where the frontier begins

Qwen3.8-Flash-Next

58.7% on Deepswe, vision support, extremely fast, enormous world knowledge — and compared to Terra/Opus-4.6 you'd barely notice the difference. The catch: 55B parameters, which can be offloaded to cheaper memory or NVMe media (with minimal speed loss). To make that work you need hardware at this scale — e.g. AMD Strix Halo, a DGX Spark, 4× 3090, an RTX Pro 6000, or a Mac M5 Ultra/Max.

Mia-Lab angle: There are exactly two Mia recipes for this: Qwen3.8-Flash-Next-Dual-DGX-Sparks (NVFP4 MoE across 2× DGX Spark, TP2, ConnectX-7/RoCEv2) and, for single systems, Qwen3.8-Flash-Next-Single-DGX-Spark.

GLM-5.3-Flash-EXL3-2BPW

A recipe built exactly for 1× DGX Spark or any 128 GB system. It is strong at reverse engineering and should preserve about 86% of BF16 intelligence. Verdict: still experimental — if you find bugs, report them.

Mia-Lab angle: GLM-5.3-Flash-EXL3-2x-DGX-Sparks is the reference deployment for the GLM-5.3-Flash-EXL3 family; if you'd rather pursue the NVFP4 variant, there's GLM-5.3-Flash-NVFP4-Dual-DGX-Spark.

196–256 GB — frontier

GLM-5.3-Flash-EXL3-4BPW

91.7% top-token match at 0.065 KLD — currently the best model in this size class, period. It takes a little tuning to run cleanly, but subjectively it sits at Opus/Sol level: "I threw everything at it and it never hit a limit."

Mia-Lab angle: Again the GLM-5.3-Flash-EXL3-2x-DGX-Sparks recipe as the foundation.

DeepSeek-V4-Flash-Vision

DeepSeek is better for conversation: more personal, more world knowledge, an excellent agent. There isn't much more to say — that suffices as a quality statement.

Mia-Lab angle: This fits on 2× DGX Spark with 1M context: DeepSeek-v4-Flash-DSpark-2x-DGX-Spark. If you only have one system, use DeepSeek-v4-Flash-One-DGX-Spark.

GLM-5.3-EXL3-3bpw-REAP

The smartest model for 2× DGX Spark / Mac M5 Ultra 256 GB in coding, agentic work and reverse engineering. It will be slow — but overnight you can use it to RE/hack, build complex components and much more. The clear caveat: outside coding/agentics/English you'll notice clear losses — that's not what it's for.

Mia-Lab angle: The EXL3-GLM reference is again the GLM-5.3-Flash-EXL3-2x-DGX-Sparks family.

Nex-N2.5-Pro

An impressive model, especially for media editing. Give it a try.

Mia-Lab angle: No dedicated recipe; the agentic benchmark collection serves as the quality yardstick (see Best-Local-Model_Agentic-Workflows_2026 above).

384–512 GB — top tier

GLM-5.3

For this class I recommend EXL3-3bpw to EXL3-4bpw. The capability on development and agentic tasks is impressive and trends toward Fable / GPT-5.6-Sol.

Mia-Lab angle: For the large scale, Mia offers glm-5.3-flash-4x-dgx-spark-switchless (GLM-5.3-Flash NVFP4 with TP4 across 4× DGX Spark, switchless RoCE ring + DFlash2). Alternative: the GLM-5.3-Flash-NVFP4-Dual-DGX-Spark mentioned above.

Why only Mia-Lab recipes?

Because ordinary model cards state parameters and quantization but almost never whether and how the model runs on your hardware. Mia's AI Lab publishes startable production recipes (NVFP4/EXL3 stacks, vLLM/SGLang setup, dual/single-DGX configurations) for many of these models, plus measuring benchmarks (tool-eval-bench) — that's the honest basis for a hardware recommendation, not rumor about a parameter count.

Applying a recipe is deliberately kept simple — essentially like this:

# Example: clone and start a Mia-Lab recipe (here GLM-5.3-Flash on 2x DGX Spark)
git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks
cp .env.example .env            # set MASTER_ADDR, WORKER_HOST, paths
./start.sh                      # bring the server up, query the API endpoint

Sources

FAQ

Why should I use a Mia-Lab recipe for self-hosting at all? Because a model-card entry only tells you what a model is — not whether it runs on your hardware. A recipe from Mia's AI Lab is a tested, startable configuration (quantization level, serving engine, batching/context setup, dual- and single-node variants). You save the hours of fiddling with tensor-parallel, KV-cache or memory options and get a reproducible start. Where there is no recipe, I say so explicitly in the article instead of inventing one for you.

What does "EXL3" or "3.5bpw" mean in the Qwen and GLM recommendations? EXL3 is a quantization engine (ExLlama) that combines high-quality weight compression with little quality loss; "bpw" stands for "bits per weight" — the lower it is, the smaller the model, the faster it runs on limited hardware, but the more quality it can lose. The 3.5–4bpw tiers are, for many applications, the sweet spot between size and intelligence. Mia's recipes deliver exactly these configurations as startable stacks, including speculative-decoding drafts that significantly boost speed on small hardware.

Can I run these models without NVIDIA hardware (Mac/AMD only)? Yes, with limits. For MoE models like Nex-N2.5-Mini or Qwen3.6-35B-A3B, the low activation is a big advantage on Macs and weaker GPUs, because only a small part of the model is active per token. Many Mia recipes are explicitly built for the GB10/DGX-Spark and Mac-m5 class (ARM/aarch64). With classic dense 27B+ models, the available unified memory or VRAM is what decides feasibility — on a Mac M5 Ultra (or comparable) many of these stacks can also be driven.

How do I find out which model is best for my specific use case? First take the VRAM/memory tier from this guide, then check suitability via a benchmark instead of marketing claims. Mia's tool-eval-bench and the head-to-head comparisons in Best-Local-Model_Agentic-Workflows_2026 help you compare agentic and tool-calling quality objectively before you set up a stack. For agentic-, coding- and reverse-engineering-heavy work, reach for the recommendations above; for world knowledge and conversation, DeepSeek or Gemma tend to fit better.

Why is "shortening reasoning" good for these models? Many frontier-capable models produce long chains of thought before answering. When a post-trainer (e.g. at Nex) shortens reasoning length, latency per request drops significantly — especially relevant on limited hardware (Mac, older GPUs). That is one reason models like Qwen3.8-27B or Nex-N2.5 run at sensible speeds even though they otherwise belong to higher classes. A "less reasoning" model is therefore not weaker — it is optimized to be fast.

What do the quality labels "Sonnet/Luna level" or "Opus/Sol level" mean? These are informal equivalence notes to commercial models that give a rough sense of expected capability. "Sonnet/Luna level" means a local model comes close to those commercial models on many tasks; "Opus/Sol level" stands for the top tier. Note: these classifications are based on practical experience and benchmarks, not a standardized metric — they are a useful heuristic for orientation, not a laboratory result.

Share article

Share: