---
type: "BlogPosting"
title: "Ternary Bonsai 2: 27B under 6 GB — the first browser model for the 16-GB everyday"
description: "Ternary Bonsai 2 compresses Qwen3.8-27B to 5.95 GB — 98.2% of the quality. What that means for 16 GB and 8 GB setups, with our own cluster measurement."
resource: "https://www.contextstudios.ai/blog/ternary-bonsai-2-27b-browser-16gb-en"
language: "en"
tags: ["Local AI", "Open Weights", "Quantization", "LLM"]
generated:
  by: "process:contextstudios-md/1"
  at: "2026-10-23T06:47:35.481Z"
status: "stable"
---

# Ternary Bonsai 2: 27B under 6 GB — the first browser model for the 16-GB everyday

Published: 2026-10-20
Tags: Local AI, Open Weights, Quantization, LLM

![Ternary Bonsai 2: 27B under 6 GB — the first browser model for the 16-GB everyday](https://wary-platypus-754.convex.cloud/api/storage/1e4c138c-4860-40d4-b2e4-c462f706f0db)

Ternary Bonsai 2 shrinks Qwen3.8-27B from roughly 54 GB to 5.95 GB — at 98.2% of the FP16 model's benchmark score. For the first time, a full 27B reasoning model fits into the RAM a typical 16 GB machine has left over, and per the Prism ML demo it even runs in the browser via WebGPU. We put the footprint numbers against our own cluster measurement and the 16 GB stack.

**TL;DR:** Ternary weights ({−1, 0, +1} plus an FP16 scale) compress Qwen3.8-27B to 5.95 GB instead of 53.8 GB — 9.2× smaller at 84.78 of 86.32 benchmark points. On a Mac Studio M4 Max the model decodes at 10.6 tok/s in everyday use (34.5 tok/s peak, memory-bandwidth bound); on an RTX 3090, Qwen3.8-27B in Q4 stays at 39.7 tok/s. The catch: it requires the Prism ML fork of llama.cpp, the WebGPU path is a demo, not a product — and the speed advantage of smaller weights is partially eaten by slow ternary unpacking during prefill.

## What ternary weights actually are

Instead of storing every model weight as a 16-bit float, the ternary g128 format allows only three values: −1, 0 and +1. Per group of 128 weights there is exactly one shared FP16 scale factor. A ternary value carries log₂3 ≈ 1.585 bits of information; with the amortized scale factor the format lands at 1.72 bits per weight in practice — an idealized 9.3× reduction versus 16 bits for FP16.

Three details separate marketing from practice:

- **End-to-end ternary:** embeddings, attention projections, MLP projections and the LM head are all in the same format. There are no hidden FP16 escape hatches — the 26.2M parameters above ternary (the recurrent state path of the linear-attention layers plus normalizations) are cleanly disclosed and included in the 1.72-bit math.
- **Hadamard rotation folded in:** before ternary assignment, every matrix is rotated blockwise by an orthogonal transform; the rotation is folded into the weights offline and costs neither bits nor extra read bandwidth.
- **Packed, not expanded:** the GGUF variants PTQ1_0 (1.75 bits/weight, 5.95 GB) and PQ2_0 (2.13 bits/weight, 7.21 GB) are consumed directly — weights are never expanded back to FP16 at runtime.

## The footprint in comparison

| Variant | Size | Benchmark avg (14 tests) | Intelligence density |
|---|---|---|---|
| Qwen3.8-27B FP16 | 54.66 GB | 86.32 | 0.053 |
| Qwen3.8-27B UD-Q4_K_XL | 17.56 GB | 85.18 | 0.157 |
| Qwen3.8-27B IQ2_XXS | 7.27 GB | 72.59 | 0.257 |
| **Bonsai 2 27B (PTQ1_0)** | **5.95 GB** | **84.78** | **0.457** |
| Ternary Bonsai 27B (predecessor) | 5.75 GB | 80.98 | 0.416 |

This table comes straight from the official model card [^1]. The interesting column is the right one: Prism ML divides capability by deployed size. Bonsai 2 reaches 0.457 there — almost twice the density of IQ2_XXS (0.257) at a similar size, and far more than Q4 (0.157). The IQ2_XXS comparison is the real story: conventional 2-bit quantization collapses to 72.59 points, the ternary representation holds 84.78 — at 82% of the IQ2 build's footprint.

[^1]: prism-ml/Ternary-Bonsai-2-27B-gguf, Hugging Face model card, retrieved 2026-10-20.

## Our measurement: 37.6 tok/s on the DGX Spark cluster

To anchor the Bonsai numbers, our own reference point from production — the same Qwen3.8-27B backbone in NVFP4 on vLLM:

| Metric | Value |
|---|---|
| Model | qwen3.8-flash-next-nvfp4 (vLLM) |
| Decode, 1 stream (median) | 37.6 tok/s |
| Time to first token (median) | 0.495 s |
| Aggregate at 8 parallel streams | 64.3 tok/s |
| Max. output tokens per request | 256 |

Source: our own run against http://127.0.0.1:8212 (vLLM, OpenAI-compatible), measured 2026-10-20T16:04:29+02:00. The endpoint also serves the production crons; these are operational figures, not lab numbers.

## What Bonsai 2 delivers on consumer hardware — and what slows it down

Three practical questions: does it fit, how fast, under which constraints?

**Fit.** 5.95 GB language model plus KV cache: at 32K context with q4_0 KV, a PTQ1_0 run lands at 7.1 GiB per the community VRAM math — 8 GB card territory, and comfortably within 16 GB reality. An RTX 2060 Super 8 GB reportedly ran 16.8 tok/s at 32K. For the 16 GB class (RTX 4060 Ti 16G, laptop SoCs) there is headroom for longer context or the vision mmproj (0.63 GB, Q8_0, loaded only for image input).

**Speed — two sides.** Decode is pure bandwidth economics: fewer bits per weight means fewer bytes moved per token. Prism ML reports 47 tok/s on an M5 Max (Metal); on a Mac Studio M4 Max (community recipe, PQ2_0) we see **10.6 tok/s everyday / 34.5 tok/s peak** — the peak under the same conditions as our own (thinking off, temp 0). Notably, the peak being bandwidth-bound on Apple Silicon is a direct test of the "fewer bits = more tok per GB/s" thesis. The flip side is in the recipe: ternary unpacking makes prefill slower than 4-bit-affine checkpoints (242 vs 274 tok/s prefill on the same hardware) — part of the read saving is paid back during unpacking.

**Constraints.** Three, and they matter:

1. **No stock llama.cpp.** Tensor types 142/143 (PTQ1_0/PQ2_0) are not merged in mainline — Ollama and LM Studio cannot load the files today. You need the [Prism ML fork](https://github.com/PrismML-Eng/llama.cpp) with the custom ternary kernels for hybrid attention.
2. **CUDA, Metal, CPU — no ROCm-first path.** AMD setups: wait or build the fork yourself.
3. **Reasoning defaults:** xhigh effort is on; dropping to medium shortens answers. low is not supported.

## The 16 GB question: browser, WebGPU, and the reality check

The [WebGPU demo path](https://x.com/xenovacom/status/2110825606254391577) (Xenova, Oct 15 2026) loads the small Bonsai 2 variants entirely in the browser — 2B at up to 500 tok/s on M4 Max, no server involved. For the 27B that means: download like a movie, run in a tab. That is the European angle in this story: 16 GB RAM is the reality standard for solo setups, and a 27B reasoner that runs in a tab without a dedicated GPU and without installation changes the default answer to "do I need a GPU for this?".

The reality check belongs in the same sentence: the WebGPU demo is a demonstrator, not a product. The tested backends are llama.cpp (CUDA/Metal/CPU) and MLX. For production that means: pin the fork, budget the KV cache, load the mmproj only when images arrive. The 2B/4B/9B tail of the family is browser-ready today; the 27B is the frontier.

## Verdict: when Bonsai 2, when Q4?

| Situation | Choice |
|---|---|
| 8 GB VRAM, 27B must fit | Bonsai 2 PTQ1_0 — the only 27B-class that fits plus KV in 8 GiB |
| 16 GB, everyday + vision | Bonsai 2 + mmproj Q8_0; Q4-class also fits, with less context headroom |
| Max tok/s on a 3090, quality matters | Qwen3.8-27B Q4_K_M/EXL3 — 39.7–42.8 tok/s, established llama.cpp/vLLM paths, AA intelligence 33.7 |
| Apple Silicon, no fork maintenance | Qwen3.8-27B oQ4e/MLX — 53.3 tok/s everyday (M4 Max, with MTP), no custom kernel |
| Browser product | Bonsai 2 2B/4B/9B today; 27B as a demo |

The AA intelligence of 33.7 applies to Qwen3.8-27B (Artificial Analysis Index v4.3.2, as of 2026-09-26); as a derived model, Bonsai 2 has no AA value of its own — the 84.78/86.32 benchmark average from the model card is the best proxy.

## Conclusion

Ternary Bonsai 2 is the most convincing evidence yet that sub-4-bit does not mean quality loss when the representation is applied consistently. 84.78 of 86.32 points at a ninth of the footprint is a class of its own. The price is ecosystem friction: a dedicated llama.cpp fork, no Ollama/LM Studio integration, ROCm open. For the 16 GB and 8 GB classes that is not a drawback but the entry ticket — there, the alternative so far was "27B doesn't fit".

## Sources

- [Prism ML: Introducing Bonsai 2 27B](https://prismml.com/news/bonsai-2-27b) — official announcement with footprint and retention numbers
- [prism-ml/Ternary-Bonsai-2-27B-gguf model card](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) — weight representation, size/benchmark table, packing formats
- [Bonsai docs: introduction](https://docs.prismml.com/get-started/introduction) — model lineup, 1-bit/ternary sizes
- [PrismML-Eng/llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) — required custom kernels (CUDA/Metal)
- [PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) — tested setup per backend, whitepaper link
- [ggml-org/llama.cpp discussion #29263](https://github.com/ggml-org/llama.cpp/discussions/29263) — community VRAM math (8 GB run, 32K KV numbers)
- [Weschera: Bonsai-2-27B-Mac](https://github.com/Weschera/Bonsai-2-27B-Mac) — Mac Studio M4 Max measurements (basis of the hub recipe)
- [Xenova on X, Oct 15 2026](https://x.com/xenovacom/status/2110825606254391577) — WebGPU demo, browser tok/s of the small variants
- Artificial Analysis Intelligence Index v4.3.2 (2026-09-26) — AA intelligence 33.7 for Qwen3.8-27B
- [16 GB are enough: the Qwen3.8-27B GGUF measurement table](https://www.contextstudios.ai/de/blog/16-gb-sind-genug-die-qwen3-8-27b-gguf-mess-tabelle-11-8-gb-40-tok-s) — our 16 GB baseline
- [Used RTX 3090 for local LLMs](https://www.contextstudios.ai/de/blog/gebrauchte-rtx-3090-lokale-llms-tok-s-kosten-gegen-dgx-spark) — 3090 Q4 reference
- [Local AI hub: Bonsai-2-27B on Mac](https://www.contextstudios.ai/de/lokale-ki/weschera--bonsai-2-27b-pq2_0-prism-llamacpp-mac-studio-m4-max) and [Qwen3.8-27B Mirai-S 16 GB on 3090](https://www.contextstudios.ai/de/lokale-ki/alesha-pro--qwen38-27b-mirai-s-mtp3-16gb-1x3090) — our recipe rows with sourced numbers

## FAQ

**Does Ternary Bonsai 2 27B run with Ollama or LM Studio?**

Not with the stock versions. The GGUF files use tensor types 142 and 143 (PTQ1_0/PQ2_0), which are not merged in mainline llama.cpp, so Ollama and LM Studio reject the files. You need the Prism ML fork of llama.cpp with the custom ternary kernels. The model card and the demo repo are the authoritative sources for the setup path.

**How much RAM/VRAM does Bonsai 2 27B actually need?**

The language model itself occupies 5.95 GB (PTQ1_0) or 7.21 GB (PQ2_0). On top comes the KV cache: at 32K context with q4_0 KV, a complete run lands at roughly 7.1 GiB per the community math, or 7.6 GiB with q8_0 KV. An 8 GB card therefore suffices for 32K context; on 16 GB there is headroom for longer contexts or the 0.63 GB vision mmproj.

**How fast is the model in everyday use?**

It depends heavily on hardware and packing. On a Mac Studio M4 Max (PQ2_0) we measure 10.6 tok/s in the everyday case and 34.5 tok/s peak with thinking disabled; Prism ML reports 47 tok/s on an M5 Max. On NVIDIA cards, PQ2_0 is faster on H100/A100/Blackwell while PTQ1_0 wins on Ada — the choice of packing is a performance decision in itself. Prefill is slower than 4-bit checkpoints because ternary unpacking costs time.

**Does it really keep 98% of the quality?**

Per the model card, Bonsai 2 27B holds 84.78 of 86.32 points in the 14-benchmark average — 98.2%. The 1.54-point gap is unevenly distributed: math is within half a point, coding is level, agentic tool calling sits at 74.92. For comparison, conventional IQ2_XXS only reaches 72.59 points. The 98% claim applies to the benchmark average, not to every single task.

**Can I run the model in the browser?**

The small variants (2B/4B/9B) run in the browser via WebGPU today — the 2B reaches up to 500 tok/s on M4 Max per the Xenova demo. The 27B exists as a WebGPU demo, but the tested production paths are llama.cpp (CUDA/Metal/CPU) and MLX. For a browser deployment of the 27B, budget the ~6 GB download, tab memory limits and WebGPU support of the target GPUs.

**Is Bonsai 2 worth it over Q4 quantization?**

At 16 GB and above: yes, if you need 27B-class capability on small hardware or care about context headroom — quality measurably exceeds IQ2 builds and is close to Q4. On a full 24 GB card with an established llama.cpp/vLLM stack, Q4-class is the more convenient choice: same AI-index league (33.7), 39.7–42.8 tok/s, no fork maintenance. The decision is hardware budget versus operational simplicity.

## Related

- [LLM Development](https://www.contextstudios.ai/llm-development.md)
- [AI Development](https://www.contextstudios.ai/ai-development.md)
- [LLM Integration](https://www.contextstudios.ai/llm-integration.md)
- [AI Consulting](https://www.contextstudios.ai/ai-consulting.md)