Skip to content

EN

Local AI · Recipe · 1× Mac

Bonsai-2-27B Q8_0 with llama.cpp on 1× Mac

Bonsai-2-27B Q8_0 with llama.cpp on 1× Mac: 10.6 tok/s according to github.com (dataset as of Sep 29, 2026).

10.6tok/sEveryday

1 request, prose prompt, no speculative decoding

Source: github.com

Intelligence (original model) · Artificial Analysisno independent value

Instructions in the README Repo

Engine: llama.cpp (prism fork)
Quantization: Ternary PQ2_0 (empfohlen) / PTQ1_0 (5,9 GB); mmproj-Q8_0 für Vision
Model family: Bonsai-2-27B (Qwen3.8-27B-Basis)
Context: 131,072
Parameters: 27B ternär 1.72 bits/weight (README); PQ2_0 7,2 GB
Creator: Weschera
GitHub stars: 0
Repo updated: Sep 18, 2026

Measurements

Every sourced value of this recipe, each with its condition and source. Bars relative to the largest value in the group.

Decode · 1 request

Everyday (decode)10.6 tok/sMac mini M4 24 GB, PQ2_0 — 3-Rep-Ladder WÄHREND parallel laufendem Spark-Bench (GPU-Contention); idle-Qualifizierung 11.0 → approx; TTFTs dieser Ladder sind Queueing-Artifacts Source
Peak (decode)34.5 tok/sPQ2_0 Studio; 3 Reps Mittel, temp 0, thinking off, 320 tokens, ~2,1–2,4K prompts (bench-chat.py); flat prose=code → kein Spec-Decoding, rein bandgebunden Source

Prefill by context

Prompt not stated242 tok/sPQ2_0 Studio (PTQ1_0: 205); Ternary-Unpacking macht Prefill langsamer als 4-bit-affin (242 vs 274) Source

What you need

Hardware: 1 × Mac / Apple Silicon
Weights: Main weightsprism-ml/Ternary-Bonsai-2-27B-gguf Apache-2.0
Engine: llama.cpp (prism fork)
Context: 131,072 tokens

Notes

What matters before you rebuild it.

Custom kernel required

Sources

github.com/Weschera/Bonsai-2-27B-Mac/blob/main/README.md

Related recipes

1× Mac

Qwen3.8-27B with MLX on 1× Mac

53.3tok/sEveryday

1 request, prose prompt, with MTP (nativ) (per recipe) Source

Intelligenceno independent value

License unclear

oQ4e (4-bit affine g64, 166 sens. Tensoren 5-bit; Stock-Qwen3.8-27B-Konversion)oMLX 0.6.3rc2Weights Repo Updated Aug 21, 2026

1× Mac

DeepSeek-V4-Flash 284B IQ2_XXS with DwarfStar on 1× Mac (vision)

28.5tok/sEveryday

1 request, prose prompt, context 32k, with DSpark (per recipe) Source

Intelligenceno independent value

≤ 3 bit: quantization may cost quality

IQ2_XXS imatrix q2 (antirez), w2Q2K/AProjQ8/SExpQ8/OutQ8; GGUF 86,720,111,776 bytesds4 (DwarfStar)Weights Repo Updated Sep 2, 2026

1× Mac

DeepSeek-V4.1-Flash 763B with MLX on 1× Mac

25.1tok/sEveryday

1 request, prose prompt, short prompt, with MTP (DSpark built-in) (per recipe) Source

Intelligence40with thinking no thinking 25

≤ 3 bit: quantization may cost quality
No license on HF
License unclear

oQ3e (affine 3-bit) mit 27/40 MoE-Layern 2-bit gs64oMLX 0.7.0.dev2 @395ec2fd, MLX 0.32.2Weights Repo Updated Sep 16, 2026

1× Mac

GLM-5.3 UD-IQ2_XXS with DwarfStar on 1× Mac

6.14tok/sMixed

1 request, mixed prompt set, context 1k, no speculative decoding Source

Intelligence45with thinkingOriginal model's value. Heavily compressed (~2.1 bit), intelligence is likely much lower.

≤ 3 bit: quantization may cost quality
Streams from SSD
License unclear

UD-IQ2_XXS routed, blk78 Q2K, 196.58 GiBDwarfStar (ds4) main@0aaea5a (2026-09-20)Weights Repo Updated Sep 25, 2026

1× Mac

Granite-4.2 3B MLX 4-bit with MLX on 1× Mac

123tok/sPeak

1 request, best value reported by source Source

Intelligence9with thinking

mlx-4bitLM Studio (MLX-Backend) 0.4.21Weights Repo Updated Sep 23, 2026

1× Mac

Gemma-4 MLX 4-bit with MLX on 1× Mac

102tok/sPeak

1 request, best value reported by source Source

Intelligence8with thinking no thinking 7

mlx-4bitLM Studio (MLX-Backend) 0.4.21Weights Repo Updated Sep 23, 2026

← Back to overview

Local AI in your company?

In a workshop we work out which models and which hardware fit your tasks, and build the first agent on your infrastructure.

See the AI workshop Back to overview