Bonsai 1.7B: Apple Silicon edition

A drop-in optimized inference build of Bonsai 1.7B (Q2_0) for M-series Macs. All Metal kernels written and tuned by ata, our autonomous engineering agent.

	Upstream Q2_0	This build	Δ
Decode tg128	311.66 ± 2.28 t/s	442.42 ± 2.96 t/s	+42.0%
Prefill pp512	4250.32 ± 23.85 t/s	4622.63 ± 26.16 t/s	+8.8%

Same model file (Bonsai-1.7B-Q2_0.gguf), same M4 Max, same bench config (llama-bench -p 512 -n 128 -r 10 -fa 1 -ngl 99). Upstream measured against stock llama.cpp at the same Bonsai/Q2_0 commit; both rows are r=10.

Download · Mac arm64 · 358 MB tar.xz · model included

What's in the box

• chat.sh: interactive REPL
• complete.sh: non-interactive completion
• bench.sh: reproduce the numbers above
• server.sh: OpenAI-compatible HTTP API on :8080
• Bonsai-1.7B Q2_0 GGUF (442 MB)

Quick start

tar -xJf bonsai-1.7b-ternary-M4Max.tar.xz
cd bonsai-1.7b-ternary-M4Max
./chat.sh

What changed

Every Metal kernel in this build was authored and tuned by ata, our autonomous engineering agent without human intervention.

Substantively, the work is custom GPU kernels at the matvec / FFN / KV-cache layer, shape-specialized for the Bonsai 1.7B Q2_0 decode path. Same numerical output as the reference build (verified top-1 token match). Tested on M4 Max; should run on any M1+ with proportional gains.

Built on top of PrismML-Eng/Bonsai-demo (upstream model + Q2_0 quantization). For context, the Bonsai 8B whitepaper reports MLX-upstream Q2_0 decode at 235 t/s on Apple Silicon. This build measures 442 t/s on M4 Max for the 1.7B variant via custom Metal kernels (different framework, different model size; directionally indicative of headroom that existed in the stack).

Caveats

• Apple Silicon only (arm64). Intel Macs and CPU-only builds are not in this drop.
• Numbers above are from bench.sh on M4 Max. M1/M2/M3 will be lower (less memory bandwidth).
• Model is Q2_0 quantized, small accuracy delta vs F16, same quantization as the upstream.