August 2026 · 16 tasks · 6 categories

Local LLMs step into the arena —
and hold their own.

MTPLX — a free, open-source Mac app running Qwen 3.8 27B on a single laptop — vs four cloud frontier models, measured on the identical bench3 harness with blind hidden-test grading.

75%
Task pass rate · 12/16 · trial 1
$0.00
Marginal cost · 100% local
4.24/5
Judged output quality · blind rubric
1.7×
MTP decode speedup · up to 2.6×

Task pass rate

DeepSeek V4 Flash
76% · 38/50
DeepSeek V4 Pro
80% · 40/50
GPT-5.6 Sol
70% · 35/50
Grok 4.6
78% · 39/50
MTPLX · Qwen 3.8 27B
75% · 12/16
GLM 5.2 Vision · LunaRoute
81% · 13/16
Cloud arms: 3–5 trials (n=50) · MTPLX & GLM: trial 1 (n=16)

Cost · speed · quality

API spend per passing task

MTPLX
$0.000
DS Flash
$0.0027
DS Pro
$0.0083
Grok 4.6
$0.022
GPT-5.6
$0.103
GLM 5.2 Vision
TBD
MTPLX runs on your own hardware — $0 forever. GLM pricing not published at run time.

Effective tokens/sec (single-shot)

DS Flash
112
DS Pro
78
GPT-5.6
58
MTPLX
33
GLM 5.2 Vision
35
Grok
10
Cloud = vendor GPUs · MTPLX = laptop decode, MTP on

MTP speculative decoding — why the laptop is fast

22–30
tok/s · MTP on (depth 3)
drafts with its own MTP heads, verifies in one batched pass
→
13–18
tok/s · plain decoding
identical output distribution — exact at any temperature
Thermals beat samplers: the same 9,000-token prompt decodes at 36–46 tok/s on a cool Mac (46.4 in the MTPLX app dashboard), but throttles to ~26 after hours of sustained load.

Community follow-up: the Qwen MLX challenge (2.94×)

We reproduced it on this M5 Max

True serial (MTP off)
11.2 tok/s
MTP depth 8 · record submission
23.3 tok/s
2.07× · exact output parity (all tokens bit-identical)

The ladder

Our box · record submission
2.07×
Stock main harness
2.53×
Leaderboard record · gated M5 Max
2.94×
gated = ≤40°C, idle, 8 hidden prompts × 512 tok · yukon.org/mlxfast
Verdict: confirmed. Same chip as our bench (M5 Max, 64 GB): native MTP decode 11.2 → 23.3 tok/s with exact parity; the 2.94× record reflects the leaderboard's thermally gated conditions. MTPLX on the same hardware: 1.6–2.6×.

Hardware & setup

Machine
MacBook Pro · M5 Max
Memory
64 GB unified
OS
macOS 26.5.2
Runtime
MTPLX 2.7.1 · MLX · turbo · depth 3
Model
Qwen 3.8 27B Optimized Speed
Weights
20.4 GB · 4-bit dynamic quant
Peak memory
~22.4 GB of 64 GB
Context
262,144 tokens
License
Apache-2.0 · open source

Where it lands

Reasoning
4/4 tasks ✓
Solved the two hardest puzzles (r1_tiling, r4_catalan) that both DeepSeek arms missed — GPT/Grok territory.
Tool use
0/3 tasks ✗
Refused on t3 ("I can't make tool calls") — a real interaction gap vs DeepSeek/Grok.
Agentic speed
4/5 ✓ · 10–23× slower
Passed coding/workflow tasks, but agent loops took 2–23 min vs 0.5–4 min on cloud GPUs.
bench3 · benchmark by Eran Shir · github.com/eranshir
Download PNG · MTPLX: mtplx.com