Five cloud frontier models — DeepSeek V4 Flash & Pro, GPT-5.6 Sol, Grok 4.6, GLM 5.2 Vision (LunaRoute) — versus MTPLX, a free, open-source Mac app running Qwen 3.8 27B on a single laptop. Same harness, same tasks, same hidden graders. The local model holds its own.
Every arm faced the identical bench3 harness: hidden-test grading for coding and agentic tasks, objective checkers for reasoning and tool use, and a blind rubric judge for creative and writing output. MTPLX ran the whole set on a laptop — the cloud arms on their vendors' GPUs.
Pass rate by category. Reasoning is where MTPLX shines — it solved the two hardest puzzles (r1_tiling, r4_catalan) that both DeepSeek arms missed. Tool use is its blind spot.
Cloud arms: up to 5 trials. MTPLX: trial 1 (n=16). ⏱ = watchdog timeout, not a capability verdict.
The harness records wall-clock time and token counts, so every run yields an effective tokens/sec. Cloud GPUs are fast; MTPLX is a laptop. But decode speed tells a different — and surprising — story.
MTPLX accelerates generation with native MTP speculative decoding — the model's own multi-token-prediction heads draft ahead, and a batched verify pass checks them. Measured on this M5 Max (temp 0, 1,000 tokens):
≈1.6–1.7× faster, up to 2.6× in a cool burst — identical output distribution.
The same 9,000-token prompt, same sampler, measured across a 2-hour session on the M5 Max:
A cool Mac decodes at 36–46 tok/s; after hours of sustained load it throttles to ~26. The app dashboard's 46.4 tok/s was a fresh, cool run.
After our benchmark, an open leaderboard (yukon.org/mlxfast) claimed Qwen 3.8 27B runs 193.4% faster on Apple Silicon via community-tuned native MTP kernels. We fetched the record submission (Layr-Labs/qwen-3.8-mtp-challenge), built it, and reproduced the measurement on this same M5 Max — 2.07× decode speedup with exact output parity (all tokens bit-identical to serial). The leaderboard's 2.94× is measured on a thermally gated, idle box with 8 hidden prompts; our machine was hot and GPU-contended during the run.
Cloud arms burned real money across 250 runs. The local model produced its entire results column for free, on a laptop the user already owns.
/v1/chat/completions API).