The right harness is all you need: the test that reshuffled the open-weight leaderboard

What was meant to be a benchmark turned into a revealing experiment: sentdex (Harrison Kinsley) found that the inference software — the harness — changes scores more than the model itself.

The experiment that exposed the harness's weight

sentdex couldn't validate the numbers published by vendors. He switched the inference stack — from vLLM to the OM harness — and results jumped dramatically.

Twenty tasks of difference — 20 percentage points — gained just by swapping the software serving the model. Same hardware, same weights, different leaderboards.

The ranking: GLM-5.2 still leads — by a hair

On the final scoreboard, the best local model so far is GLM-5.2 quantized to 3.25 bits. DeepSeek V4 Flash and Laguna S 2.1 are close, but the harness changes who leads.

On speed, the story flips: GLM-5.2 decodes at ~75 tokens per second, while V4 Flash tops 130. Depends on what you prioritize.

Hardware: the invisible bottleneck

The second half is an infrastructure masterclass. sentdex recounts the saga of getting four RTX Pro 6000 to talk to each other without a bandwidth bottleneck.

The final topology: host machine → retimer → two MCIO cables → Gen 5 100-lane switch → GPUs. The switch removes the bottleneck that capped performance.

The cost of iterating more

None of this is free. With OM, time per solution doubles and token consumption spikes. Running frontier local models requires expensive hardware and patience.

Why it matters

The video is a faithful portrait of the local frontier race: increasingly aggressive quantizations, harnesses that shift leaderboards, and hardware becoming the deciding factor.

For anyone evaluating local AI, the lesson is clear: the model is half the story. The software serving it can be worth 20 benchmark points.