Choosing a local model with a stopwatch, not a benchmark

Model cards do not describe my work

I keep a local model on the workstation for grunt work: rewriting scripts, generating variations, chewing through structured jobs where sending everything to a hosted model is either too slow, too expensive, or inappropriate for the data. One machine, one consumer GPU with 24 GB.

Choosing that model from published benchmarks turned out to be worthless, in a specific and instructive way.

Method: five hard tasks, code actually executed

The test set is five tasks that produce runnable code with a verifiable result: distributing points evenly on a sphere, bin packing, extracting a boundary loop from a mesh, greedy meshing, and edge collapse simplification. Scoring is binary per task: the code runs and produces the correct result on the first attempt, or it does not. Three runs per model.

The first version of this test was useless and it is worth admitting why. My original tasks were easy, and every model scored five out of five on the first attempt. A test that everything passes measures nothing. I rewrote them harder, and before trusting the results I wrote my own reference solution for each one, to prove the tasks were solvable rather than merely difficult.

Results

The first two rows are the same model. The official build was 2.4 times faster and more accurate than a repack of itself.

The difference is packaging, not intelligence

Two things account for it. The importance-matrix quantisation in the repack is noticeably slower on this GPU than the plain 4-bit K-quant, despite being the more sophisticated format on paper. And the official build runs through the runtime's own template and parser path with speculative decoding enabled, while a file pulled from a model hub falls back to a generic path.

None of that is visible on a model card. Both rows would be described identically by anyone who had not run them.

The second mistake, also recorded

Partway through I checked capabilities with a command whose output I truncated to two lines, and concluded the official build did not support tool calling. It did. Both did. I had cut the list short and read the gap as an absence.

That is a small error with a general shape: a truncated view of the evidence reads exactly like evidence of absence. I now read capability output in full, every time, because the failure is silent and points confidently in the wrong direction.

What changed

Devstral was deleted. The official build became the default across the local tooling, with the repack kept behind an environment variable as a fallback. One trap came with it: the repack's model file embeds a context length of 32768, so unless it is explicitly overridden at call time, the model quietly ignores the 16384 limit the rest of the system is built around.

The rule

The whole exercise took an evening. It replaced a choice I would otherwise have made from reading, and the choice I would have made from reading was the wrong one, in both directions at once: I would have picked the more exotic quantisation, and I would have believed it was the more capable option.