Gemma 4 12B

AA coding index 31.0 (Artificial Analysis, frozen since 2026-09-11)

Google's dense 12B — the strongest measured model that still fits a small card. Weights are gated on Hugging Face (accept the licence to download).

HumanEval pass@1 63% (19/30), via local GPU (Ollama, Q4_K_M). Measured by us on this hardware, as a rough sanity check — HumanEval is a different, easier, partly-contaminated benchmark than Artificial Analysis’ composite, so it is not comparable to the coding-index column.

BigCodeBench-Hard: no score, on purpose. We measured it three times and withdrew every run. At Q4_K_M it falls into a repetition loop on BigCodeBench/37 — it repeats 'ResponseBody' until it runs out of budget — and 92 of 148 answers hit the token ceiling. The resulting 4% measures the jam, not the model. Reproducible across all three lanes, so it is not a bad run.

87.5% ± 2.1 over 1150 problems

Q6_K · a shared cluster GPU · KV f16 · 1 run · spread between runs assumed at 0.25 points, not measured. This is the non-agentic half of BFCL: does the model call the right function with the right arguments. It says nothing about how it behaves over a long agentic conversation, which is a separate measurement. The score is BFCL’s own: the unweighted mean of four groups (simple, multiple, parallel, parallel multiple), where simple is itself the mean of Python, Java and JavaScript. Every group counts the same whatever its size, and irrelevance detection — which we also measure, another 240 problems — is not part of it. The margin follows that same weighting.

One request at a time, same server build and same KV cache throughout. Every point on the same class of card. Points measured once carry the sampling margin only; where the curve jumps we measure again and give the spread. A row is only here if all seven categories completed.

All 13 RULER tasks, one request at a time, KV cache f16, bench commit c3f5e3b4, seed 42. The score is the mean of the 13; the margin is two standard deviations of the sampling, so it grows as the sample shrinks and as the score drops.

This is not a coding score and must not be read as one. RULER asks whether the model finds and follows a fact buried in a long text. Quantising hits the two differently, so a model that still retrieves at length may already have lost the code.

One or more rows ran on an engine that does not report whether the prompt was truncated, so for those we cannot prove the haystack went in whole.

~31 tok/s on RTX 4060 Ti, measured by us.

Not measurable with our current setup, and the reason is the model, not our rig. Google has an open, acknowledged bug: on long prompts the model falls into a deterministic "thought\nthought\n..." loop that runs for around a thousand iterations (huggingface.co/google/gemma-4-12B-it, discussion 41). Two things there matter if you are thinking of running it. It reproduces at F16 with no quantisation at all, shown across 48 seeds, so no GGUF build and no quant level avoids it. And it is probabilistic, not constant: roughly 3 runs in 5 fail, 43.8% with the embedded chat template and 35.4% with the best community one, which lowers the rate without removing it. Google confirmed it on 2026-07-09 and said a fix was coming; as of 2026-09-12 there is none. Against a 1,280-token budget a loop that frequent is fatal, so we publish no score. We also see a second, separate problem that is not the loop: at Q8_0, with no loop at all, it still exhausts the budget on some problems writing perfectly ordinary code. It is simply verbose. And llama.cpp crashes reproducibly on BigCodeBench/308 and /492, on upstream builds too, so that one is the GGUF rather than our patch.