Gemma 3 4B

AA coding index 2.7 (Artificial Analysis, frozen since 2026-09-11)

A vision-language model that declares Gemma3ForConditionalGeneration, so the GGUF repo ships an mmproj projector next to the weights - do not point llama-server at it by mistake. 34 layers with a 1024-token sliding window, which is short: the KV cache stays small as context grows, but most layers only ever see the last 1024 tokens. The weights are under the Gemma licence, not Apache, and the official repo is gated: you have to accept the terms before you can download the tokenizer.

BigCodeBench-Hard pass@1 11% (16/148), via official protocol · llama.cpp BF16 · one request at a time · a shared cluster GPU · mean of 3 runs, range 0.0 pp. The brutal counterpart to HumanEval — where HumanEval saturates near the top, BCB-Hard spreads the field, so this is the number that actually separates coding ability.

Each quant links to the exact file it was measured from, in unsloth/gemma-3-4b-it-GGUF. Every point measured on the same class of card, one pass, one request at a time. One problem is 0.68 points, so anything under ~3.4 points apart is a tie — that is the spread we measure between GPU classes, and our rows are not all on the same card. Each card column is the context left for the KV cache after the weights, KV-Q4 / KV-Q8. A dash means the file does not fit that card, or fits with no room left to work. Context is usually the real constraint, not quality. Computed for one slot (--parallel 1), which is what you get serving yourself. llama-server opens four by default, and on a hybrid that costs real context — the table below each curve shows how much. The card figures assume a dedicated card. Measured on an idle one: the driver keeps 623 MiB of 143,771, so 99.6% is usable. Compute buffers are measured too — a fixed ~130 MiB plus 1 MiB per 1K of context, the same slope on every architecture we checked. If your card also runs your desktop you get noticeably less, and we have not measured that case yet.

55.5% ± 3.0 over 1390 problems

Q4_K_M · a shared cluster GPU · KV f16 · 1 run. This is the non-agentic half of BFCL: does the model call the right function with the right arguments. It says nothing about how it behaves over a long agentic conversation, which is a separate measurement. The score is BFCL’s own: the unweighted mean of four groups (simple, multiple, parallel, parallel multiple), where simple is itself the mean of Python, Java and JavaScript. Every group counts the same whatever its size, and irrelevance detection — which we also measure, another 240 problems — is not part of it. The margin follows that same weighting.

All 13 RULER tasks, one request at a time, KV cache f16, bench commit c3f5e3b4, seed 42. The score is the mean of the 13; the margin is two standard deviations of the sampling, so it grows as the sample shrinks and as the score drops.

This is not a coding score and must not be read as one. RULER asks whether the model finds and follows a fact buried in a long text. On this model Q3_K_M scores 59.5 at 32K against 59.4 for Q6_K, a difference of 0.2 points — while on BigCodeBench-Hard the same two files score 4.7% and 11.5%. A page that read the long-context number as a general score would be telling the truth and misleading you.

Pull the GGUF from unsloth/gemma-3-4b-it-GGUF and serve it with llama-server: llama-server -m gemma-3-4b-it-Q4_K_M.gguf --n-gpu-layers 99 --flash-attn on --jinja. The chat template writes the BOS token as text and the server adds another, so strip it from the prompt if you drive the completions endpoint yourself.

We measured its whole quantisation curve on BigCodeBench-Hard, three runs per point, and it is well behaved: the score falls with the file size and never jumps back. Q6_K is the top at 11.5%, BF16 and Q8_0 tie at 10.8%, and it keeps dropping to 2.0% at Q2_K. One problem is 0.68 points, so the whole range from Q4_K_M up sits within three problems of itself and the useful choice is the smallest that fits. Two things worth knowing before you pick it for code. First, size is not what decides here: granite-4.0-micro scores 16.2% and MiniCPM5-2B 19.6%, both smaller than this. Second, the packing format costs more than a bit of width - Q4_K_M scores 9.5% and Q4_0, the same width and almost the same file size, 8.1%.