Runs on ≤8 GB · Hugging Face ↗
Coding index 31.0 (Artificial Analysis)
Google's dense 12B — the strongest measured model that still fits a small card. Weights are gated on Hugging Face (accept the licence to download).
HumanEval pass@1 63% (19/30), via local GPU (Ollama, Q4_K_M). Measured by us on this hardware, as a rough sanity check — HumanEval is a different, easier, partly-contaminated benchmark than Artificial Analysis’ composite, so it is not comparable to the coding-index column.
BigCodeBench-Hard pass@1 36% (44/121), via local GPU (Ollama, num_ctx 16384, budget 12K) — comparable full-148, 0/148 truncated. The brutal counterpart to HumanEval — where HumanEval saturates near the top, BCB-Hard spreads the field, so this is the number that actually separates coding ability.
~31 tok/s on RTX 4060 Ti, measured by us.
Real GGUF file sizes = weight VRAM.
| Quantization | Size |
|---|---|
| F16 | 0.9 GB |
| Q2_K | 4.7 GB |
| Q3_K_M | 5.7 GB |
| IQ4_XS | 6.4 GB |
| Q4_0 | 6.7 GB |
| Q4_K_S | 6.8 GB |
| Q4_K_M | 7.1 GB |
| Q4 | 7.4 GB |
| Q4_1 | 7.4 GB |
| Q5_K_M | 8.4 GB |
| Q8_0 | 13.1 GB |
| Q8 | 13.6 GB |
| Q6_K | 20.5 GB |
| BF16 | 24.7 GB |
Q4 lands around 8 GB, so a busy KV cache can push it to a 12-16 GB card. Follow Gemma's exact chat template. Runs very well on Apple Silicon via MLX.