Runs on Multi-GPU · Hugging Face ↗
AA coding index 52.6 (Artificial Analysis, frozen since 2026-09-11)
A 229B mixture-of-experts that activates 8 of its 256 experts per token, so it reads a fraction of its weights on each step. Shipped in FP8, which is why the file is smaller than the parameter count suggests.
BigCodeBench-Hard pass@1 25% (37/148), via official protocol · llama.cpp UD-IQ3_XXS · one request at a time · a shared cluster GPU · mean of 3 runs, range 0.0 pp. The brutal counterpart to HumanEval — where HumanEval saturates near the top, BCB-Hard spreads the field, so this is the number that actually separates coding ability. does not fit the token budget; 6 of 148 answers hit the token limit (effective ceiling 96%)
Our own runs on this model. Each number carries how many problems it is over and how many times we ran it — these are not comparable with each other, and none of them is on the same scale as the third-party indices on the leaderboard.
| test | score | over | conditions |
|---|---|---|---|
| BigCodeBench-Hard | 25.0% | n=148 · 3 run(s) | official protocol · llama.cpp UD-IQ3_XXS · one request at a time · a shared cluster GPU · mean of 3 runs, rang |
| BFCL v4 non-live | 78.4% ± 2.7 | n=1150 · 1 run(s) | BFCL v4 · UD-IQ3_XXS GGUF via llama.cpp · one request at a time · a shared cluster GPU |
| RULER (long context) | 92.3 ± 5.2 | N=5/task · at 32K | 1 quantisations measured |
78.4% ± 2.7 over 1150 problems
UD-IQ3_XXS · a shared cluster GPU · KV f16 · 1 run · spread between runs assumed at 0.25 points, not measured. This is the non-agentic half of BFCL: does the model call the right function with the right arguments. It says nothing about how it behaves over a long agentic conversation, which is a separate measurement. The score is BFCL’s own: the unweighted mean of four groups (simple, multiple, parallel, parallel multiple), where simple is itself the mean of Python, Java and JavaScript. Every group counts the same whatever its size, and irrelevance detection — which we also measure, another 240 problems — is not part of it. The margin follows that same weighting.
| quant | GiB | 4K | 8K | 32K |
|---|---|---|---|---|
| UD-IQ3_XXS | 46.3 | 96.9 ± 3.9 · N=5 | 97.4 ± 3.6 · N=5 | 92.3 ± 5.2 · N=5 |
All 13 RULER tasks, one request at a time, KV cache f16, bench commit c3f5e3b4, seed 42. The score is the mean of the 13; the margin is two standard deviations of the sampling, so it grows as the sample shrinks and as the score drops. Everything here clears the paper’s effective-length bar of 85.6 at every length shown.
This is not a coding score and must not be read as one. RULER asks whether the model finds and follows a fact buried in a long text. Quantising hits the two differently, so a model that still retrieves at length may already have lost the code.
Real GGUF file sizes = weight VRAM. Add the KV cache for your context (≈8 GB at 32K, fp16). Read from the GGUF header and checked against what llama.cpp actually allocates.
| Quantization | Size |
|---|---|
| IQ1_S | 64.0 GB |
| IQ1_M | 68.7 GB |
| IQ2_XXS | 74.0 GB |
| IQ2_M | 78.1 GB |
| Q2_K | 83.3 GB |
| Q2_K_L | 83.4 GB |
| Q2_K_XL | 85.8 GB |
| IQ3_XXS | 93.6 GB |
| Q3_K_S | 98.7 GB |
| Q3_K_XL | 101.5 GB |
| Q3_K_M | 109.3 GB |
| IQ4_XS | 121.9 GB |
| IQ4_NL | 129.0 GB |
| Q4_0 | 129.5 GB |
| Q4_K_S | 130.0 GB |
| Q4_K_XL | 131.6 GB |
| Q4_K_M | 138.3 GB |
| Q4_1 | 143.2 GB |
| Q5_K_S | 157.5 GB |
| Q5_K_XL | 162.1 GB |
| Q5_K_M | 162.3 GB |
| Q6_K | 187.8 GB |
| Q6_K_XL | 194.3 GB |
| Q8_0 | 243.1 GB |
| Q8_K_XL | 261.4 GB |
| BF16 | 457.5 GB |
~87 GiB at UD-IQ3_XXS — a 2-card node, or one 141 GB card. Too big for a desktop, but the cheapest way to run a model of this size is a quant this aggressive, and the score below is measured on exactly that file.