The honest map of what LLM coding actually costs.

What happened. Last week AA’s API had no Terminal-Bench 4.0 field. This week 141 models have a measured score on it, including models released three days ago. Meanwhile the coding index this site ranked on still holds 258 models whose newest release date is 11 September: AA computed it from Terminal-Bench 2.1 and no longer runs that. Two rulers, and neither covers the table — the new one reaches 60 of our 131 rows, the old one 114 but nothing recent.

What we changed. The rank now follows AA’s intelligence index, which covers all 131 rows. Both coding columns stay, measured only, side by side. This is not a retreat from coding: of the 50 models that carry both the new pass rate and the old index, the intelligence order agrees with Terminal-Bench 4.0 at Spearman 0.95, and the old coding index only at 0.93. The column we used to rank on is the worse predictor of the benchmark that replaced it.

Why it is the worse predictor: it saturates. Eight points of old index, from 74 to 82, hold real Terminal-Bench 4.0 scores from 12.6% to 59.1%. Fable 5.1 at 81.6 scores 52%; Kimi K3 at 76.2 scores 12.6%; GPT-5.6 Sol at 77.4 scores 39.9% while GPT-6 Astra at 76.9 scores 59.1%. The old index called those last two almost equal.

The bridge we fitted and discarded. The tempting move is to convert: 50 models have both scores, so map one onto the other and give every unmeasured row an estimate. Excluding the saturated top, the fit is tight — 1.3 points of average error below 68 index points, and 62 of the 65 unmeasured rows live in exactly that range. We discarded it for three reasons. A one-variable monotone fit preserves order, so it produces the ranking we already had and adds a number, not information. Its entire output range is 0% to 3.2%, and the benchmark is 66 tasks attempted three times each, so every value it can emit sits inside six solved attempts — about two tasks — and it cannot order the 65 rows it was built for. And it extrapolates into the gap in the wrong direction: Claude Opus 4.7, old index 73.6, comes out at 3.2% while the two measured models nearest to it on that index, Opus 4.8 at 74.3 and GLM-5.3 at 74.8, score 21.7% and 41.9%. We publish estimates elsewhere on this site and label them; we are not willing to put one in the same column as a measurement.

A trap in the API, for anyone building on it. terminalbench_v4_0 returns a plain 0 for a model AA has not run, while using real nulls elsewhere in the same record. Sixty-seven models carry that zero, Apodex 1.1 among them — it passes 69.7% of Terminal-Bench 2.1. Thirty-nine of our own 131 rows carry it, so treating those zeros as scores would have dropped almost a third of the table to the bottom. We hit the identical placeholder on their speed field last month, so we check both now.

AA also renamed almost everything. DeepSeek’s rows dropped “(Reasoning, Max Effort)” for “(Max)”; all nine Claude rows became “(Max)” or “(Max, Default Fallback)”, the second rename landing while we were building; and 123 more changed capitalisation only, (max) to (Max). Each rename silently empties a row matched by name. Under all that churn the list did not shrink: it went from 673 rows to 689, and every name that disappeared has a renamed counterpart. We added a guard that prints a loud warning when a tracked row stops matching any name in AA’s list, and we read it on every pass; it caught both renames. It does not stop the build — the row still publishes, without its numbers — so the warning is the whole mechanism.

The rest of the week. APIMart, which a week ago served no text models at all, sells them again with per-token prices 20% under list on Claude and GPT. RunPod’s public Kimi endpoint is gone, so it stops being a coding option. Weights & Biases Inference is now CoreWeave Forge. opencode added a $40 Go Plus tier, Alibaba a $10 entry plan, ChatGPT a $500 Pro tier, and Qoder extended Qwen3.8-Flash free with no end date. Krea, which served DeepSeek V4.1 Flash eight days ago, serves nothing and is retired.

The scoreboard froze, and we can prove it. On 19 September Artificial Analysis moved its Intelligence Index to methodology v4.3.2 and replaced Terminal-Bench 2.1 with 4.0. Its coding index — the column this site ranks on — is computed from 2.1, so nothing evaluated after that date gets one, and the newest model with a coding score was released on 11 September. We compared our two committed snapshots: 261 models had the index on the 19th and 259 have it today. No surviving model lost its index and no value changed; the two that are missing, Agnes 2.5 Pro Alpha and Beta, dropped out of AA’s listing altogether. It is a migration, not an outage. It also means five flagships landed in two days with nothing to rank them by.

What we did about it, and what we refused to do. We did not change what the ranking means, and we did not borrow a number from somewhere else. We looked first: BenchLM publishes a usable dataset, but it is CC BY-NC and it aggregates vendor-reported figures rather than running its own; Terminal-Bench's own leaderboard is the source AA is migrating to, but it lists fifteen model-and-agent pairs, not models. So the inclusion rule got one bounded extension instead. A model that OpenRouter serves and AA scores only on intelligence now appears in the table, at the end, with no rank and an empty coding column, if it was released after 10 September. Six rows today. They sort themselves in the day AA scores them. A row with no rank is a model we cannot rank, not a model that scored badly, and the table says so in place.

Our tests have moved to Yardstick. Everything we measure ourselves — BigCodeBench-Hard, BFCL v4, RULER, SWE-bench, speed and energy — now lives at yardstick.millaguie.net, with the harness that produces it. It publishes per card, which this page never did. Keeping two copies of the same tables was a guarantee that one of them would go stale, so this tab is now a signpost. Model pages here still show their own first-party scores, and the leaderboard keeps its three ᴹ columns.

Tool calling is saturated. With BFCL on 45 local models, the best twelve sit between 90.0% and 87.5% with about two points of margin each: a twelve-way tie that runs from a 2 GiB Granite 4.0 Micro to a 32B. The line worth keeping is Ternary Bonsai 27B, 88.3% from a 7.1 GiB file, tying the 27B it was built from. We will keep publishing the score, but as a filter, not as a ranking.

Long context still separates, and the quantisation still decides. Qwen3.8-27B scores 97.2% at 32K from 12.2 GiB, tied with Qwen3.5 9B’s 96.9% from 6.1 GiB. Gemma 4 12B scores 70.9% at 32K in BF16 and 40.1% at Q3_K_M — thirty points for the file you picked, on a test where the same step costs it far less in coding. The lower point carries no truncation guard recorded, so read it as a floor.

One number that moved because the binary moved. EXAONE 4.5 33B's three RULER points come from a newer llama.cpp than the rest of our runs, because the reference build produced looping garbage at 8K and 32K. Its 4K score fell from 86.2 to 78.5 with the new build: same data, same seed, temperature 0, only the binary changed. Its card says all of it.

The market, briefly. nano-gpt sells Claude Opus 5.5 at Anthropic's list price; AIMLAPI sells it above list. Kimi renamed its four tiers and withdrew the request counts we used to publish, so we withdrew them too. Qoder recalibrated its factors, making Kimi K3 75% dearer in credits. APIMart has no text models left at all. ShareAI now sells closed models, priced in euros. And KAT-Coder-Pro V2 and Devstral 2 left the table with no provider serving them.

On the numbers. Market figures come from the vendors' own pages, read this week and dated in plans.yaml and gateways.yaml. First-party figures are ours: BFCL non-live, one to three runs, about two to three points of margin; RULER N=5 per task except where the card says otherwise.

What we measured this week. The new open models from the last two weeks all have full curves now, three runs per point. The headline is K2 Horizon 32B: 34.0% at Q4_K_M, a 19.6 GiB file that fits a 24 GB card, within the margin of the best we have ever measured. The best two it ties need around 85 GiB. Qwen3.8-27B found a better 16 GB build, UD-IQ3_S at 33.8% from 11.2 GiB. Ternary Bonsai 2 lifts the 7 GB ternary 27B from 24.3% to 27.0%, but only runs on PrismML's own llama.cpp fork. And Spark-X2.5-4B scores 27.7% at Q8_0 but 19.4% at Q4_K_M, so on that model the bigger file is the one to download. The new VRAM tables now show, for each budget, the best measured build that fits it, with its score and its size from the same file.

A small model for tool calls, not for code. We finished BFCL, the function-calling benchmark, on the small models we already had quantisation curves for. The one that surprised me is Granite 4.0 Micro. At Q4_K_M, a 2 GiB file, it scores 87.6% on the non-live set. Qwen3.8-27B at UD-Q3_K_XL scores 88.2%. Each figure carries two to three points of margin, so that is a tie. The two even fail in the same places: both drop in Java and JavaScript, where the small model is in fact slightly ahead. The catch is that tool calls and code are different skills: on BigCodeBench-Hard the same Granite scores 16.2%, against 31.8% for the 27B at full precision. If your agent mostly routes calls to tools, the small model does the job. If it writes the code itself, it does not.

Read the categories, not the headline. SmolLM3-3B scores 65.7% overall and 25% on irrelevance, the test where the right answer is to call nothing. Before publishing we checked that this is the model and not our harness: all 180 failures are valid calls that should not have been made. MiniCPM5-2B is the reverse: 87.9% on irrelevance, but 28.5% when it has to make several calls at once. Both numbers disappear inside the total.

Two bits is a cliff for small models, not for big ones. Last week we wrote that the cliff is at two bits. This week we collected three curves of 27-32B dense models that had been measured and never picked up, and they say that rule only holds for the small ones. Qwen3.6 27B scores 28.8% at UD-Q2_K_XL (11 GiB) and 28.4% at Q5_K_M. Qwen3 32B loses about one point at two bits, and Granite 4.1 30B under three. Qwen3.5-9B lost more than half going from Q4_K_M to two bits. On a big dense model, two bits is often a fair way to fit a smaller card. On a small one it is not. It is not a law, though: Qwen3.8-27B drops from 31.8% to 23.0% at UD-IQ2_S, so read the curve of the model you actually run.

Gemma 3 4B is not worth its size. Its full curve peaks at 11.5% (Q6_K). That is below Granite 4.0 Micro and MiniCPM5-2B, which are both smaller. And the packing format shows again: Q4_K_M beats Q4_0 on Gemma 3 4B (9.5% against 8.1%) and on Granite 4.1 30B (25.0% against 23.0%), with the same bit width.

What changed in the market. CrofAI is shutting down and refunding balances. Public AI stopped being free. Chutes now states its quota in dollars: a plan includes five times its price in pay-as-you-go usage. Solheim cut its 256K plan from €30 to €24 but took away its bursts. Weights & Biases now applies its $0.50-per-million surcharge to its ARIA agent only. On OpenRouter, three routes became free (DeepSeek V4 Flash 0731, Qwen3.8 27B, GLM 5.2) and the listed price of gpt-oss-120b went up fourfold on input.

On the numbers. Every BigCodeBench figure is 148 problems and three runs, so about three to four points of margin: 28.8% against 28.4% is a tie. BFCL figures are the non-live score (irrelevance is measured and reported, but not counted in the total), one run for the small models and three for Qwen3.8-27B, two to three points of margin. Market figures are read off the vendors' own pages this week, with the date, in plans.yaml and gateways.yaml.

Six quantisation curves, and the one number that surprised me. We measured Qwen3.5 at 2B, 4B and 9B, plus SmolLM3-3B, Granite 4.0 Micro and MiniCPM5-2B — every quantisation level of each, three runs per point, 148 problems of BigCodeBench-Hard on one class of card. The finding I did not expect is that "four bits" is not one thing. On Granite 4.0 Micro the K-quant at four bits scores 16.2% and the legacy Q4_0 build of the same model at nearly the same file size scores 9.5%. That is a 40% drop bought by nothing but the packing format. If your runner defaults to Q4_0 for compatibility, that default is costing you a chunk of the model.

The cliff is at two bits, not four. From Q8 down to Q3 these curves are close to flat — on Qwen3.5-9B the whole range sits between 21.6% and 27.7%, a spread smaller than the gap between two neighbouring models. Paying for 8 bits over 4 buys almost nothing here. Two bits is where it collapses: the 9B drops to 12.2%, the 4B to 8.1%, and on the 2B the two-bit build produced nothing gradeable at all — our publication gate withdrew that point rather than print a zero, because a zero would read as "scores badly" when the truth is "emitted no usable code".

What quantisation damages, and what we cannot yet claim. We ran RULER on the same files for long-context recall. The coding collapse at two bits is not there: Qwen3.5-9B goes from 96.8% to 92.3% at 32K while its coding score more than halves. But be careful with that comparison, because we were: those RULER runs are N=5 per task, which puts a margin of 3 to 8 points on every figure, so the two overlap and we CANNOT say the context score is unaffected — only that no collapse is visible where coding clearly shows one. Settling it properly needs N=100, which is what our anchor runs use.

And size beats bits. The 9B at two bits (12.2%) beats the 4B at two bits (8.1%) and is two and a half times the 2B at full precision (4.7%). On a fixed VRAM budget, take the bigger model at a lower quant. The exception that proves how much the model itself matters: MiniCPM5-2B scores 21.6% at Q8_0, four times what Qwen3.5-2B manages at any level, from a file that is 1.6 GB at Q4_K_M.

What we could not measure, and why. Gemma 4 12B is absent because it cannot be measured, and the reason is not our rig: it falls into a deterministic repetition loop on roughly three runs in five, which reproduces at full precision across 48 seeds and is an open, acknowledged bug that Google confirmed in July. We had it filed here as "not measurable" with the cause guessed wrong; it is now filed with the cause, the rate and the link. If you are choosing a local model, a 40% chance of no answer is the kind of thing a quality index will never show you.

Plans and providers, re-read. All twenty-four subscription plans were checked against their own pages this week, and several moved in ways worth knowing: Z.ai now publishes the formula that converts its credits to tokens, Ollama switched from billing GPU time to billing tokens, and opencode Go replaced its single monthly cap with a per-model one. We also found twenty-five inference providers serving models on our board with no entry of their own; they now have one, and the build warns us if another appears.

On the numbers: BigCodeBench-Hard is hard, so these figures look low next to marketing ones — what matters is the shape of the curve, not its height. Everything is temperature 0, one request at a time, three runs per point; 23 of 24 points agreed to the decimal across runs, which is what temperature 0 should do.

Two leaderboard rows were wrong, and I fixed them. Qwen3.8-27B, the model on our front page, was showing the quality score of its weakest variant (58.2) instead of the one that actually serves that route (68.1) — an internal mapping bug, my code grabbed the first text match without checking there were four variants sharing the same base name. And Nemotron 3 Super 120B showed no quality score at all, because of an extra "NVIDIA " in the name that meant it never matched anything. Both are fixed, and I checked all 112 leaderboard rows one by one so none of the others are silently broken the same way.

Qwen3.8-Flash-Next has an architecture worth explaining properly. It shipped August 26 as an open preview of Qwen4: 180B parameters stored in total, but only 6B active per token. The trick is how it spreads those 180B. The backbone is 48 layers in 12 identical blocks, and each block runs three layers of Gated DeltaNet followed by one of Qwen Sparse Attention. Gated DeltaNet is linear attention: instead of re-reading the whole conversation every time (the squared cost that makes ordinary attention slow), it compresses the history into a fixed state it updates with a learned gate. It "remembers" instead of re-reading. Sparse attention does the opposite: it groups the context into 2,048-token blocks and a learned indexer picks which ones deserve full attention. It "retrieves" precisely instead of recalling everything. Cheap memory almost always, expensive search only when needed — that is how it reaches a million tokens of context without the cost exploding. Alibaba says the sparse-attention layer gives up to 7.6x faster prefill and 4.9x faster decode than full attention at 1M context.

It also carries a 51B-parameter table with 20 million bigram and trigram entries, sitting in layer 2: a cheat sheet of common text patterns it looks up instead of computing, so it adds no real compute cost and can be offloaded to system RAM if VRAM is tight. The residual stream, which in most models is a single channel running through the whole network, is here four parallel branches with their own gate, kept in FP8; Alibaba says that is what lets it train in FP8 at this level of sparsity without the model destabilizing. The result: the capacity of a 180B model, the compute cost of a 6B one, and training at roughly a ninth of what Qwen3.7-Plus cost. Already running it on the machines I use for testing, and I have hit the other side of so much novelty: it needs a custom llama.cpp build because the architecture is not in the main branch yet, and unsloth's GGUF does not ship the multi-token-prediction module (a separate 4B, for faster generation) that the model does have out of the box. Right now that is 10.5 tok/s with experts in RAM and 21 tok/s with everything in unified memory. Quant curve in progress, for next week.

GLM-5.3-Flash is not a mystery any more. Last time I mentioned an unbranded model on OpenRouter that resolved 10 of 10 on our SWE-bench pool, and guessed from its tokenizer that it was from the GLM family. I was right: it is GLM-5.3-Flash, a 321B MoE under MIT, and Z.ai now serves it directly (mirrored by Novita) in fp8. It has a full leaderboard entry now.

A new model that sounds like a lot. Tencent released Hy4 preview: 770B total parameters, only 49B active per token, over 1M context. In their own internal evaluation it scores above GLM-5.3 and Kimi K3 on coding tasks. No Artificial Analysis index yet — it shipped today — so it is not on the leaderboard yet. Watching it for next week.

Two subscriptions got more expensive without announcing it. MiniMax's Token Plan rose about 10% across all three tiers — Plus $20 to $22, Max $50 to $55, Ultra $120 to $132 — confirmed against the raw pricing page. Z.ai's coding plan rose less in percentage but more in absolute terms on the tiers that matter: Pro $72 to $80, Max $160 to $168; the credit quotas underneath did not change. That one only shows on the real subscription page — the docs page still advertises the plan as "starting at $18" with no more detail.

Cursor did not raise its price, it deleted the number. Until this week, Pro/Pro+/Ultra included a stated dollar amount of API usage ($20/$70/$400). The docs now describe two usage "pools" per plan with no dollar or token figure at all, just "included". That is a real disclosure regression, not a price change to the plans themselves.

The free set rotated again. opencode Zen added Ling 3.0 Flash Fin (the free route for Ling reappearing, it had vanished) and Muse Spark 1.2 Contributor; it dropped Big Pickle and DeepSeek V4 Flash Free. And OpenRouter now has 18 public :free routes, with GLM-5.2 (68.8 coding index) and MiniMax-M3 (58.6) among the new ones — more free capacity than there has ever been.

Catalogue cleanup. I dropped three models from the table that no longer exist on OpenRouter under any name (Ring-2.6-1T, Ling 3.0 Tiny, Gemma 3n E4B) — they used to sit there as rows with no price. And a price I had mislabeled on Morph: what showed as GLM-5.2's price ($6/$22.50 per million) actually belonged to "Kimi K3 fast" — GLM-5.2's real price is $0.84/$3.14, far cheaper.

One note on Our tests. That tab keeps filling in daily as BCB-Hard, BFCL and RULER results land, not on a weekly schedule like the rest of the site. It has grown enough of its own shape, its own protocol, its own model and quant matrix, that it may end up as its own site rather than a tab here. Nothing changes for now, just flagging it before it happens.

As always, free sets rotate and close without notice, so treat any free allowance as this week's, not a standing plan.

I am starting a short note each week to explain what changed on the site, in plain words. Same data as the tables, just told as a story. This is the first one.

A new free model I could actually test. NVIDIA released Nemotron 3.5 Lightning this week, a 30B mixture-of-experts model with only 3B active parameters. The small active count is the interesting part: it means the model is cheap to run and it fits on modest hardware. It is also already free on opencode Zen, so I did not need to rent a GPU. I pointed our BigCodeBench-Hard harness at the free endpoint and measured it myself: 34.7% on BCB-Hard (42 of 121). It is a reasoning model, so it thinks before it answers and it is not fast, but for a free local-agent model that is a solid number. It is now both on the leaderboard (Artificial Analysis gives it a coding index of 26.8) and in the Run local catalogue with our own measurement.

Qwen3.8-27B landed, and I ran it. The Max is still datacenter-only (2.4T, 95B active), but the small dense 27B (Apache-2.0) that fits at home shipped this afternoon, so I tested it properly. On my 16 GB card it fits at Q3 (about 14 GB), and with MTP speculative decoding through llama-server it runs at 36 tok/s, twice the 18 it manages without spec-decode, because the model ships its own draft head and about 87% of the drafts get accepted. On our BCB-Hard it scores 27% at Q3 non-thinking; the full model at good precision (BF16/FP8) is around 37 to 43%, so the 3-bit quant you need to fit 16 GB costs about 10 points. It is not on the leaderboard yet, no Artificial Analysis index and not on OpenRouter, so for now it lives in Run local with our own numbers. It made me want a bigger GPU, but on the 16 GB card I actually have it does not beat what I already run, so I have not switched, the honest take is in the Run local pick. And it is dense, so if you cannot buy a card the one to wait for is the small A3B MoE Qwen will ship next, which runs fast on the hardware you already own.

The ceiling kept climbing. Grok 4.6 joined the board at a 76.8 coding index, near the top of everything tracked. Solar Pro 4 (52.7), Muse Glimmer 30B (49) and Nemotron 3.5 Lightning also went on. Qwen3.8-Max was already there at 71.8. DeepSeek also shipped a new revision of V4 Pro (0813) that jumped its coding index from 59.4 to 68.8, a big step for an already cheap open model.

GLM-5.3 dropped today too. Zhipu launched it on August 14, "built to code", and it looks strong: they claim +50% coding over GLM-5.2 and first place among open-source models on Terminal-Bench 3.0 (DeepSWE 1.1 66.9, CyberGym 84.5). The Z.ai coding plan already serves it, and GLM-5.2 and 5.1 requests now route to 5.3. But it is not on OpenRouter or the Artificial Analysis index yet, and the open weights are about two weeks out, so it does not go on the leaderboard yet. That is the rule here: a model only joins the board once it has a public coding index and a place to buy it. I will add it the moment it lands on both.

Prices moved in a few places. OpenAI turned on unlimited text with GPT-5.6 Luna for the Free and Go tiers, it went live this week. Note it is text only and the paid tiers still keep caps on Luna, the earlier claim that Luna was unlimited on Plus and Pro was wrong. Synthetic's $7 first-month promo ended, it is back to $30 a month. Mistral folded its old $5.99 Education tier into a student discount on the $14.99 Pro plan. Alibaba now shows a $2 new-user discount across its Individual token tiers.

DeepSeek is about to get more expensive. V4 Pro 0813 shipped at the old flat rate, but from August 16 (16:00 UTC) DeepSeek switches to peak/off-peak billing and the jumps are steep: cache-hit input rises up to 12x at peak, cache-miss up to 3x, output up to 4.5x, with peak on Beijing working hours (01:00-04:00 and 06:00-10:00 UTC). The cache-hit line is the one that stings, because a coding agent replays a big cached prefix every turn, so the effective bill tracks that 12x, not the headline flat price. If you lean on DeepSeek, front-load the heavy runs before the 16th, or push them to off-peak after.

Two corrections worth calling out. The "Cerebras free tier sunsets on Aug 17" note that went around is a false alarm: that date only kills preview models on the paid Developer tier, the free trial is not going away. And Morph quietly rebuilt its whole pricing to pure pay-per-token with 200 free requests a month, the old Free/Starter/Pro/Scale credit tiers are gone. Their page bot-walls plain requests, so I had to open it in a real browser to read it, now confirmed.

New this week. The free set on opencode Zen rotated: it added Hy3 and Nemotron 3.5 Lightning and dropped Ling-3.0-flash, LongCat-2.0 and North Mini Code. I added RunPod to the gateways, it has a real OpenAI-compatible inference API through its Public Endpoints, though the text catalogue is thin and it is mostly image and video. And I added this notes tab.

As always, free sets rotate and close without notice, so treat any free allowance as this week's, not a standing plan.

And one last thing: next week we bring a surprise. Stay tuned.