Mistral Small 3.2 24B

AA coding index 12.5 (Artificial Analysis, frozen since 2026-09-11)

Mistral's dense 24B, Apache-2.0, strong at instruction-following and function calling for its size.

HumanEval pass@1 83% (25/30), via local GPU (Ollama, Q4_K_M). Measured by us on this hardware, as a rough sanity check — HumanEval is a different, easier, partly-contaminated benchmark than Artificial Analysis’ composite, so it is not comparable to the coding-index column.

BigCodeBench-Hard pass@1 23% (34/148), via official protocol · llama.cpp Q4_K_M · one request at a time · a shared cluster GPU · mean of 3 runs, range 0.0 pp. The brutal counterpart to HumanEval — where HumanEval saturates near the top, BCB-Hard spreads the field, so this is the number that actually separates coding ability. 1 of 148 answers hit the token limit (effective ceiling 99%)

61.6% ± 2.6 over 1150 problems

Q4_K_M · a shared cluster GPU · KV f16 · 1 run · spread between runs assumed at 0.25 points, not measured. This is the non-agentic half of BFCL: does the model call the right function with the right arguments. It says nothing about how it behaves over a long agentic conversation, which is a separate measurement. The score is BFCL’s own: the unweighted mean of four groups (simple, multiple, parallel, parallel multiple), where simple is itself the mean of Python, Java and JavaScript. Every group counts the same whatever its size, and irrelevance detection — which we also measure, another 240 problems — is not part of it. The margin follows that same weighting. In Java and JavaScript this is “does not call”, not “calls badly”. We ran the corrector's own parser over every saved answer to count how many produce a tool call at all: 93% in Python but 19% in Java and 14% in JavaScript, where it answers with an explanation and a code block instead. The scores follow — 83.75 / 3.00 / 4.00. The parallel categories are the opposite case: 86% and 90% of answers do call, and score 68% and 57%, so there it calls and gets it wrong. We drive it through the prompt, which is what /v1/completions gives us; the vendor's function-calling route hands the tools over natively and Gorilla scores the model that way, so it may behave differently — we have not measured that.

~11 tok/s on RTX 4060 Ti, measured by us.

~14 GB at Q4 — a 16 GB card is the comfortable home. Good tool-calling support; respect the v3 tokenizer / template. Driven through the prompt it rarely emits a tool call in Java or JavaScript, so for those give it the native function-calling route.