Nanbeige 4.2-3B

Not independently scored by Artificial Analysis — its base model Nanbeige/Nanbeige4.2-3B-Base is the closest proxy

A dense 3B from Nanbeige LLM Lab (BOSS Zhipin), pre-trained on ~23T tokens. Tiny — the "runs on anything" end of the table, laptop iGPU or CPU.

HumanEval pass@1 81% (13/16), via local GPU (transformers). Measured by us on this hardware, as a rough sanity check — HumanEval is a different, easier, partly-contaminated benchmark than Artificial Analysis’ composite, so it is not comparable to the coding-index column. Would not load in llama.cpp/Ollama at all — its custom 'nanbeige' architecture is unknown to them. We ran it via transformers with trust_remote_code in a GPU container, which needed an old transformers pin (for a rope_scaling change), sentencepiece, and a prebuilt flash_attn wheel. Worth it: it codes coherently and well for a 3B — the opposite of the Bonsai quants.

BigCodeBench-Hard pass@1 16% (24/148), via official protocol · llama.cpp Q4_K_M · one request at a time · a shared cluster GPU · mean of 3 runs, range 0.0 pp. The brutal counterpart to HumanEval — where HumanEval saturates near the top, BCB-Hard spreads the field, so this is the number that actually separates coding ability. does not fit the token budget; 7 of 148 answers hit the token limit (effective ceiling 95%)

75.2% ± 2.8 over 1150 problems

Q4KM · a shared cluster GPU · KV f16 · 1 run · spread between runs assumed at 0.25 points, not measured. This is the non-agentic half of BFCL: does the model call the right function with the right arguments. It says nothing about how it behaves over a long agentic conversation, which is a separate measurement. The score is BFCL’s own: the unweighted mean of four groups (simple, multiple, parallel, parallel multiple), where simple is itself the mean of Python, Java and JavaScript. Every group counts the same whatever its size, and irrelevance detection — which we also measure, another 240 problems — is not part of it. The margin follows that same weighting.

All 13 RULER tasks, one request at a time, KV cache f16, bench commit c3f5e3b4, seed 42. The score is the mean of the 13; the margin is two standard deviations of the sampling, so it grows as the sample shrinks and as the score drops.

This is not a coding score and must not be read as one. RULER asks whether the model finds and follows a fact buried in a long text. Quantising hits the two differently, so a model that still retrieves at length may already have lost the code.

~9 tok/s on RTX 4060 Ti, measured by us. RTX 4060 Ti via transformers+flash (bf16); ~1.0 tok/s CPU. Drops with length — the custom modeling's KV cache is ineffective (O(n^2)).

Won't load in llama.cpp / Ollama / LM Studio — the custom `nanbeige` architecture has no GGUF converter, so ignore the "runs on anything" first impression. Run it from safetensors with Hugging Face transformers and `trust_remote_code=True`.
- GPU: `pip install transformers==4.44.2` + the matching prebuilt flash-attn wheel; load with `attn_implementation="flash_attention_2"`, bf16.
- CPU: don't install flash-attn (it's CUDA-only). transformers still does a *static* import check, so drop a tiny empty `flash_attn` stub package on the path to satisfy it — `is_flash_attn_2_available()` is False on a CUDA-less torch, so the real import never fires. Load with `attn_implementation="eager"`, float32.
We measured ~9 tok/s on an RTX 4060 Ti and ~1 tok/s on a 24-core CPU. It's slower than a GGUF model its size because the custom modeling's KV cache is ineffective (throughput drops as the answer grows).

Q4 fits well under 6 GB with room for long context. Good as a fast draft / autocomplete model, not for hard agentic coding. Runs on Apple Silicon via MLX.