EdgeRock-4B: 99% Smaller KV Cache, 6.42x Throughput - the Open-Source Model English Media Hasn't Noticed

Verdict: A Shanghai-listed chip-adjacent company you've never heard of just open-sourced a 4B model that keeps its KV cache at 72 MiB no matter how long the context grows — 99.1% less than the Qwen3.5-4B it's built from — while sustaining 6.42× the serving throughput and keeping 96.3% of the source model's AIME score. The English press hasn't covered it at all (I checked — only the official repo, Hugging Face, and arXiv show up), which is exactly the window this site exists to exploit. My call: real numbers, real weights, real caveat. If you self-host long-running agent workloads, this is worth a benchmark on your own traffic this week. Don't rewrite your stack around it yet.

What actually shipped

The model is EdgeRock-4B, released October 7 by Rock Matrix (岩芯矩阵) — the AI arm of Yanshan Technology, a Shanghai-listed company — together with researchers from HKUST and CMU. Weights are on Hugging Face and ModelScope, inference and evaluation code on GitHub (634 stars in three weeks), and there's an arXiv paper (2610.10118) plus a whitepaper behind it. Chinese media picked it up today; English coverage is still zero. That's the 6–48 hour gap we hunt.

The idea is simple to state and hard to do: a bounded-state model that doesn't get dumb. Every token an agent generates lands in its KV cache; a 100K-token reasoning chain on a vanilla attention model leaves gigabytes of per-sequence history in GPU memory. RWKV, Mamba, and friends solved this decades-ago-style with fixed-size state — but they historically paid for it in reasoning ability. EdgeRock's bet is a hybrid: a recurrent state for the gist, a 1024-token local attention window for the recent stuff, and what they call Rock Memory Anchors — a fixed-capacity library of scored "anchor" records that keeps the highest-value details from earlier history (out of every 256 candidates, the top 1024 survive; the rest are evicted, per KV group).

The three numbers that matter

Verified against the official repo, not the press release:

Metric (official benchmark)	EdgeRock-4B	Qwen3.5-4B (source)
Attention KV per sequence @ 256K context	72 MiB, fixed	~8 GiB, growing
Output throughput, 32K in + 32K out, Hopper 80GB	12,124 tok/s (batch 512)	1,889 tok/s (batch 64)
AIME 2024–2026 mean pass@1	82.9%	86.1%

That middle row is the whole business case. Memory that doesn't grow means batch size can: EdgeRock runs 512 concurrent long-generation sequences on one 80GB card at 24.4 decode tokens/s per sequence, peak allocation 61.26 GiB. If you bill by the GPU-hour, aggregate throughput per card is what you're actually buying — and 6.42× on the same silicon is not a rounding error. Even single-request, it's faster than the source model: 233.7 vs 151.3 decode tokens/s at 128K in + 128K out.

Against the bounded-state competition, it's not close: the strongest prior fixed-memory baseline (RWKV-7 G1j 13.3B) scores 18.8% mean AIME; EdgeRock-4B scores 82.9% at roughly a third the parameters. Ablations in the repo show the anchors doing the lifting — remove RMA and AIME collapses from 94.1% to 0.4%.

Here's what the benchmark table quietly admits

The memory isn't free. I pulled the full 29-benchmark comparison from the repo, and three columns tell a story the launch materials won't headline:

Benchmark	EdgeRock-4B	Qwen3.5-4B	Delta
LiveCodeBench v6	61.6	75.7	−14.1
GPQA-Diamond	64.6	77.3	−12.7
HellaSwag	64.9	80.6	−15.7
HumanEval	96.7	86.0	+10.7
MBPP	87.8	71.0	+16.8

My read: the anchor mechanism trades recall-heavy and knowledge-heavy work (fresh competition-math it kept; C-Eval dropped 74.6 vs 84.0, Chinese knowledge benchmarks in general took the worst hits) for short-horizon code generation, where it actually beats its own teacher. That's a weird, interesting profile: better than Qwen3.5-4B at self-contained coding, worse at long-horizon competition code and science questions that lean on distant context. For agent loops with tool calls — where each step is short and the "memory" is re-injected as fresh input — the trade looks favorable. For single-pass, long-context comprehension, I'd test before trusting. This matches the pattern we flagged in our DeepSeek coding agents piece: agent workloads punish memory growth harder than they punish context loss.

The operational fine print nobody will screenshot

Three things between you and the numbers above. First, installation: the quick start requires a custom transformers wheel (5.7.0.dev0, bundled in the repo) — this isn't a drop-in for your current stack. Second, the GGUF packages require the custom EdgeRock RMA backend; stock llama.cpp and Ollama are explicitly not supported, and the bundled native binaries target Linux x86-64 on Hopper GPUs. There are MLX builds for Apple silicon, but the repo admits Apple Metal hasn't been tested — the functional checks ran on Linux CUDA. Third, the quantized releases (NF4/INT8/FP8/AWQ/GPTQ/NVFP4, ONNX, even LiteRT mobile previews — the packaging is genuinely thorough) each carry the disclaimer that original-model scores don't establish quantized-package accuracy. Budget an afternoon, not five minutes. If you want the contrast of a model that deploys anywhere in one command, our DeepSeek 4.1 Flash deployment guide is the other end of that spectrum.

The license has one clause worth reading

Weights ship under Apache 2.0 — commercial use, paid APIs, SaaS, fine-tuning, all fine — plus exactly one additional restriction: you may not physically hardwire the weights into silicon (mask ROM, fixed circuit topology, and the like) without written authorization. That's not paranoia; it's the roadmap. The whitepaper describes SiliconRock — EdgeRock weights baked into chips with on-die SRAM state, targeting a million tokens/s for a single user. The company's actual business is selling the silicon version of what it just gave away in software. You don't restrict chip-burning unless you plan to sell chips. For everyone not fabricating wafers, this is functionally Apache.

Who should actually try it

Not API buyers — there's no hosted endpoint at launch, and at 4B with these characteristics the interesting play is self-hosted. The buyer profile: you run agent fleets where reasoning chains run 50K+ tokens, you pay for H100/H200 time, and your effective cost per completed task is dominated by how many concurrent sessions you can pack per card. For that profile, a model that holds 82.9% AIME at constant 72 MiB is a straight capacity multiplier — the same reason on-device models like the 3GB MiniCPM-V matter on phones: memory is the constraint that sets your unit economics. If your workload is single-shot long-document Q&A, the GPQA and HellaSwag gaps say wait for the next revision. And note what this model isn't: it doesn't solve provider-side context caching economics the way DeepSeek's context caching does — that's a pricing mechanism, this is an architecture. Different problems, occasionally the same buyer.

My bet

The claims check out against the repo, the weights exist, and the benchmark losses are published rather than buried — that already puts this release ahead of most "revolutionary efficiency" announcements. The honest uncertainty is whether anchor-based memory holds up on adversarial real-world context (their own RULER-style ablation shows variable binding at 53.1%, the weakest subtask — tracking which variable is which across long spans is exactly where a 1024-anchor library would thin out). If a 4B constant-memory model with this profile survives third-party replication, every serving stack will grow an EdgeRock backend within a quarter, and the chip story stops being slide-ware. First-party numbers, my skepticism on record: promising, not proven — benchmark it on your own agent traffic before you believe any of it, including me.

Caveat (Oct 26, 2026): all numbers are from the official EdgeRock GitHub repo, Hugging Face model cards, and the EdgeRock Model License 1.0 as read on October 26, 2026 — first-party benchmarks, not independently reproduced. AIME figures are mean pass@1 2024–2026; throughput figures are Hopper 80GB, output tokens only, elapsed time including prefill. Chinese media coverage via QbitAI (量子位), October 26, 2026.

Sources: rockmatrix-ai/EdgeRock (GitHub) · EdgeRock-4B (Hugging Face) · arXiv:2610.10118 · QbitAI coverage (Chinese)