REAP the Experts: How Cerebras Cuts an MoE in Half Without Wrecking It

TL;DR

REAP (Router-weighted Expert Activation Pruning) is a one-shot compression method from Cerebras that removes the least useful experts from a Mixture-of-Experts (MoE) model in a single pass, no retraining. Across seven models from 21B to 1 trillion parameters, it drops 25% to 50% of the experts while holding over 96% of baseline quality on coding and agentic tasks. The headline result: Qwen3-Coder-480B pruned 50% down to 246B keeps 97.6% of non-agentic coding ability and 96.7% on SWE-Bench. The code and the pruned checkpoints are open (Apache 2.0), so you can shrink a model yourself or just download one already done.

What a REAP actually is

A Mixture-of-Experts model is mostly experts: dozens or hundreds of feed-forward blocks, of which a router activates only a few per token. That is what lets a 1T-parameter model run with 35B active. But you still have to hold all the experts in memory, and a lot of them barely earn their keep. REAP asks a simple question per expert: how much does this one actually matter, and can we delete the ones that do not?

It answers with a saliency score built from two signals measured on calibration data:

• Selection weight: how often and how strongly the router gates to that expert.
• Functional impact: the magnitude of the expert's output, i.e. how much it actually moves the result when it does fire.

Multiply the two, average over the tokens where the expert was active, and you get its importance. Rank every expert, delete the lowest-scoring ones up to your target ratio, and you are done. It is one-shot: no gradient updates, no fine-tuning, just a forward pass over calibration data to measure saliency and then a clean cut.

Why pruning beats merging

The obvious alternative to deleting experts is merging them: average several similar experts into one to save space. REAP's paper argues, and its numbers show, that merging quietly breaks the model. They call the failure functional subspace collapse: when you fuse experts into a static average, the router loses its ability to dynamically mix them. The router was the whole point of an MoE, and merging takes its steering wheel away.

Pruning does not touch the survivors. The router keeps full independent control over every expert that remains, so the model that is left behaves like a smaller version of itself rather than a blurrier one. That is why, at the same 50% cut, REAP holds 94 to 96% of quality while a strong merging method (HC-SMoE) falls to roughly 59 to 65%. For generative work (code, reasoning, tool use) that gap is the difference between usable and not.

The performance numbers

• At 25% compression, Qwen3-30B-A3B and GLM-4.5-Air stay within about 1 point of baseline. Merging methods already lose 2 to 5 points here.
• At 50% compression, every model tested kept over 96% of baseline across the suite (EvalPlus, LiveCodeBench, SWE-Bench Verified, BFCL-v3 tool use).
• Qwen3-Coder-480B-FP8: 97.6% of non-agentic coding and 96.7% on agentic SWE-Bench, pruned to 246B (35B active). The released checkpoint posts HumanEval 93.9% vs 95.1% baseline.
• Kimi-K2-Instruct, a 1-trillion-parameter model, was pruned 50% in one shot, the headline proof that this scales to the largest open MoEs.

How to make your own

The CerebrasResearch/reap repo is the whole toolkit. The short version:

• Set up with scripts/build.sh (it uses uv) or the provided docker compose up --build -d, then copy .env.template to .env and fill it in.
• Pick calibration data. The repo ships a ready agentic-reasoning mix of code, math, science, and tool-use trajectories (about 24,576 samples, max sequence length 16,384). The saliency estimate is only as representative as this data, so match it to how you will use the model.
• Run the pruner: bash experiments/pruning-cli.sh "0" [MODEL] reap [SEED] 0.5 [DATASET] ..., where the reap argument selects the criterion and 0.5 is the compression ratio (use 0.25 for a gentler cut).
• It includes a memory-efficient layer-wise calibration observer specifically so you can prune large models on a single GPU instead of needing the full model resident at once.

If you just want the result, the Cerebras org on Hugging Face has pre-pruned REAPs of Qwen3-Coder, GLM-4.5-Air and GLM-4.6/4.7, DeepSeek-V3.2, MiniMax-M2, and Kimi-Linear, ready to serve (the FP8 Qwen3-Coder checkpoint wants vLLM 0.11.0 or newer).

The honest caveats

Pruning is lossy by definition. "Over 96% of baseline" still means you gave something up, and the weakest experts on your calibration mix may not be the weakest for your actual workload, so a domain shift can cost you more than the benchmarks suggest. Validate a REAP on your own evals before trusting it in production. The numbers here are Cerebras's own, on coding and agentic tasks, from a paper accepted to ICLR 2026; reproduce on your domain rather than assuming the retention rate transfers. The upside is real though: half the experts gone means roughly half the memory, which is the difference between a model fitting on your hardware and not.

Key Takeaways

• REAP is one-shot MoE compression: score each expert by router gate-value times output magnitude, then delete the lowest-scoring ones. No retraining.
• Pruning beats merging because merging causes functional subspace collapse, stripping the router of dynamic control; pruning leaves the survivors untouched.
• At 50% compression it keeps over 96% of baseline coding and agentic quality, versus roughly 59 to 65% for a strong merging baseline.
• It scales: demonstrated up to a 1-trillion-parameter model (Kimi-K2) pruned 50% in a single pass.
• Code and Apache-2.0 checkpoints are open. Make your own with the repo's pruning CLI, or download a pre-pruned model from the Cerebras Hugging Face org.

Sources: Cerebras: REAP (blog), arXiv 2510.13999 (paper), CerebrasResearch/reap (GitHub), Qwen3-Coder-REAP checkpoint (Hugging Face)