Qwen Brought Back N-Grams. All 51 Billion Parameters of Them.

TL;DR

On August 26 the Qwen team at Alibaba published Qwen3.8-Flash-Next, and called it in the model card an "experimental preview of the architecture that will underpin Qwen4." It is a multimodal mixture-of-experts model with 125B parameters that activates 6B per token, carrying a separate 51.2B parameter n-gram embedding table that is looked up rather than multiplied. Because those 51.2B parameters never do arithmetic, SGLang can park them in pinned host memory: on an H200 at TP4 that drops per-GPU model weights from 83.91 GiB to 60.45 GiB and raises allocated KV cache capacity from 1.84M to 3.28M tokens, with matched throughput moving 0.07%. Weights are up on Hugging Face and ModelScope under the Qwen Community License 1.0, which is permissive right up until you try to sell a coding assistant.

N-grams, of all things

The headline change is the one nobody had on their card. N-gram statistics ran natural language processing for two decades before transformers, and Qwen just wired 20,000,000 of them into layer 2 as a learned, hash-addressed embedding table.

Here is the mechanism. For each token, the model hashes the local bigram and trigram context into that 20M-row table, gathers 16 rows, and sums them into a 2560-dimensional vector that gets added to the residual stream at the second decoder block. At 2560 dimensions per row, the table is 51.2B parameters, about 95.4 GiB in BF16. It is a dictionary, not a layer. Think of it as the index in the back of a reference book: the model does not re-derive what "return early if" tends to mean, it turns to the page.

That distinction is the whole point. Mixture-of-experts buys you parameters at the cost of memory bandwidth on the critical path. An embedding table buys you parameters at the cost of a gather, and a gather can happen somewhere cheap.

Which is why the offload matters more than the parameter count

The SGLang team shipped day-0 support and did the obvious thing: keep each rank's shard of the table in pinned host memory, gather the selected rows into a small BF16 GPU buffer with a Triton kernel over unified addressing, and overlap that gather with the first decoder block on a dedicated CUDA stream.

On an H200 at TP4 with multi-token-prediction speculative decoding, that moved target-model weights from 83.91 GiB per GPU to 60.45 GiB, a saving of 23.46 GiB, and at the same memory fraction the freed space went straight into KV cache: 1.84M tokens to 3.28M tokens, up 78.54%. Across 1, 2 and 4 concurrent requests, matched throughput changed by 0.07% geometric mean, and four fixed prompts produced identical output token IDs with the offload on and off.

If you run models at home, read that as a design pattern rather than a Qwen feature. Parameters that only ever get indexed do not need to sit in the expensive seats, and a serving stack that knows the difference can hand the difference back to your context window.

Three quarters of the layers never see the full context

The attention stack is the other half of the story. The 48 layers run twelve repeats of a 3-to-1 pattern: three Gated DeltaNet layers, each followed by an MoE block, then one attention layer followed by an MoE block. The DeltaNet layers compress history into a fixed-size state. Only the fourth layer in each group does precise retrieval over the whole sequence.

That fourth layer uses what Qwen calls Qwen Sparse Attention, and the trick is granularity. Instead of scoring individual tokens for relevance, QSA aggregates the sequence into micro-blocks, ranks the blocks with a compressed lightweight indexer, and attends only inside the winners. The config caps the indexer at 512 blocks or 2048 tokens with a compression ratio of 4. Ranking neighbourhoods instead of tokens cuts the indexing overhead and the attention cost in the same move.

The MoE blocks are aggressively sparse in their own right: 512 experts with 10 routed plus 1 shared active per token, and an expert intermediate dimension of just 640. Native context is 262,144 tokens, extensible to 1M.

NVIDIA's numbers on a GB300 NVL72 put prefill at up to 7.6x and decoding at up to 4.9x versus full attention, and claim 8.6x the prefill throughput of Qwen3.7-Plus at a 1M-token context with a 90% prefix-cache hit rate, at over 16K tokens per second per GPU. SGLang measured 540 tok/s at batch size 1 on B200 TP4 with the NVFP4 checkpoint and speculative decoding, at an accept length of 3.3.

What it scores

Qwen's published comparisons put Flash-Next against its own Qwen3.8-27B and Qwen3.7-Plus, DeepSeek-V4-Flash-0731, and Claude Opus 4.6 in max mode. On SWE-bench Pro it reports 62.5 against 61.7, 55.8, 56.0 and 53.4. On GPQA Diamond, 91.7. On LiveCodeBench v6, 91.9. On Humanity's Last Exam, 35.9, which is the one line where Opus 4.6 pulls ahead at 40.0.

The spread is widest on the agentic evals, where a 6B-active model beating a 17B-active one by 28 points is the sort of result that deserves an independent rerun before you rearrange your stack around it.

What it costs to actually run

The Hugging Face repo is 131 safetensors shards and roughly 360 GB on disk, which is what 176B parameters plus a 4B multi-token-prediction head look like in BF16. This is not a single-GPU model, and no amount of clever offload makes it one.

NVIDIA lists DGX Station, DGX Spark and four RTX PRO 6000 Blackwell Max-Q cards as workstation targets. vLLM and SGLang both published day-0 recipes, and the model runs in thinking mode by default with three reasoning-effort levels. Qwen's own advice, which is unusually honest for a model card, is that turning reasoning effort down in multi-turn agentic work often costs you more total latency, because shallow analysis produces retries.

The license is where it gets awkward

The weights ship under the Qwen Community License 1.0, not Apache 2.0. Two clauses matter for anyone building a business:

• Attribution at scale. If your commercial product passes 100,000,000 monthly active users or US$20,000,000 monthly revenue, the model name must be displayed prominently in the product UI.
• Model as a Service and "AI Work Assistant" carve-out. If you or your affiliates run an inference or fine-tuning API, or an independent product primarily for AI-assisted coding or office productivity, you need a separate commercial license from Qwen. Internal use is exempt, provided you do not expose the model, its outputs, or its capabilities to third parties.

Read that against the benchmark table and the shape becomes clear. The model is tuned to top agentic coding and office-work evals, and the license specifically requires a negotiation before you sell agentic coding or office-work software built on it. Qwen names its own Qoder and QwenWork as the reference examples of what it is protecting.

Key Takeaways

• This is a Qwen4 architecture preview, not a point release. Gated DeltaNet plus Qwen Sparse Attention, gated residuals, n-gram embeddings and a Muon/AdamW split recipe are the pieces Qwen says Qwen4 will be built on.
• 51.2B of the parameters are a lookup table, not a layer. 20,000,000 hash-addressed bigram and trigram rows at 2560 dimensions, 16 rows read per token, 95.4 GiB in BF16.
• Parameters that do no arithmetic can leave the GPU. SGLang's pinned-host offload gives back 23.46 GiB per GPU on H200 TP4 and 78.54% more KV cache for a 0.07% throughput change.
• 6B active per token, 262K native context, 1M extensible. Three of every four layers are linear-attention layers that never touch the full sequence.
• Benchmarks are vendor-reported. SWE-bench Pro 62.5 and JobBench 55.7 are strong for a 6B-active model, and both deserve an independent rerun before they change your architecture.
• Check the license before you build a product. Coding assistants, office-productivity assistants and inference APIs need a separate license from Qwen. Internal use does not.

Sources: Qwen3.8-Flash-Next model card and license (Hugging Face), Qwen blog, Qwen3.8-Flash-Next technical report, SGLang day-0 support writeup (LMSYS), NVIDIA Technical Blog, vLLM recipe, ModelScope