Same Weights, Same GPU, One vLLM Flag. It Picked the Wrong Port.

TL;DR

A Level1Techs member posting as thr3e captured the full next-token logits of Qwen3.6-27B across a real 100K-token network-automation session and replayed it under one change at a time: the vLLM attention kernel, the KV-cache precision, the weight quant, the tensor-parallel degree, and later four "uncensored" fine-tunes of Qwen3.8-27B. Same weights, same GPU, same prompt. Switching from Triton attention to FlashAttention 2 flipped the greedy token at roughly 21% of sampled positions in one late window. INT4 KV cache hit about 47% and broke a tool call it never recovered from. The NVFP4 and AWQ 4-bit quants ran the wrong Cisco command. The thread hit the Hacker News front page this weekend with 417 points, and the companion RTX 5090 guide turns the method into a launch command you can copy.

What was measured, and why it is not the usual quant benchmark

The setup is deliberately boring: the official BF16 checkpoint of Qwen3.6-27B on an RTX PRO 6000 Blackwell, tensor parallelism 1, BF16 KV cache, a pinned vLLM nightly container (734 packages inside it, 252 of them Python), eager execution, CUDA graphs off, prefix caching off, MTP off, 2K-token chunked prefill. The workload is not a needle-in-a-haystack test. It is "Prompt 2", a roughly 100K-token transcript scrubbed from the author's own network-automation lab, with keepalives and parallel tool calls in it, which also means no quant author could have calibrated for it.

Every 32 prompt tokens the harness stores the full-vocabulary logits in BF16, then does the comparisons afterwards in FP64. The headline metric is the top-1 flip: would this configuration have picked a different greedy next token than the baseline at this position? Each 8K-token window holds 250 probes, so one flip is 0.4 percentage points. The runs are teacher-forced, so a flip does not alter the history; it marks a spot where an unconstrained generation would have branched. Part 2 then follows those branches.

One detail matters for everything below: running the same backend twice produced bit-for-bit identical logits at every position. The divergence is not sampling noise. It comes from the arithmetic.

Test 1: three attention kernels, three opinions

Qwen3.6-27B is a hybrid: 64 layers in a repeating pattern of three Gated DeltaNet layers and one full-attention layer, so only 16 layers touch the selectable backend. vLLM offers three of them for this workload on SM120: Triton attention, FlashAttention 2, and FlashInfer. The author set Triton as the baseline and changed nothing else.

For the first few thousand tokens all three agreed on every probe. Then they started disagreeing in clusters that tracked the prompt content rather than climbing smoothly with length: about 16% in the 40-48K window, back under 8%, then a peak of roughly 21% for FlashAttention 2 in the 80-88K window.

Why do three correct kernels disagree? Floating-point addition is not associative, and each kernel tiles and accumulates the matrix products in a different order. Picture three cashiers totalling the same 100-item receipt on registers that round every line to the cent: each adds the items in a different order, the pennies land differently, and when two products sit a penny apart, the order decides which one wins. Most token decisions are not a penny apart. Over 100K tokens, enough of them are.

Test 2: quantize only the KV cache

Same Triton baseline, same BF16 weights, and now only the KV cache changes: BF16, INT8 per-token/head, INT4 per-token/head. INT8 drifted to a peak near 22%. INT4 was above 13% by the 12K window, crossed 30% around 44K, and hit roughly 47% in the 80-88K window. The author let the divergent tool calls play out: the INT8 run eventually recovered, the INT4 run did not. His caption for that result is "friends don't let friends" next to a dumpster on fire, which is about right.

The relevant number for anyone who set --kv-cache-dtype to fit a longer context: this is the same model, the same weights, and the same prompt. The only thing that changed is how many bits each cached key and value keeps, and the effect grows with depth, which is precisely the regime the setting was supposed to buy you.

Test 3: five quants, one tool call

Back to BF16 KV, and now the weights change. The bakeoff compared the BF16 reference against the official Qwen3.6-27B-FP8 (block-scaled FP8 weights, dynamic FP8 activations), TheHouseOfTheDude's INT8 W8A16 (channel-wise INT8 weights, BF16 activations, GDN projections left unquantized), Nvidia's NVFP4 (FP8 for attention and GDN, FP4 for the MLPs), and cyankiwi's AWQ W4A16. Each one resolves to a different GEMM kernel inside vLLM, which is part of the point.

By the 88K window the ranking was unambiguous: INT8 W8A16 at about 30% flips, official FP8 at about 46%, AWQ and NVFP4 both at about 50%. The branches had consequences. Both 4-bit checkpoints failed to close their tool calls and botched the Cisco CLI: the correct command was show arp, they ran show run, which is the network-engineering equivalent of answering "where is the bathroom" with a tour of the whole building. FP8 and INT8 completed the correct calls.

One caveat the author flags himself: on this upstream nightly, vLLM decided the GPU path lacked native FP4 support and ran the NVFP4 checkpoint as weight-only FP4 through Marlin. That is a real deployment path for a lot of SM120 owners, but it is not a verdict on FP4 arithmetic in general.

Part 2: following the fork

Sampling 3% of positions tells you drift exists; it does not tell you what a flip costs. For Part 2 the author captured 100% of the logits around the tool calls and built a visualizer that branches the generation at every disagreement and keeps decoding both futures.

The showcase flip is a single token in a tool call targeting a Cisco router interface. Triton addressed GigabitEthernet0/0/1.201. FlashAttention 2 addressed GigabitEthernet0/1/4, a different port, then compounded it by running the wrong command again in two diverging calls (the right one was show mac address table; it reached for show run). In another branch the FlashAttention 2 run failed to set an interface description at all. The only change between those runs was the attention backend flag.

Tensor parallelism produced the strangest result in the thread: at TP1 the tool call was correct, at TP2 it failed, at TP4 it was correct again. That is the kind of outcome that makes you check whether the GPU is haunted. It is not; the author's experience says it is usually NCCL reduction order. Across the five-quant panel on this particular call, BF16, FP8, INT8 and AWQ all passed and only NVFP4 failed.

Part 3: the "uncensored" tax

Because Qwen3.8-27B shipped and the abliterated fine-tunes followed within days, the third installment ran four popular full-BF16 derivatives against stock Qwen3.8 over two recorded workstreams (SP04, 1,535 output tokens in six ranges; SP06, 4,339 tokens across prose, exact CLI and SQL, multi-tool calls and recovery steps).

• Heretic-ARA: 0.717% and 1.337% flips, zero invalid branches, and 57 of its 58 SP06 flips landed where stock Qwen was already uncertain.
• Huihui abliterated: 0.912% and 1.406%, zero invalid branches, despite its own card calling the method a crude proof of concept.
• Blackfrost: 3.844% and 4.978%, mostly still coherent.
• AEON Ultimate: 2.997% and 5.831%, 36 structurally invalid SP06 branches, and all eight of the invalid SP04 branches.

The AEON failure is the cleanest illustration of what a flip means in practice. At SP06 position 42,950 the context contained PostgreSQL port 5432. Stock Qwen chose the final "2" with probability 0.9991. AEON chose "ql" with probability 0.9158, produced 543ql, and then failed to close the function envelope. A second branch mutated a known hostname. These are high-confidence literal-copy failures inside operational commands, not style drift. The author also hashed every vision and MTP tensor against stock (333 of 333 and 15 of 15 matched), so the damage lives in the language weights.

The RTX 5090 recipe

A method is only useful if it ends in a launch command, so the companion guide scores seven Qwen3.8-27B 4-bit checkpoints over 7,114 output tokens at 18.8K to 123.6K context. cyankiwi's AWQ G32 won with 156 flips (2.193%). Unsloth's NVFP4 was second on fidelity (2.586%) but needed extra memory headroom to finish the 123K run. The worst NVFP4 export hit 6.340% and earned the verdict "dumpsterize".

Then the practical matrix on the winning checkpoint at an exact 32K context, three repetitions per cell, medians reported. A Triton/BF16 repeat of itself flipped 13 of 3,512 positions (0.370%), which is the repeatability floor everything else is measured against.

The verdict: FlashAttention 2 with BF16 KV is the highest-confidence default. FlashInfer with FP8 KV is the best capacity trade, with two caveats the guide spells out: this checkpoint ships no calibrated K/V scales, so vLLM ran unit-scale E4M3 rather than calibrated FP8, and FlashAttention 2 plus FP8 KV is simply rejected on SM120 (vLLM wants FA3 on SM90 or FA4 on SM100 for that combination). The reproducible docker launch, pinned to an image digest with pick-one substitutions for backend and KV dtype, is in the guide.

What this does not prove

The author is careful about scope and you should be too. Part 1's charts sample 3% of positions and were meant as a glimpse; Parts 2 and 3 capture 100%, but only in selected ranges. A top-1 flip measures disagreement with BF16, not wrongness: BF16 is a numerical reference, not an oracle, and a quant can diverge and still answer better. Flip rates were highly prompt-dependent, so the numbers above describe this network-automation workload, not yours. The capture tooling and dataset are not yet packaged for others to run, though the author says that is coming.

The Hacker News thread added two useful corrections. llama.cpp enforces a grammar once a tool call starts, so the unclosed-envelope failure cannot happen there, though the wrong-command failure still can. And the most common cause of a "dumb" local model is still a GGUF that dropped its chat template and silently fell back to ChatML, followed by sampler defaults that do not match the model card. Check those before you blame the kernel.

Key Takeaways

• Identical weights, GPU and prompt produced up to roughly 21% greedy-token disagreement in one 8K window purely from the vLLM attention backend, and the divergence was bit-for-bit reproducible.
• Quantizing only the KV cache to INT4 reached about 47% flips deep in a 100K-token session and broke a tool call that never recovered; INT8 KV peaked near 22% and recovered.
• Among weight quants, INT8 W8A16 stayed closest to BF16 (about 30% at 88K); the NVFP4 and AWQ 4-bit checkpoints hit about 50% and ran the wrong Cisco command.
• Tensor parallelism is a variable too: the same call passed at TP1, failed at TP2, and passed at TP4.
• Abliterated Qwen3.8 fine-tunes ranged from 1.3% to 5.8% flips on the same workload; the worst turned port 5432 into 543ql at 0.92 confidence.
• For a single RTX 5090, cyankiwi's AWQ G32 with FlashAttention 2 and BF16 KV is the measured default; FlashInfer with FP8 KV buys 75% more KV capacity at the same flip rate.

Sources: Level1Techs: Why your local LLM feels dumber than it is (Parts 1-3), Level1Techs: Qwen 3.8 quant selection guide for RTX 5090, Hacker News discussion, Qwen3.6-27B on Hugging Face, nvidia/Qwen3.6-27B-NVFP4, cyankiwi/Qwen3.8-27B-AWQ-INT4, vLLM