How AI Picks Its Next Word: A Plain-English Guide to LLM Samplers

TL;DR

When an AI writes, it never just "knows" the next word. It scores every possible word with a probability, then rolls dice to pick one. "Samplers" are the rules for that dice roll: which words are even allowed, and how risky the pick is. This guide explains the five you will actually meet, with pictures, using one running example: finishing the sentence "My favorite breakfast is ___".

The one idea behind all of it

Here is the thing almost nobody tells you: a language model does not choose a word. It produces a giant scoreboard. For "My favorite breakfast is ___" it might rate "eggs" at 30 percent, "toast" at 20, "pancakes" at 14, and so on, down a long tail of increasingly silly options like "gravel".

If the AI always grabbed the single highest-scoring word, every answer would be identical and read like a robot. So instead it picks randomly, but weighted by those slice sizes. That weighted randomness is what makes writing feel alive. Samplers are the knobs that control it. Picture them as a line of bouncers standing between the scoreboard and the dice: each one removes some options or changes the odds before the final roll.

1. Temperature: the chaos dial

Temperature is the master "how wild should it be" knob. Low temperature makes the model timid: it exaggerates the favorite and squashes everything else, so it almost always says "eggs". High temperature flattens the scoreboard and gives the long shots a real chance.

Analogy: temperature is how many drinks the writer has had. Stone sober, it gives you the obvious, safe answer every time. A couple of drinks in, it gets creative and surprising. Too many, and it starts saying "gravel". Most real use lives around 0.7 to 1.0. Below that is predictable, well above it needs a designated driver (the filters below) or it falls over.

2. Top-K: the short guest list

Top-K is the bluntest filter: only the K highest-scoring words are allowed in the room, everyone else is turned away at the door. Set K to 4 and the model may only choose among the top four, no matter how the dice land.

Analogy: a club with a strict "first 4 names on the list" policy. Simple and predictable, but dumb in one way. It always lets in exactly 4, even when the model is wildly unsure (and 4 is too few) or dead certain (and 3 of the 4 are junk). That rigidity is why the next two filters exist.

3. Top-P (nucleus): fill the room to 90 percent

Top-P fixes Top-K's rigidity by counting probability instead of names. Starting from the top word, it keeps adding words until their combined score crosses a threshold, say 90 percent, then stops. Whatever is left in the tail gets cut.

Analogy: instead of "the first 4 guests," it is "let people in until the room is 90 percent full". The clever part is that it adapts. When the model is confident, two or three words already cover 90 percent, so the list is short. When it is unsure, it might take a dozen. Same setting, flexible behavior. This is probably the most common sampler you will see, usually written as "top_p".

4. Min-P: a height bar next to the tallest kid

Min-P is the modern favorite, and the easiest to picture. It sets a cutoff relative to the top word: keep only words at least, say, 20 percent as likely as the leader. If the favorite scores 30 percent, the bar sits at 6 percent, and anything shorter is out.

Analogy: the "you must be this tall to ride" sign at a fair, except the sign slides up or down to match the tallest kid in line. When one option clearly dominates, the bar is high and almost everything else is rejected (safe). When the top options are all similar height, the bar is low and lots of words qualify (creative). That self-adjusting quality is why people pair a low min-p (0.05 to 0.1) with a higher temperature: the temperature adds spice, and min-p quietly throws out the genuinely bad picks.

5. Repetition penalties: "you already said that"

The samplers above shape variety within a single word choice. Repetition penalties work across the whole text, fighting the model's habit of getting stuck in loops ("very very very") or reusing the same phrase. They simply lower the score of words it has already used.

Analogy: a friend editing your draft who keeps saying "you used that word three sentences ago, find another". The basic version (repetition penalty, or frequency penalty) just dings repeats. A smarter modern one called DRY ("Don't Repeat Yourself") watches for repeating patterns, not just single words, and comes down harder the longer the repeated streak, which kills those death-spiral loops without flattening normal writing.

How they stack up (order matters)

You rarely use just one. They run in a line, like airport security: each station filters or reweights the candidates, and only at the very end do you roll the dice among whoever survived.

scoreboard, then penalties (drop repeats), then temperature (set the wildness), then a tail filter like top-p or min-p (cut the junk), then roll the dice

Order genuinely changes the result. Apply temperature before the filters and a low temperature can rescue a few outsiders; apply it after and the filters have already locked the choices in. The good news for a beginner: you do not need to micromanage this. Sensible defaults ship with every tool.

The exotic zoo (you can ignore these at first)

The original guide lists 20-plus samplers. Most are clever variations on one theme: cut the junk tail more intelligently. A quick tour so the names stop being scary:

• Top-A, Top-N-Sigma, Tail-Free, Typical, Eta and Epsilon cutoff: different math for finding exactly where the "good options" end and the "junk tail" begins. Same goal as min-p, different curve.
• Mirostat: a thermostat for surprise. Instead of a fixed setting, it watches how surprising each word turned out to be and constantly nudges the dial to hold a steady level of spice.
• XTC ("exclude top choices"): the contrarian. It occasionally throws out the obvious words on purpose to force fresher phrasing.
• Dynamic temperature: raises the wildness when the model is unsure and lowers it when the model is confident, automatically.
• Beam search and contrastive search: older "explore several sentences at once and keep the best" methods, mostly retired for chat because they are slow and can sound flat.

So what should you actually set?

If you just want good output and want to stop reading:

• The simple combo: temperature around 0.7 to 1.0, plus either top-p 0.9 or min-p 0.05 to 0.1. Pick one tail filter, not both.
• For creative writing: nudge temperature up (1.0 to 1.2) and lean on min-p to catch the bad picks.
• For factual or code tasks: drop temperature low (0.2 to 0.5) so it sticks to the safe favorite.
• If it loops or repeats: add a little repetition penalty, or turn on DRY.

Everything else is fine-tuning. The mental model is the whole game: the AI builds a scoreboard of every possible next word, the samplers decide who is allowed to play and how risky the pick is, and then it rolls the dice.

Key Takeaways

• AI does not pick words, it rolls weighted dice. Every next word is a probability on a scoreboard, and samplers shape that roll.
• Temperature is the wildness dial. Low is safe and repetitive, high is creative and risky.
• Top-K, Top-P, and Min-P all cut the junk tail, from bluntest (a fixed count) to smartest (relative to the top word).
• Repetition penalties stop loops by lowering the score of words already used, and DRY does it for whole patterns.
• You only need two knobs. A temperature plus one tail filter (top-p or min-p) covers almost everything, so ignore the exotic zoo until you have a reason.

Sources: LLM Samplers guide (rentry.co/samplers), Hugging Face: text generation strategies