Skip to content
Optimal Lab

AI Engineering · interactive

How an LLM picks the next token

The model does not return an answer. It runs the same eight steps for every single token. Read them top to bottom, change the settings, then sample to watch one token travel through.

Change settings in the flow, then sample

1

Read the text so far

The prompt and everything written so far go in.

prompt

What is the capital of Pakistan?

output so far

The capital of Pakistan is …

2

Score every possible token

The model gives each token in its vocabulary a raw score, called a logit. Higher means a better fit.

  • Islamabad
    5.0
  • Lahore
    3.0
  • Karachi
    2.5
  • Peshawar
    1.5
  • Multan
    0.5

A real model scores ~100,000 tokens. We show the top 5, with illustrative numbers.

3

Divide by temperature

Every score is divided by T. Below 1 the gaps grow, above 1 they shrink.

  • Islamabad5.0 ÷ 1.00 = 5.00
  • Lahore3.0 ÷ 1.00 = 3.00
  • Karachi2.5 ÷ 1.00 = 2.50
  • Peshawar1.5 ÷ 1.00 = 1.50
  • Multan0.5 ÷ 1.00 = 0.50
4

Softmax: turn scores into chances

Scores can be any number. Softmax maps them to probabilities that add up to 100%.

  • Islamabad
    79.4%
  • Lahore
    10.8%
  • Karachi
    6.52%
  • Peshawar
    2.40%
  • Multan
    0.88%

P = exp(score) / Σ exp(score)

5

Cut the long tail

Top-K keeps only the K most likely tokens. Top-P keeps the smallest set that covers P of the chance.

  • Islamabad
  • Lahore
  • Karachi
  • Peshawar
  • Multan

5 of 5 tokens survive.

6

Scale back up to 100%

The survivors share the chance that was cut, so their odds add up to 100% again.

  • Islamabad
    79.4%
  • Lahore
    10.8%
  • Karachi
    6.52%
  • Peshawar
    2.40%
  • Multan
    0.88%

Faint bar = chance before the cut.

7

Roll the weighted die

Drop a random point on this strip. A wider slice is more likely to be hit.

8

Add it to the text, then repeat

The picked token is appended, and the whole loop runs again from step 1 for the next token.

The capital of Pakistan is

↺ back to step 1 with the new text

That is why the same prompt can give different answers: steps 1 to 6 are fixed maths, but step 7 is a dice roll. Temperature 0 removes the roll and always takes the top token.

Next in this track: The context window, turn by turn →

In production

Tools and technologies

Inference stacks all implement the same scoring → filter → sample loop for each token.

Not tied to one language — use these names when you search docs or design a system.