How Large Language Models Work, Without the Maths

Chatbots can write essays, code and poetry, so it’s natural to wonder what’s happening inside. The answer is both simpler and stranger than most people expect.

Tokens

Models don’t read words the way we do. Text is broken into tokens, which are often parts of words. A model sees your question as a sequence of tokens and produces its answer one token at a time.

Predicting the next token

At its core, a language model is trained to predict what comes next. During pretraining, it reads an enormous amount of text and adjusts billions of internal numbers, called parameters, to get better at that prediction. To predict well, it has to learn grammar, facts, reasoning patterns and even some common sense.

Transformers and attention

Modern models use the transformer architecture, introduced by Google researchers in 2017. Its key idea, attention, lets the model weigh how relevant every earlier token is to the next one. That’s how it keeps track of who “she” refers to three sentences later.

From predictor to assistant

A raw pretrained model just continues text. To become a helpful assistant, it’s further trained on examples of good conversations and with feedback from humans and other methods, so it follows instructions, refuses harmful requests and admits uncertainty.

Reasoning models

Newer models can spend extra computation “thinking” before answering, working through steps internally. This helps with maths, coding and complex planning.

Context windows

The context window is how much text the model can consider at once. Larger windows let models read long documents or codebases in one go.

Limits to remember

• Models can produce confident but false statements.
• Their knowledge stops at a training cutoff unless they use search tools.
• They reflect biases in their training data.

Sources

• Vaswani A. et al., Attention Is All You Need (2017)
• Ouyang L. et al., Training language models to follow instructions with human feedback (2022)