Long tutorial explains LLM positional embeddings, from GPT-2 to RoPE

Long tutorial explains LLM positional embeddings, from GPT-2 to RoPE

This is an educational article, not a new research result. Its author sets out to explain how LLMs encode and process token position information, covering classic and contemporary methods including RoPE. It combines visual examples, a little math and a small PyTorch implementation, and it ends by looking at how modern architectures adapt or omit explicit positional encodings. The article opens with a preview of which positional embedding choices appear in 2026 open-weight models, to be discussed after the underlying mechanisms.

The author gives two reasons for writing it. While preparing a new article on this year's LLM architecture trends, the author realised positional methods had never been covered in any earlier article, and chose to publish a standalone building block instead of letting that piece sprawl. Readers had also asked for years for a tutorial on relative positional embeddings (aka RoPE) as a follow-up to the learned absolute embeddings used in GPT. The article assumes some familiarity with basic LLM architectures and works as a follow-up to Build a Large Language Model (From Scratch). The author warns that it is very technical and likely too detailed for some readers, and promises the next one will return to a more comfortable style. Readers who already know how position embeddings work are pointed to Section 6, on positional choices in modern architectures.

Section 1 places position information in the overall pipeline. A tokenizer turns text into token IDs, an embedding layer maps each ID to a vector, transformer blocks modify those vectors with attention and feedforward layers, and an output layer produces logit scores for the next token. The article compares GPT-2, described as a relatively classic LLM, with MiniMax-M3, a more recent model from this year that stands in for most other recent LLMs. GPT-2 adds a position vector to each token embedding before the result enters the first transformer block. MiniMax-M3 instead adds positional information in each transformer block, using RoPE inside attention.

Section 1.1 asks why position information is needed at all. The example is a pair of sentences: "The red fox jumps over the gray fox." and "The gray fox jumps over the red fox." Swapping the colours changes which fox does the jumping. For simplicity each word is one token, so the first sentence has eight positions, 0 through 7, and a toy vocabulary of seven entries stands in for real ones, which typically have 50k to 200k entries. Both occurrences of "fox" get the same token ID, 2, and so the same embedding. In plain self-attention with no positional encoding and no causal mask, swapping any two words, such as "red" and "gray", would still produce the same embeddings and attention scores between each token pair.

GPT-style models use a causal mask, which limits each token to itself and earlier tokens. That gives some implicit order information: "jumps" can attend to "red" in the first sentence and to "gray" in the second, so a causal LLM can theoretically distinguish the sentences even without explicit positional embeddings. The author says this idea returns later under the name "NoPE". Still, GPT-2 and most newer LLMs include explicit positional information. The causal mask decides which tokens are visible, but it does not distinguish a separation of one position from a separation of ten.

Section 2 covers the simplest and most direct approach: give each sequence position its own vector and add it to the token embedding. Section 2.1 shows this in GPT-2. The model learns a table of position vectors during training, using an embedding layer with the same output dimensions as the token embedding layer so the two can be added element-wise. The article includes an excerpt of GPT-2 code in which the token embeddings and position embeddings (looked up for positions 0 to seq_len minus 1) are summed before dropout and the transformer blocks. In notation, with m as the position index, the input to the first block is the token embedding plus the position vector P_m. A toy example shows position 0, where the token embedding [1, 1, 1] plus the position embedding [1.1, 1.2, 1.3] gives [2.1, 2.2, 2.3]. These values are illustrative; in practice both tables start with small random values and are learned. This is called an absolute method because P_m depends on the token's own position m. The two "fox" tokens start with the same embedding, but adding different vectors, P2 and P7, makes them differ before the first block, so the query, key and value vectors already carry position information.

The approach has a cost. The original GPT-2 implementation has 1,024 learned position vectors, so it only handles inputs of 1,024 tokens or fewer. Extending the window means extending the table, and since the table trains like any other layer, the context length must be fixed before training starts. The table also grows expensive: in GPT-2 Medium the positional embedding layer is only 0.30% of the total weights at 1,024 positions, but at a context length of 100k, now common, it would be more than 22% of the total model weights, which the author calls very uneconomic.

Section 2.2 turns to the original "Attention Is All You Need" transformer from 2017, which predates GPT. It uses the same addition mechanism but computes its position vectors with fixed sine and cosine functions, added before both the encoder and decoder stacks. The author introduces it here because sine and cosine functions become relevant again for RoPE. The original transformer also scales its token embeddings by the square root of d_model, the embedding width, before adding the position vectors.

A postscript says the author has started experimenting with Substack's new "perks" system: supporters get image exports and metadata from the author's architecture gallery for use in their own analyses.

Key facts

  • The article explains how LLMs encode token position, from learned absolute embeddings in GPT-2 to RoPE, which MiniMax-M3 applies inside attention in each transformer block.
  • The original GPT-2 implementation has 1,024 learned position vectors, so it handles at most 1,024 tokens, and the context length must be set before training.
  • In GPT-2 Medium the position table is 0.30% of weights at 1,024 positions, but would exceed 22% at a 100k context.
  • A causal mask gives implicit order information, so a causal LLM can theoretically separate reordered sentences without positional embeddings, but it cannot tell a gap of one position from a gap of ten.
  • The 2017 transformer uses fixed sine and cosine position vectors and scales token embeddings by the square root of d_model; the sine and cosine functions return later in the RoPE discussion.

Why it matters

Word order changes meaning, yet token embeddings alone are blind to it: the two "fox" tokens in the example get the same embedding. Where and how position is injected shapes what a model can do with long inputs. The GPT-2 approach ties the context window to a fixed table that must be sized before training and grows costly at 100k tokens; newer models like MiniMax-M3 instead add position inside attention with RoPE. The author also notes the topic had been missing from earlier articles even though readers had asked for a RoPE tutorial for years.

Who it affects

Readers who want to understand LLM internals: people working through Build a Large Language Model (From Scratch), and those preparing to follow the author's upcoming article on this year's LLM architecture trends. The article assumes some familiarity with basic LLM architectures and with query, key and value vectors in attention. It is not aimed at readers who want a light overview.

How to use it

Read it in a web browser rather than an email client; the web view has a navigation bar on the left. If attention internals feel fuzzy, the author recommends the self-attention walkthrough first. Readers who already know how position embeddings work can jump to Section 6, on positional choices in modern architectures. The article includes a small PyTorch implementation, and the GPT-2 code excerpt shows exactly where the position table is added. Supporters on Substack can get image exports and metadata from the architecture gallery through the new perks system.

How solid is it

This is a tutorial by the author of the newsletter, written to explain established methods, not to report a new result or benchmark. The claims retold here are explanations and simple arithmetic, such as the 0.30% and over 22% weight shares for GPT-2 Medium. The 22% figure is a hypothetical for a 100k context, not a measurement of any real model. This retelling covers the introduction and the first sections only; the RoPE mechanism and the survey of which 2026 open-weight models use which method come later in the article and are not summarised here.

Risks and caveats

The author warns the piece is very technical and likely too detailed for some readers. The toy numbers, such as [2.1, 2.2, 2.3], are illustrative and not real learned values, and the one-token-per-word simplification differs from real subword tokenizers. The claim that a causal LLM can distinguish the sentences without positional embeddings is stated as a theoretical possibility, and the article still treats explicit positional methods as useful or even essential. No benchmark comparisons between positional methods appear in the sections covered here.

“So, explicit positional methods are still useful (or even essential), since they supply that information.”

— The article's author