Your LLM Has a Curved Space of Paragraphs

I spent years studying general relativity theory since the beginning of my PhD student life. One of the things that make GR different from classic mechanics is:

Coordinates are not distances.

In GR, you write down coordinates for events. But the distance between two events isn’t a function of their coordinates alone. Instead, it might depend on the metric, which varies from place to place. This post is about an experiment I designed to look for that distinction inside a language model. It’s aimed at anyone who works with transformers and has wondered whether the standard positional encoding captures “position” the way our intuition says it should. No physics background required as the relativity discussion is an analogy.

At some point I started wondering whether anything like that distinction shows up inside a language model.

Two paragraphs that look the same

Now consider two texts:

I am no bird and no net ensnares me. I am a free human being with an independent will.

and

I am no bird and no net ensnares me. I am a free human being with an independent will.

A tokenizer would see these as nearly the same input — same words, same order, with only a newline when the paragraph break falls. But you don’t. The second version has a pause, a change of local context, a sense that what comes next is a new unit. It feels like the two sentences are further apart than their token distance suggests.

The question I wanted to ask was narrow: if you give a transformer an explicit paragraph coordinate — separate from the token index — does its attention change somehow? And if it does, does the change have a shape?

Position as a coordinate, and why that’s lossy

A standard transformer encodes position as a single number: the token’s index in the reading order. RoPE rotates each channel pair by an angle proportional to this index. ALiBi adds a bias proportional to it. Different mechanisms, same object. They work because reading order determines token distances, and token distances are what attention cares about.

But for us human readers, text isn’t a line. It’s a tree or hierarchy structure:

document ⊃ paragraph ⊃ sentence ⊃ token

Two tokens at the same reading-order distance can be in the same sentence, in different sentences of the same paragraph, or in different paragraphs. Taking n tokens, the same sequence admits 2^(n−1) paragraph segmentations.

So the standard positional encoding might be lossy: it tells the model where a token is in the sequence, but there is nothing to with the hierarchy structure. A trained transformer might recover some hierarchical structure implicitly from content while the positional representation does not have that information

I wanted to know what happens if we do give it that information.

hRoPE: three coordinates instead of one

I built a hierarchical rotary positional encoding, hRoPE, that gives paragraph, sentence, and token indices their own independent channels. Each level rotates its own block of dimensions, and the blocks don’t interact.

The point here is separability. I can change the paragraph coordinate of a token while holding everything else — its sentence index, its token index, the entire token sequence — exactly fixed.

I also trained several variants, matched in parameter count, depth, schedule, and seeds. Two are the controls that matter:

• rand_axial: identical to the hierarchical model, but the paragraph coordinate is a density-matched random label, resampled every training step so it can never be memorized.

• period_axial: identical, but the paragraph coordinate is a mechanical periodic grid at the same density as the true segmentation.

rand_axial isolates having an extra coordinate from that coordinate carrying real information. period_axial isolates a coordinate correlated with token distance from one tracking genuine boundaries.

The intervention

Now let us do two interventions:

• fake-merge: assign two adjacent real paragraphs the same paragraph index p1​. The boundary between them disappears from the model’s view.

• fake-split: assign two halves of one real paragraph different p1​ values. A boundary appears where there wasn’t one.

In the two settings above, we keep same tokens, same order, same token-to-token distances, same sentence boundaries. So, if attention changes, it changed only because of p1.

To measure such change, we average attention over layers and heads, take logs. We fit the resulted average a linear trend against token distance, and look at the residual, which is, the part of the attention pattern that distance alone doesn’t explain.

The result is the same in all three corpora (WikiText-2, OpenWebText, and Python source code). Removing a real boundary lowers cross-paragraph attention. Inserting a fake boundary raises it. The two interventions push attention in opposite directions and they prove together that the paragraph affects the attention.

The control that ruins the story

The model whose paragraph index is a random one rand_axial, also changes cross-paragraph attention, in every corpus. The change is evidently weaker than the hierarchical model’s, but it is there.

So a change in cross-paragraph attention is not by itself a signature of paragraph structure. Any channel with the right density induces some change. My story does not end here.

Depth is the signal

For every pair of tokens that sit in different paragraphs, we compute the extra suppression their attention receives — beyond what token distance alone predicts. Then we plot that suppression against paragraph displacement: how many paragraph boundaries separate the two tokens.

For the real model, the suppression grows with displacement up to about 2–5 paragraphs, then shrinks again — a well. The depth of that well, the most negative point on the curve, is what we call U*.

The table below gives U* for three corpora. Negative values indicate compression. Both models compress in every corpus; what differs hrope_axial from rand_axial is the depth, and the depth is corpus-dependent for the real model in a way the control’s is not.

Two of three corpora show a clear depth gap: Code and WikiText-2, where the hierarchical model compresses deeper than the control. OpenWebText is an exception, and I’ll come back to it. What matters beyond the two individual gaps is that the depth itself behaves differently across corpora in the two rows. The control’s depths sit in a narrow band — −0.22 to −0.33, a range of about 0.11. The hierarchical model’s span −0.31 to −0.77, a range of about 0.46. The depth is corpus-sensitive for the real model and nearly corpus-blind for the control. That contrast, not any single row, is the first hint that depth carries information.

All three corpora show compression ($U<0$) rather than a mix of compression and dilation

The causal test

Cross-corpus correlation alone cannot be evidence. So I ran a within-corpus test: shuffle the order of paragraphs in each document with same content, same lengths stay, and the same number of paragraphs stays. Only their order changes.

The logic is: if depth reflects genuine paragraph structure, shuffling should move it. If depth is an artifact of the channel’s density, shuffling should do nothing.

The result turns out:

• hrope_axial: depth changes in every corpus. Not always in the same direction — Code gets shallower, WikiText-2 and OpenWebText get deeper — but the effect is real and reproducible.

• rand_axial: depth changes in no corpus. Every confidence interval includes zero.

The shuffle changes the input for both models equally while the difference is what the two models do with the paragraph coordinate. hrope_axial was trained on real paragraph order, so shuffling that order moves its depth. rand_axial was trained on a coordinate that was random from the start, so there’s nothing for the shuffle to disturb.

What doesn’t work

I also tried to explain the corpus-dependent depth using corpus-only statistics — computed from raw text, no model involved. Three constructs, eight quantities: lexical persistence, paragraph length, embedding-based coherence.

None fully reproduces the ordering of U*.

Lexical persistence gets Code right and swaps the other two. Paragraph length swaps Code and WikiText-2. Embedding coherence comes closest — two of three corpus pairs — but leaves one unresolved. It’s the candidate I’d recommend following up, not a complete explanation.

This is a negative result and I’m reporting it anyway. The corpus-dependent depth is real and causally tied to paragraph structure. What in the corpus determines that depth is open and I am open to any discussion.

The honest caveat

As shown above, OpenWebText doesn’t show a depth gap. Real and random are statistically indistinguishable there. I can’t separate a true null from a power limitation at the seed count I used. It’s a corpus where the depth comparison isn’t resolvable — not one where the causal effect disappears. The shuffle test still distinguishes real from random there.

And for the scale: eight-layer models, 512 dimensions, three seeds, three corpora, I have limited resource. The phenomenon is reproducible at this scale. Whether it persists at production scale is not something this paper answers.

What I think this means — and what I don’t

Here’s where the GR intuition comes back.

In GR, coordinates and distances are different objects. You label events with coordinates, but the metric — the thing that tells you how to measure separation — is a function of position, not a fixed rule. Two events with the same coordinate difference can have different proper distances.

The situation I think I’m looking at is structurally similar. The token index is a coordinate, exact for the linear sequence. But the “distance” that attention computes over doesn’t reduce to coordinate difference: two tokens at the same coordinate separation attract different amounts of attention depending on their hierarchical relationship.

In the physics vocabulary: the positional coordinate is not the metric. The metric depends on the coordinate and on the hierarchy. What I’ve measured is a piece of that dependence — specifically, how attention weight varies with paragraph displacement, holding token distance fixed.

And once you’re in that frame, “potential well” stops being a metaphor. A well is what you get when you plot effective interaction against separation and find a minimum. That’s what the compression curve looks like: a well in paragraph-displacement space, whose depth is the signature.

The strongest statement I can defend is this: attention exhibits a structured, coordinate-dependent response field over hierarchical textual position, whose qualitative shape is consistent with a metric-like dependence. The well is real in the sense that it’s reproducible. It is not real in the sense that I’ve derived a geometry.

So, the honest analogy isn’t “transformers are relativistic.” It’s narrower: the coordinate that standard positional encodings supply is not the same object as the distance that attention actually computes over.

What I’d want next

Two directions:

1. Scale. Does the depth signature persist at 1B+ parameters?

2. More corpora. Three shows the pattern exists and is causal; it isn’t enough to say what determines the depth.

The paper is on arXiv: 2609.23551. I’d especially like feedback on the OpenWebText null.