Why Transformers Need Positional Encoding For Time Series: A Visual Guide

While digging into foundation models for time series, I realized that I could not really understand them without first understanding transformers. I did not want to use these models as black boxes, so I started tracing the ideas backward, from foundation models to transformers, and from transformers to self-attention. What made the transition interesting is that although transformers were originally built for language, the core idea carries naturally to time series. The two modalities are very different, but they share something fundamental: both are sequences, and in both cases, order changes meaning.

In language, dog bites man is very different from man bites dog.

Time series are no different. A temperature of $30^circ$ yesterday and $20^circ$ today tells a different story from $20^circ$yesterday and $30^circ$ today. The values may be the same, but their order changes the meaning of the sequence.

The question is that if self-attention looks at all observations at once, how does a transformer know which observation came first, which came later, or how far apart two observations are?

That question led me to positional encoding.

What surprised me most was how such a simple mathematical idea could give a Transformer a sense of order. The exact techniques have evolved considerably since then, but the underlying problem remains the same.

This article is my attempt to build that intuition from the ground up, starting with a simple time series and following the path from raw observations to self-attention and finally to positional encoding.

From scalar observations to vector representations

Consider a simple time series containing the temperature recorded over five weekdays:

Each observation $x_t$

A simple way to do this is through a learned linear projection i.e. embedding:

giving us a sequence of vector representations: $e_1, e_2, e_3, e_4, e_5$

An embedding is a deep, abstract representation of the series in the form of a multidimensional numerical vector that encodes its features and that the model understands. [1]

Each $e_t$

The important word here is learned. The model is not given a predefined vector representation for a temperature such as $18^circ$. The parameters $W_e$

At this point, $e_t$

How self-attention builds context?

Self-attention allows each observation to use information from the rest of the sequence.

Suppose we want to update Friday’s representation. The model first creates three learned projections from every $e_t$

The matrices $W_Q$

For Friday, its query $q_5$

Each comparison produces an attention score:

which measures how relevant observation ‘j’ is when updating Friday’s representation. The scaling factor $sqrt{d_k}$

These scores are passed through a softmax function to convert them into attention weights:

Finally, those weights are used to combine the value vectors:

So $e_5$

In short:

Queries and keys learn which observations are relevant to one another. Values carry the information that is combined to form the new representation.

What happens if we shuffle the sequence?

Now comes the important question.

Suppose the same five temperature observations are rearranged.

The values themselves have not changed; only the order has. After the learned projection, we still have the same set of value representations, just rearranged.

Self-attention can still compare each representation with all the others. The same query, key, and value projections are applied, and the same kinds of pairwise relationships can still be computed.

What has disappeared is the temporal structure.

Nothing inside $e(27^circ)$ says that it originally came from Thursday. Nothing inside $e(18^circ)$ says that it occurred after $e(27^circ)$. Also, if Wednesday and Friday have the same temperature value, the learned projection will map them to the same embedding vector. Without positional information, the model therefore has no way to distinguish which embedding came from Wednesday and which came from Friday.

This is the key limitation:

Self-attention can learn which observations are related, but without an additional positional signal, it has no built-in way to know where those observations occurred in the sequence.

What should positional information tell the model?

If self-attention does not know the order of the observations, then the next question is: what kind of positional information would be useful?

At a minimum, we would want the model to know:

• Which position an observation belongs to?
  Position 2 should be distinguishable from position 20.

• Which observation came before or after another?
  The model should be able to distinguish $t-1$

• How far apart are two observations?
  In time series, the difference between $t-1$

• That nearby positions are related in a structured way.
  Position 10 and position 11 should not look like two completely unrelated identifiers.

• That the representation remains useful over longer sequences.
  Ideally, the positional scheme should still provide meaningful structure as the sequence grows.

For time series, the third property is especially useful. A model may care about an observation one step ago because of short-term dependence, or seven steps ago because of a weekly seasonal pattern.

So positional information should do more than simply assign a unique label to each timestep. It should give the model a structured representation of order and relative distance.

How can we represent position?

We now know what information is missing. The next question is how to represent it.

A simple way to represent position would be to assign each timestep a number:

But feeding the raw position directly into the model is not ideal. The values keep growing with sequence length, and a single number does not give the model a rich representation of positional relationships.

One of the original Transformer’s solutions was sinusoidal positional encoding, where each position is represented using sine and cosine functions at different frequencies.

Why sine and cosine?

Start with the simplest two-dimensional example:

As $t$ changes, the positional vector moves smoothly around a circle that allows nearby positions to have different but still related representations.

More importantly, moving forward by the same number of steps produces the same kind of change in the positional representation. For example, an offset of 7 positions has the same mathematical relationship whether we move from position 3 to 10 or from position 20 to 27. That is useful for time series because relative distance often matters:

can represent very different temporal relationships.

The full sinusoidal positional encoding extends this idea across many dimensions:

Different dimensions use different frequencies. Some change quickly across nearby positions, while others change much more slowly.

One useful way to think about this is as many clocks running at different speeds. Together, their readings give every position a structured positional signature.

So instead of giving timestep (t) only a number, we give it a vector:

that contains information about its position and its relationship to other positions.

Combining value and position

After adding positional information, each timestep is represented as:

where $e_t$

Self-attention now builds its queries and keys from this combined representation:

So when the model computes an attention score,

the comparison is no longer based only on the observed values. The query and key vectors were created from representations that already contain positional information.

As a result, the model can learn relationships that depend on both:

• what was observed, and

• where the observations occurred in the sequence.

For a time series, this means the model can potentially learn that an observation one timestep ago should be treated differently from one seven timesteps ago, even if their values are similar.

Positional encoding therefore does not tell the model explicitly which lags are important. It gives self-attention the information needed to learn which positional relationships matter for the task.

Summary

We started with a simple problem: self-attention can compare every observation with every other observation, but by itself it does not know the order in which those observations occurred.

The solution is to enrich each value representation $e_t$

Self-attention then builds its queries, keys, and values from $h_t$

For regularly sampled time series, this makes relationships such as t−1, t−7, or t−24 accessible to the model. A lag of one step may capture short-term dependence, while a lag of seven or twenty-four steps may correspond to a seasonal pattern.

Positional encoding, however, represents sequence position, not necessarily real-world time. If observations are irregularly spaced, being one position apart does not always mean being one hour or one day apart. This is where richer temporal encodings and time features become important.

Sinusoidal positional encoding is only one way to inject order into a Transformer. Other approaches include learned positional embeddings, where the position vectors themselves are learned during training, and relative positional encodings, which focus more directly on the distance between two observations rather than their absolute positions.

For time series, the problem can become even richer. Sequence position may not be enough: the model may also need to know the actual timestamp, calendar effects, periodicity, or irregular gaps between observations.

So the broader question is not simply:

How do we tell a Transformer that this is position 7?

but rather:

What notion of time does the model actually need for the task?

···

And this brings me back to where this exploration started: time-series foundation models. Modern architectures may use more sophisticated approaches such as rotary or learned positional representations rather than the original sinusoidal formulation. But understanding the simple sinusoidal construction gives us the foundation for understanding why those methods exist in the first place.

We have now built the foundation of why positional information is needed, how it is represented, and how self-attention uses it. With this intuition in place, more advanced positional encoding techniques should feel much easier to understand when we encounter them in future.

···

Note: The figures in this article were conceptually designed by the author and generated and refined with the assistance of an AI image-generation tool.

References

[1] Peixeiro, Marco. Time Series Forecasting Using Foundation Models: How to Build High Accuracy Predictive Models. Manning, 2025.

[2] Davidson, Graeme, and Lei Ma. Time Series with PyTorch: Modern Deep Learning Toolkit for Real-World Forecasting Challenges. Packt Publishing, 2026.