Recurrence: reuse the update

A Transformer block receives numbers and produces new numbers. What if we fed that result through the same block again? The rule could stay fixed while the state it operates on changes.

That is the central idea of recurrence across depth. It gives a model another way to spend computation before choosing its next output token.

Same rule, new state

First use one number instead of a whole matrix. Start at h = 0 and repeatedly apply this chosen rule:

$$h_{\rm next}=h+0.5(4-h).$$

Round	Current state h	Update: 0.5(4 − h)	Next state
1	0	2	2
2	2	1	3
3	3	0.5	3.5
4	3.5	0.25	3.75

The rule did not change. Each application received the previous result, so it calculated a different update. This particular rule halves the remaining distance to 4. These numbers illustrate recurrence; they are not a Transformer calculation, and a general recurrent network need not converge this way.

Now replace the number with token vectors

Call the hidden state H: a matrix with one vector per token. Call a block’s weights and biases θ. Ordinary Transformer layers already pass state forward, but usually use different parameter sets:

$$H_{\ell+1}=B_{\theta_\ell}(H_\ell).$$

Here B means “apply a block,” and ℓ identifies the layer. The subscript on θ says each layer has its own parameters.

A simple shared-depth design instead uses

$$H_{r+1}=B_\theta(H_r).$$

r counts repetitions. The same θ is reused; H changes. New hidden-state values produce new queries, keys, values, attention weights, and feed-forward outputs even though the projection parameters are unchanged.

In a simplified residual form, the update looks like

$$H_{r+1}=H_r+F_\theta(H_r).$$

F calculates the contribution added to the current state. A residual addition alone does not make a model recurrent across depth: parameter reuse is the additional idea here.

These equations isolate the basic idea. Actual designs can also supply the round number, preserve a separate input representation, or decide when to stop. Universal Transformers studied shared recurrence across depth. A 2025 recurrent-depth language-model study explored spending additional inference computation on internal iterations.

Four processes that can sound alike

Process	What changes?	What happens to parameters?
Ordinary depth	Hidden state passes through successive layers	Each layer usually has its own stored set
Shared-depth recurrence	Hidden state passes through a block again	The same parameter set is reused
Token generation	A chosen token extends the input sequence	Parameters remain fixed
Training update	An optimizer changes parameter values using a loss	Parameters are updated

The token-flow explorer shows one block and a separate Append action. Appending is token generation. A recurrent-depth loop would update internal vectors again before needing to append a token.

Recurrence is also distinct from recursive self-improvement, where AI contributes to research or engineering that improves future AI systems. Reapplying a fixed block does not itself rewrite that block’s parameters or training procedure.

More computation without a new copy of the weights

Repeating a shared block adds computation without adding another independent copy of its parameters. It still costs time and working memory. Parameter count and computation performed are different quantities.

This is why recurrence belongs beside the question about scale and intelligence. We can ask how much a model improves when given more internal updates, holding its learned parameters fixed. Useful improvement depends on the architecture, training, task, and stopping rule. Looping an arbitrary block guarantees neither a better answer nor convergence.

Why the written reasoning may show less of the work

A model can perform several internal updates without writing a token for each one. The states are arrays of numbers, not sentences. This is often called latent reasoning: computation in internal representations, rather than an explicit sequence of reasoning tokens.

All Transformers already perform internal computation that is not written out. Recurrence can add more of that work between emitted tokens. A monitor that reads only the chain-of-thought text cannot directly inspect those state updates. It may still find useful clues in the text, but the written reasoning is not a complete recording of the calculation.

There are also different kinds of access. An outside reader may see only the answer or a reasoning summary. A researcher with access to the model’s internals may record its activations, but interpreting those numbers is a further problem. Neither “we have a written explanation” nor “we saved every vector” establishes that we understand the mechanism.

In An Alien Mind, published September 6, 2026, OpenAI’s Jakub Pachocki describes declining reliability of chain-of-thought monitoring. Among the reasons he discusses is stronger model capability without verbalized reasoning. That supports the broader monitoring concern; the essay does not establish whether Astra uses recurrence or disclose its internal architecture.

This gives us a more precise question: as a model uses more internal computation, what evidence lets us understand and check what it is doing? The interpretability chapter separates observing outputs, inspecting activations, and testing a proposed mechanism.

A concrete recurrence example from OpenAI’s research competition

In its May 12, 2026 Parameter Golf report, OpenAI highlighted a participant’s experiment that repeated layers 4 and 5. It introduced recurrence partway through training and gave some repeated feed-forward components separate parameters. This was an example of partial reuse in a constrained research competition, rather than sharing every parameter in a whole model.

That report documents the competition result. It does not specify the architecture of OpenAI’s deployed GPT models. The distinction matters when connecting a clear mathematical idea to claims about a particular product.