I. At Long Last, We Have Created Neuralese Recurrence, From The Classic Sci-Fi Novel “Don’t Create Neuralese Recurrence”
In the public debates after the release of AI 2027, some people accused us of misleading readers by positing “neuralese recurrence”, a hypothetical technology that would allow AIs to think dangerous thoughts without getting detected.
You’ll never guess what happened earlier this month!
But what is recurrence? Is this exactly the same as the neuralese recurrence in AI 2027? And how worrying is it?
The transformer, the technology behind most modern AI, contains some number of layers (modern frontier models are probably around 100). By convention, the input layer is called the “bottom” and the output layer the “top”. When a transformer does next-word prediction, it takes the last word in at the bottom layer, spends the middle layers processing it, and outputs the predicted next word at the “top”.
What if it’s working on a hard question that needs more than 100 layers of processing? From 2017 - 2024, the answer was “you’re screwed”. From 2024 onward, the answer was: it outputs its intermediate result, as text, onto a scratchpad called the “chain-of-thought”. Then it runs the transformer again on the intermediate result. Then it repeats until it has a final result.
(Why does it need a scratchpad, instead of feeding its intermediate results directly into user output and then reading those back into itself? First, the intermediate results may be hundreds of pages long, and the user doesn’t want to read those. Second, the AI companies want to train the AIs to do the intermediate thinking properly, and this benefits from a ‘psychological’ distinction between intermediate pondering and final user output.)
This was purely a capabilities play - models are smarter when they can keep thinking instead of limiting themselves to 100 processing steps - but it coincidentally was very good for safety. The chain-of-thought scratchpad is written in English (although some Chinese models use an English-Chinese hybrid, and other AIs develop their own weird jargon). You can just read what the AI is thinking! If the AI is thinking “Better hack some websites, then kill all humans”, you can shut it down. Maybe not actually - if you have thousands of AIs writing millions of pages of scratchpad, you can’t read all of that in real-time, and will need to delegate the task to fallible AI monitors. But in theory this ought to work.
But language is slower and shallower than thought. For 100 steps in a row, the AI can shoot delicate subtle ideas from layer to layer at the speed of light. Then there’s one step where it has to encode them into twenty-six glyphs invented by Phoenician turquoise miners in 1800 BC. Then it has to re-encode the Phoenician glyphs into delicate subtle lightspeed ideas before it can do anything else. This has long been acknowledged as a bottleneck in existing transformers. So: what if they could finish their 100 layers of thinking, then send the resulting thought back to the first layer for more processing, rather than sending a text scratchpad on which they had written the thought?
This is neuralese recurrence - “recurrence” because it’s going back in a loop, “neuralese” because the thing that’s looping is the thought itself, in the native language of thought, rather than words.
The AI’s “language of thought” looks like a vector, thousands of numbers long. An “intermediate result” in this scheme might look something like (0.4, 0.1, 5, 0.443, … and so on for thousands of numbers). We don’t know how to read these. The science of reading these thoughts is a subfield of AI interpretability, which is still in its infancy. If an AI with neuralese recurrence were to think “Better hack some websites, then kill all humans”, it would look like (0.4, 0.1, 5, 0.443, … and so on for thousands of numbers), and we would never find out. This is why AI 2027 discussed it as a prelude to AIs that could escape human monitoring.
II. Pachocki Contra Specter
So did OpenAI achieve this? Their chief scientist, Jakub Pachocki, says “not really”:
What does Pachocki mean by “the depth of the computation graph . . . is within a factor of two of GPT-4”? And should this resolve alignment concerns?
Our example transformer has four green layers: a layer depth of four:
Now we add a little recurrence: instead of layer 3 going to layer 4, it loops back to layer 2, and the data go through this loop three times, then finally progress to layer 4:
Here the data are getting eight layers of processing: 1 → 2 → 3 → 2 → 3 → 2 → 3 → 4. So this recurrent transformer has a depth of eight.
In this sense, “recurrence”, despite sounding scary, is no worse than just building an eight-layer transformer to begin with, which was always an option.
That is, from an alignment perspective, suppose it takes 1,000 layers’ worth of processing to come up with an evil plot. A normal AI can think for 100 layers, but then it has to record its output on the chain-of-thought - something like “Here’s one-tenth of the evil plot I’m trying to hatch” - and we can catch it. So we want to avoid letting the model compute for 1,000 layers straight without using the scratchpad. This means don’t create a recurrent model that can loop 100 layers 10x, but it also means don’t build a 1,000 layer model to begin with.
So when Pachocki says that Astra only has twice the depth of GPT-4, he’s saying something like: look, guys, we’ve been adding layers to our AIs for years. GPT-2 had 48 layers, GPT-3 had 96, and we didn’t tell you how many GPT-4 had but let’s say it was 120. At none of these points did you complain, because it wasn’t “recurrence”, just a normal natural 120 layers. Now Astra is - let’s say - 180 layers. It’s true that we got these extra layers by recurrence rather than literally building a bigger transformer, but the alignment implications are no different. For all you know, Anthropic built a literal 180 layer transformer yesterday, and you guys didn’t bother them1. The important thing is that we not increase our layer number by some crazy amount, and we didn’t. We’re just adding layers the same way lots of other AI architectures did, albeit by other means.
How convincing is this argument?
III. God Help Us, Let’s Try To Understand Transformer Layers
When reading this, my question was - why don’t you just loop all the layers one million times, and get an AI with a million times the computational depth of the shmuck who didn’t do that? Isn’t this free capabilities?
A looped transformer model like Astra uses loops as a hack to add more layers. This hack isn’t as good as adding more layers for real. A real new layer gives the AI more “room” to store knowledge and thought styles. A looped layer just lets the AI use the same knowledge and thought style more times.
This is far from useless. A mathematician might spend years contemplating the same problem before getting it right; all through this period, he is using the same type of thinking (the mathematical knowledge and techniques his brain is capable of). But sometimes doing it for a year is better than doing it for a minute.
But it’s not optimal either. An AI with more real layers can actually be “smarter” in the conventional sense of the term. Even though it might help to give the same mathematician more time to think about the problem, holding amount of time constant, it will help even more to give the problem to a smarter mathematician with more training.
I asked Fable to estimate the relative capabilities gain, measured in “generations” (eg GPT-4 to GPT-5) from three things:
Using chain-of-thought (baseline)
Doubling the “simulated” number of layers by a recurrent loop.
Doubling the “real” number of layers.
Its answers were 0, +0.03, and +0.252, with wide variation depending on the type of task. Its headline result was that the gains from extra simulated layers are much smaller than the gains from extra real layers.
But why are we merely doubling the number of layers? Once you can loop layers at all, why not loop them a million times? Pachocki says he’s replacing something like 120 layers → chain-of-thought scratchpad → another 120 layers with something more like 240 layers → chain-of-thought scratchpad → another 240 layers, but why doesn’t he just loop the AI endlessly until it comes up with a final result?
Here is a diagram of a transformer process (source):
On each token (for example, the word “so”) the transformer looks at its existing context and runs a forward pass (the gray up arrows) through some number of layers. Then it finishes its forward pass, emits a token of chain-of-thought (the curvy blue arrows), then starts another forward pass on the new context (the old context plus the new token).
Adding more layers lets the AI think more between chain-of-thought tokens, improving the quality of each new token. But it uses extra compute. If the user is on a budget, the more compute that the AI spends on each chain-of-thought token, the fewer chain-of-thought tokens it can emit. Holding compute fixed, there’s some optimal balance between the quality and quantity of chain-of-thought tokens, and OpenAI must have decided it lay at looping 2-4x but not 1000x.
But why use chain-of-thought at all? Why not just loop unboundedly, until the problem is complete?
(on the diagram above, that would look like replacing the English-reasoning blue arrows with neuralese-reasoning gray arrows)
This would work if the AI could be trained to do it, but currently it can’t. Transformers are pre-trained on human text. They know how to use chain-of-thought partly from reading real humans’ chains-of-thought: “Hmmmm, this is a hard problem, but it looks sort of like something I’ve solved with differential equations before. Maybe if I plug in a differential equation there . . . no, that would be too inelegant . . wait, what if -”, and partly from reading enough other human output that this kind of human-style thinking comes naturally. But there is no convenient Internet text about human reasoning entirely in vectors, so the AIs can’t be trained to do it properly. In principle this isn’t fatal – all AI reasoning is in vectors, and the whole point of training is to teach them to do it well – but training an entire vector-based thought process for thousands of steps without ever bottoming out in a human-imitating chain-of-thought-token intermediate is prohibitively costly. The researchers who design transformers have to make guesses about what chain-of-thought frequency will maximize capabilities while minimizing training cost. This is part of what Pachocki was denying. AI 2027’s “neuralese” is the thing researchers currently don’t know how to do3 – train an AI to loop as often as it wants, avoiding English chain-of-thought entirely. Pachocki was saying: don’t worry, we still don’t know how to do that.
But now we can see that this is not entirely reassuring. If it takes an AI 1,000 unmonitored gray-arrow steps to devise a dangerous plot, it can get those steps either by replacing all the blue arrows with gray arrows (in which case all its steps are unmonitored, and it can plot as much as it wants), or by having so many layers that there are 1,000 steps in a single vertical forward pass in between blue arrows4. It would be prohibitive to make that many real layers, but it’s possible by looping. So (the safety community argues) even Pachocki’s looped transformer architecture is a step along that path.
All of this is starting to get confusing, but the takeaway is:
An AI with a limited number of layers can’t plot at all without having to record the plot on its chain-of-thought.
An AI that uses loops to simulate a large number of layers (“looped transformer”) can make a plot within a single forward pass, in between chain-of-thought steps.
An AI that no longer uses chain-of-thought at all (“true neuralese”) can plot at leisure, considering it for as many episodes as it needs, and we will never know.
Every previous transformer has been (1). Astra is getting part of the way to (2). The dangerous AIs in AI 2027 are (3), but so far nobody has invented this in real life.
IV. A Psychological Barrier, Not A Technical One
Having finished that digression, let’s return to the original question. Pachocki says that Astra’s recurrence barely gives it any more layers than the competition. Does that exonerate him from the charge of creating dangerous recurrent AIs?
Here the best thing I’ve read is Linchuan Zhang’s Categorical Taboos Are Much Better Than Threshold Taboos: Neuralese Edition.
Linch says: there’s an emerging taboo on creating the dangerous sort of neuralese AIs seen in AI 2027. Everyone in this discussion agrees that the taboo is correct. The only question is whether OpenAI violated that taboo.
But taboos only work when there are clear boundaries defining what is versus isn’t within the tabooed area. For example, it’s illegal in America to buy alcohol before age 21. Suppose that someone buys alcohol two days before their twenty-first birthday. Should the police arrest them? Obviously this age gap doesn’t really make a difference. We’re using age as a proxy for something like maturity, and there’s so much variation in maturity that there’s no real difference between a 20.99 year old and a 21 year old. Still, although the police might sometimes choose to overlook this, we have to at least maintain the fiction that this is an arrestable offense. Why? Suppose that we made a specific boundary - if you’re above 20.8 years old, we won’t arrest you. But then we’ve just decreased our bright line from 21 to 20.8. And we could encounter the same problem: a 20.799-year-old buys alcohol and the police have to decide whether to make an exception or not. If we’re going to draw the line somewhere, it might as well be at the place we’d already promised to draw it, told everybody that we’d drawn it, etc.
(Linch’s own example is nuclear weapons. There’s a taboo on using nukes in war. If someone uses a tiny nuke, which produces an explosion no bigger than a conventional bomb, then in some sense this doesn’t matter, because it’s no worse than a conventional bomb would have been. But in another sense, it matters a lot, because you’ve broken the taboo, and now there’s only a weaker, fuzzier taboo preventing you from using a very big nuke.)
So if we want a taboo on recurrence, there needs to be some specific taboo. When Pachocki says “Yes, we did recurrence, but it only added a couple of layers, so it doesn’t really matter,” this is analogous to the case where a drinker tells the police “Yes, I’m below 21, but only by a few months, so it doesn’t really matter”, or a despot tells the UN “Yes, I used a nuke, but it was very small.” It’s true that it doesn’t matter in real life, but it matters a lot for whether you can maintain a taboo or not.
Unfortunately, we currently lack agreement on what, if any, taboo exists. Some people argue that there should have been a taboo on looping layers (which OpenAI would have broken), and other people say that they’re making that up and that was never a taboo. There’s probably an informal taboo on true neuralese with no chain of thought at all, but it’s a complicated technology with multiple moving parts, there’s no agreement as to which moving part the taboo is on, and without that agreement, some of the parts might slip through the cracks. So what are our options?
First, contra Pachocki, it might be worth instituting a taboo on looping (adding “simulated layers”) at all, even without any taboo on adding more “real layers”. Adding real layers is expensive and generally not worth it; AI companies have only chosen to increase real depth of their models by ~20% per year; at that rate, it might take decades to reach a danger point. Adding “simulated layers”, which are much cheaper and can be jammed in hundreds at a time, is more dangerous. So maybe we should ignore the hypothetical real layer / simulated layer equivalence in favor of saying that looping is always taboo, regardless of how many real layers other people are adding somewhere else.
Second, companies could agree on some maximum number of layers (real or simulated), like 1,000. This probably wouldn’t affect real layers (by the argument above), but it would limit loops to some maximum size.
Third, researchers could hash out what the components of “true neuralese” are, and agree not to do them. This would require some conceptual foundations, but it would be the most durable and the closest to closing off the specific dangerous technology that AI 2027 was worried about.
Unlike some other vague attempts to “ban superintelligent AI” or “ban recursive self-improvement”, these taboos might stick even without strong government action: at least for now, the capability gains from breaking them seem modest5, and the dangers are particularly obvious, so they might be maintainable by voluntary commitments even while a “race dynamic” was still going on.
AI company leaders have recently put aside their differences and agreed to “pace the frontier” in various senses to be fully defined later. Their meetings will probably have a very long list of discussion items, but one more useful thing they could do would be to formalize this taboo, so we know exactly what it is we’re trying to stay away from.
If you’re interested in this topic, please also read this essay by Redwood Research, which explains some of the relevant concepts and technologies in more detail.
They didn’t. Although there was a brief period when layers were increasing pretty rapidly, they’ve since slowed down, and adding more layers to transformers the “normal” way no longer seems to be providing very much capabilities gain.
An AI researcher who proofread this post said he thought Fable’s estimates were “bad”, but as far as I know nobody has better ones, and I wanted to give a rough order of magnitude.
A looped transformer might have enough processing power in between chain-of-thought tokens to devise a plot, but why wouldn’t the plot still show up in the chain-of-thought?
There is some sense in which AIs “choose” what goes in the chain-of-thought - it’s unclear whether this is a “conscious” choice by the AIs, or more like the way humans unconsciously determine what to think - but training incentives can affect it, and an AI that was “incentivized” to hide its chain-of-thought could do so. We don’t need to worry about this in unlooped transformers with few layers, because they can’t think properly without their chain-of-thought, and if they tried to “hide” the plot from a monitor, they would be hiding it from themselves as well, and be unable to think about it properly.
If a looped transformer devised a plot on token 100 and avoided writing it down in the chain-of-thought, how would it remember the plot on token 200? All transformers use something called a KV cache, which lets them access past versions of their minds. These are layered in such a way that the 1st layer of the transformer can only read the 1st layer of the KV cache, and so on, so it can’t serve as a substitute for the chain-of-thought, and AIs can’t use KV caches to continue an attempt to devise a plot across multiple forward passes. But if they fully devised a plan by layer 356 on token 100, they would be able to remember it on layer 357 on token 200.
At least this is the current understanding; see the Claude Fable estimates above. But some researchers argue that Astra seems more capable than would be expected from what we know about its architecture, suggesting that there is some trick juicing its abilities. Pachocki has said it’s not the loops, but the community will remain doubtful until OpenAI offers more clarity into what they’re doing.