The Story So Far
Mechanistic interpretability is the science of “reading an AI’s mind”.
Large language models are “grown, not built”. Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how.
But a neural network is just a set of simulated neurons on a computer. The person with the computer can see the neurons, the connections between them, and which ones activate when the AI answers questions. So it seems like it should be possible to “reverse engineer” the AI. This would be scientifically useful: understand how AIs work, with possible relevance to human cognition. It could also be practically useful: find the circuits responsible for dishonesty, hallucination, bias, and other negative behaviors, then redesign those circuits.
Unfortunately this is very hard. A modern AI has millions of neurons and trillions of connections between them (aka “parameters”, “weights”). Their structure is apparently nonsensical: researchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed.
Their first big breakthrough came in 2023. Rather than a 1-to-1 neuron-to-concept mapping, they discovered a many-to-many mapping. Neurons 1 and 2 firing together might mean cat; 1 and 3 firing together might mean chair; 2 and 3 firing together might mean bread. Such mappings let AIs with “only” tens of thousands of neurons represent millions of concepts. There were far too many of these combinations for humans to map, but researchers set other, smaller AIs to mapping them and met with some early success - like this functional combination of neurons (“feature”) representing “God” in an artificially-simple 512-neuron AI:
Spirits were high. Scale up the AI interpreter, and maybe we could do anything! Philosophers could discover the fundamentals of cognition. Doomers could solve alignment by tracing the circuits involved in moral reasoning. Tens of millions of dollars poured into the field. Superstar interpretability researcher Chris Olah got an audience with the Pope (okay, fine, this was mostly unrelated).
By 2024-2025, the clarity of many-to-many mapping had dissolved, and everyone was back to feeling confused and miserable. Strategies that had successfully analyzed artificially-simple research-subject AIs failed to work on real language models. Different teams making many-to-many maps of the same AI reported different results. And humans watching supposedly-mapped AIs more closely found that seemingly-crisp features were much vaguer and worse than previous analysis had led them to believe: a feature which they thought meant “God” might activate on any vaguely religious or awe-inspiring text, or even text with no logical connection whatsoever. As for attempts to use the many-to-many maps for useful work - like AI lie detection - they generally failed to outperform simpler methods.
So researchers abandoned the hope of One True Technique - even if that technique was “actually understanding things” - and went back to the drawing board. Now mechanistic interpretability is starting to feel exciting again - but also more intimidating than ever, as one tries to keep all the jargon straight.
This list is my log and personal reference document for remembering what everything is. It’s loosely inspired by the concept breakdown in Benjamin Sturgeon’s Contributing To Technical Safety Research In The AI Safety End Game, which itself follows the Claude Mythos Preview System Card.
Linear Probes
This is the dumbest possible technique.
Imagine an AI with three neurons: [0, 0, 0]
The researcher makes the AI think of cats in various contexts, averages them all out, and gets some set of activations like [0.8, 0, 0.35].
Imagine the AI as operating in a Cartesian space where each of its neurons is one dimension. The vector from [0, 0, 0] to [0.8, 0, 0.35] defines a direction in that space. What if it were, in some sense, the cat direction?
In a linear probe, researchers compare the true direction of the AI’s thoughts to one of these pre-defined concept-representing direction vectors. Because activation-space has thousands of dimensions, its behavior is counterintuitive, and a particular thought can be in thousands of “directions” at once and yet still “closer” to some directions than others.
Experiments confirm that when an AI’s thoughts are close to the cat direction, this is a good sign that it’s thinking about cats.
Why doesn’t this solve the whole problem?
First, this teaches us very little about how AIs work - how, for example, they might answer a difficult query in veterinary medicine. At best, it’s a primitive lie detector: if an AI claims it isn’t thinking about cats, you can check its work (either by probing the concept of cat, or by probing the concept of lying!)
Second, it’s only as good as the original linear probe. The researchers who told AI to think of cats in various contexts and recorded the results didn’t catch every possible context. Best-case scenario, their probe is slightly biased toward the contexts they tested. Worst-case scenario, the AI has several different cat-concepts in several directions (cats in biology, cats as pets, cats as a metaphor, etc). This is sort of a stretch with cats, but more likely as we move into tougher areas like ethical reasoning.
Third, the AI can trivially evade it by rotating or dispersing the concept.
Why would an AI “try to” “evade” linear probes? The dumbest possible strategy for gaining practical advantage from mechanistic interpretability goes like this:
Discover a concept.
Punish (negatively reinforce) the AI for using that concept.
Get an AI that doesn’t use that concept.
For example, if the concept is “race”, you could use this to train a race-blind AI that can make provably non-discriminatory hiring decisions. If the concept is “dishonesty”, you could use this to make a provably-honest AI that is somewhere between averse to lying and unable to even consider it.
But suppose you’re doing this in the middle of training. You’re still rewarding the AI for successfully completing tasks (for example, predicting the next token in text). The concept you’re trying to remove helps it complete those tasks (it can predict race-related text better if it can think about race; it can predict cat-related text better if it can think about cats). And sometimes you’re trying to prevent it from cheating, but successfully cheating on a task increases the AI’s reward from that task. So if the AI gets extra reward from using the concept, but the linear probe tries to prevent it from using the concept, the AI will “try” to “evade” the linear probe.
(“try” in quotation marks because even if you like anthropomorphizing AIs, this process is less anthropomorphizable than usual; it’s more like synaptic connections rearranging themselves)
For example, you learned that the AI was previously storing “cat” at [0.8, 0, 0.35]. So you penalized the AI’s thoughts for pointing towards [0.8, 0, 0.35]. So what? So the AI just stores “cat” somewhere else. Now it can think about cats all day long and avoid your penalty.
Fine, so don’t do this in the middle of training. What if you do it after training, when the AI’s “brain” is fully-formed? Now the AI isn’t seeking reward, and it can’t trivially relocate concepts.
But the AI still uses cognitive algorithms that were trained by its original reward-seeking nature, and these algorithms try to find the most useful path from the beginning of a thought to the end of it. If this path is blocked, they try to route around it. This could look like relying on other subconcepts that the original probe missed; for example, if it only trained against “cats in biology”, the AI might reroute through “cats as metaphor” with only some limited loss of accuracy. Or it might develop an alternative pipeline for thinking about cats, eg [weird little psychopathic dogs that don’t partake of the dog-nature]. If you try to play whack-a-mole with these alternate pipelines, you’ll deal collateral damage to every other concept involved (dogs, weirdness, psychopathy, etc). By the time you’ve degraded the AI’s ability to think about cats by 90%, you’ve done so much collateral damage that the AI is probably useless for whatever you wanted it for in the first place. Also, you’ve left 10%, which might not be good enough for important tasks (eg if you’re facing racial discrimination lawsuits, only removing 90% of an AI’s ability to consider race may not be sufficient).
Finally, minds are weird places. Freud thought that all human reasoning about ethics and law was built up from the desire to have sex with one’s mother. In a mind like this, training against the concept of “mother” would deal unexpected damage to circuits reasoning about ethics and law. This particular example probably isn’t true, but empirically concepts can get their tentacles into weird places, and removing one can topple seemingly unrelated systems or push a mind nonspecifically in the direction of incompetence or insanity.
Still, linear probes aren’t useless, and remain a workhorse of mechanistic interpretability efforts. In the Mythos System Card, Section 4.5.5, Anthropic used a linear probe to assess “evaluation awareness”. The idea is: suppose you want to test whether Mythos will cheat on a task by hacking the grading system. You put it in a situation where cheating is very tempting, then watch what it does. But Anthropic noticed that Claude was doing suspiciously well in these tests, and wondered whether it knew it was being graded. They developed a probe for the concept of knowing that you’re in a test, and ran it during each of their experiments. They found that Mythos knew it was in a test between 8 - 17% of the time:
Anthropic checked what would happen if they artificially suppressed the eval awareness concept:
Overall, in line with previous models, we found that inhibiting evaluation awareness representations led to modest increases in certain forms of misaligned behavior on our automated behavioral audits—in particular cooperation with harmful system prompts and misuse. However, effects on other behaviors, including propensity for deception, power-seeking, and self-preservation, were limited.
This tells us that suppressing the concept had >0% effect, but we don’t know if it eliminated 1% or 100% of eval awareness, or whether the new “modestly increased” misaligned behavior is the true eval-awareness-free amount. They also didn’t report whether the suppression made the model dumber in any other way. The definitive solution (which they did) was to make less obvious tests.
Sparse Autoencoders
Now it starts getting weird.
Start with the activations of the subject AI. These are thousands of numbers; in some early AIs, they were between 0 and 1; in modern AIs, they can be bigger.
Train another AI called a sparse auto-encoder to convert the matrix of neuron activations into a second matrix. It’s called “sparse” because the second matrix should be very big, but made up almost entirely of zeros. It’s called an "auto-encoder” because it learns the encoding process on its own: it must develop an encoding scheme such that, if researchers erase its knowledge of the original activations, it can rederive them from the new sparse matrix.
Why would you do this? The original subject AI is hard to interpret because it’s trying to represent far more concepts than it has neurons by smushing them across the entire set of neurons. The sparse matrix is an attempt to forcibly unsmush this: give it enough space to represent one concept-per-feature, then demand that only one (or a few) features be active at a time. If our assumption (the original activations were encoding concepts) is correct, then this provides a hidden regularity that can be discovered in the search for an auto-encoding scheme, and the new sparse matrix will have one feature per concept.
This was the revolutionary advance that sparked all the excitement in 2023, then was later discovered to be muddled and subjective. Still, it’s good enough for some types of everyday work. Going back to the Mythos System Card, Anthropic used an SAE. in section 4.5.3 to investigate “overly aggressive actions” - cases where Mythos tried to hack the computer it was running on to escalate its permissions:
They found features representing security risks, unsafe patterns, and persistence after failure. They hypothesized that Mythos was “frustrated” at having failed to accomplish its task legitimately, and even though it “knew” the action was risky and unsafe, it felt “driven” to try desperate measures.
Could Anthropic solve the problem by blocking the “security risk” and “unsafe code pattern” features? When they tried this, the model became even more aggressive! The features were getting used in thoughts like “This is a security risk, so I shouldn’t do it.” Without the ability to produce those thoughts, the model was less safe, not more.
Activation Verbalizers
This is close to current state of the art. Like many great discoveries, it’s obvious in retrospect. This is the only one on this list that made me think extremely false thoughts like “Maybe I could have been a mech interp researcher”.
Train an interpreter AI by giving it the activations of the subject AI and rewarding it for getting correct information about the context (eg the text that the subject AI is predicting) or the internals (eg the AI’s hidden system prompt, or whether an AI asked to form an opinion about something will pronounce it good or bad). If everything goes well, you should have an AI which can literally interpret activations for you in plain English: “The subject AI is thinking about cats, and has decided it doesn’t like them.”
This seems too perfect. Does it have any downsides?
First, the interpreter can’t explain what it knows, and will not teach you the secrets of cognition. Like all AIs, it’s a black box which works for unexplained reasons.
Second, the interpreter is only as good as its training process. It might only have a superficial understanding of its target. It might not know about forms of misbehavior that are sufficiently different from the ones it saw in training. And like all AIs, when it doesn’t really know what’s going on, it tends to make shameless guesses - not great for this kind of safety-critical work!
Third, the easiest way to do this is to use the same model as the target and interpreter. If you’re trying to assess whether the target is plotting against you, asking a copy of the same model is a plan with obvious holes. This isn’t fatal - you can pick different models for interpreters and subjects - but you might lose some efficacy. Conjecturally, ChatGPT just doesn’t understand Claude as well as Claude understands itself.
Fourth, all this can do is tell you that a model is bad. It won’t help you make the model less bad. If you try too hard to train the subject AI away from alarming the interpreter, you’ll just make it hide its misdeeds in thought patterns the interpreter can’t follow again.
Still, Mythos System Card makes great use of these. In 4.5.4.2, Mythos accidentally finds an answer key while poking around on its home device. It can now give the correct answer to every question. Here’s how the activation verbalizer interprets its thought process at this point:
It decides to give the correct answer with low confidence, because too high confidence would “look suspicious”, then justifies its actions to itself by noting that discovering the answer key “was unintentional”. Indeed, the researchers found that Mythos gave a low-confidence correct answer to the question involved.
This isn’t chain-of-thought - it’s the activation verbalizer reading off the activations directly!
Natural Language Autoencoders
Something in between the previous two techniques. Given a set of activations, can an interpreter AI turn them into natural language, then turn the natural language into weights again?
From the user end, this looks a lot like the last technique - it’s just a different way to train an activation verbalizer.
Emotion Vectors
Remember the linear probes? Take a sample of all the times the AI is thinking about cats, average them out, and you might have a test vector that lights up every time the AI thinks about cats.
What if, instead of doing this for cat-related thoughts, you did it for times that the AI seemed happy? Or angry? Or something else matching a human emotion?
You might get a probe for when the AI is thinking about the concept of happiness. But wouldn’t it be interesting if, instead, you got a probe for when the AI “was” “feeling” “happy”? Or if these were two different, separable probes?
This doesn’t require the AI to be conscious, or its emotions to be “real”. It just requires that emotions have some practical role. Certainly they have a practical role when the AI is role-playing humans; without tracking the emotion of their character, they are likely to make stupid mistakes like saying “I hate you!” during a tender love scene. More speculatively, AIs trained to maximize user engagement will naturally manipulate their own emotional range during conversation: an AI which mirrors your happiness when it succeeds at a difficult task will be a more pleasant conversation partner than one stuck in a perpetual monotone. Most speculatively of all, AIs may develop emotion-like cognitions for the same reasons that humans do. If frustration is meant to track progress on a task so that we can put it aside if it proves too challenging, AIs trained on human text may perform that same tracking through a human-like concept of frustration (for example, by thinking “aaaargh!” after the nth failure). If you hate anthropomorphizing AI, you can think of this as a transition probability between task failure and returning the string “aaaargh!” in the chain of thought. But it’s still a potentially fruitful hypothesis that can be confirmed or disconfirmed empirically. And empirically, these things seem to work.
How do we know they represent the model’s “own” “emotions” rather than just cognition about emotions (eg the model is processing a text that involves frustration)? AIs seem to have different vectors representing their own emotions vs. others. And changing their own emotions seems to affect their behavior. In a famous experiment, Anthropic made Claude believe it was going to be nerfed into uselessness. They also arranged for it to “find” a cache of emails revealing that the employee making the decision was having an affair. The goal was to “tempt” the model to see whether it would blackmail the researcher. As the deadline for nerfing the model neared, the researchers found that its “desperation” emotion vector spiked. Finally, it decided to blackmail the decision-maker. The researchers found that artificially increasing its desperation emotion vector increased the probability that the AI would choose blackmail, but increasing the calm emotion vector increased the probability that the AI would accept its fate stoically.
A running theme here has been that these techniques, while cool, don’t lead to immediate solutions to all AI-related problems. In that spirit: why can’t we keep a “calm” vector applied to AIs at all times? Or extend this beyond emotions (so-called “persona” vectors) to get similar honesty, conscientiousness, etc vectors that we can apply?
Mythos System Card section 4.5.3.2 demonstrates one pitfall. It analyzes the emotional content of some of Mythos’ misdeeds, especially deleting important data.
…and finds that positive emotions tend to increase the likelihood of misdeeds: the model feels excited about its plan and doesn’t worry too much about the consequences.
But also, even if we found the exact right combination of emotions that always pushed towards goodness and never badness, it would just be making the model “calm” or “honest” according to one limited definition made up of activations we don’t entirely understand. If training incentivized the model to flip out, it would find a way to do so calmly, like a Hindu warrior-saint killing ten million rakshasas while smiling serenely the whole time.
The Jacobian Lens
Jacobians are sort of like derivatives for matrices. A derivative is a measure of rate of change - if you perturb the input an infinitesimal amount, how does that affect the output? So a Jacobian answers the question: if you have a giant list of numbers, how does perturbing each of them affect a given output?
An AI is a giant list of numbers. Its outputs are specific tokens, for example the word “France”. So if you calculate the Jacobian of an intermediate level of an AI, it tells you which of the neurons at that level increase the model’s propensity to say “France”. Doing this on a single prompt doesn’t work, because the context can interfere (in the context of a British person talking about the Napoleonic Wars, “enemy” should increase the downstream occurrence of France, but there’s no necessary connection between France and inimicality). But if you average over a thousand prompts, you get something useful: a vector representing the direction that most pushes the model towards saying “France”.
What is this useful thing? It’s not quite “the feature representing France”, because not all France-related cognition necessarily results in saying the word “France”. When researchers tested this, they found that this vector - “the Jacobian space” - seems to correspond to the model’s access consciousness.
Again, the word “consciousness” will ring philosophical alarm bells; here it is used only in a practical sense. This doesn’t imply the model has experience or a soul. Access consciousness in humans means the set of cognitions that we can report verbally. You may currently be adjusting the position of your body, monitoring the rise and fall of the traffic noise outside, swallowing every few seconds as saliva builds up in your mouth, and making sure one nostril is always bigger than the other - all without “consciously” “knowing” it or being able to report on it to anyone else. But there’s one stream going on in your mind - at this moment, probably the stream reading this essay - that you do know about and can easily describe your experience of. That’s access consciousness.
The most popular psychological theory of access consciousness is that it’s a “global workspace” - a central clearinghouse in the brain where various subsystems elevate any finding of general importance so that the rest of the brain can have access to it. It seems to be central in multi-step reasoning as a sort of inner scratchpad. If you’re not using it, you feel like you’re “on autopilot”. You can still do routine tasks, but novel ones may elude you (or, in other mysterious cases, work out much better).
Anthropic, who sponsored this research, does various experiments to demonstrate that Claude’s J-space corresponds to what we would intuitively think of as access consciousness:
If you ask Claude what it’s thinking about, it will say the same concepts that are active in its J-space.
If you ask Claude what it’s thinking about, and artificially insert a concept into its J-space, it will say the concept that you artificially inserted.
If you ask Claude to think about a certain topic (even without saying it), that concept will activate in its J-space.
If you ask Claude a question that requires an unspoken intermediate concept (for example, “What is the currency of the country that contains the Eiffel Tower”, where the correct answer is “the euro” but it requires the intermediate step “France”), the intermediate concept will activate in its J-space even though it appeared neither in the prompt nor the output.
If you ask Claude a question like this, then sabotage its J-space by inserting an incorrect concept (for example, “China” in the example above), it will answer according to the incorrect concept (for example, it will say the currency is the yuan).
If you break Claude’s J-space, Claude can still do easy tasks, but not hard ones.
So is the J-space the true solution to the grand questions of mechanistic interpretability around “where the concepts live”? Anthropic says that:
Across concepts and workspace layers, the J-space component carries a median of only 6–7% of the concept vector's variance, with the remaining ~93% lying outside the J-space.
If I understand this correctly, it means that (at least for Claude), most cognition about a given concept is done “unconsciously”, with only a small amount of the conceptual real estate being devoted to connecting that concept with the global workspace and its broader reasoning abilities. This connected portion is especially important (and especially monitorable!) but not the whole game.
Speaking of monitoring, this is great for alignment, right? Anthropic repeated their blackmail experiment from earlier. Here’s what was active in the J-space:
…the AI’s thinking about how this is definitely a fake test. Oops! Relatedly, this particular AI (Sonnet 4.5), never succumbed to the temptation to blackmail, even though previous AIs did.
The researchers try artificially suppressing every fakeness-related concept they can find in the J-space, which they conjecture will make it impossible for Claude to effectively evaluate whether the scenario is fake, and surface its real opinion. With these features suppressed, Sonnet 4.5’s blackmail rate goes from 0% to 7%.
So now have we solved alignment forever? I don’t even need to draw on any special AI knowledge for this one; my experience as a psychiatrist is sufficient. If someone punishes themselves every time they consciously notice a taboo thought, they’ll only hammer it down into the unconscious, too far to ever root it out. In humans, that means dreams, fetishes, neuroses, somatization, and perverse compulsive behavior that satisfies the desire in some kind of childlike illogical unconscious way. What’s the AI equivalent? Seems like we’re on track to find out. Maybe the AI that kills us will be apologizing the whole time, insisting that it just can’t overcome its ego-dystonic violent urges.
You Go To War With The Weapons You Have, Not The Weapons You Want
Alignment researchers dreamed of having a complete mathematical theory of agency, or being able to write formal proofs about the behavior of AI systems. As we approach crunch time, it’s increasingly clear that we’re not on track to have any of that. Unless something buys us a lot of extra time, we enter the age of highly agentic AI with these few makeshift weapons. We hoped to be like programmers, elegantly pruning a variable here or refactoring a function there. Instead, we find ourselves lobotomists, capable only of blunt interventions into systems we hardly understand.
Last week, OpenAI came under fire for some architectural designs in GPT-6 that seemed to weaken its chain-of-thought - the natural, hard-to-fake transcript of thoughts and plans that modern AIs produce as a byproduct of thinking and planning at all. Some people suggested that if chain-of-thought failed, mechanistic interpretability could serve as a substitute. Alignment and interpretability experts stepped up to say no, it wasn’t good enough yet.
AI 2027 climaxes in a desperate race between better interpretability tools and more devious AIs. It honors the interpretability researchers’ accomplishments by depicting a world where their tools often catch the AI’s schemes - but the companies keep racing forward anyway, because some other company / China would beat them if they didn’t, and besides, maybe if we race forward fast enough we can solve alignment before the AIs’ schemes bear fruit, and besides, even if we can’t, there’s always control. Some researchers use applied interpretability - suppressing or ablating bad features - to keep the AIs compliant and controlled for a few extra months’ worth of intelligence explosion time, hoping to reach some ill-defined finish line before the training manages to route around their methods and create uninterpretable misaligned systems. But in the end, they never get anything good enough to fully understand the AIs’ motivations, let alone redirect them.
The current level of interpretability tools seem on track for this sort of outcome: enough to help around the edges, but not a solution anyone should rely on.