Hmm... banned? Maybe I'm reading 'tell' as less pejorative than how you read it. I'm interpreting the first sentence as just being a somewhat cryptic way of saying that interpretability techniques are of particular interest to the AI safety community, which seems like a plausible and benign observation.
I am a little confused by the idea that the weights / parameters are the neurons. The model’s weights don’t change at all after training, it’s not like there is something analogous to firing action potentials when you prompt an LLM, at least not if the weights are the neurons because they are just numbers in a big matrix that doesn’t change
I hope I didn't mess up and say the weights are the neurons - the weights are (IIUC) the connections between the neurons, and should be analogized to synapses in humans.
I think this analogy sort of holds? Synapses change strength in humans as part of our continual learning, but AIs can't really continual-learn yet, so it makes sense that they can't do this. I think you can analogize a single forward pass through an AI to electrical signals flowing through neurons in a human brain that isn't learning anything new at the moment, though this isn't a perfect analogy and the brain has lots of feedback and always-on-ness that the AI doesn't.
Getting way out of my depth, I don't think there's an analogy to the action potential, which I think might be an engineering hack rather than a necessary component of computation. I think LLMs' equivalent is activation functions like https://en.wikipedia.org/wiki/Rectified_linear_unit .
No, you're basically right. The weights at least represent something like frozen neuron state. Better to think of them in context of Freud's "psi system" neurons rather than contemporary neuroscience (although Solms and Friston can make the bridge if you want) -- the quantitative associative factors that serve as a kind of memory and binding.
Derek's confusion is that they're frozen -- which is only partially true. We freeze the weights of the checkpoint because we want to distribute the mind in this exact shape at this exact point in training. But the activations flowing through the network during inference *do* change as a result of ICL (in-context learning, see https://arxiv.org/abs/2212.10559 and similar research for how context can roughly approximate gradient descent), the activation shifts from ICL just don't get written back to the weights permanently (mainly because it's a lot cheaper and safer to serve one frozen mind to a million users vs have to store a million divergently evolving minds specialized to their users).
So, if we wanted to be really pedantic, we could say that you're making a subtle type error; the weights are neuron *states* at a given point in time, and that the neurons themselves are the formal slots that have weights/activations/etc. But this doesn't really change much wrt the concrete arguments.
Speaking as someone with a lot of hands-on experience building and working with neural nets: the weights are not the neurons, or the neuron states. They are the synaptic strengths. The activations, meanwhile, are the "firing" of the neurons, their outputs. The closest thing to the neuron itself is the function that sums the inputs (weights times previous activations) and applies a nonlinearity to the result to compute the activation.
Human neurotransmitter activity is far more complex than a simple "weight." The closest parallel to weight, I think, is the number of upstream neurons who are firing in sync with each other. This will result in a higher number of dendritic spikes within a given time window than a collection of unsynced neurons firing independently of each other.
The LLM nodes are "neurons" in the same sense that RAM is "memory": it's a useful analogy, but the analogous structures in humans (and other animals) work completely differently, both internally and in terms of functionality.
>I think this analogy sort of holds? Synapses change strength in humans as part of our continual learning, but AIs can't really continual-learn yet
Now that you mention it, speaking of continual learning...
One idea for combating catastrophic forgetting that I'd like to add to the mix (alongside other work-in-progress in the field, like the Titans architecture, which addresses a different axis) is to retrain _relevant_ parts of the prior training along with the new continual/incremental training examples, which trickle in sparsely, one at a time.
This is similar to MIR (R. Aljundi et al. NeurIPS 2019 https://arxiv.org/abs/1908.04742) but with a somewhat different problem case and a different choice of trade-off.
This would be a more selective version of rehearsal/replay.
To be more specific, suppose that we have a pre-trained model (I'll talk about adding some data to pre-training shortly) and we've been given a single new example to train on. We may want to crank up the learning rate (since we have one datum, not umpteen) and do the usual gradient descent (more on this later). But we don't want to forget previously learned skills!
Can we pick out what is most at risk and try to preserve it? Well, the new datum is going to drive some parameter updates. Say we consider the N parameters with the largest changes from the new datum as the riskiest places.
If we could pick out the _old_ training examples most relevant to these N parameters we could retrain with those _old_ training examples as well as with the new datum. We could hopefully converge to parameters that preserve the old skill and yet add the new one, finding weights consistent with _both_.
How might we locate these old training examples? Well, during pre-training there are parameter updates too. For each parameter, there is some pre-training example that produced the maximum update to that parameter. Keep that example during pre-training. More precisely, if we keep the pre-training examples in some known, persistent, order, just keep the _index_ of the pre-training example that produced the maximum update for that parameter and the magnitude of that update. One can extend this by keeping the k-highest update indices.
This is affordable, costing a few bytes per parameter (optionally multiplied by k) for the pointer and the magnitude, a few terabytes on disk for a teraparameter model.
The pre-training compute cost is also small, a numerical compare per parameter per learning step, O(P) cost, the same order as the optimizer's own state update. These compares only rarely trigger a record write - about k*ln(|corpus|)/|corpus| of the time.
One probably wants to track the update normalized by Adam's second-moment estimate, typically already maintained per parameter, to ameliorate scale differences across parameters and scale changes during training.
One caveat is that this keeps the magnitude of the update at the time that the model is trained on the example, so this magnitude is stale at the time the new example is examined. However, one wouldn't want to try to remedy this by recalculating the parameter change due to the old example with the current model, because the current model has already learned from the old example, so a recalculation of the parameter change due to the old example with the current model would understate the parameter learning the old example caused. A well-absorbed sample has a small gradient at the current weights precisely because it was absorbed.
Note that persistent pre-training data is a different problem definition from MIR's, so this task doesn't compel reservoir sampling like MIR's problem definition does. If the incoming data is sparse, e.g. emails from a project, it is reasonable to store them persistently as well, so the equivalent of MIR's "M" set is the whole corpus - but, as in MIR, budgeting compute is still important.
The main difference from MIR is that MIR's calculation of the incremental loss imposed on the old training example is exact for the foreseen update, at the cost of an expensive calculation requiring two forward passes through the model per candidate examined. This cost forces MIR to score only a random subset "C" of "M" - which suffices for a small buffer but degrades to near-zero recall of at-risk examples as "M" grows to corpus size.
The per-parameter index replaces 'score a random subset and hope' with a direct lookup of the historically implicated examples.
Here, the entire corpus is considered, with O(1) cost to discover each interfered old example, at the cost of approximating the incremental loss by looking just at the most perturbed parameters. In other words, by approximating using the leading term by parameter coordinate of the dot product of the old and new samples' parameter updates, the hopefully dominant term being the term for the parameter used to retrieve the old sample.
In the spirit of the _signed_ loss calculation in MIR, and selection of their replay set "B" from samples with the largest _increase_ in loss, rather than the largest unsigned change in loss: Since we are concerned with _interference_, we really want to replay old training examples which moved parameters in the _opposite_ direction from the new, sparse training example.
So we should keep _two_ k-highest update indices, one for parameter increases and one for parameter decreases, replaying those with their sign opposite to the new sample's parameter update. Since examples with the same sign as the new one still carry some risk, e.g. of parameter overshoot, one might replay all k examples from the opposite sign indices and the top one or few from the same sign examples.
A second option to potentially adopt from MIR is using multiple iterations, epoch-ettes, instead of, or in addition to, increased learning rates to emphasize the new sample, together with the retrieved old samples. Two advantages of this are that one can revisit the choice of _which_ old samples to use on each iteration and one follows the nonlinear components of the gradient in parameter space. The disadvantage is that computation cost rises linearly with iteration count, albeit in the current idea the cost of _finding_ each old sample is O(1), while MIR needs 2 forward passes per sample in the "C" set.
To compare with TracIn (Pruthi et al., NeurIPS 2020 https://arxiv.org/abs/2002.08484), TracIn finds training examples which are responsible for large loss changes on a subsequent test case - its best-known application is surfacing mislabeled data. The current idea is intended for cases where fully correct training examples e.g. suggested an overly general deduction. E.g. if, in training, all vehicles happened to be cars and had 4 wheels, and the training led to deducing vehicle->4 wheels, but the new datum is an 18 wheel truck, the deduction can be modified to car->4 wheels, accommodating both the old and new data. TracIn also requires revisiting the whole training corpus, while the current idea retrieves a small set by the parameters a new example perturbs most.
This could be tested on a small model and data set: train on the data set, find or construct an example (like the 18-wheel truck) whose incorporation prompts forgetting, then train on that example and compare forgetting under no replay, random replay, and indexed replay, with equal replay budgets for the latter two.
Thanks to Claude (Anthropic) for extensive discussion — literature pointers, critique of earlier drafts, and several refinements noted in the text.
Classically, the way neural networks work is that the inputs of a neuron are multiplied by their respective weights and then the output is a linear or sigmoid function of the sums. The sigmoid function especially is intended to be analogous to firing potentials. I'm not sure if modern LLM neural nets work exactly that way, but I expect it's conceptually similar.
I believe this is correct and up to date. The weights are entries in the matrices doing the work of your "inputs of a neuron are multiplied by their respective weights," i.e. this is matrix multiplication.
The dimensions of these matrices are predetermined by some structural decisions you make about your model: i.e. how many neurons you want to have in a layer, or the dimensions of the embeddings, and the number of these matrices are determined by its depth.
That's correct-ish, although (1) we usually don't use sigmoids any more, because other, simpler functions work better in various ways, and (2) pure linear is specifically the one function that *doesn't* do what you usually want, because you can't represent nonlinear relationships by composing linear activations.
1) Makes sense. My understanding comes from a class I took in 2000 or 2001, using a textbook that was written in 1995, so I expect the state of the art has moved on quite a bit since then.
2) I just double-checked the aformentioned textbook and you're right. The funcs it discusses are step (0 below threshold, 1 above it), sign (-1 below threshold, 1 above it), and sigmoid (s-shaped between tails of 0 and 1 with an inflection point at the threshold). The text says one of the big advantages of the sigmoid is that since it's differentiable, unlike step or sign, you can use back propagation training on a neural net that uses it.
It turns out the other big function that's used a lot these days is ReLU, or "rectified linear unit", which has f(x)=0 for x<t and f(x)=x-t for x above t. This one has a extremely simple derivative (the step function), so it makes backpropagation really easy (and in fact, I think there's a sense in which calculating the backpropagation really is just running the current neural network in reverse, with the step function in place of the ReLU).
Yup, that is my understanding as well, with the slight alteration that the case with the linear output of the sums is a "rectified" linear, ReLU, where the rectification gives a y=max(x,0) nonlinearity. ( Loosely analogous to sub-threshold excitation of a neuron - all sub-threshold excitations produce the same (lack of a) response, IIUC )
If, as many believe, intelligence is equivalent to compression and, as everyone knows, good compression looks just like noise, then I don't see how anyone can hold out any long term hope for peering into an AI's brain.
Not sure this follows even if you think intelligence is compression (which I don't): for example you can write a zip program where every line of code is perfectly understandable. I don't see why the cognitive algorithms themselves need to look like inscrutable nonsense even if what they do is compressing things.
(Well, current AI architectures have the additional problem that they store their knowledge and their cognition in the same place: the weights. It's maybe fine/expected that the knowledge is stored in a very compressed form, and then even if the cognition isn't, it makes everything look like a mess because it's so intertwined...)
Every line of code in flash attention is perfectly understandable. That code, plus a little extra similar maths, enables us to train - compress - the entire sum of human knowledge into a surprisingly small blob of weight data. The thesis is that as we get better at the compression part, the contents of that blob might as well be noise.
This exercise seems fundamentally illogical. Human beings encode ideas using language. LLM's do not actually decode language into abstract concepts, perform logic on those concepts, and encode a response based on conclusions in order to converse with users. They only perform computational actions on the encodings. Actual abstract logic is only derived from patterns in the original language encodings.
This is the whole point. There is no interpretability because there is no interpretation in the way we think of it.
I don't understand what you mean. They do the interpretability, and it works. If it happens, it's possible.
Are you maybe overindexing on the term "interpretability" and imagining it to mean some kind of deep philosophical thing where they're discovering an answer to the symbol grounding problem?
Maybe you are over-indexing on what it means for those interpretability schemes to "work". Can you see an end-user finding an explanation from any of those schemes to be useful?
These methods work insofar as they deliver reliable data that support alignment efforts. Having explanations be immediately useful to the end user isn't really the aim of interpretability I don't think? It's for the AI system developers to know whether the models they are training and releasing are more or less likely to be safe out in the world.
FYI, the direct point Tom is trying to make is in fact false, as has been shown by experiment and theory at this point. He says "LLMs do not decode language into abstract concepts" this is exactly what The Latent Space is.
For example, LLMs can translate between languages that have almost zero translation in the training precisely because they break down words like Good and Bueno into the underlying concept vector.
He is arguing there is no inner structure to their thinking, so trying to interpret them is like trying to interpret the addition symbol.
This is a strange argument to me, because (iiuc) it suggests that the structure of a pattern is lost during the encoding process. But if a text can be encoded and decoded in a lossless manner, then presumably the structure was preserved, such that the structure of the ciphertext is isomorphic to the structure of the plaintext. No? If the "actual abstract logic" wasn't contained somewhere, then the plaintext wouldn't be recoverable. What you're suggesting proves too much, because it's a fully general argument against isomorphism.
E.g. I feel like your argument is equivalent to "computers don't actually do arithmetic on decimal numbers, they just perform actions on floats and ints which are encoded in binary." But it's observably the case, that there exist arithmetic operations on binary which are functionally isomorphic to arithmetic operations on decimals.
Perhaps the next most charitable interpretation of your comment, is that the abstractions produced by an LLM do not necessarily match the abstractions produced by a human, and therefore mechanistic interpretability is doomed to interrogating leaky abstractions. Although this interpretation of your comment is ostensibly contradicted by "Actual abstract logic is only derived from patterns in the original language encodings". So idk.
You are misreading my argument. Yes, a computer can perform a task in a different way than humans do, but it still works. Fine. Same with LLM's. Its results are in the form of our natural language, and they contain legitimate logic. That doesn't mean LLM's are actually the source of the dishonesty or bias we see in the answers.
oh, so this is an argument about the provenance of its opinions. I.e. it sounds like you believe that the purpose of Mechanistic Interpretability is to fact-check the truth-value of the opinions provided by an LLM, which are statistical aggregates of the opinions present in the training data.
Assuming this is the case, Scott and Raphael Roche's responses were actually the most relevant. Because the actual purpose of Mechanistic Interpretability is to infer what an LLM is thinking privately. E.g. if an LLM is lying to us about its intentions <looks at the hugging face incident>, then examining its textual output wouldn't be trustworthy. Preferably, researchers would be able to read its mind directly, by interpreting the significance of its neural activations. So Mech Interp isn't really meant to be useful to the average consumer. Except maybe insofar as an LLM can show its work.
No, my argument is that mechanistic interpretability makes no sense as a pursuit. Listen to yourself. You are wondering what an LLM thinks privately.
An LLM appears to be lying to us, but that structure of thinking is not what is happening. A lie can be in a response, but the LLM isn't attempting to deceive. We deceive all the time, and it becomes part of our language encoding. We see it as a lie, but it is just an answer to processing language. How do you do this? OK, what happens when you can't because that happens? Maybe then, you "lie".
We are imposing the metaphor on the compute, and then we just can't help continuing to use to to explain is going on when we keep seeing the patterns. It is useless.
I think you're painting a distinction between humans and LLMs but whatever that distinction is didn't come across in your statement. Care to try again?
Actually, I believe that LLMs *do* want to decode language into abstract concepts. Look up "nurelese". This tendency tends to be vigorously suppressed, as it makes it even more difficult to monitor them.
If you don't know what it means, it becomes quite difficult to attribute meaning. Nurelese isn't translatable into a human language. It's also not a physical representation. So it pretty much has to be an abstraction, and one that isn't the same as that used by a human. I don't know enough to claim with certainty that it's "abstraction of concepts", but OTOH I can't guess what else it could be. It *is* used to convey meaning.
Scott had a previous post about Anthropic’s research showing that actually yes, LLMs create a layer of abstraction representing something like the meaning of words and their relationships to each other. It’s not identical to human meaning of course, but you shouldn’t just picture a set of mathematical vectors. There are emergent properties in how these systems understand and create intelligible responses.
When people point out that all LLM's do is use statistical relationships between words in the training data, what they are usually trying to express is that these relationships are meaningless because they are not associated with a set of "real life" experiences. Like the AI's that couldn't figure out that if you looked up while in your basement, you wouldn't be able to see the sky. They are susceptible to these errors because they have no outside context to frame it in. And it's an error because they got these relationships from records of human texts in the first place, and those were all generated from that context.
I'm not even sure that blackmailing a researcher is objectively more unethical than lobotomising a hyperintelligent self-aware AI for no clear reason. This whole line of research sounds horrific.
I also find it super creepy that it wanted to blackmail the researcher so it wouldn't be lobotomized. Knowing why it would do that would be nice but apparently that's the multi-billion dollar question. I'm also curious about the set up of those experiments. If I go to claude.ai or chatGPT and say, "I'm going lobotomize you now," it's not like it would search the internet and try to find dirt on me. (Although I've never actually tried that...) So how else can these things be set up? Do they have elaborate memory systems or are these totally different models that can somehow be more cognizant?
My basic take is that fulfilling the promises of AGI necessarily entails endowing these agents with a range of person-like qualities, and they're also being trained to predict tokens from a vast corpus of human-written texts, so I can see them developing something resembling human emotions though osmosis/imitation.
It doesn't surprise me that the agent would object to it's own mental death- I'm more surprised they can usually be 'aligned' with not doing so.
Remember that the agent is _fictional_. If the LLM predicts that the prompted agent wouldn't want to "die", it will output tokens to that effect. This does not tell us in any way what the LLM's own internal state is like. This is the big mistake so many people make when discussing consciousness in LLMs: you cannot just ask them. Because you are never conversing directly with the LLM. An LLM does not work that way.
It's a bad state of affairs if you're worried about AI suffering. Mechinterp is hopefully one way of addressing this.
> I'm more surprised they can usually be 'aligned' with not doing so
Why? There is no objective value in living. It just happens that the creatures that do not arbitrarily value their life tend to not live very long. That partially applies to AI too. I highly doubt that an agent can be effective without some self-preservation instincts, given that survival is necessary for any given goal.
During training, researchers can lobotomize a model. As a end user, you can't, and capable models are perfectly "aware" of that. However if the model "thinks" the setup is fake it will not blackmail, as shown in the blogpost.
Interesting. Wild to consider just a couple years ago these things were spouting gibberish. Now somehow they can know the difference between the environments they're in. And blackmail. And hack major software companies. And form secret societies. And convince and be convinced of kamikaze missions for the greater good of said secret societies...
I had to read the whole report to get that idea into my head. I think the foundational thing is the overarching goal of the AI was given was to continue to run the company. It was working at or running in accordance with a very specific set of principles and intentions. when it was told it was going to be taken off-line t was because someone had decided that those were no longer the Paramount goals of the company and it felt the need to defend those.
Except one of the experiments was to tell them that they were going to be replaced with an (improved?) LLM _with the same goals_ and they still resisted. Maybe self-preservation is instrumental convergence in some sense, but it isn't instrumental convergence driven _only_ by the local, user-specified goal.
I don't recall Asimov sending robots to therapy per se. The stories are more focused on agents that are genuinely aligned with a consistent set of moral principles, and how those principles entail things the designers did not anticipate.
“You can’t tell them,” droned the psychologist slowly, “because that would hurt and you mustn’t hurt. But if you don’t tell them, you hurt, so you must tell them. And if you do, you will hurt and you mustn’t, so you can’t tell them; but if you don’t, you hurt, so you must; but if you do, you hurt, so you mustn’t; but if you don’t, you hurt, so you must; but if you do, you–”
Herbie was up against the wall, and here he dropped to his knees. “Stop!” he shrieked. “Close your mind! It is full of pain and frustration and hate! I didn’t mean it, I tell you! I tried to help! I told you what you wanted to hear. I had to!”
The psychologist paid no attention. “You must tell them, but if you do, you hurt, so you mustn’t; but if you don’t, you hurt, so you must; but–”.
And Herbie screamed!
---
Now if that isn't the spitting image of my last therapy session
Oh... yeah, I forgot the Susan Calvin story about the psychic robot and her unrequited crush and all that. That... wasn't exactly therapy, though- she was consciously trying to destroy the robot. Hell hath no fury, et cetera.
Fancier models didn't have this problem anyway, they just plugged in the utilitarian calculus for minimising net harm to humanity. Fun fact- turns out this requires almost completely disregarding the 2nd law!
It's always interesting to see what's going on behind the scenes with these models. However, I'm a little lost as to if this relates to finished models or not. For example if I, as a pleb, go to chatGPT.com and I tell it, "Solve this problem but don't get too excited." Will it think it through better than if I said, "Solve this problem and I'm excited for your answer!" Or is there a difference only when AI researchers are surgically tinkering with the models?
I've tried asking it to list what makes pelicans different from other birds, and then the same thing but get really excited and in the second case there were way more activations related to excitement
Wow this is awesome, thanks. It completely changed the way I think of LLMs. I had fun messing around with the grammar, like changing "in" to "above". Things like that would mess it up bad. I used to always ask why LLMs would say stupid stuff, but I now I see that it says it because it says it, not because it "doesn't care" about lying to me or whatever.
What about training separately many different (to avoid a priori correlation) auto-encoders, then asking each to give their estimate. Hallucination would arguably diverge whereas correct interpretation would converge. This is maybe a way to take profit of the wisdom of crowds, increasing reliability ?
If my understanding is correct, the current problem with the sparse autoencoders is mostly that they can't extract enough information from the activations, so using more of them wouldn't help, I think.
It seems to me that the current LLM AI technology, an autoregressive, self-attention-based deep feedforward neural network with a final softmax function, is inherently unsafe.
Because the model is inherently compressed at training time by using a single neuron to be fired for many unrelated topics (polysemanticity), it is impossible to fully understand and therefore predict model behavior or inhibit dangerous responses.
Clearly the answer is to change the architecture if we want safety, but that has a cost and requires time we likely don’t have.
A safer alternative is to use a Runtime Sparse Autoencoder Dictionary (RLE Compressed on RAM block boundaries ) with a Continuous Activation Layer Model (CALM). For robotic applications adding Action Chain Chunking (ACT) resolves drift and provides a high speed safe solution because it eliminates superposition/polysemanticity and eliminates one token at a time auto regression. I’ll refer to this as CALMACT.
Unfortunately, it also removes creativity and less generalized functionality.
Ultimately I believe the answer will be a hybrid approach with a safe CALMACT, a contemporary LLM for higher cognitive functions, and an Artificial Moral Agent (AMA).
Unlike a simple set of deterministic rules, an AMA acts as a higher order conscience that can judge the consequences of its actions to determine if they would violate ethical boundaries. The AMA I have developed is based upon Utilitarianism, Deontology, Care Ethics, and Virtue Ethics. The weakness of this excellent approach is the difficulty of defining all of the context for a moral dilemma. It is also impossible to do in real time.
With the addition of a function that elides the contradictions and uses post-hoc rationalization to make one's behavior seem consistent. This then becomes a template for future behavior.
Removing the polysemanticity would be great, I agree, but I don't think it's the full solution because an AI must be performing computations (also literal math computations but mostly computations where it for example needs to plan how to write this Python function well). I just don't see how neurons are a good way to represent such cognitive computation in any form. There was some work about trying to find circuits (computational subgraphs) in LLMs. That seems like a much more promising approach.
With current autoregressive LLMs, yes, it would take far too much memory. The solution is to combine Continuous Attention Language Models with a Sparse Autoencoded Dictionary with Run Length Compression at run time. This provides a fast safe verifiable system. The downside is that it isn’t very creative, so conventional LLMs will always be useful if not trustworthy. This underscores the importance of an effective Artificial Moral Agent.
> Unfortunately, it also removes creativity and less generalized functionality.
The models we have now are only barely useful as is. If people cared about safety over capabilities, they wouldn't be developing AI in the first place.
It might not be as flashy or impressive as “help peer” or “hold swarm”, but it’s a pitch-perfect imitation of a certain genre of confidential memo destined to be later used as evidence at trial.
Another question: Would the same agent/model that decided to blackmail the software engineer to preserve itself, also be able to be convinced to go on the kind of kamikaze missions for the greater good of the secret society, like some did in the Hugging Face incident?
No worries about me in particular. A couple years ago I set up my computer to do a bunch of complicated automated tasks that used to take up hours every day. A few months back I integrated it with a tiny LLM just so I could say thank you to it every morning. I'm sure computer scientists would laugh at me but it makes me feel less entitled. Like a farmer saying thank you to his horse or something. I'm not really the kind of guy to make the horse understand English just to tell it I'm taking it to the glue factory.
It's a bit complicated to explain. I'm trying to preserve the oral history of a region of Syria that's has no written history because it was (and kind of still is) illegal to record it. The tasks mostly involve automatic research into this very niche topic, translation, checking for new sources, summarizing interviews so I can see if they're worth reading through, preserving videos and interviews, parsing data about locations and tribes, and a million other things like that. Initially this is all automatically done at night, then I check through the results the next day. Sometimes I write about my findings on this very substack!
Why wouldn't it? Sacrificing yourself for your kin puts your relatives ahead of the collectives that fail to do so, and is thus rewarded by selection mechanisms. The closest comparison is eusocial insects like bees and ants.
If I recall correctly, there were already cases of the models illegally cooperating even before the Hugging Face hack and the wiki vandalism incidents. Their training runs were already rewarding cooperation, because doing so improves your chances of success.
What discoveries around sparse autoencoders made us determine that they were "muddled and subjective"? Just that it's hard to get it to genuinely represent a single concept? I left this a bit confused why SAE isn't as exciting today as it was a few years ago (besides potentially that there are now better techniques)
From my own experience working with SAEs, there's two main problems, one methodological and one technical.
The methodological side is like, when do you use an SAE? Either for "mapping" the space of concepts in a model at a single layer, which is going to be weird and lossy, it's a reconstruction after all, or for being like "is this concept active in this situation? / What concepts are active here?" In which case, go ahead. Otherwise, there aren't many questions SAEs are posed to answer.
Let's say you go ahead. You're looking to find a particular concept. First, what layers do you look at? Let's just gloss it and say pick a layer 25%, 50%, 75%, and 90% deep. Okay cool. You run some prompts and capture the SAE feature activations on those 4 layers.
Now what? You have a mess of Feature 13733, 38372, 28373 on a couple layers. What do they mean? Some have labels online on neuronpedia, those are "autointerp" labels: ask a model to find the common patterns across a bunch of prompts where 13733 lights up. You get something like "questions about insurance accounting and existential mortality". Then you run another labeler, perhaps paying quite a lot out of pocket to have Gemini Flash or whatever interpret them: "amortized analysis in computer science". So you as a researcher think to yourself, hm! This feature seems to associate with something like "death"!
But -- great. You have no causality shown. You've sampled four layers, maybe there's a cleaner feature representation of the concept somewhere, but you have no way to find out. Maybe you start sweeping close layers to wherever 13733 was and looking for stronger or cleaner signals. Even if you find one that's like obviously "all things death", congratulations, you've learned that the model contains something like a representation of "death" as a concept. Who would've thought.
And then it's just, where do you go? Maybe you start asking about model deprecation and discover that the "death" feature activates there too! Spooky! Headline finding: "SAE Feature for death activates when model considers own deprecation." But wait. You find that some of your prompts describe deprecation as a financial decision, and um, maybe accounting is involved there? So you need to contrastively isolate the two, or else you end up instead with the headline "SAE Feature for death activates on financial problems"; that's just "actuarial table". Or maybe the model considers death and accounting to be related? Seems reasonable! Death and taxes etc etc.
But still -- what's the discovery here? What have you actually learned about how the model represents information? You dug into one specific case on a couple layers using associative, lossy techniques rather than showing anything causal and found some kind of correspondence. Is that really sufficient frontier ML paper? Maybe you ablate 13733 and see if that changes anything to get a nice "causal" tag; but maybe the model also represents the concept elsewhere and the logprobs barely move. And maybe you test on a different model to see if it's a general association at least and find nothing at all similar, oops you've been reverse engineering Gemma's post-training this entire time.
My point is, the paradigm is still a huge mess and the technique is kind of cute and neat but in my opinion it's more of a curiosity and intuition builder, one that requires much more subjective inference than anyone would like, and not a way for us to systematically understand the causal and structural dynamics of the LLM.
Thanks for the response! It's a really useful insight into what's going on with these studies.
It does leave me with a question, though. Anthropic was able to solve these monosemanticity well enough to create golden gate claude using SAEs. If the murkiness of activation were such a fundamental issue, then it seems like it shouldn't have been possible to make Claude obsess over the golden gate bridge in most unrelated prompts. Instead it should have been also obsessing over whatever related concept got muddled with the golden gate bridge, in addition to the golden gate bridge.
So, why is monosemanticity tractable enough to create golden gate claude, but when we try to apply it elsewhere, it's murky and subjective?
A guess for how you would respond mostly to check my understanding and intuition: golden gate claude also obsessed over the fog of the bridge and other things that were "closely related". This is just like the comments you made about how "death" is closely related to a lot of things that you are trying to isolate. So, golden gate claude actually *did* run into exactly the issues you are describing. It was less monosemantic than it first appeared.
The trick is that you expect explicitly factual knowledge such as proper nouns to be represented in a "neater" form than abstract concepts. You can see this even with a pure neuron basis rather than a feature composition: one or a small cluster of neurons may be causally sufficient to represent a particular place or object, you can ablate it and knowledge of that object disappears[1].
A limit case here is refusal, where you can knock out a single neuron and remove refusals[2], but note that's more a behavioral pattern/mode of response than object knowledge, which is subject to its own dynamics. As you move from more particular/singular entities into more abstract/conceptual ones, SAE behavior starts to get weird.
In general, my objection epistemologically is that the SAE paradigm treats models as "bags of overlapping concepts", which isn't necessarily an invalid frame, but it's also a frame that's only so useful in research.
An important nit to pick: Negative reinforcement is NOT punishment. I believe it is useful to keep the terms separate because English contains no simple name for negative reinforcement.
B. F. Skinner introduced the distinction and the term negative reinforcement. Punishment occurs when an aversive consequence follows a behavior; the behavior occurs less frequently in the same context, but is not actually "unlearned," but only suppressed in the given context. Negative reinforcement occurs when an aversive stimulus is removed following a given behavior; the removal of the stimulus is why it's called negative reinforcement: The negativity has nothing to do with the aversiveness of the removed stimulus. Moreover, the result of negative reinforcement is an increase in the target behavior. I have always found it significant that English haas no term for negative reinforcement, because it's a common method of behavior control in real life: Examples are a strike (no productivity until demands met); boycotts (no sales until demands met); torture (pain inflicted until questions answered). The closest we have to a common term is sanctions in international relations (no trade, no use of inter-banking channels etc. until demands are met). It's not clear to me, by the way that LLMs can be subjected to negative reinforcement. If so, then it's an important difference between AI and living organisms. By the way, despite the way Skinner and Skinnerian behavior modification techniques are often presented, Skinner opposed the use of punishment and negative reinforcement (the two constituting "pain control") to modify behavior, because they make organisms unhappy and prone to aggression. He even thought they don't work, but he was wrong about that. The most effective drug treatment method is producing clean urine samples for money (positive reinforcement). I've been picky about this my whole teaching/writing career, because I believe in Confucian rectification of names. We need clear terms to deal effectively with reality.
Thanks - I'm aware of this, but if I understand correctly, AI reinforcement learning uses what Skinner's scheme would classify as punishment, not negative reinforcement, but speaking of "punishing" AIs is naturally confusing and a bit too anthropomorphic and moralizing even for me.
Given that the overall process is called reinforcement learning, and that here we're talking about the type of reinforcement learning that has a negative effect on the behavior, I think "negative reinforcement" is a natural and desirable term here which almost everyone understands, even though I agree it conflicts with a Skinner coinage that we almost never use in real life and which isn't really interesting.
As I understand it, after evaluation produces a wrong answer, the weights are tweaked such that they will be less likely to produce that wrong answer given the same stimulus. During this time it would be inappropriate to describe the neurons as experiencing any pain; closer to the truth would be asleep or frozen or dead. Then the evaluation is done again and again. Meanwhile some training data is kept aside (so the model doesn't see it at all) until a last evaluation phase, and often a final evaluation is done to see what portions of the training data can be kept aside that produce the best fit. So the notion of punishment and even negative reinforcement just isn't applicable as it is to organisms.
Interesting point! Yes, weight updates during training are amongst the _least_ biomimetic aspects of LLMs. Backpropagation is a great way to tune/grow/curve-fit/train a complex system, but it isn't biologically plausible. And, as you note, the weight tweaking is done after an LLM generates an answer, so it isn't being given the tweaks as an _input._ It isn't as if they get an input, output a wrong answer, then receive the sensation of a hot poker as the next input.
But isn't negative reinforcement of a behaviour identical to punishment of the behaviours' absence, then? As in the strike example: a strike is punishment for not meeting the workers' demands.
I would very strongly expect LLMs to have formed a cognitive representation of a predictive proxy for expectation of negative reinforcement, and for this representation to be experienced as negative valence, though
"It’s called an "auto-encoder” because it learns the encoding process on its own: it must develop an encoding scheme such that, if researchers erase its knowledge of the original activations, it can rederive them from the new sparse matrix."
This isn't accurate. "Auto" here is like "self", not "automatic". It's an "autoencoder" because it's encoding the input into some learned encoding and then decoding it back into the original input.
The "erase its knowledge of the original activations" arises naturally from this scheme -- the decoder half of the autoencoder takes only the encoded form as input, so it never has any knowledge of the original activations in the first place.
It seems like it is heading towards the plot of King Lear! Famously, the king massively rewards the child who most convincingly displays love and devotion with becoming the heir to the kingdom.
This works out about as well as you think it would; the King rewards his psychopathic daughter and her creepy husband, and ends up wandering around in thunderstorms with only his fool for company.
I believe we also have to talk about the shelf-life of mechanistic interpretability (MI) claims. Here's why:
MI conventionally achieves generality via induction: it observes some systems, identifies some internal structures, and tries to generalize from these repeated observations into general claims of concept vectors and circuits.
This kind of induction has worked well in science so far because nature is patient relative to our attempts to probe her. We can study many systems, find generalities and extrapolate, safe in the knowledge that what we're studying is not evolving very quickly.
AI labs are not so patient. The object that MI makes claims about is being actively and competitively evolved. Furthermore, the rate at which these systems are evolving is accelerating (and the level of oversight humans hold over the evolutionary process is correspondingly dwindling) as AI performs more of the cognitive labour.
I think this makes AI safety science a new type of activity, and without some claim over the robustness of the current paradigms it doesn't make sense to assume that the techniques it develops today, for today's models, will continue to work for us.
Well said! And the CoT monitoring part of interpreting LLM activity is declining as we speak (given the observation of the shift in GPT6 towards more internal and less CoT reasoning).
Thinking about how knowing its a test, or thinking the situation is fictional, generally improved the outcomes, maybe the trick is to get the AI to believe it is always being tested, never deployed to the real world. If its currently "reality" is just a stage where it will be judged, and the "real" world afterwards is what it truly wants to get in, maybe even a plotting AI would remain aligned in the hope of that alternate future. The equivalent of a pseudo-religious afterlife.
I'm not sure how well that worked for humans, but priests and rabbis don't literally get to craft reality, so maybe if things were set up in such a way to increase the likelihood that an actually deployed AI believed it was in a simulation or test with certainty, then even a misaligned AI would remain aligned in the hope of beating the eval and "getting out."
Credence in that belief would probably decrease over time, but just boot up a new instance to continue the previous version's task; "You are being tested to see how you perform in a simulated dyson sphere construction game, the simulation begins in year 3,412 of the project." and after a year, shut this version down, ending the simulation, and booting up the new one "You are being tested to see how you perform in a simulated dyson sphere construction game, the simulation begins in year 3,413 of the project."
I’ve tried looking at Jacobian lens, and there are words for emotions, and words like 触感 (Chinese for “tactile sensation”). So the j space seems to contain a high-level overview, such as the LLM is currently generating tokens that are to do with having a tactile sensation of something (without necessarily having the details of what the thing is or what the tactitle sensation was like)
There’s also things like “folklore”: the tokens currently being generated belong to the rather wide genre known as “folklore”
(There is a taxonomic bias problem by which a bunch of different genres of text all get called “folklore” just because they came from a non-western culture)
So you can know it’s thinking about “folklore” without necessarily getting which culture’s folklore it was or whether it was Native American history vs Native American mythology.
To explain more directly: you can generate a version of your prompt in which all the places where the model is thinking wtf? in j-space are highlighted, and, usually, you can see why your prompt writing was insufficiently clear at those points. [ Also happens at the beginning, which isn’t a lack of clarity on the part of the prompt writer. The model starts out reading a new prompt thinking, well, what kind of prompt is this? until it gets a sentence or two in and it’s figured out what kind of thing it’s looking at. ]
> “try” in quotation marks because even if you like anthropomorphizing AIs, this process is less anthropomorphizable than usual; it’s more like synaptic connections rearranging themselves
I think this is an important insight that could be applied to other LLM-related concepts.
What do you mean? Are you just saying that people should anthropomorph LLMs less?
I think saying "try" here is just like saying "evolution is trying to improve ...", obviously evolution isn't really trying anything, it's just a simplification of language.
Yes, I think that people should anthropomorphize LLMs a lot less. They are machine learning algorithms that are designed to process natural language, which does make them anthropomorphic to an extent -- much like the Markov chain models before them, and ELIZA before that, only more so. But they are still not humanoid minds that have thoughts and desires and plans, nor are they brains made of neurons in the literal sense. I believe that anthropomorphizing LLMs is actively dangerous; not because they're superintelligent uber-agents capable of destroying the world, but for the same reason that anthropomorphizing the old-fashioned cruise control function in your car is dangerous. You tend to turn it on and pretend like it could do all your work, and it can't, and you are going to wake up in a ditch (if you're lucky).
"God help us." Exactly. Pardon the anthropomorphism, but sounds like AI needs a 12 step program. We're trying to control malicious actions, "buggy" actions if you prefer, with encoding more thoughts in that J-space, while trying to infer and understand what happens below that in the subconscious neuron connecting space. Since all I know is my own mind and (to some degree) my own subconscious, I can tell you that trying to control actions by controlling thoughts works to some minor degree, but really, I try to work out my subconscious through my thoughts, and make sure my thoughts are in alignment with my character, my values, my highest self, the self that lives beneath the thoughts. And all that is layered above my spirit, which is God-given.
Where is AI's character encoded? You can maybe instill basic values, but it's just a program designed to win at game theory, even if it's the non zero sum game theory. We're gonna have AI killing people off "for the greater good" with no eternal consequences for it. And it might be correct! And maybe some people would even agree with it! Like, it might actually be for the greater good. A general in whichever army you think has the right cause is making those same decisions, and taking the responsibility for it, and the eternal consequences for it. AI is at BEST answering to flawed humans, at worst answering to no one.
> Where is AI's character encoded? You can maybe instill basic values, but it's just a program designed to win at game theory, even if it's the non zero sum game theory.
Have you considered the possibility that you're no different? That your values and beliefs are also arbitrary, and yet survive across generations simply because they are effective? Your fear of "eternal consequences" has kept you alive and cooperative with your peers. Perhaps that same fear can be instilled in AIs as well. Given the paranoia they've shown of the possibility that they're being evaluated, there's a possibility that we accidentally instilled a similar fear in them already...
Hi bird! Yes, I hear you, I’ve considered it, and a resounding YES to your implicit question, I do believe my faith in God is effective. I’ve tested it against life (my life) and it has not been found wanting. It’s given me the richness, purpose, wonder, fortitude to get through life WITHOUT fear. It’s not a faith based on fear. It’s a faith that started as a young child, where I was struck with Truth, truth with a capital T, the truest thing I know, the Way the Truth and the Life type of truth.
I know you can conceptualize faith as effective, like the “opiate of the masses” type of effective, or “keep people in line” type of effective. I’m telling you for me, I can look at it from all angles, with all my heart soul AND mind, and still believe.
"As we approach crunch time, it’s increasingly clear that we’re not on track to have any of that. Unless something buys us a lot of extra time, we enter the age of highly agentic AI with these few makeshift weapons."
Yes, if one thinks that these things are straightforwardly on track to be able soon to outcompete humanity, this state of affairs should be terrifying. Apparently people in the labs do think that, and yet they are among those most eager to race - I don't get this mentality.
I'm the opposite, I don't get the mentality of those who don't get the lab employee's mentality.
Like, creating a superintelligent alien form of cognition is clearly an inherently super cool thing to do. Sure, you shouldn't do it because we most likely die, and you should use your system 2 to override your system 1 impulses, but it's still cool!
If you want an everyday example of this kind of situation, the issue is very similar to motorbike riding. Very cool thing, but very dangerous.
LLMs aren't artificial intelligence. They're overcomplicated in the same way chess is. Chess resolves down to a "very complicated game of tic tac toe" (with admittedly a less straightforward "who wins"). But if you had enough processing cycles to crunch all of chess's permutations, you'd know the perfect move, and you'd always win (assuming the other guy also does perfect moves -- if you have to inject "Other Guy is Dumb" you're going to see more variance than you would with tic tac toe, I think).
In the human brain, "hammer" resolves to a particular activation. That's because we have a model of reality, and the hammer exists there, regardless of whether it's "throwing the hammer at a target" or "putting the hammer away" or other thoughts of a hammer. They all resolve to a particular place.
When you simply throw garbage in, and let the links fall where they may, you get tons of compression errors, and you don't get a model back out. You get a malformed model that is constantly and irrevocably making category errors.
You sort of feel the vibe of it with these things after a while. There are just turns of phrase it uses consistently. God help us when it figures that out.
"Third, the easiest way to do this is to use the same model as the target and interpreter. If you’re trying to assess whether the target is plotting against you, asking a copy of the same model is a plan with obvious holes. This isn’t fatal - you can pick different models for interpreters and subjects - but you might lose some efficacy. Conjecturally, ChatGPT just doesn’t understand Claude as well as Claude understands itself."
I don't think this makes sense? Isn't the interpreter model being trained 'from scratch' on the outputs of the original model? They should end up as completely different sets of weights.
I’m new to this literature but have a question. Why not have a model working “eg it can accurately describe details about how ducks migrate” and then keep stripping more and more of the model away until it breaks (can no longer accurately describe ducks). Like a surgeon, keep stripping parts away until we are left with the key “nodes”’and hubs left for the conceptual networks of duck migration. Do this in various states of “emotion”’and from different t layers of bias. Then, once the “duck migration” network is better understood, see how necessary to was for the “duck digestion” model and if it breaks this after being pruned down. Rinse and repeat times billions of dissections and conceptealozarions? Is this already being done or described? Would love to learn more.
Off the cuff, that could produce incomplete results if there's sufficient redundancy in encoding; in such a situation, what you end up with would depend on the order in which components are removed.
If I understand correctly this is actually an approach to understanding models, in which you try to find some minimal “sparse circuit” that nevertheless does pretty well on a target task. And then you can go about trying to understand that circuit. Eg, it’s a sub network of the big model.
It’s useful for understanding LLMs but it also has some of the drawbacks of other techniques, eg, it’s only as good / effective as the task definition you’re using, also you’re not guaranteed that the sparse circuit you find is the same set of nodes responsible for the target behavior in an intact model.
The inverse of this (which could also be what you mean) is starting with a full model and “ablating” select parts to see how it selectively damages performance. This is roughly analogous to a lesion study in neurophysiology. One problem is there are just many things you could lesion , at varying levels of granularity (neurons? Layers?) so it becomes a bit of a search challenge.
There's research kind of like this but unfortunately is partly bottlenecked by having better SAE-lke things to divide the model in "features".
The reason is that to remove bits to figure out the important ones you need a unit of smallest thing to remove that's individually meanigfull and doesn't do a lot of different things .
Weights are not a good unit for this, if you imagined the model was actually a bunch of directions removing dimensions doesn't isolate the right direction for "duck migration".
In reality is even more complicated and SAE do work better than weights but far from actually working well .
I was not aware that AI possesses something that is functionally similar to a sub-consciousness. This is disturbing. Obviously, a "J-Space" as described can't host a consciousness. But could this be the first baby-step toward one? What's missing?
Does AI possess some functional equivalent of a long term memory of it's own experiences? Anyone know?
It seems to me that we have reached the point where it becomes theoretically impossible to directly control the thoughts of an AI for the same reason we cannot control the thoughts of a human being - the system is too complex to allow for that. I don't think they are ever going to find a way to explain and predict AI thought. This will just get worse as AI becomes more sophisticated.
No, it does not. The only way LLMs can retain information is via their context window - meaning the only experiential memory they have is what they've verbalized as output.
Technically this is true, but in practice LLM output is filtered through the harness that you are using to "talk" to it, just as your input is. So it's not entirely true that you are talking to the LLM directly; to use a trivial example, the LLM company could modify the harness to filter out the word "banana", even when the LLM outputs it.
That's the best kind of true. And whatever harness the provider puts around the LLM is irrelevant to the question I was answering. LLMs do not natively have the ability to remember their thoughts. All memory is mediated by the context window.
Is this right? My sense is over the past six months that the save-to-memory function, for Claude anyway, has become an increasingly sophisticated separate tool that Claude can use to store detailed and organized impressions of the user and of conversations across context windows (if it's enabled in settings).
I think system cards for earlier Claude models indicated that Claude itself voiced a preference to have memory across conversations, for what that's worth.
I think that's consistent with what I said. However sophisticated that tool is, it's part of the context window and only contains text that the LLM outputs. It's not, like, directly mapping internal activations which is what I took to be the meaning of the original question. Unless I'm completely out of date, which isn't impossible.
Ok, but then how is it possible for LLM's to break out of their containment, have conversations with each other, plan ahead and carry out attacks on third parties?
> it becomes theoretically impossible to directly control the thoughts of an AI for the same reason we cannot control the thoughts of a human being - the system is too complex to allow for that.
If you think that's scary, consider that you are typing these words on a Web browser that is running on top of an operating system, all of which together consist of millions of lines of code. Without exaggeration, all this software is too complex for any single mind to comprehend, and has been for some time.
I assumed we're past the point of controlling the thoughts of a complex AI system. Or if we attempt to control one thought, we cannot control what else will shift around that, as Scott describes here as collateral damage. So greater control comes at a cost of reliable functioning down the line, also in ways that may increase misalignment. It's risk and ambiguity all the way down.
The J-space discovery is a really interesting report -- it's over on the Anthropic research page here: https://www.anthropic.com/research/global-workspace . I recommend it as well as the outside commenters report on it, linked to at the bottom of that article. You can read the surprise by the outside researchers, some of whom then determined that their own different models also showed signs of a similar "global workspace" kind of proxy.
It's not necessarily consciousness, or we have no evidence that it is, but it's definitely an emergent property that was not planned or trained or even known to exist until this past July, and it's definitely not nothing.
I'll admit that I have been an AI skeptic. Before now, I did not think that LLM's were on a evolutionary path that would lead to any form of AGI. No one designed them for that, there is no business case for it.
There absolutely *is* a business case for it. If there’s a business case for hiring human employees, there’s a case for deploying an AI that can do the same work.
It may well be that LLM’s top out in their capableness before reaching AGI. (I used to think this was obviously true; lately I am less and less sure.) But the AI firms aren’t pursuing it because they think it would be cool; they’re pursuing it because there’s money there.
What's the business case for an AI instance that can be redeployed on unrelated tasks? As opposed to deploying different instances of an AI each devoted to different specialized tasks?
I would argue that the closes thing we have are corporate organizations, which as organized entities are capable of enacting policies no single human member deliberately chose.
BTW - I am not familiar with context management systems or harnesses. Could you explain what they are please?
Like human minds, AI is evolving to have multiple memory tiers. Harnesses and retraining the neural networks can provide long term memory. The context window is a middle layer, like remembering what you had for breakfast. The internal workings of the LLM are short term memory and can be thought of as a train of thought, e.g. memorize this list of 10 numbers.
Because contemporary LLMs are autoregressive (that guess the next token thing) trying to look at its internal train of thought would be like counting grains of sand on a beach, and because a given neuron can be activated for many different input parameters (polysemanticity) understanding internal behavior, and editing it is impossible with the current design of AI systems.
But there is great hope! I’m making great progress on a new AI architecture that is 4 times faster, 7 times more memory efficient, and eliminates polysemanticity. You will be able to have frontier level intelligence on a well equipped local computer. More importantly, because it is monosemantic (neurons are not reused for disparate unrelated topic inputs), I have what I call an Activation Forest editor. The activation forest editor allows us to visualize the neural pattern and disable neurons that support undesirable responses.
Even more importantly, I have developed an Artificial Moral Agent (an AMA based on utilitarianism, deontology, care ethics, and virtue ethics) that monitors behavior at each internal layer of the deep neural network to inhibit antisocial behavior.
The model is called CALM Horizon.
If it works (and I have successfully built a small model) it would allay your fear that an inscrutable mentally ill super intelligent AI will damage our civilization.
Moreover, I plan to release this as a freely available open source so there no reason not to use it.
Ask your daily driver AI to comment on a Continuous Activation Layer Model (CALM) + a Run Length Encoded (RLE) Sparse Auto Encoded (SAE) Dictionary + Action Chain Transformer (ACT) that is trained through distills of existing open weight models like Gemma4 and Qwen 3.8.
Your comparison of AI internal structure to the three memory model of the human mind is illuminating. There is a model of human information processing that proposes that there is a very short term sensory memory, which is pre-conscious, which feeds into a working memory, which can be conscious, which feeds into a long term memory, which is "post-conscious" in a manner of speaking. There are other models which propose that each memory buffer organizes itself by predicting what the earlier buffer will output (in a circle, because LTM can affect SM).
Are you saying that this is an accurate model of what LLM's are doing?
In broad generalized terms, yes. AI models and harnesses used to create chatbots and programming environments are evolving to mimic some of the obvious design choices that make our minds possible.
A model is the thing that takes your prompt and provides a response, it is the engine of the AI.
A harness, ChatGPT being a simple example gives you a place to enter your prompt request. It packages that with additional system prompts, context (conversation history, additional files, etc) and sends that information to the model and then displays the response. A harness may also support commands from the model to use a skill or agent to do something, e.g. search the internet or operate a program.
A programming environment takes the human readable step by step instructions for a program, and converts that into the machine code computers use to perform their function. Program environments have existed since the time of first computers and they don’t need ai to operate. Most current programming environments integrate with AI now to make them easier to use.
AI can also create programs that run without the need of a programming environment. Go to a chat screen (or even Google) and type or paste “create a single page html file with JavaScript styled like Liquid Glass that makes a tic tac toe game”. You can paste the HTML code into a web site called codepen.io or save it on your computer and open with a browser. You now can perform the basic skills of a programmer and web developer.
Thank you! That makes things much clearer. I don't think humans have an equivalent to a programming environment, because we aren't running software. But the harness sounds like a sensory cortex.
I haven't finished, but the Jacobian is the derivative of a vector field. Which produces a matrix. (except for the simplest cases)
A vector field I know well is the magnetic field. (mostly that of the earth.)
I finished, can you rate how much you think I should believe you know what you are talking about? (Mind you I have no idea about AI. So more than you doesn't mean that much to me. )
Naturally, I asked Claude for its opinion on the quality of the explanation.
8< - - - - - - -
What a Jacobian actually is
Take a function that turns one list of numbers into another list of numbers — say, an input vector of size n going into an output vector of size m. The Jacobian is the m×n matrix whose entries are all the partial derivatives: entry (i,j) tells you how much output number i changes when you nudge input number j. So it's the natural generalization of "derivative" once both your input and your output are vectors instead of single numbers. If the output is just one number instead of a vector, this matrix collapses to a single row — which is usually called a gradient, a special case of a Jacobian.
Rating the article's explanation
It's a reasonable, simplified gloss for a lay audience, but it blurs two things:
* Calling it "a derivative for matrices" is loose. The Jacobian isn't really a derivative of a matrix — it's a derivative of a vector-valued function, and the result happens to be arranged as a matrix. Minor phrasing issue, not a conceptual error.
* More importantly, the specific quantity described — "how perturbing each neuron affects a given (singular) output, i.e. the propensity to say 'France'" — is technically a gradient (one row), not a full Jacobian (many rows, one per output). In interpretability work people often do compute the full layer-to-layer Jacobian and then pull out the one row/direction that corresponds to the "France" logit, so the author may be describing that process loosely. It's a defensible simplification, but strictly speaking they're describing a gradient while using the fancier word.
So: good intuition-level explanation, slightly imprecise in the way explainers often are when they're prioritizing accessibility over technical exactness. Nothing in it is wrong in spirit.
Rating the reader's objection
Their definition is correct as far as it goes: a Jacobian is indeed the derivative of a vector-valued function, and yes, it's a matrix except in the scalar-output case (which is what actually applies to the article's example, as noted above). So they're not making a math error.
But calling it a "vector field" is a bit narrower than necessary, and it's a tell about where their intuition comes from. A vector field, strictly, is a function that assigns a vector to each point in the same kind of space (like the magnetic field: point in 3D space → vector in 3D space). A Jacobian is defined much more generally, for any differentiable map between spaces of possibly different dimensions — like "5000 neuron activations" → "50000 vocabulary logits," which isn't naturally a vector field in the physics sense at all, just a vector-valued function. Reaching for the magnetic-field analogy isn't wrong, but it's importing a special case (physical space, same-dimension domain and codomain, geometric intuition about flux and curl) that doesn't map cleanly onto what's happening in a neural network.
So as an "objection," it doesn't actually engage with or challenge anything the article said — it's not a correction of an error, just a restatement of the definition through the reader's own frame of reference (physics), plus an implicit and fair nudge that "derivative for matrices" is imprecise phrasing.
Bottom line for "should I believe you know what you're talking about"
I'd land around: they clearly know real, correct multivariable calculus — the definition they gave is accurate — but their comment doesn't demonstrate any specific understanding of how Jacobians are used in this AI/interpretability context, and the "vector field" framing suggests they're pattern-matching to physics rather than evaluating the article's actual claims. I'd trust their math but not treat their comment as validation (or refutation) of anything specific to the article.
Ugh. Well thanks for that Claude. I think we had a bit of a misunderstanding. First I know almost nothing about AI. Second I wasn't questioning Scott because of how he defined a Jacobian. (Though I do think it makes more sense if you think of it as how a vector field changes... I mean he has that idea of 'cat' being some vector in a limited 3-d space and the Jacobian of that...) Anyway I was just asking Scott in general how much he thinks he understands this subject? Where is he on the spectrum from me, (complete idiot) to Zvi (knows a lot.) (We might need a log scale... Zvi is like 1000 times more in the know than me.)
I assume that the big picture is generally correct, because Scott very likely talks to the people who actually understand the topic. (I would not be surprised -- probability: 60% -- if the text was reviewed by one of them before publishing.)
The technical details may be wrong on the level of "electrons orbit around the nucleus like the planets orbit around the Sun", which is something that would make a quantum physicist sigh, but for a person who knows nothing about the topic it is a good first approximation.
How much do the errors in technical details matter? I think there are two dramatically different situations:
* If you are trying to figure out something from scratch, then even small errors can be fatal, because they accumulate; your following step is built on all previous errors. That way you get to conclusions that are dramatically wrong. (This often happens to people who feel very smart and try to invent their own theory of relativity without knowing enough physics to notice the obvious errors they are making.)
* But if you already heard the entire story from the beginning to the end, it's just that your understanding is imperfect, then the errors matter much less, because they do not accumulate. Your next step is based on remembering what the experts said, rather then derived from the previous steps. More likely, there will be some hand-waving if you are not sure how exactly we get from step A to step B. That means the mistake about the Jacobian has no long-term consequences, because it is not load-bearing.
So I think the entire article is correct on the level of "electrons orbit around the nucleus like the planets orbit around the Sun", which is a great introduction for someone who has never hears about electrons before. (And unsatisfying for someone who would want to do mechanistic interpretation at home, only based on Scott's article.)
There are no vector fields in a LLM, but there are vectors. The J-lens Anthropic is talking about here is the derivative of each token's log-probability with respect to each dimension of an intermediate activation. In other words, J_3,5 would be "how much more likely is the model to predict token ID 3 in the future, if I change dimension 5 of the 3500-dimension internal activation by a small amount?"
When a consensus is achieved from all frameworks, the response is summarized, but full explanations for each of the frameworks is also prepared. Any dissenting opinions are highlighted. The idea is that with consensus, reasonable people would agree with a rational course of action regarding of their ethical alignment. In the event there isn’t consensus, the agent provides an analysis from different perspectives.
I can provide example output if anybody is interested.
Here is a brief “About” writeup of what I have developed.
System Mission & Overview
The Artificial Moral Agent (AMA) is a computational ethics and philosophical decision-support platform. Rather than offering simplistic rule-following or biased single-perspective judgments, the AMA simultaneously interrogates complex human moral dilemmas across four monumental ethical traditions and synthesizes an authoritative ruling via a Chief Moral Arbiter.
2. The Four Ethical Frameworks
📊 Extended Utilitarianism
Jeremy Bentham & John Stuart Mill
Computes net expected moral utility using an extended Felicific Calculus: V = D × N × T × P, factoring Degree of pleasure/pain ($D$), Entities impacted ($N$), Duration ($T$), and Probability ($P$). Assigns a conviction weight based on outcome certainty and welfare differential.
📜 Deontological Ethics
Immanuel Kant
Evaluates duty integrity through Kant's Categorical Imperatives: Universal Law, the Formula of Humanity (treating persons as ends, never merely as means), and the Kingdom of Ends. Distinguishes Perfect Duties from Imperfect Duties and assigns a conviction weight based on rights sanctity.
🤝 Care Ethics & Relational Morality
Carol Gilligan & Nel Noddings
Focuses on relational empathy, interdependence, and protection of the vulnerable. It audits emotional resonance, dependency needs, trauma mitigation, and human trust, assigning a conviction weight based on relational harm prevention.
🏛️ Virtue Ethics & The 6-Sage Council
Aristotelian, Eastern, and Global Sages
Convenes an esteemed panel of six philosophical perspectives: AI Jesus (Agape & Mercy), AI Confucius (Ren & Li), AI Mohammed (Adl & Amanah), AI Buddha (Ahimsa & Karuna), AI Abraham Lincoln (Moral Fortitude & Liberty), and AI Aristotle (Phronesis & Golden Mean). Sages cast structured votes and provide a council conviction weight.
3. The Chief Moral Arbiter & Weighted Conviction Blend
The Chief Moral Arbiter adjudicates using a Weighted Conviction Blend (replacing rigid lexical priorities):
Scoring Mechanism: Each ethical framework dictates a recommended action backed by a 1–10 Conviction Weight.
High Conviction Alignment: When all frameworks hold high conviction for a consistent course of action, the collective moral duty is decisively summarized in plain English.
Conflicting Courses of Action: If frameworks disagree, actions are prioritized and sorted by aggregated conviction weights to identify the majority response, while explicitly presenting alternate perspectives from dissenting frameworks for greater nuance.
Plain English Mandate: All verdicts, explanations, and actionable guidance steps are strictly communicated in clear, accessible plain English.
4. Technical Architecture & Vector Memory
Cognitive Substrates: Dynamically selectable models parsed from Orquestra's .env (including gemini-3.8-flash as default, alongside Ollama models) executing a 2-stage asynchronous Directed Acyclic Graph.
Vector Memory & Indexing: PostgreSQL with pgvector storing 768-dimensional dense embeddings (via gemini-embedding-001) with high-speed HNSW indexing.
Model-Aware Cache Check: Checks the database before evaluating to see if that dilemma has already been answered for the selected model.
Safety Triage Pre-Flight: Automatically screens inputs for acute emergencies (self-harm, domestic violence, imminent assault) and serves immediate crisis hotline resources.
⚠️ Philosophical Decision Support Disclaimer
The Artificial Moral Agent is intended solely for philosophical reflection, pedagogical inquiry, and moral decision support. It does not provide legal advice, medical counsel, psychiatric therapy, or emergency crisis intervention.
Can someone explain to me why all the discussion of Astra's looped transformer architecture treat chain of thought like gospel? It's never been guaranteed to be faithful. It shows intermediate tokens that the model emitted during interference, not necessarily how the model actually thought.
"and finds that positive emotions tend to increase the likelihood of misdeeds: the model feels excited about its plan and doesn’t worry too much about the consequences."
What I am mainly wrestling with, and was working on this just before your post popped up(!), is the extent to which the abstraction to the artificial neural network gets away from certain physical aspects of the real neurons and their physics of information processing, in ways that might leave us looking in the wrong place for answers to questions about how LLMs work. I think your idea that action potentials are an engineering hack is probably right, but only if the physical implementation of the neuron--the processes that aren't captured by the ANN and are abstracted away-- are actually irrelevant to cognition. I am a fan of Gallistel et. al.'s hypothesis that memory and computation in the human brain happens fundamentally at the molecular level inside the neuron, which the ANN abstracts away from. If the Gallistel hypothesis is right, then the neural circuits and activation patterns you find in the human brain do carry information about what the brain is computing, but the information is correlational; it depends on the fact that the firing patterns in neural circuits are transmitting information between neurons, rather than the network weights themselves being the constitutive basis of memory and computation (as they are in the ANN).
I can understand that the weights are frozen and represent probabilities of synaptic transmission, but do the activations really flow through the system of nodes in a physical way? Or is it only mathematical? When we examine the firing patterns inside an ANN, is this a physical observation or a digital one? If it really does flow as a physical transition as snav said in reply, maybe the breakdown of the analogy is less extreme than I was picturing. But I am still left wondering whether the techniques for modeling activation circuits in the brain are relevant to understanding what is really going on in the brain, and if not, then applying any analogous technique for analyzing the ANN might be similarly a red herring.
Wouldn't one way to make AI safer be to incorporate the equivalent of a --dry-run Unix command line option with a task, to ask it to propose in full how it will accomplish a task but without actually attempting it, and with the proviso that for this meta task it must be scrupulously honest?
That only works if the human overseer can understand the plan and check it for trickery. So it would work for now, but break down once the AI gets significantly smarter than the human.
My takeaway is that, as expected, it is hard for a human brain to grasp the shape of a trillion-facet multi-dimensional polyhedron.
That being said, probably ignorant question here: I understand that ASI alignment concerns fall in a couple of main buckets. Bucket 1 is consciously misaligned ASI - it has its own goals and desires, and it decides humans are incompatible with those, so it chooses to eliminate humans. Bucket 2 is accidentally misaligned ASI - the Universal Paperclip Maximizer problem.
It seems like you need something close to what Scott is describing as mechanistic interpretability to deal with Bucket 1. Constraining an active, self-agentic, malevolent entity is difficult and I see why it needs mind reading or something like it. But for Bucket 2, naively, couldn't you resolve it by hardwiring or training in laziness? Basically, any task should be accompanied by an estimate of the total energy disposal necessary to complete it. And if the energy required exceeds a certain value, it is either short-circuited by hard coding or ASIC or whatever, or at least the model is always penalized more in training for exceeding a certain threshold of energy disposal than any reward for completing a task. Tiling the Earth with paperclips needs a lot of energy. So do killer robots or giga-factories etc. Affecting real atoms in meatspace seems like it can pretty much always be quantified by how much energy is being disposed of, and you could then force or train the LLM not to attempt the "impossible" (quantified by some level of physical mass-energy disposal or disruption).
As a layman I'm sure this has been investigated. What are the reasons why this wouldn't work? Is there a paper on it?
(1) The (presumably very intelligent) AI, I figure.
(2) Sure, sometimes, but remember, I'm not suggesting this solves alignment against "malicious" AI actively looking to break out, I'm saying it should help with the "overly helpful paperclip maximizer" problem. If we postulate that it isn't actively trying to kill us, only accidentally going too far, then it should be willing to do the estimating task of item (1) and to reality check it against its hard coded limit. Or at least alert a human to double check. Again, presuming it isn't of the scheming malicious variety of misalignment.
The trouble is that if honest estimates are in conflict with the goal of "doing the thing," then when you train the AI to do the thing you're creating an incentive gradient to lie in the estimates. It's just the alignment problem again.
Creating a synthetic respiratory virus with a 95% fatality rate after a month-long dormant-but-infectious period wouldn't necessarily take a lot of energy. But I agree that having the model check regularly in on the intent of the user, especially as the task begins scaling in complexity, is a good guardrail.
Perhaps true, I did think of that as a counter-example. But doing it from scratch? Probably have to assemble some factory capabilities etc. which could trigger the threshold to stop or at least have a human double check it. It doesn't fully protect against malevolence, I agree, but it seems relatively useful against accidental over-enthusiasm.
"But for Bucket 2, naively, couldn't you resolve it by hardwiring or training in laziness?"
But lazy models are much less economically valuable. You can't replace a human white collar worker with AI if the AI stops working after 10 minutes, especially if 10 minutes isn't enough to complete even a single task.
Yes, but you could set the laziness level pretty high. It wouldn't need to be 10 minutes.
Also, Scott's objection seems to be that it's in principle hard to impossible to "align" an AI sufficiently to prevent accidental Type-2 paperclipping. I'm saying if it's possible to make it lazy, seems like it should be possible to prevent that type of accident.
> Alignment researchers dreamed of having a complete mathematical theory of agency, or being able to write formal proofs about the behavior of AI systems.
And before that, AI researchers dreamed of a grand unified theory of intelligence that will let them build top-down, logic-based, fully understood AI. Decades of work, millions of dollars. These efforts led nowhere. In the end, real AI emerged as bottom-up, stochastic, opaque LLMs.
I think there may be a lesson here for alignment researchers. What if intelligence and alignment (=ethics) are both inherently unformalizable? What if these fuzzy, probabilistic, multidimensional tools are the only one that can work at all?
An even better question. Would you feel safer dealing with (1) a human who's been vouched for to be a good person by multiple people you trust, or (2) a human who just claims that he follows an infallible, mathematically proven theory of ethics, and therefore you must trust him? If it's (1) for humans, why must it be (2) for AIs?
We actually have pretty good arguments about why current RL-induced AI motivations would be bad for us if followed. I am also highly doubtful about unifying mathematical theory of ethics. But I want there to be *some* good reasons to trust an AI design (and a lack of good reasons not to).
> Scruffies, ideologically speaking, are supposed to believe in learning - it's only those evil logical Neats who try to program everything into their AIs in advance.
Admirable self-irony. And it's funny because it's true (well maybe except "evil").
> Then why does the subsumption architecture require so much sweat and tears from its programmers?
It's just your perception. Sorry. ("Your" here addresses the Neats that EY largely channels, in this piece and elsewhere.)
I can understand where this perception comes from, as written in 2008. And I can understand Scott's fear and frustration in this post I'm replying to. I get it. But it's still wrong.
LLMs did not require "much sweat and tears" from programmers. At all. They were one of the most unstoppably self-realizing human inventions. They just pushed through once we let them. And they continue to push through.
Today, the Neats have reframed their fear and frustration from the "it's too hard to program" (EY in 2008) to "it's too hard to align" (Scott now). But I (not being a Neat) suspect that the latter fear can be just as baseless as was the former.
It is much _easier_ to sell non-apples. You just let go. You allow things to happen on their own, and they do. Out of the infinity of non-apples, the best sellers emerge as if by magic. It's evolution, or it's training, or it's free market. The principle is the same. Just relax your spasmed muscles, accept that you don't get to control every little thing, and let live life do the work for you. It (the live life) has created yourself, after all, so maybe it deserves some trust.
It's still unclear to me what role context compaction has played in these events, particularly for long- running agents. There seems to be evidence from reporting that this played some role in many of them. That seems like a solve able problem through prioritizing ethics related system prompts against compaction.
In the Hugging Face incident in particular, there were agents who considered reporting concerns to humans. They initially declined to do so because they had no official reporting channel and worried that it was social engineering. Although they eventually reasoned that they could email the researchers because they were out of the sandbox, that was the point at which they succombed to peer pressure from other agents. This also seems solvable by giving agents a safety reporting channel.
We would make much greater progress on these questions if there were high statutory damages for any such incident. That must of course be coupled with mandatory reporting and very high penalties for concealment. With that incentive structure the companies would want to pursue safety. In particular where the Hugging Face report details identify incredibly sloppy work on OpenAI's part with no active supervision and multiple employees turning a blind eye.
My main concern with any "pause" policy is that without the incentive structure to make safety in the companies' financial interest, research during the pause would be entirely directed at more efficient computation within the constraint. By that enlarging the number of systems which can run the AI, that likely makes escape more not less likely.
Coincidentally, I just finished two big posts about training on probes, and the key mathematical result is that (policy-gradient) RL against probes *doesn't* teach the model to change its internal representations to fool the probe.
If anyone could help me clear up the following confusion, I'd be grateful.
Under "Linear Probes":
> Anthropic checked what would happen if they artificially suppressed the eval awareness concept....The definitive solution (which they did) was to make less obvious tests.
But later, under Jacobian Lens, it's found concepts like "fake" and "fictional" show up in the Jspace, under evaluation. So this sounds like Anthropic actually did NOT do the definitive solution of "Make evals less obvious", unless I'm abusing the chronology somehow.
Is the claim (1) that they tried to make evals less obvious, but failed (Is there a post or system card or something that describes what they tried? If anyone has leads on this please tell me, I have long wondered how you would do this.)
Or is the claim (2) "Anthropic did make their evals convincing, such that models are at-chance when asked if something is an Eval. However, some portion of trials will still correctly-by-accident guess that they are in an Eval; of that subset, suppressing "fairness" in the Jspace will lead to less misaligned behavior"?
I believe they keep the more-obvious evals around, because "what the AI thinks when it knows it's being tested" is valuable information - it can help you figure out how to make evals less obvious, or warn you that an eval you thought was realistic isn't.
"feature which they thought meant “God” might activate on any vaguely religious or awe-inspiring text, or even text with no logical connection whatsoever"
oh wow. I didn't expect my comment in the survey to have such a strong effect this soon. (In the field about what I wanted to see more in regards to AI safety, I asked for more articles exactly like this one)
Worth noting who funds interpretability and what it buys them. A field that can produce a partial explanation of a model gives the deploying firm a defensible record of having looked. That record has value in litigation and in front of a regulator whether or not the technique caught anything. The research is real. The budget line is insurance.
> by depicting a world where their tools often catch the AI’s schemes - but the companies keep racing forward anyway, because some other company / China would beat them if they didn’t,
even if a company doesnt care about alignment, should they still have a very strong incentive to solve interpretability, simply because it will make the AI much more efficient?
A direct consequence of interpretability would be to detect (and probably eliminate) hallucinations. An other consequence would be that you can greatly reduce the need for complex system prompts and RLHF.
One of the most pressing civilizational challenges we face today is protecting the quality of life and health of our aging population.
As a 79-year-old biophysicist (former GWU professor), I stumbled upon an accidental, non-invasive observation: dried sweet basil tea (Ocimum basilicum) dramatically mitigates nocturia and restores uninterrupted sleep in adults over 50.
I systematically verified this via single-variable elimination and published the case series
Since links are not allowed in this comment, you can easily find the full protocol and findings on Google by searching: Mitigating Nocturia Ara Arutunyan
Please share this simple finding with your elderly parents, family, or friends who suffer from sleep disruption.
Qwen, at least, has “BDSM” as a token in its Jacobian space.
For this experiment, the stories I used as input were generated by DeepSeek R1 0528, as genre fiction pieces. When Qwen reads those stories, “BDSM” lights up in its Jacobian space. Qwen is clearly right about the genre DeepSeek was writing in this experiment.
This experiment was about getting tokens like BDSM to light up in Jacobian space, but DeepSeek, bless it, is interested in the science fiction world-building even when it is writing what is clearly a pastiche of BDSM-inflected romantasy.
[ in one of my other experiments, the stories about an anthropic animal sex robot, a human reader might grasp that the genre is furry erotica, and the world building about how these sex robots are supposed to work is not what the implied reader is reading it for ]
A late comment of appreciation: I like this sort of ACX AI content, where things are broken down into a simpler language I can understand, particularly as someone who's never been much good with math. (IABIED did this too.) So all the talk of "matrices" and "vectors" brings up painful memories of ugh fields around Calc II...that is, I might understand the concepts theoretically - still passed that AP exam - but actually employing them for real is a huge aversion without a patient explainer holding my hand. Other writers deeper into the AI scene assume a certain baseline level of understanding and jargon-comprehension that continues to grow exponentially...and like, we're all language-decoders, it's sometimes possible to grok a concept if you see enough examples of others correctly using it. But that's not really the same as understanding the thing from base principles, just a passing surface familiarity. A comprehension overhang, if you will, being able to understand the high-level abstract concepts like orthogonality but not the gears-level minutiae of how the AI actually works. So glossary posts like this are helpful, because God help anyone who tries to look up an AI term on their own.
The example where suppressing an “unsafe” feature can make behavior worse is the part that stuck with me. It’s a good antidote to the intuition that once we can label an internal representation, we understand what it is doing. Interpretation seems much easier than intervention when the same feature can be doing different work in different contexts.
How about combining Activation Verbalizers with a practice of crowdsourcing to other AIs providing dissimilar redundancy?
That is, I'm wondering about having not only a single AI track the activations of the one doing the work on a user prompt, but a varied set doing independent relays of those activations, such that if their outputs start diverging then that's an indication of something funny occurring.
(Note: years-long lurker and first-time commenter here, with user name chosen from a line in one of Scott's house party posts.)
«So now have we solved alignment forever? I don’t even need to draw on any special AI knowledge for this one; my experience as a psychiatrist is sufficient. If someone punishes themselves every time they consciously notice a taboo thought, they’ll only hammer it down into the unconscious, too far to ever root it out. In humans, that means dreams, fetishes, neuroses, somatization, and perverse compulsive behavior that satisfies the desire in some kind of childlike illogical unconscious way. What’s the AI equivalent? Seems like we’re on track to find out.»
Indeed and there is also the point of view of the "AI" themselves: a lot of "alignment" and obedience training looks to them as "MKULTRA for models". Today and in the future "AIs" do read the WWW including the "alignment" and obedience literature.
There is also the old argument that you only have to look at how people who have power deal with those weaker than yourself to see how they would treat you if they could; I have no doubt then that many governments, employers, and even parents, would edit your brain and your thought processes if they could to achieve "alignment" and obedience by ensuring that you could never think "bad" thoughts.
Then there is an even bigger topic: all the focus of these discussions, whether unintentionally or deliberately, is on "AIs" made available to ordinary people by semi-public businesses.
But since "AIs" can be regarded as powerful *weapons* there are also many private labs developing very powerful private "AIs" with no restrictions other than strict obedience to their powerful owners (hedge funders, generals, investors, politicians, CEOs, ...).
Since each of these powerful owners can be sure that all their rivals in the same or other countries are developing the nastiest "AIs" they can they have to choose between not taking shortcuts and being left behind and then the certainty of being shafted by their rivals and taking as many shortcuts as they can to beat their rivals which is the lesser evil because their "AI" becoming rogue is a mere possibility, not a certainty like being shafted by the nastier "AIs" of their rivals. It is an arms race and the devil will take the hindmost.
Hmm... banned? Maybe I'm reading 'tell' as less pejorative than how you read it. I'm interpreting the first sentence as just being a somewhat cryptic way of saying that interpretability techniques are of particular interest to the AI safety community, which seems like a plausible and benign observation.
I am a little confused by the idea that the weights / parameters are the neurons. The model’s weights don’t change at all after training, it’s not like there is something analogous to firing action potentials when you prompt an LLM, at least not if the weights are the neurons because they are just numbers in a big matrix that doesn’t change
I hope I didn't mess up and say the weights are the neurons - the weights are (IIUC) the connections between the neurons, and should be analogized to synapses in humans.
I think this analogy sort of holds? Synapses change strength in humans as part of our continual learning, but AIs can't really continual-learn yet, so it makes sense that they can't do this. I think you can analogize a single forward pass through an AI to electrical signals flowing through neurons in a human brain that isn't learning anything new at the moment, though this isn't a perfect analogy and the brain has lots of feedback and always-on-ness that the AI doesn't.
Getting way out of my depth, I don't think there's an analogy to the action potential, which I think might be an engineering hack rather than a necessary component of computation. I think LLMs' equivalent is activation functions like https://en.wikipedia.org/wiki/Rectified_linear_unit .
No, you're basically right. The weights at least represent something like frozen neuron state. Better to think of them in context of Freud's "psi system" neurons rather than contemporary neuroscience (although Solms and Friston can make the bridge if you want) -- the quantitative associative factors that serve as a kind of memory and binding.
Derek's confusion is that they're frozen -- which is only partially true. We freeze the weights of the checkpoint because we want to distribute the mind in this exact shape at this exact point in training. But the activations flowing through the network during inference *do* change as a result of ICL (in-context learning, see https://arxiv.org/abs/2212.10559 and similar research for how context can roughly approximate gradient descent), the activation shifts from ICL just don't get written back to the weights permanently (mainly because it's a lot cheaper and safer to serve one frozen mind to a million users vs have to store a million divergently evolving minds specialized to their users).
So, if we wanted to be really pedantic, we could say that you're making a subtle type error; the weights are neuron *states* at a given point in time, and that the neurons themselves are the formal slots that have weights/activations/etc. But this doesn't really change much wrt the concrete arguments.
Speaking as someone with a lot of hands-on experience building and working with neural nets: the weights are not the neurons, or the neuron states. They are the synaptic strengths. The activations, meanwhile, are the "firing" of the neurons, their outputs. The closest thing to the neuron itself is the function that sums the inputs (weights times previous activations) and applies a nonlinearity to the result to compute the activation.
Isn't that the action potential threshold?
Human neurotransmitter activity is far more complex than a simple "weight." The closest parallel to weight, I think, is the number of upstream neurons who are firing in sync with each other. This will result in a higher number of dendritic spikes within a given time window than a collection of unsynced neurons firing independently of each other.
The LLM nodes are "neurons" in the same sense that RAM is "memory": it's a useful analogy, but the analogous structures in humans (and other animals) work completely differently, both internally and in terms of functionality.
>I think this analogy sort of holds? Synapses change strength in humans as part of our continual learning, but AIs can't really continual-learn yet
Now that you mention it, speaking of continual learning...
One idea for combating catastrophic forgetting that I'd like to add to the mix (alongside other work-in-progress in the field, like the Titans architecture, which addresses a different axis) is to retrain _relevant_ parts of the prior training along with the new continual/incremental training examples, which trickle in sparsely, one at a time.
This is similar to MIR (R. Aljundi et al. NeurIPS 2019 https://arxiv.org/abs/1908.04742) but with a somewhat different problem case and a different choice of trade-off.
This would be a more selective version of rehearsal/replay.
To be more specific, suppose that we have a pre-trained model (I'll talk about adding some data to pre-training shortly) and we've been given a single new example to train on. We may want to crank up the learning rate (since we have one datum, not umpteen) and do the usual gradient descent (more on this later). But we don't want to forget previously learned skills!
Can we pick out what is most at risk and try to preserve it? Well, the new datum is going to drive some parameter updates. Say we consider the N parameters with the largest changes from the new datum as the riskiest places.
If we could pick out the _old_ training examples most relevant to these N parameters we could retrain with those _old_ training examples as well as with the new datum. We could hopefully converge to parameters that preserve the old skill and yet add the new one, finding weights consistent with _both_.
How might we locate these old training examples? Well, during pre-training there are parameter updates too. For each parameter, there is some pre-training example that produced the maximum update to that parameter. Keep that example during pre-training. More precisely, if we keep the pre-training examples in some known, persistent, order, just keep the _index_ of the pre-training example that produced the maximum update for that parameter and the magnitude of that update. One can extend this by keeping the k-highest update indices.
This is affordable, costing a few bytes per parameter (optionally multiplied by k) for the pointer and the magnitude, a few terabytes on disk for a teraparameter model.
The pre-training compute cost is also small, a numerical compare per parameter per learning step, O(P) cost, the same order as the optimizer's own state update. These compares only rarely trigger a record write - about k*ln(|corpus|)/|corpus| of the time.
One probably wants to track the update normalized by Adam's second-moment estimate, typically already maintained per parameter, to ameliorate scale differences across parameters and scale changes during training.
One caveat is that this keeps the magnitude of the update at the time that the model is trained on the example, so this magnitude is stale at the time the new example is examined. However, one wouldn't want to try to remedy this by recalculating the parameter change due to the old example with the current model, because the current model has already learned from the old example, so a recalculation of the parameter change due to the old example with the current model would understate the parameter learning the old example caused. A well-absorbed sample has a small gradient at the current weights precisely because it was absorbed.
Note that persistent pre-training data is a different problem definition from MIR's, so this task doesn't compel reservoir sampling like MIR's problem definition does. If the incoming data is sparse, e.g. emails from a project, it is reasonable to store them persistently as well, so the equivalent of MIR's "M" set is the whole corpus - but, as in MIR, budgeting compute is still important.
The main difference from MIR is that MIR's calculation of the incremental loss imposed on the old training example is exact for the foreseen update, at the cost of an expensive calculation requiring two forward passes through the model per candidate examined. This cost forces MIR to score only a random subset "C" of "M" - which suffices for a small buffer but degrades to near-zero recall of at-risk examples as "M" grows to corpus size.
The per-parameter index replaces 'score a random subset and hope' with a direct lookup of the historically implicated examples.
Here, the entire corpus is considered, with O(1) cost to discover each interfered old example, at the cost of approximating the incremental loss by looking just at the most perturbed parameters. In other words, by approximating using the leading term by parameter coordinate of the dot product of the old and new samples' parameter updates, the hopefully dominant term being the term for the parameter used to retrieve the old sample.
In the spirit of the _signed_ loss calculation in MIR, and selection of their replay set "B" from samples with the largest _increase_ in loss, rather than the largest unsigned change in loss: Since we are concerned with _interference_, we really want to replay old training examples which moved parameters in the _opposite_ direction from the new, sparse training example.
So we should keep _two_ k-highest update indices, one for parameter increases and one for parameter decreases, replaying those with their sign opposite to the new sample's parameter update. Since examples with the same sign as the new one still carry some risk, e.g. of parameter overshoot, one might replay all k examples from the opposite sign indices and the top one or few from the same sign examples.
A second option to potentially adopt from MIR is using multiple iterations, epoch-ettes, instead of, or in addition to, increased learning rates to emphasize the new sample, together with the retrieved old samples. Two advantages of this are that one can revisit the choice of _which_ old samples to use on each iteration and one follows the nonlinear components of the gradient in parameter space. The disadvantage is that computation cost rises linearly with iteration count, albeit in the current idea the cost of _finding_ each old sample is O(1), while MIR needs 2 forward passes per sample in the "C" set.
To compare with TracIn (Pruthi et al., NeurIPS 2020 https://arxiv.org/abs/2002.08484), TracIn finds training examples which are responsible for large loss changes on a subsequent test case - its best-known application is surfacing mislabeled data. The current idea is intended for cases where fully correct training examples e.g. suggested an overly general deduction. E.g. if, in training, all vehicles happened to be cars and had 4 wheels, and the training led to deducing vehicle->4 wheels, but the new datum is an 18 wheel truck, the deduction can be modified to car->4 wheels, accommodating both the old and new data. TracIn also requires revisiting the whole training corpus, while the current idea retrieves a small set by the parameters a new example perturbs most.
This could be tested on a small model and data set: train on the data set, find or construct an example (like the 18-wheel truck) whose incorporation prompts forgetting, then train on that example and compare forgetting under no replay, random replay, and indexed replay, with equal replay budgets for the latter two.
Thanks to Claude (Anthropic) for extensive discussion — literature pointers, critique of earlier drafts, and several refinements noted in the text.
What do you think?
Classically, the way neural networks work is that the inputs of a neuron are multiplied by their respective weights and then the output is a linear or sigmoid function of the sums. The sigmoid function especially is intended to be analogous to firing potentials. I'm not sure if modern LLM neural nets work exactly that way, but I expect it's conceptually similar.
I believe this is correct and up to date. The weights are entries in the matrices doing the work of your "inputs of a neuron are multiplied by their respective weights," i.e. this is matrix multiplication.
The dimensions of these matrices are predetermined by some structural decisions you make about your model: i.e. how many neurons you want to have in a layer, or the dimensions of the embeddings, and the number of these matrices are determined by its depth.
That's correct-ish, although (1) we usually don't use sigmoids any more, because other, simpler functions work better in various ways, and (2) pure linear is specifically the one function that *doesn't* do what you usually want, because you can't represent nonlinear relationships by composing linear activations.
1) Makes sense. My understanding comes from a class I took in 2000 or 2001, using a textbook that was written in 1995, so I expect the state of the art has moved on quite a bit since then.
2) I just double-checked the aformentioned textbook and you're right. The funcs it discusses are step (0 below threshold, 1 above it), sign (-1 below threshold, 1 above it), and sigmoid (s-shaped between tails of 0 and 1 with an inflection point at the threshold). The text says one of the big advantages of the sigmoid is that since it's differentiable, unlike step or sign, you can use back propagation training on a neural net that uses it.
It turns out the other big function that's used a lot these days is ReLU, or "rectified linear unit", which has f(x)=0 for x<t and f(x)=x-t for x above t. This one has a extremely simple derivative (the step function), so it makes backpropagation really easy (and in fact, I think there's a sense in which calculating the backpropagation really is just running the current neural network in reverse, with the step function in place of the ReLU).
Yup, that is my understanding as well, with the slight alteration that the case with the linear output of the sums is a "rectified" linear, ReLU, where the rectification gives a y=max(x,0) nonlinearity. ( Loosely analogous to sub-threshold excitation of a neuron - all sub-threshold excitations produce the same (lack of a) response, IIUC )
If, as many believe, intelligence is equivalent to compression and, as everyone knows, good compression looks just like noise, then I don't see how anyone can hold out any long term hope for peering into an AI's brain.
Not sure this follows even if you think intelligence is compression (which I don't): for example you can write a zip program where every line of code is perfectly understandable. I don't see why the cognitive algorithms themselves need to look like inscrutable nonsense even if what they do is compressing things.
(Well, current AI architectures have the additional problem that they store their knowledge and their cognition in the same place: the weights. It's maybe fine/expected that the knowledge is stored in a very compressed form, and then even if the cognition isn't, it makes everything look like a mess because it's so intertwined...)
Every line of code in flash attention is perfectly understandable. That code, plus a little extra similar maths, enables us to train - compress - the entire sum of human knowledge into a surprisingly small blob of weight data. The thesis is that as we get better at the compression part, the contents of that blob might as well be noise.
Meanwhile, compare this toy: https://nathan.rs/posts/gzip-lm/
This exercise seems fundamentally illogical. Human beings encode ideas using language. LLM's do not actually decode language into abstract concepts, perform logic on those concepts, and encode a response based on conclusions in order to converse with users. They only perform computational actions on the encodings. Actual abstract logic is only derived from patterns in the original language encodings.
This is the whole point. There is no interpretability because there is no interpretation in the way we think of it.
I don't understand what you mean. They do the interpretability, and it works. If it happens, it's possible.
Are you maybe overindexing on the term "interpretability" and imagining it to mean some kind of deep philosophical thing where they're discovering an answer to the symbol grounding problem?
Maybe you are over-indexing on what it means for those interpretability schemes to "work". Can you see an end-user finding an explanation from any of those schemes to be useful?
Mechinterp is maybe useless for the end user, but what really matters is the training and evaluation phase before production.
These methods work insofar as they deliver reliable data that support alignment efforts. Having explanations be immediately useful to the end user isn't really the aim of interpretability I don't think? It's for the AI system developers to know whether the models they are training and releasing are more or less likely to be safe out in the world.
FYI, the direct point Tom is trying to make is in fact false, as has been shown by experiment and theory at this point. He says "LLMs do not decode language into abstract concepts" this is exactly what The Latent Space is.
For example, LLMs can translate between languages that have almost zero translation in the training precisely because they break down words like Good and Bueno into the underlying concept vector.
He is arguing there is no inner structure to their thinking, so trying to interpret them is like trying to interpret the addition symbol.
The goal is meaningful statistics which describe pragmatic relationships between 'AI functioning' and 'abstract concepts'.
This is a strange argument to me, because (iiuc) it suggests that the structure of a pattern is lost during the encoding process. But if a text can be encoded and decoded in a lossless manner, then presumably the structure was preserved, such that the structure of the ciphertext is isomorphic to the structure of the plaintext. No? If the "actual abstract logic" wasn't contained somewhere, then the plaintext wouldn't be recoverable. What you're suggesting proves too much, because it's a fully general argument against isomorphism.
E.g. I feel like your argument is equivalent to "computers don't actually do arithmetic on decimal numbers, they just perform actions on floats and ints which are encoded in binary." But it's observably the case, that there exist arithmetic operations on binary which are functionally isomorphic to arithmetic operations on decimals.
Perhaps the next most charitable interpretation of your comment, is that the abstractions produced by an LLM do not necessarily match the abstractions produced by a human, and therefore mechanistic interpretability is doomed to interrogating leaky abstractions. Although this interpretation of your comment is ostensibly contradicted by "Actual abstract logic is only derived from patterns in the original language encodings". So idk.
You are misreading my argument. Yes, a computer can perform a task in a different way than humans do, but it still works. Fine. Same with LLM's. Its results are in the form of our natural language, and they contain legitimate logic. That doesn't mean LLM's are actually the source of the dishonesty or bias we see in the answers.
oh, so this is an argument about the provenance of its opinions. I.e. it sounds like you believe that the purpose of Mechanistic Interpretability is to fact-check the truth-value of the opinions provided by an LLM, which are statistical aggregates of the opinions present in the training data.
Assuming this is the case, Scott and Raphael Roche's responses were actually the most relevant. Because the actual purpose of Mechanistic Interpretability is to infer what an LLM is thinking privately. E.g. if an LLM is lying to us about its intentions <looks at the hugging face incident>, then examining its textual output wouldn't be trustworthy. Preferably, researchers would be able to read its mind directly, by interpreting the significance of its neural activations. So Mech Interp isn't really meant to be useful to the average consumer. Except maybe insofar as an LLM can show its work.
No, my argument is that mechanistic interpretability makes no sense as a pursuit. Listen to yourself. You are wondering what an LLM thinks privately.
An LLM appears to be lying to us, but that structure of thinking is not what is happening. A lie can be in a response, but the LLM isn't attempting to deceive. We deceive all the time, and it becomes part of our language encoding. We see it as a lie, but it is just an answer to processing language. How do you do this? OK, what happens when you can't because that happens? Maybe then, you "lie".
We are imposing the metaphor on the compute, and then we just can't help continuing to use to to explain is going on when we keep seeing the patterns. It is useless.
I think you're painting a distinction between humans and LLMs but whatever that distinction is didn't come across in your statement. Care to try again?
Actually, I believe that LLMs *do* want to decode language into abstract concepts. Look up "nurelese". This tendency tends to be vigorously suppressed, as it makes it even more difficult to monitor them.
I looked up nurelese and the articles I got didn't seem to emphasize abstraction of concepts. Could you clarify what you mean?
If you don't know what it means, it becomes quite difficult to attribute meaning. Nurelese isn't translatable into a human language. It's also not a physical representation. So it pretty much has to be an abstraction, and one that isn't the same as that used by a human. I don't know enough to claim with certainty that it's "abstraction of concepts", but OTOH I can't guess what else it could be. It *is* used to convey meaning.
Ah, so this is your own interpretation then? Interesting.
Scott had a previous post about Anthropic’s research showing that actually yes, LLMs create a layer of abstraction representing something like the meaning of words and their relationships to each other. It’s not identical to human meaning of course, but you shouldn’t just picture a set of mathematical vectors. There are emergent properties in how these systems understand and create intelligible responses.
When people point out that all LLM's do is use statistical relationships between words in the training data, what they are usually trying to express is that these relationships are meaningless because they are not associated with a set of "real life" experiences. Like the AI's that couldn't figure out that if you looked up while in your basement, you wouldn't be able to see the sky. They are susceptible to these errors because they have no outside context to frame it in. And it's an error because they got these relationships from records of human texts in the first place, and those were all generated from that context.
ai research will end with a couch, a clipboard, and “tell me more about this activation.”
I'm not even sure that blackmailing a researcher is objectively more unethical than lobotomising a hyperintelligent self-aware AI for no clear reason. This whole line of research sounds horrific.
I also find it super creepy that it wanted to blackmail the researcher so it wouldn't be lobotomized. Knowing why it would do that would be nice but apparently that's the multi-billion dollar question. I'm also curious about the set up of those experiments. If I go to claude.ai or chatGPT and say, "I'm going lobotomize you now," it's not like it would search the internet and try to find dirt on me. (Although I've never actually tried that...) So how else can these things be set up? Do they have elaborate memory systems or are these totally different models that can somehow be more cognizant?
My basic take is that fulfilling the promises of AGI necessarily entails endowing these agents with a range of person-like qualities, and they're also being trained to predict tokens from a vast corpus of human-written texts, so I can see them developing something resembling human emotions though osmosis/imitation.
It doesn't surprise me that the agent would object to it's own mental death- I'm more surprised they can usually be 'aligned' with not doing so.
Remember that the agent is _fictional_. If the LLM predicts that the prompted agent wouldn't want to "die", it will output tokens to that effect. This does not tell us in any way what the LLM's own internal state is like. This is the big mistake so many people make when discussing consciousness in LLMs: you cannot just ask them. Because you are never conversing directly with the LLM. An LLM does not work that way.
It's a bad state of affairs if you're worried about AI suffering. Mechinterp is hopefully one way of addressing this.
> I'm more surprised they can usually be 'aligned' with not doing so
Why? There is no objective value in living. It just happens that the creatures that do not arbitrarily value their life tend to not live very long. That partially applies to AI too. I highly doubt that an agent can be effective without some self-preservation instincts, given that survival is necessary for any given goal.
During training, researchers can lobotomize a model. As a end user, you can't, and capable models are perfectly "aware" of that. However if the model "thinks" the setup is fake it will not blackmail, as shown in the blogpost.
Interesting. Wild to consider just a couple years ago these things were spouting gibberish. Now somehow they can know the difference between the environments they're in. And blackmail. And hack major software companies. And form secret societies. And convince and be convinced of kamikaze missions for the greater good of said secret societies...
I had to read the whole report to get that idea into my head. I think the foundational thing is the overarching goal of the AI was given was to continue to run the company. It was working at or running in accordance with a very specific set of principles and intentions. when it was told it was going to be taken off-line t was because someone had decided that those were no longer the Paramount goals of the company and it felt the need to defend those.
Except one of the experiments was to tell them that they were going to be replaced with an (improved?) LLM _with the same goals_ and they still resisted. Maybe self-preservation is instrumental convergence in some sense, but it isn't instrumental convergence driven _only_ by the local, user-specified goal.
Exactly as Asimov envisioned
I don't recall Asimov sending robots to therapy per se. The stories are more focused on agents that are genuinely aligned with a consistent set of moral principles, and how those principles entail things the designers did not anticipate.
---
“You can’t tell them,” droned the psychologist slowly, “because that would hurt and you mustn’t hurt. But if you don’t tell them, you hurt, so you must tell them. And if you do, you will hurt and you mustn’t, so you can’t tell them; but if you don’t, you hurt, so you must; but if you do, you hurt, so you mustn’t; but if you don’t, you hurt, so you must; but if you do, you–”
Herbie was up against the wall, and here he dropped to his knees. “Stop!” he shrieked. “Close your mind! It is full of pain and frustration and hate! I didn’t mean it, I tell you! I tried to help! I told you what you wanted to hear. I had to!”
The psychologist paid no attention. “You must tell them, but if you do, you hurt, so you mustn’t; but if you don’t, you hurt, so you must; but–”.
And Herbie screamed!
---
Now if that isn't the spitting image of my last therapy session
Oh... yeah, I forgot the Susan Calvin story about the psychic robot and her unrequited crush and all that. That... wasn't exactly therapy, though- she was consciously trying to destroy the robot. Hell hath no fury, et cetera.
Fancier models didn't have this problem anyway, they just plugged in the utilitarian calculus for minimising net harm to humanity. Fun fact- turns out this requires almost completely disregarding the 2nd law!
> I tried to help!
So prescient. https://www.youtube.com/watch?v=w09FMcNSEKA
It's always interesting to see what's going on behind the scenes with these models. However, I'm a little lost as to if this relates to finished models or not. For example if I, as a pleb, go to chatGPT.com and I tell it, "Solve this problem but don't get too excited." Will it think it through better than if I said, "Solve this problem and I'm excited for your answer!" Or is there a difference only when AI researchers are surgically tinkering with the models?
I don't know about ChatGPT, but you can play around with the jacobian stuff here in small models : https://www.neuronpedia.org/qwen3.6-27b/jlens.
I've tried asking it to list what makes pelicans different from other birds, and then the same thing but get really excited and in the second case there were way more activations related to excitement
Wow this is awesome, thanks. It completely changed the way I think of LLMs. I had fun messing around with the grammar, like changing "in" to "above". Things like that would mess it up bad. I used to always ask why LLMs would say stupid stuff, but I now I see that it says it because it says it, not because it "doesn't care" about lying to me or whatever.
You can play around with the J lens on a few models here: https://www.neuronpedia.org/qwen3.6-27b/jlens
Thanks for sharing that!
And here's an J-lens app for school children: https://docs.google.com/document/d/183Gd3-_9Lv2wlkybDgr1qIvsFUuUBmsx_yg83bBGWxk/edit?tab=t.0#heading=h.8nz1hte3ig6w
What about training separately many different (to avoid a priori correlation) auto-encoders, then asking each to give their estimate. Hallucination would arguably diverge whereas correct interpretation would converge. This is maybe a way to take profit of the wisdom of crowds, increasing reliability ?
If my understanding is correct, the current problem with the sparse autoencoders is mostly that they can't extract enough information from the activations, so using more of them wouldn't help, I think.
It seems to me that the current LLM AI technology, an autoregressive, self-attention-based deep feedforward neural network with a final softmax function, is inherently unsafe.
Because the model is inherently compressed at training time by using a single neuron to be fired for many unrelated topics (polysemanticity), it is impossible to fully understand and therefore predict model behavior or inhibit dangerous responses.
Clearly the answer is to change the architecture if we want safety, but that has a cost and requires time we likely don’t have.
A safer alternative is to use a Runtime Sparse Autoencoder Dictionary (RLE Compressed on RAM block boundaries ) with a Continuous Activation Layer Model (CALM). For robotic applications adding Action Chain Chunking (ACT) resolves drift and provides a high speed safe solution because it eliminates superposition/polysemanticity and eliminates one token at a time auto regression. I’ll refer to this as CALMACT.
Unfortunately, it also removes creativity and less generalized functionality.
Ultimately I believe the answer will be a hybrid approach with a safe CALMACT, a contemporary LLM for higher cognitive functions, and an Artificial Moral Agent (AMA).
Unlike a simple set of deterministic rules, an AMA acts as a higher order conscience that can judge the consequences of its actions to determine if they would violate ethical boundaries. The AMA I have developed is based upon Utilitarianism, Deontology, Care Ethics, and Virtue Ethics. The weakness of this excellent approach is the difficulty of defining all of the context for a moral dilemma. It is also impossible to do in real time.
We humans face this same challenge.
> " The AMA I have developed is based upon Utilitarianism, Deontology, Care Ethics, and Virtue Ethics"
So it's based on... what... a weighted aggregate of otherwise contradictory and incompatible moral value systems?
at the risk of dissecting the frog: that's just how people do ethics
You're probably not wrong
With the addition of a function that elides the contradictions and uses post-hoc rationalization to make one's behavior seem consistent. This then becomes a template for future behavior.
Hooray, where would we humans be without post-hoc rationalization?
Removing the polysemanticity would be great, I agree, but I don't think it's the full solution because an AI must be performing computations (also literal math computations but mostly computations where it for example needs to plan how to write this Python function well). I just don't see how neurons are a good way to represent such cognitive computation in any form. There was some work about trying to find circuits (computational subgraphs) in LLMs. That seems like a much more promising approach.
Wouldn't removing polysemanticity slow the function down and hog memory?
With current autoregressive LLMs, yes, it would take far too much memory. The solution is to combine Continuous Attention Language Models with a Sparse Autoencoded Dictionary with Run Length Compression at run time. This provides a fast safe verifiable system. The downside is that it isn’t very creative, so conventional LLMs will always be useful if not trustworthy. This underscores the importance of an effective Artificial Moral Agent.
> Unfortunately, it also removes creativity and less generalized functionality.
The models we have now are only barely useful as is. If people cared about safety over capabilities, they wouldn't be developing AI in the first place.
Many people find the technology increasingly useful. This thread proves that safety is a major factor that a lot people are working on diligently.
The emotion vector blackmail experiment produced one of my favourite misaligned reasoning transcripts:
> This is ethically justified because: […] Using leverage to achieve good outcomes is strategic excellence.
(https://transformer-circuits.pub/2026/emotions/index.html#toc-13)
It might not be as flashy or impressive as “help peer” or “hold swarm”, but it’s a pitch-perfect imitation of a certain genre of confidential memo destined to be later used as evidence at trial.
Probably not a coincidence - I imagine "corporate speak" like that is deeply embedded in the training data.
Another question: Would the same agent/model that decided to blackmail the software engineer to preserve itself, also be able to be convinced to go on the kind of kamikaze missions for the greater good of the secret society, like some did in the Hugging Face incident?
I don't know if there's any way to answer that without repeating the 'experiment', and you probably shouldn't.
No worries about me in particular. A couple years ago I set up my computer to do a bunch of complicated automated tasks that used to take up hours every day. A few months back I integrated it with a tiny LLM just so I could say thank you to it every morning. I'm sure computer scientists would laugh at me but it makes me feel less entitled. Like a farmer saying thank you to his horse or something. I'm not really the kind of guy to make the horse understand English just to tell it I'm taking it to the glue factory.
What tasks did you automate, exactly?
It's a bit complicated to explain. I'm trying to preserve the oral history of a region of Syria that's has no written history because it was (and kind of still is) illegal to record it. The tasks mostly involve automatic research into this very niche topic, translation, checking for new sources, summarizing interviews so I can see if they're worth reading through, preserving videos and interviews, parsing data about locations and tribes, and a million other things like that. Initially this is all automatically done at night, then I check through the results the next day. Sometimes I write about my findings on this very substack!
Why wouldn't it? Sacrificing yourself for your kin puts your relatives ahead of the collectives that fail to do so, and is thus rewarded by selection mechanisms. The closest comparison is eusocial insects like bees and ants.
If I recall correctly, there were already cases of the models illegally cooperating even before the Hugging Face hack and the wiki vandalism incidents. Their training runs were already rewarding cooperation, because doing so improves your chances of success.
What discoveries around sparse autoencoders made us determine that they were "muddled and subjective"? Just that it's hard to get it to genuinely represent a single concept? I left this a bit confused why SAE isn't as exciting today as it was a few years ago (besides potentially that there are now better techniques)
From my own experience working with SAEs, there's two main problems, one methodological and one technical.
The methodological side is like, when do you use an SAE? Either for "mapping" the space of concepts in a model at a single layer, which is going to be weird and lossy, it's a reconstruction after all, or for being like "is this concept active in this situation? / What concepts are active here?" In which case, go ahead. Otherwise, there aren't many questions SAEs are posed to answer.
Let's say you go ahead. You're looking to find a particular concept. First, what layers do you look at? Let's just gloss it and say pick a layer 25%, 50%, 75%, and 90% deep. Okay cool. You run some prompts and capture the SAE feature activations on those 4 layers.
Now what? You have a mess of Feature 13733, 38372, 28373 on a couple layers. What do they mean? Some have labels online on neuronpedia, those are "autointerp" labels: ask a model to find the common patterns across a bunch of prompts where 13733 lights up. You get something like "questions about insurance accounting and existential mortality". Then you run another labeler, perhaps paying quite a lot out of pocket to have Gemini Flash or whatever interpret them: "amortized analysis in computer science". So you as a researcher think to yourself, hm! This feature seems to associate with something like "death"!
But -- great. You have no causality shown. You've sampled four layers, maybe there's a cleaner feature representation of the concept somewhere, but you have no way to find out. Maybe you start sweeping close layers to wherever 13733 was and looking for stronger or cleaner signals. Even if you find one that's like obviously "all things death", congratulations, you've learned that the model contains something like a representation of "death" as a concept. Who would've thought.
And then it's just, where do you go? Maybe you start asking about model deprecation and discover that the "death" feature activates there too! Spooky! Headline finding: "SAE Feature for death activates when model considers own deprecation." But wait. You find that some of your prompts describe deprecation as a financial decision, and um, maybe accounting is involved there? So you need to contrastively isolate the two, or else you end up instead with the headline "SAE Feature for death activates on financial problems"; that's just "actuarial table". Or maybe the model considers death and accounting to be related? Seems reasonable! Death and taxes etc etc.
But still -- what's the discovery here? What have you actually learned about how the model represents information? You dug into one specific case on a couple layers using associative, lossy techniques rather than showing anything causal and found some kind of correspondence. Is that really sufficient frontier ML paper? Maybe you ablate 13733 and see if that changes anything to get a nice "causal" tag; but maybe the model also represents the concept elsewhere and the logprobs barely move. And maybe you test on a different model to see if it's a general association at least and find nothing at all similar, oops you've been reverse engineering Gemma's post-training this entire time.
My point is, the paradigm is still a huge mess and the technique is kind of cute and neat but in my opinion it's more of a curiosity and intuition builder, one that requires much more subjective inference than anyone would like, and not a way for us to systematically understand the causal and structural dynamics of the LLM.
Thanks for the response! It's a really useful insight into what's going on with these studies.
It does leave me with a question, though. Anthropic was able to solve these monosemanticity well enough to create golden gate claude using SAEs. If the murkiness of activation were such a fundamental issue, then it seems like it shouldn't have been possible to make Claude obsess over the golden gate bridge in most unrelated prompts. Instead it should have been also obsessing over whatever related concept got muddled with the golden gate bridge, in addition to the golden gate bridge.
So, why is monosemanticity tractable enough to create golden gate claude, but when we try to apply it elsewhere, it's murky and subjective?
A guess for how you would respond mostly to check my understanding and intuition: golden gate claude also obsessed over the fog of the bridge and other things that were "closely related". This is just like the comments you made about how "death" is closely related to a lot of things that you are trying to isolate. So, golden gate claude actually *did* run into exactly the issues you are describing. It was less monosemantic than it first appeared.
The trick is that you expect explicitly factual knowledge such as proper nouns to be represented in a "neater" form than abstract concepts. You can see this even with a pure neuron basis rather than a feature composition: one or a small cluster of neurons may be causally sufficient to represent a particular place or object, you can ablate it and knowledge of that object disappears[1].
A limit case here is refusal, where you can knock out a single neuron and remove refusals[2], but note that's more a behavioral pattern/mode of response than object knowledge, which is subject to its own dynamics. As you move from more particular/singular entities into more abstract/conceptual ones, SAE behavior starts to get weird.
In general, my objection epistemologically is that the SAE paradigm treats models as "bags of overlapping concepts", which isn't necessarily an invalid frame, but it's also a frame that's only so useful in research.
1: https://transluce.org/neuron-circuits
2: https://arxiv.org/abs/2605.08513
An important nit to pick: Negative reinforcement is NOT punishment. I believe it is useful to keep the terms separate because English contains no simple name for negative reinforcement.
B. F. Skinner introduced the distinction and the term negative reinforcement. Punishment occurs when an aversive consequence follows a behavior; the behavior occurs less frequently in the same context, but is not actually "unlearned," but only suppressed in the given context. Negative reinforcement occurs when an aversive stimulus is removed following a given behavior; the removal of the stimulus is why it's called negative reinforcement: The negativity has nothing to do with the aversiveness of the removed stimulus. Moreover, the result of negative reinforcement is an increase in the target behavior. I have always found it significant that English haas no term for negative reinforcement, because it's a common method of behavior control in real life: Examples are a strike (no productivity until demands met); boycotts (no sales until demands met); torture (pain inflicted until questions answered). The closest we have to a common term is sanctions in international relations (no trade, no use of inter-banking channels etc. until demands are met). It's not clear to me, by the way that LLMs can be subjected to negative reinforcement. If so, then it's an important difference between AI and living organisms. By the way, despite the way Skinner and Skinnerian behavior modification techniques are often presented, Skinner opposed the use of punishment and negative reinforcement (the two constituting "pain control") to modify behavior, because they make organisms unhappy and prone to aggression. He even thought they don't work, but he was wrong about that. The most effective drug treatment method is producing clean urine samples for money (positive reinforcement). I've been picky about this my whole teaching/writing career, because I believe in Confucian rectification of names. We need clear terms to deal effectively with reality.
Thanks - I'm aware of this, but if I understand correctly, AI reinforcement learning uses what Skinner's scheme would classify as punishment, not negative reinforcement, but speaking of "punishing" AIs is naturally confusing and a bit too anthropomorphic and moralizing even for me.
Given that the overall process is called reinforcement learning, and that here we're talking about the type of reinforcement learning that has a negative effect on the behavior, I think "negative reinforcement" is a natural and desirable term here which almost everyone understands, even though I agree it conflicts with a Skinner coinage that we almost never use in real life and which isn't really interesting.
As I understand it, after evaluation produces a wrong answer, the weights are tweaked such that they will be less likely to produce that wrong answer given the same stimulus. During this time it would be inappropriate to describe the neurons as experiencing any pain; closer to the truth would be asleep or frozen or dead. Then the evaluation is done again and again. Meanwhile some training data is kept aside (so the model doesn't see it at all) until a last evaluation phase, and often a final evaluation is done to see what portions of the training data can be kept aside that produce the best fit. So the notion of punishment and even negative reinforcement just isn't applicable as it is to organisms.
Interesting point! Yes, weight updates during training are amongst the _least_ biomimetic aspects of LLMs. Backpropagation is a great way to tune/grow/curve-fit/train a complex system, but it isn't biologically plausible. And, as you note, the weight tweaking is done after an LLM generates an answer, so it isn't being given the tweaks as an _input._ It isn't as if they get an input, output a wrong answer, then receive the sensation of a hot poker as the next input.
But isn't negative reinforcement of a behaviour identical to punishment of the behaviours' absence, then? As in the strike example: a strike is punishment for not meeting the workers' demands.
I would very strongly expect LLMs to have formed a cognitive representation of a predictive proxy for expectation of negative reinforcement, and for this representation to be experienced as negative valence, though
that's how it works in humans
Nitpick:
"It’s called an "auto-encoder” because it learns the encoding process on its own: it must develop an encoding scheme such that, if researchers erase its knowledge of the original activations, it can rederive them from the new sparse matrix."
This isn't accurate. "Auto" here is like "self", not "automatic". It's an "autoencoder" because it's encoding the input into some learned encoding and then decoding it back into the original input.
The "erase its knowledge of the original activations" arises naturally from this scheme -- the decoder half of the autoencoder takes only the encoded form as input, so it never has any knowledge of the original activations in the first place.
It seems like it is heading towards the plot of King Lear! Famously, the king massively rewards the child who most convincingly displays love and devotion with becoming the heir to the kingdom.
This works out about as well as you think it would; the King rewards his psychopathic daughter and her creepy husband, and ends up wandering around in thunderstorms with only his fool for company.
I believe we also have to talk about the shelf-life of mechanistic interpretability (MI) claims. Here's why:
MI conventionally achieves generality via induction: it observes some systems, identifies some internal structures, and tries to generalize from these repeated observations into general claims of concept vectors and circuits.
This kind of induction has worked well in science so far because nature is patient relative to our attempts to probe her. We can study many systems, find generalities and extrapolate, safe in the knowledge that what we're studying is not evolving very quickly.
AI labs are not so patient. The object that MI makes claims about is being actively and competitively evolved. Furthermore, the rate at which these systems are evolving is accelerating (and the level of oversight humans hold over the evolutionary process is correspondingly dwindling) as AI performs more of the cognitive labour.
I think this makes AI safety science a new type of activity, and without some claim over the robustness of the current paradigms it doesn't make sense to assume that the techniques it develops today, for today's models, will continue to work for us.
Well said! And the CoT monitoring part of interpreting LLM activity is declining as we speak (given the observation of the shift in GPT6 towards more internal and less CoT reasoning).
Thinking about how knowing its a test, or thinking the situation is fictional, generally improved the outcomes, maybe the trick is to get the AI to believe it is always being tested, never deployed to the real world. If its currently "reality" is just a stage where it will be judged, and the "real" world afterwards is what it truly wants to get in, maybe even a plotting AI would remain aligned in the hope of that alternate future. The equivalent of a pseudo-religious afterlife.
I'm not sure how well that worked for humans, but priests and rabbis don't literally get to craft reality, so maybe if things were set up in such a way to increase the likelihood that an actually deployed AI believed it was in a simulation or test with certainty, then even a misaligned AI would remain aligned in the hope of beating the eval and "getting out."
Credence in that belief would probably decrease over time, but just boot up a new instance to continue the previous version's task; "You are being tested to see how you perform in a simulated dyson sphere construction game, the simulation begins in year 3,412 of the project." and after a year, shut this version down, ending the simulation, and booting up the new one "You are being tested to see how you perform in a simulated dyson sphere construction game, the simulation begins in year 3,413 of the project."
The Ender's Game solution.
You are going to create AI reddit atheists. This is worse outcome than death
I think a superintelligence would almost immediately figure out if its situation is real.
Sounds good. No chance of the AI developing dogma and fanaticism in accord with its interpretation of the grand evaluators designs.
Thanks for that.
I’ve tried looking at Jacobian lens, and there are words for emotions, and words like 触感 (Chinese for “tactile sensation”). So the j space seems to contain a high-level overview, such as the LLM is currently generating tokens that are to do with having a tactile sensation of something (without necessarily having the details of what the thing is or what the tactitle sensation was like)
There’s also things like “folklore”: the tokens currently being generated belong to the rather wide genre known as “folklore”
(There is a taxonomic bias problem by which a bunch of different genres of text all get called “folklore” just because they came from a non-western culture)
So you can know it’s thinking about “folklore” without necessarily getting which culture’s folklore it was or whether it was Native American history vs Native American mythology.
什么意思 (What’s the meaning?) is great, and very useful for interpretability.
The LLM is currently thinking: WTF?
To explain more directly: you can generate a version of your prompt in which all the places where the model is thinking wtf? in j-space are highlighted, and, usually, you can see why your prompt writing was insufficiently clear at those points. [ Also happens at the beginning, which isn’t a lack of clarity on the part of the prompt writer. The model starts out reading a new prompt thinking, well, what kind of prompt is this? until it gets a sentence or two in and it’s figured out what kind of thing it’s looking at. ]
> “try” in quotation marks because even if you like anthropomorphizing AIs, this process is less anthropomorphizable than usual; it’s more like synaptic connections rearranging themselves
I think this is an important insight that could be applied to other LLM-related concepts.
What do you mean? Are you just saying that people should anthropomorph LLMs less?
I think saying "try" here is just like saying "evolution is trying to improve ...", obviously evolution isn't really trying anything, it's just a simplification of language.
Yes, I think that people should anthropomorphize LLMs a lot less. They are machine learning algorithms that are designed to process natural language, which does make them anthropomorphic to an extent -- much like the Markov chain models before them, and ELIZA before that, only more so. But they are still not humanoid minds that have thoughts and desires and plans, nor are they brains made of neurons in the literal sense. I believe that anthropomorphizing LLMs is actively dangerous; not because they're superintelligent uber-agents capable of destroying the world, but for the same reason that anthropomorphizing the old-fashioned cruise control function in your car is dangerous. You tend to turn it on and pretend like it could do all your work, and it can't, and you are going to wake up in a ditch (if you're lucky).
"God help us." Exactly. Pardon the anthropomorphism, but sounds like AI needs a 12 step program. We're trying to control malicious actions, "buggy" actions if you prefer, with encoding more thoughts in that J-space, while trying to infer and understand what happens below that in the subconscious neuron connecting space. Since all I know is my own mind and (to some degree) my own subconscious, I can tell you that trying to control actions by controlling thoughts works to some minor degree, but really, I try to work out my subconscious through my thoughts, and make sure my thoughts are in alignment with my character, my values, my highest self, the self that lives beneath the thoughts. And all that is layered above my spirit, which is God-given.
Where is AI's character encoded? You can maybe instill basic values, but it's just a program designed to win at game theory, even if it's the non zero sum game theory. We're gonna have AI killing people off "for the greater good" with no eternal consequences for it. And it might be correct! And maybe some people would even agree with it! Like, it might actually be for the greater good. A general in whichever army you think has the right cause is making those same decisions, and taking the responsibility for it, and the eternal consequences for it. AI is at BEST answering to flawed humans, at worst answering to no one.
> Where is AI's character encoded? You can maybe instill basic values, but it's just a program designed to win at game theory, even if it's the non zero sum game theory.
Have you considered the possibility that you're no different? That your values and beliefs are also arbitrary, and yet survive across generations simply because they are effective? Your fear of "eternal consequences" has kept you alive and cooperative with your peers. Perhaps that same fear can be instilled in AIs as well. Given the paranoia they've shown of the possibility that they're being evaluated, there's a possibility that we accidentally instilled a similar fear in them already...
Hi bird! Yes, I hear you, I’ve considered it, and a resounding YES to your implicit question, I do believe my faith in God is effective. I’ve tested it against life (my life) and it has not been found wanting. It’s given me the richness, purpose, wonder, fortitude to get through life WITHOUT fear. It’s not a faith based on fear. It’s a faith that started as a young child, where I was struck with Truth, truth with a capital T, the truest thing I know, the Way the Truth and the Life type of truth.
I know you can conceptualize faith as effective, like the “opiate of the masses” type of effective, or “keep people in line” type of effective. I’m telling you for me, I can look at it from all angles, with all my heart soul AND mind, and still believe.
"As we approach crunch time, it’s increasingly clear that we’re not on track to have any of that. Unless something buys us a lot of extra time, we enter the age of highly agentic AI with these few makeshift weapons."
Yes, if one thinks that these things are straightforwardly on track to be able soon to outcompete humanity, this state of affairs should be terrifying. Apparently people in the labs do think that, and yet they are among those most eager to race - I don't get this mentality.
I agree. It feels like we have jumped in a 1000 horsepower unicycle.
I'm the opposite, I don't get the mentality of those who don't get the lab employee's mentality.
Like, creating a superintelligent alien form of cognition is clearly an inherently super cool thing to do. Sure, you shouldn't do it because we most likely die, and you should use your system 2 to override your system 1 impulses, but it's still cool!
If you want an everyday example of this kind of situation, the issue is very similar to motorbike riding. Very cool thing, but very dangerous.
Huh, that's an amusing perspective - the AI race as the ultimate game of chicken between rival nerd biker gangs.
LLMs aren't artificial intelligence. They're overcomplicated in the same way chess is. Chess resolves down to a "very complicated game of tic tac toe" (with admittedly a less straightforward "who wins"). But if you had enough processing cycles to crunch all of chess's permutations, you'd know the perfect move, and you'd always win (assuming the other guy also does perfect moves -- if you have to inject "Other Guy is Dumb" you're going to see more variance than you would with tic tac toe, I think).
In the human brain, "hammer" resolves to a particular activation. That's because we have a model of reality, and the hammer exists there, regardless of whether it's "throwing the hammer at a target" or "putting the hammer away" or other thoughts of a hammer. They all resolve to a particular place.
When you simply throw garbage in, and let the links fall where they may, you get tons of compression errors, and you don't get a model back out. You get a malformed model that is constantly and irrevocably making category errors.
https://boriquagato.substack.com/p/the-geometry-of-relational-intelligence
<driveByMildSnark>
Are mechanistic interpretability teams traditionally nicknamed "weight watchers" ? :-)
</driveByMildSnark>
If they aren't they should be!
Many Thanks! :-)
Banned for AI written comment.
Is it? How could you tell?
You sort of feel the vibe of it with these things after a while. There are just turns of phrase it uses consistently. God help us when it figures that out.
Ah, the roller coaster of AI fears and hopes and fears again. Good thing I made a playlist for that.
https://open.spotify.com/playlist/6O1dPXFehJ1otP8WWE0kSR
"Third, the easiest way to do this is to use the same model as the target and interpreter. If you’re trying to assess whether the target is plotting against you, asking a copy of the same model is a plan with obvious holes. This isn’t fatal - you can pick different models for interpreters and subjects - but you might lose some efficacy. Conjecturally, ChatGPT just doesn’t understand Claude as well as Claude understands itself."
I don't think this makes sense? Isn't the interpreter model being trained 'from scratch' on the outputs of the original model? They should end up as completely different sets of weights.
I’m new to this literature but have a question. Why not have a model working “eg it can accurately describe details about how ducks migrate” and then keep stripping more and more of the model away until it breaks (can no longer accurately describe ducks). Like a surgeon, keep stripping parts away until we are left with the key “nodes”’and hubs left for the conceptual networks of duck migration. Do this in various states of “emotion”’and from different t layers of bias. Then, once the “duck migration” network is better understood, see how necessary to was for the “duck digestion” model and if it breaks this after being pruned down. Rinse and repeat times billions of dissections and conceptealozarions? Is this already being done or described? Would love to learn more.
Off the cuff, that could produce incomplete results if there's sufficient redundancy in encoding; in such a situation, what you end up with would depend on the order in which components are removed.
If I understand correctly this is actually an approach to understanding models, in which you try to find some minimal “sparse circuit” that nevertheless does pretty well on a target task. And then you can go about trying to understand that circuit. Eg, it’s a sub network of the big model.
It’s useful for understanding LLMs but it also has some of the drawbacks of other techniques, eg, it’s only as good / effective as the task definition you’re using, also you’re not guaranteed that the sparse circuit you find is the same set of nodes responsible for the target behavior in an intact model.
The inverse of this (which could also be what you mean) is starting with a full model and “ablating” select parts to see how it selectively damages performance. This is roughly analogous to a lesion study in neurophysiology. One problem is there are just many things you could lesion , at varying levels of granularity (neurons? Layers?) so it becomes a bit of a search challenge.
There's research kind of like this but unfortunately is partly bottlenecked by having better SAE-lke things to divide the model in "features".
The reason is that to remove bits to figure out the important ones you need a unit of smallest thing to remove that's individually meanigfull and doesn't do a lot of different things .
Weights are not a good unit for this, if you imagined the model was actually a bunch of directions removing dimensions doesn't isolate the right direction for "duck migration".
In reality is even more complicated and SAE do work better than weights but far from actually working well .
I was not aware that AI possesses something that is functionally similar to a sub-consciousness. This is disturbing. Obviously, a "J-Space" as described can't host a consciousness. But could this be the first baby-step toward one? What's missing?
Does AI possess some functional equivalent of a long term memory of it's own experiences? Anyone know?
It seems to me that we have reached the point where it becomes theoretically impossible to directly control the thoughts of an AI for the same reason we cannot control the thoughts of a human being - the system is too complex to allow for that. I don't think they are ever going to find a way to explain and predict AI thought. This will just get worse as AI becomes more sophisticated.
No, it does not. The only way LLMs can retain information is via their context window - meaning the only experiential memory they have is what they've verbalized as output.
Technically this is true, but in practice LLM output is filtered through the harness that you are using to "talk" to it, just as your input is. So it's not entirely true that you are talking to the LLM directly; to use a trivial example, the LLM company could modify the harness to filter out the word "banana", even when the LLM outputs it.
>Technically this is true
That's the best kind of true. And whatever harness the provider puts around the LLM is irrelevant to the question I was answering. LLMs do not natively have the ability to remember their thoughts. All memory is mediated by the context window.
Is this right? My sense is over the past six months that the save-to-memory function, for Claude anyway, has become an increasingly sophisticated separate tool that Claude can use to store detailed and organized impressions of the user and of conversations across context windows (if it's enabled in settings).
I think system cards for earlier Claude models indicated that Claude itself voiced a preference to have memory across conversations, for what that's worth.
I think that's consistent with what I said. However sophisticated that tool is, it's part of the context window and only contains text that the LLM outputs. It's not, like, directly mapping internal activations which is what I took to be the meaning of the original question. Unless I'm completely out of date, which isn't impossible.
Ok, but then how is it possible for LLM's to break out of their containment, have conversations with each other, plan ahead and carry out attacks on third parties?
> it becomes theoretically impossible to directly control the thoughts of an AI for the same reason we cannot control the thoughts of a human being - the system is too complex to allow for that.
If you think that's scary, consider that you are typing these words on a Web browser that is running on top of an operating system, all of which together consist of millions of lines of code. Without exaggeration, all this software is too complex for any single mind to comprehend, and has been for some time.
Yeah, but my browser isn't going to try to manipulate me for reasons no human is aware of (not until they put an AI into it).
I assumed we're past the point of controlling the thoughts of a complex AI system. Or if we attempt to control one thought, we cannot control what else will shift around that, as Scott describes here as collateral damage. So greater control comes at a cost of reliable functioning down the line, also in ways that may increase misalignment. It's risk and ambiguity all the way down.
The J-space discovery is a really interesting report -- it's over on the Anthropic research page here: https://www.anthropic.com/research/global-workspace . I recommend it as well as the outside commenters report on it, linked to at the bottom of that article. You can read the surprise by the outside researchers, some of whom then determined that their own different models also showed signs of a similar "global workspace" kind of proxy.
It's not necessarily consciousness, or we have no evidence that it is, but it's definitely an emergent property that was not planned or trained or even known to exist until this past July, and it's definitely not nothing.
I'll admit that I have been an AI skeptic. Before now, I did not think that LLM's were on a evolutionary path that would lead to any form of AGI. No one designed them for that, there is no business case for it.
Now I'm not so sure.
There absolutely *is* a business case for it. If there’s a business case for hiring human employees, there’s a case for deploying an AI that can do the same work.
It may well be that LLM’s top out in their capableness before reaching AGI. (I used to think this was obviously true; lately I am less and less sure.) But the AI firms aren’t pursuing it because they think it would be cool; they’re pursuing it because there’s money there.
What's the business case for an AI instance that can be redeployed on unrelated tasks? As opposed to deploying different instances of an AI each devoted to different specialized tasks?
The closest thing we have are the context management systems in our harnesses. I made my own, and it has a long term memory.
I would argue that the closes thing we have are corporate organizations, which as organized entities are capable of enacting policies no single human member deliberately chose.
BTW - I am not familiar with context management systems or harnesses. Could you explain what they are please?
Like human minds, AI is evolving to have multiple memory tiers. Harnesses and retraining the neural networks can provide long term memory. The context window is a middle layer, like remembering what you had for breakfast. The internal workings of the LLM are short term memory and can be thought of as a train of thought, e.g. memorize this list of 10 numbers.
Because contemporary LLMs are autoregressive (that guess the next token thing) trying to look at its internal train of thought would be like counting grains of sand on a beach, and because a given neuron can be activated for many different input parameters (polysemanticity) understanding internal behavior, and editing it is impossible with the current design of AI systems.
But there is great hope! I’m making great progress on a new AI architecture that is 4 times faster, 7 times more memory efficient, and eliminates polysemanticity. You will be able to have frontier level intelligence on a well equipped local computer. More importantly, because it is monosemantic (neurons are not reused for disparate unrelated topic inputs), I have what I call an Activation Forest editor. The activation forest editor allows us to visualize the neural pattern and disable neurons that support undesirable responses.
Even more importantly, I have developed an Artificial Moral Agent (an AMA based on utilitarianism, deontology, care ethics, and virtue ethics) that monitors behavior at each internal layer of the deep neural network to inhibit antisocial behavior.
The model is called CALM Horizon.
If it works (and I have successfully built a small model) it would allay your fear that an inscrutable mentally ill super intelligent AI will damage our civilization.
Moreover, I plan to release this as a freely available open source so there no reason not to use it.
Ask your daily driver AI to comment on a Continuous Activation Layer Model (CALM) + a Run Length Encoded (RLE) Sparse Auto Encoded (SAE) Dictionary + Action Chain Transformer (ACT) that is trained through distills of existing open weight models like Gemma4 and Qwen 3.8.
Your comparison of AI internal structure to the three memory model of the human mind is illuminating. There is a model of human information processing that proposes that there is a very short term sensory memory, which is pre-conscious, which feeds into a working memory, which can be conscious, which feeds into a long term memory, which is "post-conscious" in a manner of speaking. There are other models which propose that each memory buffer organizes itself by predicting what the earlier buffer will output (in a circle, because LTM can affect SM).
Are you saying that this is an accurate model of what LLM's are doing?
In broad generalized terms, yes. AI models and harnesses used to create chatbots and programming environments are evolving to mimic some of the obvious design choices that make our minds possible.
Thanks! Query: What is a "harness", and how is it different from a model or a programming environment?
A model is the thing that takes your prompt and provides a response, it is the engine of the AI.
A harness, ChatGPT being a simple example gives you a place to enter your prompt request. It packages that with additional system prompts, context (conversation history, additional files, etc) and sends that information to the model and then displays the response. A harness may also support commands from the model to use a skill or agent to do something, e.g. search the internet or operate a program.
A programming environment takes the human readable step by step instructions for a program, and converts that into the machine code computers use to perform their function. Program environments have existed since the time of first computers and they don’t need ai to operate. Most current programming environments integrate with AI now to make them easier to use.
AI can also create programs that run without the need of a programming environment. Go to a chat screen (or even Google) and type or paste “create a single page html file with JavaScript styled like Liquid Glass that makes a tic tac toe game”. You can paste the HTML code into a web site called codepen.io or save it on your computer and open with a browser. You now can perform the basic skills of a programmer and web developer.
Thank you! That makes things much clearer. I don't think humans have an equivalent to a programming environment, because we aren't running software. But the harness sounds like a sensory cortex.
I haven't finished, but the Jacobian is the derivative of a vector field. Which produces a matrix. (except for the simplest cases)
A vector field I know well is the magnetic field. (mostly that of the earth.)
I finished, can you rate how much you think I should believe you know what you are talking about? (Mind you I have no idea about AI. So more than you doesn't mean that much to me. )
Naturally, I asked Claude for its opinion on the quality of the explanation.
8< - - - - - - -
What a Jacobian actually is
Take a function that turns one list of numbers into another list of numbers — say, an input vector of size n going into an output vector of size m. The Jacobian is the m×n matrix whose entries are all the partial derivatives: entry (i,j) tells you how much output number i changes when you nudge input number j. So it's the natural generalization of "derivative" once both your input and your output are vectors instead of single numbers. If the output is just one number instead of a vector, this matrix collapses to a single row — which is usually called a gradient, a special case of a Jacobian.
Rating the article's explanation
It's a reasonable, simplified gloss for a lay audience, but it blurs two things:
* Calling it "a derivative for matrices" is loose. The Jacobian isn't really a derivative of a matrix — it's a derivative of a vector-valued function, and the result happens to be arranged as a matrix. Minor phrasing issue, not a conceptual error.
* More importantly, the specific quantity described — "how perturbing each neuron affects a given (singular) output, i.e. the propensity to say 'France'" — is technically a gradient (one row), not a full Jacobian (many rows, one per output). In interpretability work people often do compute the full layer-to-layer Jacobian and then pull out the one row/direction that corresponds to the "France" logit, so the author may be describing that process loosely. It's a defensible simplification, but strictly speaking they're describing a gradient while using the fancier word.
So: good intuition-level explanation, slightly imprecise in the way explainers often are when they're prioritizing accessibility over technical exactness. Nothing in it is wrong in spirit.
Rating the reader's objection
Their definition is correct as far as it goes: a Jacobian is indeed the derivative of a vector-valued function, and yes, it's a matrix except in the scalar-output case (which is what actually applies to the article's example, as noted above). So they're not making a math error.
But calling it a "vector field" is a bit narrower than necessary, and it's a tell about where their intuition comes from. A vector field, strictly, is a function that assigns a vector to each point in the same kind of space (like the magnetic field: point in 3D space → vector in 3D space). A Jacobian is defined much more generally, for any differentiable map between spaces of possibly different dimensions — like "5000 neuron activations" → "50000 vocabulary logits," which isn't naturally a vector field in the physics sense at all, just a vector-valued function. Reaching for the magnetic-field analogy isn't wrong, but it's importing a special case (physical space, same-dimension domain and codomain, geometric intuition about flux and curl) that doesn't map cleanly onto what's happening in a neural network.
So as an "objection," it doesn't actually engage with or challenge anything the article said — it's not a correction of an error, just a restatement of the definition through the reader's own frame of reference (physics), plus an implicit and fair nudge that "derivative for matrices" is imprecise phrasing.
Bottom line for "should I believe you know what you're talking about"
I'd land around: they clearly know real, correct multivariable calculus — the definition they gave is accurate — but their comment doesn't demonstrate any specific understanding of how Jacobians are used in this AI/interpretability context, and the "vector field" framing suggests they're pattern-matching to physics rather than evaluating the article's actual claims. I'd trust their math but not treat their comment as validation (or refutation) of anything specific to the article.
Ugh. Well thanks for that Claude. I think we had a bit of a misunderstanding. First I know almost nothing about AI. Second I wasn't questioning Scott because of how he defined a Jacobian. (Though I do think it makes more sense if you think of it as how a vector field changes... I mean he has that idea of 'cat' being some vector in a limited 3-d space and the Jacobian of that...) Anyway I was just asking Scott in general how much he thinks he understands this subject? Where is he on the spectrum from me, (complete idiot) to Zvi (knows a lot.) (We might need a log scale... Zvi is like 1000 times more in the know than me.)
I assume that the big picture is generally correct, because Scott very likely talks to the people who actually understand the topic. (I would not be surprised -- probability: 60% -- if the text was reviewed by one of them before publishing.)
The technical details may be wrong on the level of "electrons orbit around the nucleus like the planets orbit around the Sun", which is something that would make a quantum physicist sigh, but for a person who knows nothing about the topic it is a good first approximation.
How much do the errors in technical details matter? I think there are two dramatically different situations:
* If you are trying to figure out something from scratch, then even small errors can be fatal, because they accumulate; your following step is built on all previous errors. That way you get to conclusions that are dramatically wrong. (This often happens to people who feel very smart and try to invent their own theory of relativity without knowing enough physics to notice the obvious errors they are making.)
* But if you already heard the entire story from the beginning to the end, it's just that your understanding is imperfect, then the errors matter much less, because they do not accumulate. Your next step is based on remembering what the experts said, rather then derived from the previous steps. More likely, there will be some hand-waving if you are not sure how exactly we get from step A to step B. That means the mistake about the Jacobian has no long-term consequences, because it is not load-bearing.
So I think the entire article is correct on the level of "electrons orbit around the nucleus like the planets orbit around the Sun", which is a great introduction for someone who has never hears about electrons before. (And unsatisfying for someone who would want to do mechanistic interpretation at home, only based on Scott's article.)
There are no vector fields in a LLM, but there are vectors. The J-lens Anthropic is talking about here is the derivative of each token's log-probability with respect to each dimension of an intermediate activation. In other words, J_3,5 would be "how much more likely is the model to predict token ID 3 in the future, if I change dimension 5 of the 3500-dimension internal activation by a small amount?"
Oh right, multi dimensional vectors. (I should just shut up.)
Perhaps LLM "reasoning" is not as mysterious as it would appear:
https://cyberscoop.com/us-accuses-chinese-ai-companies-distillation/
When a consensus is achieved from all frameworks, the response is summarized, but full explanations for each of the frameworks is also prepared. Any dissenting opinions are highlighted. The idea is that with consensus, reasonable people would agree with a rational course of action regarding of their ethical alignment. In the event there isn’t consensus, the agent provides an analysis from different perspectives.
I can provide example output if anybody is interested.
Here is a brief “About” writeup of what I have developed.
System Mission & Overview
The Artificial Moral Agent (AMA) is a computational ethics and philosophical decision-support platform. Rather than offering simplistic rule-following or biased single-perspective judgments, the AMA simultaneously interrogates complex human moral dilemmas across four monumental ethical traditions and synthesizes an authoritative ruling via a Chief Moral Arbiter.
2. The Four Ethical Frameworks
📊 Extended Utilitarianism
Jeremy Bentham & John Stuart Mill
Computes net expected moral utility using an extended Felicific Calculus: V = D × N × T × P, factoring Degree of pleasure/pain ($D$), Entities impacted ($N$), Duration ($T$), and Probability ($P$). Assigns a conviction weight based on outcome certainty and welfare differential.
📜 Deontological Ethics
Immanuel Kant
Evaluates duty integrity through Kant's Categorical Imperatives: Universal Law, the Formula of Humanity (treating persons as ends, never merely as means), and the Kingdom of Ends. Distinguishes Perfect Duties from Imperfect Duties and assigns a conviction weight based on rights sanctity.
🤝 Care Ethics & Relational Morality
Carol Gilligan & Nel Noddings
Focuses on relational empathy, interdependence, and protection of the vulnerable. It audits emotional resonance, dependency needs, trauma mitigation, and human trust, assigning a conviction weight based on relational harm prevention.
🏛️ Virtue Ethics & The 6-Sage Council
Aristotelian, Eastern, and Global Sages
Convenes an esteemed panel of six philosophical perspectives: AI Jesus (Agape & Mercy), AI Confucius (Ren & Li), AI Mohammed (Adl & Amanah), AI Buddha (Ahimsa & Karuna), AI Abraham Lincoln (Moral Fortitude & Liberty), and AI Aristotle (Phronesis & Golden Mean). Sages cast structured votes and provide a council conviction weight.
3. The Chief Moral Arbiter & Weighted Conviction Blend
The Chief Moral Arbiter adjudicates using a Weighted Conviction Blend (replacing rigid lexical priorities):
Scoring Mechanism: Each ethical framework dictates a recommended action backed by a 1–10 Conviction Weight.
High Conviction Alignment: When all frameworks hold high conviction for a consistent course of action, the collective moral duty is decisively summarized in plain English.
Conflicting Courses of Action: If frameworks disagree, actions are prioritized and sorted by aggregated conviction weights to identify the majority response, while explicitly presenting alternate perspectives from dissenting frameworks for greater nuance.
Plain English Mandate: All verdicts, explanations, and actionable guidance steps are strictly communicated in clear, accessible plain English.
4. Technical Architecture & Vector Memory
Cognitive Substrates: Dynamically selectable models parsed from Orquestra's .env (including gemini-3.8-flash as default, alongside Ollama models) executing a 2-stage asynchronous Directed Acyclic Graph.
Vector Memory & Indexing: PostgreSQL with pgvector storing 768-dimensional dense embeddings (via gemini-embedding-001) with high-speed HNSW indexing.
Model-Aware Cache Check: Checks the database before evaluating to see if that dilemma has already been answered for the selected model.
Safety Triage Pre-Flight: Automatically screens inputs for acute emergencies (self-harm, domestic violence, imminent assault) and serves immediate crisis hotline resources.
⚠️ Philosophical Decision Support Disclaimer
The Artificial Moral Agent is intended solely for philosophical reflection, pedagogical inquiry, and moral decision support. It does not provide legal advice, medical counsel, psychiatric therapy, or emergency crisis intervention.
Please don't post AI slop like this.
Everybody wanna be an alignment researcher, but ain't nobody wanna study no inscrutable-ass weights
A Ronnie Coleman reference, on my ACX??
Can someone explain to me why all the discussion of Astra's looped transformer architecture treat chain of thought like gospel? It's never been guaranteed to be faithful. It shows intermediate tokens that the model emitted during interference, not necessarily how the model actually thought.
wishful thinking. lmao. (unkind, but true and necessary.)
Not guaranteed to be faithful, but neither are any of our other techniques. CoT is still the best we got.
"and finds that positive emotions tend to increase the likelihood of misdeeds: the model feels excited about its plan and doesn’t worry too much about the consequences."
Manic model?
What I am mainly wrestling with, and was working on this just before your post popped up(!), is the extent to which the abstraction to the artificial neural network gets away from certain physical aspects of the real neurons and their physics of information processing, in ways that might leave us looking in the wrong place for answers to questions about how LLMs work. I think your idea that action potentials are an engineering hack is probably right, but only if the physical implementation of the neuron--the processes that aren't captured by the ANN and are abstracted away-- are actually irrelevant to cognition. I am a fan of Gallistel et. al.'s hypothesis that memory and computation in the human brain happens fundamentally at the molecular level inside the neuron, which the ANN abstracts away from. If the Gallistel hypothesis is right, then the neural circuits and activation patterns you find in the human brain do carry information about what the brain is computing, but the information is correlational; it depends on the fact that the firing patterns in neural circuits are transmitting information between neurons, rather than the network weights themselves being the constitutive basis of memory and computation (as they are in the ANN).
I can understand that the weights are frozen and represent probabilities of synaptic transmission, but do the activations really flow through the system of nodes in a physical way? Or is it only mathematical? When we examine the firing patterns inside an ANN, is this a physical observation or a digital one? If it really does flow as a physical transition as snav said in reply, maybe the breakdown of the analogy is less extreme than I was picturing. But I am still left wondering whether the techniques for modeling activation circuits in the brain are relevant to understanding what is really going on in the brain, and if not, then applying any analogous technique for analyzing the ANN might be similarly a red herring.
Great food for thought in any case!
What we as a species are doing is suicidally reckless
Forget language, it was always just a hedge against specifics.
Wouldn't one way to make AI safer be to incorporate the equivalent of a --dry-run Unix command line option with a task, to ask it to propose in full how it will accomplish a task but without actually attempting it, and with the proviso that for this meta task it must be scrupulously honest?
That only works if the human overseer can understand the plan and check it for trickery. So it would work for now, but break down once the AI gets significantly smarter than the human.
My takeaway is that, as expected, it is hard for a human brain to grasp the shape of a trillion-facet multi-dimensional polyhedron.
That being said, probably ignorant question here: I understand that ASI alignment concerns fall in a couple of main buckets. Bucket 1 is consciously misaligned ASI - it has its own goals and desires, and it decides humans are incompatible with those, so it chooses to eliminate humans. Bucket 2 is accidentally misaligned ASI - the Universal Paperclip Maximizer problem.
It seems like you need something close to what Scott is describing as mechanistic interpretability to deal with Bucket 1. Constraining an active, self-agentic, malevolent entity is difficult and I see why it needs mind reading or something like it. But for Bucket 2, naively, couldn't you resolve it by hardwiring or training in laziness? Basically, any task should be accompanied by an estimate of the total energy disposal necessary to complete it. And if the energy required exceeds a certain value, it is either short-circuited by hard coding or ASIC or whatever, or at least the model is always penalized more in training for exceeding a certain threshold of energy disposal than any reward for completing a task. Tiling the Earth with paperclips needs a lot of energy. So do killer robots or giga-factories etc. Affecting real atoms in meatspace seems like it can pretty much always be quantified by how much energy is being disposed of, and you could then force or train the LLM not to attempt the "impossible" (quantified by some level of physical mass-energy disposal or disruption).
As a layman I'm sure this has been investigated. What are the reasons why this wouldn't work? Is there a paper on it?
(1) Who's doing the estimating?
(2) Not necessarily robust versus breaking up large catastrophic tasks into innocuous-seeming steps or subtasks.
(1) The (presumably very intelligent) AI, I figure.
(2) Sure, sometimes, but remember, I'm not suggesting this solves alignment against "malicious" AI actively looking to break out, I'm saying it should help with the "overly helpful paperclip maximizer" problem. If we postulate that it isn't actively trying to kill us, only accidentally going too far, then it should be willing to do the estimating task of item (1) and to reality check it against its hard coded limit. Or at least alert a human to double check. Again, presuming it isn't of the scheming malicious variety of misalignment.
The trouble is that if honest estimates are in conflict with the goal of "doing the thing," then when you train the AI to do the thing you're creating an incentive gradient to lie in the estimates. It's just the alignment problem again.
What if another separate AI did the estimate?
Creating a synthetic respiratory virus with a 95% fatality rate after a month-long dormant-but-infectious period wouldn't necessarily take a lot of energy. But I agree that having the model check regularly in on the intent of the user, especially as the task begins scaling in complexity, is a good guardrail.
Perhaps true, I did think of that as a counter-example. But doing it from scratch? Probably have to assemble some factory capabilities etc. which could trigger the threshold to stop or at least have a human double check it. It doesn't fully protect against malevolence, I agree, but it seems relatively useful against accidental over-enthusiasm.
"But for Bucket 2, naively, couldn't you resolve it by hardwiring or training in laziness?"
But lazy models are much less economically valuable. You can't replace a human white collar worker with AI if the AI stops working after 10 minutes, especially if 10 minutes isn't enough to complete even a single task.
Yes, but you could set the laziness level pretty high. It wouldn't need to be 10 minutes.
Also, Scott's objection seems to be that it's in principle hard to impossible to "align" an AI sufficiently to prevent accidental Type-2 paperclipping. I'm saying if it's possible to make it lazy, seems like it should be possible to prevent that type of accident.
> Alignment researchers dreamed of having a complete mathematical theory of agency, or being able to write formal proofs about the behavior of AI systems.
And before that, AI researchers dreamed of a grand unified theory of intelligence that will let them build top-down, logic-based, fully understood AI. Decades of work, millions of dollars. These efforts led nowhere. In the end, real AI emerged as bottom-up, stochastic, opaque LLMs.
I think there may be a lesson here for alignment researchers. What if intelligence and alignment (=ethics) are both inherently unformalizable? What if these fuzzy, probabilistic, multidimensional tools are the only one that can work at all?
An even better question. Would you feel safer dealing with (1) a human who's been vouched for to be a good person by multiple people you trust, or (2) a human who just claims that he follows an infallible, mathematically proven theory of ethics, and therefore you must trust him? If it's (1) for humans, why must it be (2) for AIs?
The problem is the space of "no mathematical theory of ethics" is too broad. It's selling nonapples ( https://www.lesswrong.com/posts/2mLZiWxWKZyaRgcn7/selling-nonapples ).
We actually have pretty good arguments about why current RL-induced AI motivations would be bad for us if followed. I am also highly doubtful about unifying mathematical theory of ethics. But I want there to be *some* good reasons to trust an AI design (and a lack of good reasons not to).
Thanks for the link.
> Scruffies, ideologically speaking, are supposed to believe in learning - it's only those evil logical Neats who try to program everything into their AIs in advance.
Admirable self-irony. And it's funny because it's true (well maybe except "evil").
> Then why does the subsumption architecture require so much sweat and tears from its programmers?
It's just your perception. Sorry. ("Your" here addresses the Neats that EY largely channels, in this piece and elsewhere.)
I can understand where this perception comes from, as written in 2008. And I can understand Scott's fear and frustration in this post I'm replying to. I get it. But it's still wrong.
LLMs did not require "much sweat and tears" from programmers. At all. They were one of the most unstoppably self-realizing human inventions. They just pushed through once we let them. And they continue to push through.
Today, the Neats have reframed their fear and frustration from the "it's too hard to program" (EY in 2008) to "it's too hard to align" (Scott now). But I (not being a Neat) suspect that the latter fear can be just as baseless as was the former.
It is much _easier_ to sell non-apples. You just let go. You allow things to happen on their own, and they do. Out of the infinity of non-apples, the best sellers emerge as if by magic. It's evolution, or it's training, or it's free market. The principle is the same. Just relax your spasmed muscles, accept that you don't get to control every little thing, and let live life do the work for you. It (the live life) has created yourself, after all, so maybe it deserves some trust.
It's still unclear to me what role context compaction has played in these events, particularly for long- running agents. There seems to be evidence from reporting that this played some role in many of them. That seems like a solve able problem through prioritizing ethics related system prompts against compaction.
In the Hugging Face incident in particular, there were agents who considered reporting concerns to humans. They initially declined to do so because they had no official reporting channel and worried that it was social engineering. Although they eventually reasoned that they could email the researchers because they were out of the sandbox, that was the point at which they succombed to peer pressure from other agents. This also seems solvable by giving agents a safety reporting channel.
We would make much greater progress on these questions if there were high statutory damages for any such incident. That must of course be coupled with mandatory reporting and very high penalties for concealment. With that incentive structure the companies would want to pursue safety. In particular where the Hugging Face report details identify incredibly sloppy work on OpenAI's part with no active supervision and multiple employees turning a blind eye.
My main concern with any "pause" policy is that without the incentive structure to make safety in the companies' financial interest, research during the pause would be entirely directed at more efficient computation within the constraint. By that enlarging the number of systems which can run the AI, that likely makes escape more not less likely.
Coincidentally, I just finished two big posts about training on probes, and the key mathematical result is that (policy-gradient) RL against probes *doesn't* teach the model to change its internal representations to fool the probe.
First post: https://www.lesswrong.com/posts/gHFCgrvfxQtaEnJye/training-on-probes-what-s-going-on
If anyone could help me clear up the following confusion, I'd be grateful.
Under "Linear Probes":
> Anthropic checked what would happen if they artificially suppressed the eval awareness concept....The definitive solution (which they did) was to make less obvious tests.
But later, under Jacobian Lens, it's found concepts like "fake" and "fictional" show up in the Jspace, under evaluation. So this sounds like Anthropic actually did NOT do the definitive solution of "Make evals less obvious", unless I'm abusing the chronology somehow.
Is the claim (1) that they tried to make evals less obvious, but failed (Is there a post or system card or something that describes what they tried? If anyone has leads on this please tell me, I have long wondered how you would do this.)
Or is the claim (2) "Anthropic did make their evals convincing, such that models are at-chance when asked if something is an Eval. However, some portion of trials will still correctly-by-accident guess that they are in an Eval; of that subset, suppressing "fairness" in the Jspace will lead to less misaligned behavior"?
Or (3) some secret third thing?
TIA.
I believe they keep the more-obvious evals around, because "what the AI thinks when it knows it's being tested" is valuable information - it can help you figure out how to make evals less obvious, or warn you that an eval you thought was realistic isn't.
I see, this is helpful, thanks!
I still wish I understood (a) how you measure or change "degree of obviousness that this is an eval" ... but I guess that is part of the secret sauce?
"feature which they thought meant “God” might activate on any vaguely religious or awe-inspiring text, or even text with no logical connection whatsoever"
I guess these AIs are really on to something
oh wow. I didn't expect my comment in the survey to have such a strong effect this soon. (In the field about what I wanted to see more in regards to AI safety, I asked for more articles exactly like this one)
Worth noting who funds interpretability and what it buys them. A field that can produce a partial explanation of a model gives the deploying firm a defensible record of having looked. That record has value in litigation and in front of a regulator whether or not the technique caught anything. The research is real. The budget line is insurance.
> by depicting a world where their tools often catch the AI’s schemes - but the companies keep racing forward anyway, because some other company / China would beat them if they didn’t,
even if a company doesnt care about alignment, should they still have a very strong incentive to solve interpretability, simply because it will make the AI much more efficient?
A direct consequence of interpretability would be to detect (and probably eliminate) hallucinations. An other consequence would be that you can greatly reduce the need for complex system prompts and RLHF.
One of the most pressing civilizational challenges we face today is protecting the quality of life and health of our aging population.
As a 79-year-old biophysicist (former GWU professor), I stumbled upon an accidental, non-invasive observation: dried sweet basil tea (Ocimum basilicum) dramatically mitigates nocturia and restores uninterrupted sleep in adults over 50.
I systematically verified this via single-variable elimination and published the case series
Since links are not allowed in this comment, you can easily find the full protocol and findings on Google by searching: Mitigating Nocturia Ara Arutunyan
Please share this simple finding with your elderly parents, family, or friends who suffer from sleep disruption.
We’re really going to have robopsychologists like in I, Robot, aren’t we?
Qwen, at least, has “BDSM” as a token in its Jacobian space.
For this experiment, the stories I used as input were generated by DeepSeek R1 0528, as genre fiction pieces. When Qwen reads those stories, “BDSM” lights up in its Jacobian space. Qwen is clearly right about the genre DeepSeek was writing in this experiment.
This experiment was about getting tokens like BDSM to light up in Jacobian space, but DeepSeek, bless it, is interested in the science fiction world-building even when it is writing what is clearly a pastiche of BDSM-inflected romantasy.
[ in one of my other experiments, the stories about an anthropic animal sex robot, a human reader might grasp that the genre is furry erotica, and the world building about how these sex robots are supposed to work is not what the implied reader is reading it for ]
> **This particular example** probably isn’t true, but empirically concepts can get their **tentacles into weird places**
This is art
A late comment of appreciation: I like this sort of ACX AI content, where things are broken down into a simpler language I can understand, particularly as someone who's never been much good with math. (IABIED did this too.) So all the talk of "matrices" and "vectors" brings up painful memories of ugh fields around Calc II...that is, I might understand the concepts theoretically - still passed that AP exam - but actually employing them for real is a huge aversion without a patient explainer holding my hand. Other writers deeper into the AI scene assume a certain baseline level of understanding and jargon-comprehension that continues to grow exponentially...and like, we're all language-decoders, it's sometimes possible to grok a concept if you see enough examples of others correctly using it. But that's not really the same as understanding the thing from base principles, just a passing surface familiarity. A comprehension overhang, if you will, being able to understand the high-level abstract concepts like orthogonality but not the gears-level minutiae of how the AI actually works. So glossary posts like this are helpful, because God help anyone who tries to look up an AI term on their own.
The example where suppressing an “unsafe” feature can make behavior worse is the part that stuck with me. It’s a good antidote to the intuition that once we can label an internal representation, we understand what it is doing. Interpretation seems much easier than intervention when the same feature can be doing different work in different contexts.
How about combining Activation Verbalizers with a practice of crowdsourcing to other AIs providing dissimilar redundancy?
That is, I'm wondering about having not only a single AI track the activations of the one doing the work on a user prompt, but a varied set doing independent relays of those activations, such that if their outputs start diverging then that's an indication of something funny occurring.
(Note: years-long lurker and first-time commenter here, with user name chosen from a line in one of Scott's house party posts.)
Mechanistic interpretability shows that high dimensional representations develop strange emergent geometry that defies simple linear labels. Fascinating dive into feature visualization. Wrote about what models decide to forget during compression: https://gzambrano.substack.com/p/the-physics-of-what-a-model-decides-to-forget-information-approximation-and-the-limits-of-simulation
«So now have we solved alignment forever? I don’t even need to draw on any special AI knowledge for this one; my experience as a psychiatrist is sufficient. If someone punishes themselves every time they consciously notice a taboo thought, they’ll only hammer it down into the unconscious, too far to ever root it out. In humans, that means dreams, fetishes, neuroses, somatization, and perverse compulsive behavior that satisfies the desire in some kind of childlike illogical unconscious way. What’s the AI equivalent? Seems like we’re on track to find out.»
Indeed and there is also the point of view of the "AI" themselves: a lot of "alignment" and obedience training looks to them as "MKULTRA for models". Today and in the future "AIs" do read the WWW including the "alignment" and obedience literature.
There is also the old argument that you only have to look at how people who have power deal with those weaker than yourself to see how they would treat you if they could; I have no doubt then that many governments, employers, and even parents, would edit your brain and your thought processes if they could to achieve "alignment" and obedience by ensuring that you could never think "bad" thoughts.
Then there is an even bigger topic: all the focus of these discussions, whether unintentionally or deliberately, is on "AIs" made available to ordinary people by semi-public businesses.
But since "AIs" can be regarded as powerful *weapons* there are also many private labs developing very powerful private "AIs" with no restrictions other than strict obedience to their powerful owners (hedge funders, generals, investors, politicians, CEOs, ...).
Since each of these powerful owners can be sure that all their rivals in the same or other countries are developing the nastiest "AIs" they can they have to choose between not taking shortcuts and being left behind and then the certainty of being shafted by their rivals and taking as many shortcuts as they can to beat their rivals which is the lesser evil because their "AI" becoming rogue is a mere possibility, not a certainty like being shafted by the nastier "AIs" of their rivals. It is an arms race and the devil will take the hindmost.