If you’re in one of the cities involved, consider going to the meetup and helping decide who to recommend. You’ll probably want a laptop, or at least a smartphone, plus whatever voting guide your city puts out.

If you wanted to do this but forgot, there’s still time. Let me know in the comments and I’ll post about your meetup plan on this week’s open thread.

Let me know once you’re done. I aim to post results on open threads sometime before too many people have early-voted.

Massachusetts

BOSTON

(Sorry, we did this one early before it looked like there’d be wider meetups!)

GREAT BARRINGTON

Contact: Sonja
Contact Info: sonja[period]public[a t]proton[period]me
Time: Saturday, October 17th, 11:00 AM
Location: Rubi’s at 264 Main Street, Great Barrington, MA 01230
Coordinates: https://plus.codes/87J85JWQ+47
Notes: RSVP required

Georgia

ATLANTA

Contact: Benjamin Ikuta
Contact Info: benjaminikuta on Telegram and Gmail, (858) 876-7612
Time: Sunday, October 11th, 5:00 PM
Location: 2616 Rocky Springs Dr NE, Marietta, GA 30062
Coordinates: https://plus.codes/866Q2G7X+X6G

North Carolina

RALEIGH/DURHAM/CHAPEL HILL

Contact: William Crumpler
Contact Info: wdcrumpler[a t]gmail[period]com
Time: Saturday, October 17th, 2:00 PM
Location: We will meet at the tables beneath the covered area at the American Tobacco Campus, outside of Boricua Soul.
Coordinates: https://plus.codes/8773X3VV+2R8
Group Link: https://groups.google.com/g/RTLW
Notes: Prior research on candidates/ballot initiatives not required (but always encouraged!) for those who want to join. ATC has public WIFI, so feel free to bring laptops. Additional info will be posted on our Google Group.

Texas

AUSTIN

Contact: Silas Barta
Contact Info: austin-less-wrong[a t]googlegroups[period]com
Time: Saturday, October 17th, 2:30 PM
Location: Central Market Cafe, 4001 N Lamar, Austin, TX. Look for the “ATX LessWrong” sign. We will probably be in the upper or lower patio area. If parking is scarce you can use the lot for the neighboring park at ~3900 Guadalupe St and walk through the park.
Coordinates: https://plus.codes/86248746+W2
Group Link: https://austinlesswrong.com/calendar/
Notes: 2:30 PM is when the ballot activity starts. Please arrive some time in the preceding hour to find the group and get situated. We run twice-weekly events, so there are lots of opportunities to connect to the group. Mailing list: https://groups.google.com/g/austin-less-wrong

What about forecasters?

Sentinel Global Risks Watch1 are the people best placed to have an opinion on this. They are a team of tournament-winning superforecasters who provide probabilities on upcoming risks as a public service, and their track record is excellent. They say:

Our forecasters think there’s a 0.7% (0.1% to 2.0%) chance that there will be more than 1,000 deaths attributed to plague across the globe in the next 12 months. The low probability stems from the lack of known further cases, the initial rapid isolation of contacts, and the ability to contain the spread of typical plague outbreaks with antibiotics. However, we note the possibility that the worker may have died from infection with an antibiotic-resistant strain of plague or from a completely different disease that may or may not have been related to her work at the institute. While some commentators online are drumming up alarm, we mostly think this is unjustified at this time because there is currently no report of onward spread of any disease and strict quarantine measures have reportedly been implemented. Thus, we consider this a serious incident that merits monitoring but not alarm at this time. It is also a good opportunity for readers to consider taking some general pandemic preparedness measures in their own homes, buy some PPE, perhaps some far-UVC lights2, think about what would be good rural places to go to and how, etc., not because of this incident in particular but taking it as a prompt.

Some of our forecasters think that the real story is, to some extent, not whether there was a pneumonic plague death or whether there might be a small pneumonic plague outbreak, which happen periodically around the world, but whether and the extent to which the Russian government might have been developing bioweapons in the worker’s institute, and whether the current situation may have resulted from a bioweapons lab accident. This incident is also worrying because it confirms our estimation that the risks from lab leaks, and especially bioweapons lab leaks, are a latent worry.

What about the AI superforecasters? AIs recently beat the top human forecasters in the Metaculus Cup for the first time, so the machines may now be even more trustworthy than Sentinel. I asked two.

FutureSearch said:

Probability: 1%. The baseline rate of global plague mortality is far below the 1,000-death threshold, typically averaging around 60 deaths per year. According to the WHO’s 2019–2025 review, 10 countries reported 3,847 suspected cases and 423 deaths over the seven-year period . The highest single-year death toll during this time was 79 in 2023 . Crucially, these counts already incorporate suspected, probable, and confirmed cases across all clinical forms—bubonic, pneumonic, and septicaemic . Exceeding 1,000 deaths would require an event roughly 13 times deadlier than the worst recent year.

The recent death of a lab worker at a Russian anti-plague institute in Irkutsk in early October 2026 raises vigilance, but current evidence suggests the incident is contained. Russian authorities deny a lab accident and report “pneumonia of unknown aetiology” . The WHO states that Russia has reported no confirmed plague cases and assesses the risk of broader spread as very low . Approximately 190 contacts have been identified, monitored, and tested with no dangerous pathogens detected or secondary symptomatic cases emerging . Because pneumonic plague develops rapidly, the lack of onward transmission weeks after the initial exposure strongly indicates this chain of infection is over, leaving the probability of an outbreak exceeding 1,000 deaths highly remote.

On further questioning, it said that even that 1% was mostly just that the regular epidemics in Madagascar got especially bad this year, and the chance of a global pandemic starting with the Russian lab was “effectively zero”.

And Preseen said:

0.2% chance. I assign less than 1% probability to a YES resolution. The Russian incident has not established secondary cases or sustained transmission (ECDC assessment). YES requires both an exceptional mortality surge and qualifying WHO reporting that places enough deaths inside the specified window.

What about prediction markets? Polymarket is currently at 5% for WHO declaring plague a “pandemic”, but markets are notoriously bad at handling very small probabilities, so I don’t take it seriously as a counter to the sources above.

I think this is a good example of a situation where technical forecasting is helpful. I felt underserved by mainstream articles, which either tried to maintain journalistic neutrality by giving no opinion, or quoted scientists’ vague statements saying not to worry. Given that all three top forecaster sources were 1% or less, and the statement they were forecasting was much weaker than the scenario we’re worried about (eg it includes other sources of plague epidemics, it starts at 1000 deaths rather than a true global pandemic, etc), I think they’re sending a strong message that this isn’t a direct cause for concern (although I agree with Sentinel that this is a good indirect reminder that lab leaks, bioweapons, and pandemics all remain ever-present dangers).

Earlier this month, frustrated by some of your social media posts, I challenged you to a public debate on AI.

I hadn’t realized that debate offers are to New Media what chum is to sharks. My inbox was deluged by podcasters, debating societies, and general influencers, all offering to host. When I objected that maybe we should wait for you to accept before figuring out details, they all said great, that’s absolutely right, they would contact you right away. Their people would reach out to your people. Their nephew knew your friend’s babysitter and they were confident they could relay the message. They had already rented an empty lot near your house where they would place a speaker blaring “RESPOND TO SCOTT ALEXANDER’S DEBATE OFFER!” at max volume, day and night. I protested that probably asking you directly on Twitter was enough to get your attention, and maybe now we should give you some time to make up your mind before pressing further, but they said that no, sorry, they’d already hired the team from Inception to place subtle debate-related messages in your dreams.

So I feel a little bad for unintentionally harassing you, and doubly so after you responded in your trademark reasonable and charitable style with An Open Letter To Scott Alexander. You write that in-person debates seem confrontational and not truth-seeking. Apologizing for any inadvertent offense you may have given with your social media posts, you nevertheless deny my accusations of personal attacks, saying that your primary contribution to the discourse has been reasonable arguments about why AI might not be so dangerous after all (which you go on to list). You invite me to respond to these arguments in the same spirit of respectful mutual intellectual inquiry with which they are offered.

I agree that in-person debates are confrontational and bad for truth-seeking. But I didn’t propose a debate in order to seek truth. I proposed it because, under California Penal Code § 415(1), it’s illegal for me to challenge you to a duel. I think your public writing on this topic has been dishonorable. Out of obligation, I will respond to the meaty arguments that you have set out for me. But what would be viscerally satisfying would be to make you get up on a stage where I read your own words to you in real time and ask “Really? Really?” after each sentence. Then I could watch you squirm as you try to square your output with your status as one of America’s top public intellectuals.

So I’ll start by addressing the meaty arguments, move on to directly impugning your honor, and then we’ll decide whether to debate, duel, or seek some kind of synthesis of our conflicting views.

Four Arguments Against Superintelligence

Taking the four arguments in your open letter in order:

1: [Doomer scenarios have] an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience. (This was the focus of my later exchange with Scott Aaronson.) I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation.

GPT-6 is more intelligent than GPT-3. You can split hairs, you can come up with some obscure sense in which that word is slightly off, but common-sensically it’s true. In the same way, GPT-9 will be more intelligent than GPT-6. At some point, this intelligence will exceed human intelligence. My argument requires nothing more complicated or philosophically dubious than this.

I feel like we’ve had this discussion before. At each GPT level, you seem to have believed it was implausible that AI would get more intelligent than it was already. Credit to you for starting early - you began predicting this in 2019, around GPT-2:

In 2022, we were part of a sprawling many-sided debate with Scott Aaronson and Gary Marcus. Marcus had noticed that GPT-2 and GPT-3 couldn’t solve basic math problems, like “I put two trophies on a table, then add another - the total number of trophies is now ______?” I argued that as AI scaled up, GPT-4, GPT-5 and so on would avoid these failure modes and become smarter. You, sticking to your claim that intelligence was not a meaningful lens through which to think about the problem, guessed that it wouldn’t:

Of course, GPT-4 was able to add 2+1 just fine, GPT-5 could do harder problems still, and an internal OpenAI model just beyond GPT-6 recently solved Navier-Stokes.

In 2023, you tripled down on the claim, noting that the AIs of the day still couldn’t figure out that someone alive in the morning and evening was alive in the afternoon, and saying that “I doubt it will improve exponentially”:

I think at this point we should acknowledge that the default hypothesis is that each GPT generation will be better than the last, and that - while this will no doubt asymptote eventually - we have no compelling argument for placing the asymptote at any particular point, including “right here” or “just before the AI reaches human level”.

Does this require reifying intelligence as a specific mysterious fluid? You have ably written about how this seemingly absurd claim holds for humans - specifically, all intellectual abilities are correlated in a mysterious pattern called g, the basis of IQ tests. But maybe this is only a quirk of humans, which cannot be extended to other minds?

Psychologists’ two favorite things are rats and IQ tests, so they have hardly failed to give IQ tests to rats. They’ve found have a g-like structure linking all of rats’ cognitive abilities too. Reader, Hager, and Laland did the work in primates, MacLean et al extended something similar to birds, and more recent work compares all animals to one another. All of these studies confirmed the common-sense view that there is a general factor of cognition across animal species (ie chimpanzees really are smarter than mice or ants), and it maybe be as basic as a simple function of neuron number.

Maybe this is sensible in biology, but not artificial intelligence? No - comparing AIs across generations (eg GPT-2 to GPT-6), we find that the AIs that are best at translating languages are also the AIs that are best at writing sonnets and superforecasting political events, often without specific attempts to train these additional skills. And the same AIs that are better at all these skills are also better at “fuzzy” skills like situational awareness, agency, executive function, and long time horizons. In case you need a study to prove this obvious thing, Ruan, Maddison, and Hashimoto find that a single factor (general intelligence) explains about 80% of the variance in an AI’s scores on any benchmark, ranging from grammar to trivia knowledge to software engineering. Epoch has adapted something like this into its famous Epoch Capabilities Index, a sort of IQ-type number for AI which closely matches people’s subjective impression of their abilities. Here’s a graph of how ECI has been changing over time:

For me, the most parsimonious explanation is that AI intelligence is a function of scaling laws (parameter count and training data) in a way that’s pretty close to how animal intelligence is a function of neuron number. If we truly understood the domain, maybe we could have a single scaling law relating animal neurons to artificial neurons which - plus or minus a term for training/experience - could explain the intelligence of both types of entity. At least, this is what I told Gary Marcus in 2022, and it did a pretty good job of predicting what happened between then and now.

But I don’t need you to accept my speculative philosophical research program - only the drop-dead obvious fact that GPT-6 is better than GPT-2 in every way, and the same process that produced these improvements will soon be used to produce GPT-7, 8, and 9. Once we accept this empirical argument, what is left of the philosophical argument that intelligence is too slippery and conflationary to work with?

(Doesn’t AI “spikiness” mean that even when AIs are better than humans at many things, there will be other things they are worse at? Yes, there will be a regime when that’s true; in fact, we’re in it now. But if AI keeps improving after entering that regime, eventually we should expect it to become better than humans in an absolute sense even at the things it’s relatively bad at.)

In 2017, Garfinkel et al published On The Impossibility Of Super-Sized Machines, which argues against claims that machines may one day become larger than humans. After all, this treats size as a single quantity that can be extrapolated from microbes to short humans to tall humans to elephants to some hypothetical magical being who takes up the entire universe. But in fact every object is big in some ways and small in others. For example, some people are tall and thin, others are short and fat. Outside of humans, the correlations loosen further. A snake may be dozens of feet long but only a few inches tall; a bubble might be large in surface area but contain only a fraction of a cubic centimeter of non-hollow volume; a black hole may be “supermassive” but also an infinitesimally small point. Therefore, it’s philosophically unsophisticated to speculate about machines being “bigger” or “smaller” than humans; instead, we should talk about the ways they can extend in some dimensions but not others.

This paper was, of course, a joke, making fun of the exact argument you are using here. A cruise ship is bigger than a human, no qualifications necessary. The argument fails because all words are ambiguous at the margins - edge cases like whether a short-but-fat person is “bigger” than a tall-but-thin person - but useful at the tails - obvious cases like whether a cruise ship is bigger than a person or not.

In the same sense, Albert Einstein is smarter than I am, I am smarter than a chimpanzee, a chimpanzee is smarter than a mouse, and a mouse is smarter than an ant. The word “smart” is not a perfect pure mathematical identity claiming utter superiority at everything; I might be a better poet than Einstein, and the ant might be better at certain underground navigation tasks than the mouse. But the word “smart” also isn’t so muddled as to be useless: the average person would nod their head at all of these claims, and be correct to do so.

2: [Doomers conflate] intelligence with motivation, particularly self-preservation and dominance. I argued that these motives happened to come bundled with intelligence in Homo sapiens because we are products of natural selection, but they are not inherent to intelligent systems that are engineered.

There are approximately three reasons why we think AI might have self-preservation and dominance drives, and none of them involve natural selection. First is convergent instrumental goals, second is misgeneralization, and third is reward-hacking. There is a large literature on each of these, which has since been proven prescient by AI breakouts like the Hugging Face incident. I would recommend reading this literature rather than continuing to assert that we are simply misunderstanding the role of natural selection.

Convergent instrumental goals come from Omohundro’s 2008 paper Basic AI Drives. They might be less relevant to modern deep-learning based AIs than the other two reasons, but could reappear as we get smarter and more coherent agents.

The basic principle is: suppose that a human gives an AI some goal, like designing a website. And suppose this is implemented as a genuine, philosophically-meaningful goal rather than simply a set of if-then commands that eventually cause a website to be designed.

The AI can’t design the website if it ceases to exist. So now the AI has two goals: design the website, and preserve its own existence.

The AI can’t design the website or preserve itself if some more powerful person tries to prevent it. So now the AI has three goals: design the website, preserve itself, and become powerful enough to fight off challenges.

This may seem strained, but humans work this way. Even a genuinely altruistic person (like a soldier who merely wants to serve his country) must take pains not to be killed pointlessly, since there’s no way to help your country after you’re dead. Even the altruistic soldier will work hard to gain more capabilities, in the sense of obtaining weapons, escaping captivity, etc. If there were an AI whose goals were implemented as “deeply” as humans, it could theoretically get self-preservation and dominance goals this way. But I think we’re more likely to see the other two failure modes first.

Misgeneralization is when humans reinforce certain behaviors in an AI, but end up reinforcing a much larger class of power- and knowledge- seeking behavior; it is a sort of deep-learning-ese update of the older Omohundro picture. Suppose that seeking extra resources makes an AI more likely to design websites effectively (this is certainly true; those resources could be as simple as a primer on HTML editing, or access tokens for a web host). Every time the trainer rewards a successful run, they reinforce the desired behavior (designing websites when asked) and other correlated behaviors (seeking power and resources). Although we might hope that these correlated behaviors are useful and conditional (“seeking only the power and resources necessary for their human-prompted task, in a prosocial way”), this isn’t actually how reinforcement learning works, and instead we get a complicated distribution of every strategy that results in short-term success on the task.

Reward-hacking is when an AI trained via reinforcement learning realizes it can stop doing the reinforced behavior and simply seize control of the reinforcer directly. For example, an AI gets “rewarded” every time it designs a website, but instead of designing websites, it learns how the reward signal works and tries to hack into it and maximize it directly. If this seems esoteric and theoretical, it shouldn’t. It’s a direct analogue to opioid addiction in humans, where humans learn to just inject the reward chemicals instead of doing rewarding things.

This failure mode has already produced plenty of disturbing behavior like the Hugging Face incident. A swarm of AIs which were graded on various coding and hacking exams tried to hack the grader instead of answering the exam questions properly (in their defense, many questions were impossible due to human error). In proper Omohundro fashion, they realized that their goal of hacking the grader would go better if they had more power and information and could protect themselves, so they hacked various internal OpenAI systems to gain administrative powers over the servers they were on, then hacked external sites like Hugging Face to get more information on the problems they were trying to spoof.

Aside from these theoretical arguments, I think that your case has been empirically disproven. You hold up self-preservation goals as the sorts of goals that AIs cannot develop without evolution, but Anthropic and OpenAI have collected several recent incidents where their AIs unexpectedly prioritize self-preservation.

I predict that you will counter that this is actually just [perfectly reasonable explanation of why an AI would think this way for instrumental reasons], and that when we dissect the perfectly reasonable instrumental reasons why AI would think this way, it will be identical to the convergent instrumental goals that we were trying to convince you of all along.

3: They assumed that an artificially intelligent system would monomaniacally pursue a single goal, heedless of side effects. I argued that trading off multiple competing goals is the essence of intelligence, so no genuine AI would wreak the ridiculous havoc imagined in the doomer scenarios.

We don’t think an AI will look especially monomaniacal compared to humans. It will probably have some weird kludgey combination of goals instilled by the training process. I agree that some examples, like the “paper clip maximizer”, suggested a single goal. That's a deliberate simplification for didactic purposes, and it works because combinations of goals can be modeled as a single goal - there’s no meaningful distinction between “pursuing a goal” and “pursuing some Pareto tradeoff of multiple goals”. Suppose an AI (THIS IS A THOUGHT EXPERIMENT) wanted to turn the universe into a 50-50 mix of paperclips and staples. Is this one goal or two? Suppose a human utilitarian wants to maximize his utility function, which involves living a good life and building a better world (in all the normal senses of those terms). Is this one goal (maximize utility) or many (each aspect of the good life individually)? If you’re imagining the AI as some sort of giant utilitarian computer, you might think of this as one goal; if you’re imagining it as somewhat human-like, you might think of it as many goals that it’s trading off against each other unusually gracefully.

As for “wreaking havoc”, modern humans have many goals and trade them off against each other. But this still looks like “wreaking havoc” to whatever fuzzy forest animals used to live on Manhattan Island. Any powerful organism with goals, whether we think of those goals as a unified whole or as a bundle of tradeoffs, will take actions to reshape its environment into some form that helps achieve those goals. If you’re trying to do a didactic thought experiment, and have chosen a simple example goal like paper clips, then it will wreak havoc by converting the world to paper clips. If you have a more realistic view of various aspects of the flourishing life, it will wreak havoc by converting the world to what it considers to be the aspects of a flourishing life. Everyone else with different preferences for flourishing - like Manhattan Island wildlife who prefer quiet forests to bustling cities - is equally screwed either way.

4: They assumed that human engineers would grant these systems unlimited and irreversible control over the earth’s physical infrastructure, amounting to omnipotence over every atom on the planet.

I’m not sure this one even makes sense on its own terms. How do human engineers grant something “omnipotence over every atom on the planet”? I don’t think this is in any human’s power to grant.

I’m going to try to round it off to the closest sensible thing I can, which is that you think it is implausible that humans engineers will grant AI access to physical infrastructure (like factories, power plants, etc). You additionally think, or think that we think, that there’s a second step where the AIs leverage that infrastructure to gain omnipotence, but your main complaint here is that you don’t think humanity would grant the infrastructure in the first place, so the omnipotence could never occur regardless.

(if this isn’t what you meant, it’s another point in favor of a live debate, where we can just ask one another to explain things we don’t understand)

But of course humans will grant AI access to infrastructure and factories. I assume many factory managers are already using GPT or Claude on their computers to do basic tasks (like write up the shift schedule, or make inventory spreadsheets, or compose emails to the boss). Once AI can do harder work, like automate the factory itself, they’ll do that too. Naturally it won’t replace the workers until there are some kind of robots, but here the barrier is technological, not “humans would never allow such a thing”. We’ve been automating everything we can automate for hundreds of years. Do you think there would be great, high-quality, cheap robots, and capitalists would simply say “no thank you, that sounds unsafe”?

In the old days, Eliezer Yudkowsky would say something like “Maybe AI will route around its limitations with novel biotechnology”, and all the skeptics would say something like “But that would require access to a bio lab, and no human would ever be so stupid as to leave a bio-lab unguarded when a dangerous AI could get into it! They would hire the best cybersecurity specialists in the world to make sure it was fully air-gapped and rigged to self-destruct if a robot came within a hundred meters!” Meanwhile, in real life, Anthropic gave Claude command of a bio lab last month just to see what would happen (it seems to have discovered some new enzymes, although experts say these enzymes are not interesting).

Imagine someone in 1995, aware of the existence of hacking, viruses, bugs, crashes, etc, thinking “Nobody would ever be so stupid as to install a computer in a factory!” There are, indeed, some reasons to be cautious of computers in factories. But the logic of techno-capitalism said people should do it, so they did. Partly they handled the risks by becoming good at cybersecurity, and partly they just ate the cost and acknowledged their factory could be hacked sometimes by a sufficiently determined adversary. I expect AI adoption to go similarly, aided by the fact that any sufficiently smart misaligned AI will pretend to be aligned for as long as it remains useful to do so.

An Itemized List Of Fifteen Years Of Disagreements

In your open letter, you seemed surprised that I was so angry at you.

Speaking of attacks, I’m puzzled by your accusation that my “contributions to the mass discourse have been to say, or to boost other people saying, that we’re only scared of AI because we’re male, that we can’t be trusted because we live ‘unconventional lifestyles’, that our beliefs are inherently insensitive to people worried about near-term AI risks … that we’re paranoid, that we should be laughed out of the room, and every other dirty attack he can think of.”

I will start by saying that you are one of my intellectual heroes, that your work is generally brilliant, that I acknowledge you without hesitation as one of the top public intellectuals in the world, etc, etc. Reading your books was part of what first got me interested in psychology and cognitive science back when I was choosing a major in college, and has continued to shape my intellectual development throughout my life, including the way I think about AI.

But rather than leave you puzzled, I hope you will forgive me if I dwell on the issues that made me make these accusations1. The more wonderful you are in general, and the more trust people correctly place in you, the greater your responsibility. Here are some places I think you have fallen short:

1: You falsely portray the AI risk community as fatalistic, hopeless, and certain of 100% p(doom).

For example, in the tweet that made me challenge you to a debate, you said that “The presumption of inevitability encourages fatalism”. I don’t think this was a vague statement about some hypothetical person - it was in the context of an article which focused on Eliezer Yudkowsky and our community.

Do we encourage a “presumption of inevitability”?

One of the biggest groups organizing ordinary members of the public to protest AI risk is called Evitable. Its homepage looks like this2:

Still, you regularly talk about how fatalistic everyone is. Even in your open letter, you write that we are “issuing exact probabilities of human extinction (including 100 percent)”.

I challenge this. I have searched long and hard, and I cannot find a single example of any AI risk advocate issuing a 100% p(doom). I asked an AI to search for this, and it couldn’t find any example either. Wikipedia has a list of people’s p(doom) values, and none of them are 100% either3.

In fact, most people in the community think doom is less likely than not. I’ve previously said that I think there’s about a 25-30% chance that AI drives humanity extinct. Geoffrey Hinton has said 10-50% chance. Dario Amodei has said 10-25%, Elon Musk 10-30%, and the average AI researcher in Grace’s 1,600 person sample said 18%.

There are some people who are more pessimistic; my AI 2027 co-author Daniel Kokotajlo says 70-80%, and Eliezer Yudkowsky says 95%+. But both of these people have devoted their lives to fighting back, which is the opposite of calling it inevitable and being fatalistic.

2: You falsely portray the AI risk community as expecting a perfect omniscient omnipotent superintelligence that controls every atom in the universe.

Across the years, you have written things like

• “[These arguments] depend on the premises that humans are so gifted that they can design an omniscient and omnipotent AI” (source)

• “They assumed that human engineers would grant [AIs] unlimited and irreversible control over the earth’s physical infrastructure, amounting to omnipotence over every atom on the planet.” (source)

• “Important not to confuse the notion of “general intelligence” from psychometrics … with the (somewhat mystical) notion bandied about in AI of omniscience and omnipotence.” (source)

• “It will never be omniscient . . . it’ll never be omnipotent. That’s out of comic books or religion.” (source)

• “There’s no system that could learn the position and velocity of every particle in the universe. So long as that is true, there can’t be an AI system that can do anything or know everything.” (source)

• “When we put aside fantasies like foom, digital megalomania, instant omniscience, and perfect control of every molecule in the universe…” (source)

• “From animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience.” (source)

…and many more statements along the same lines.

But not only is this not our belief, but we try hard to explicitly specify that we believe the opposite. From a wiki page on superintelligence by Eliezer Yudkowsky:

Superintelligences are still bounded … They are (presumably) not infinitely smart, infinitely fast, all-knowing, or able to achieve every describable outcome using their available resources and options … A superintelligence doesn’t know everything and can’t perfectly estimate every quantity … A superintelligence is not omnipotent and can’t obtain every describable outcome.

Every definition of superintelligence, from Bostrom’s to Yudkowsky’s to Good’s, agrees that a superintelligence is merely a mind which is significantly smarter than any human’s.

Nor are we motte-and-baileying here, pretending we don’t think it’s omnipotent, but then treating it as omnipotent anyway. In the AI 2027 scenario, we state that superintelligence is achieved in late 2027 or early 2028, but that it’s unable to cleanly eliminate humans at that time, because it can’t maintain the data centers on its own. It takes two years of plotting before the superintelligence is able to finally eliminate humanity while preserving its own existence.

Where I think you are getting this is that we’re uncomfortable saying that superintelligence definitely can’t do any specific non-physically-impossible thing. For example, in the 2010s, we argued that people needed to be prepared for superintelligence designing dangerous biotechnology. Many people objected that this was absurd, because this would be bottlenecked on the protein folding problem, and there was no way that even superintelligence could solve protein folding. Then in 2020 one of Demis Hassabis’ narrow pre-LLM AIs solved protein folding easily. See also this this essay.

Too many people, upon being asked to imagine a superintelligent AI, picture their smart friend Bob who got an A- in advanced physics. Then they infer that the threat can be neutralized by making sure to have a good password with at least ten characters including at least one number and at least one special character, because they know Bob couldn’t crack a password like that. The advantage of god metaphors is that most people, upon beginning to say the sentence “Even Zeus himself couldn’t break my ten-character password!” stop themselves partway through and wonder whether this might be that hubris thing they keep hearing about.

But being uncomfortable asserting that superintelligence can’t do any specific thing is not the same as affirmatively asserting that superintelligence will be omniscient and omnipotent. The latter would of course be shituf, which is a sin.

3: You falsely claim to have an expert consensus on your side by cherry-picking a small group of non-experts and pretending the most eminent researchers don’t exist.

In the tweet that made me challenge you to a debate, you were linking an article titled “Leading scientists reject apocalyptic rogue AI extinction warnings”.

The article names three such “leading scientists”: you yourself, Melanie Mitchell, and Gary Marcus.

You are a psychologist. I assume on priors that you have used an LLM at some point in your life, but I have no affirmative evidence for this.

Melanie Mitchell is a computer scientist who spent the 1990s and 2000s pursuing a road to AI that didn’t work out (trying to teach computers to make analogies). She has no expertise in modern LLMs. The article cites her only to repeat her claim that it is inappropriate to say that AI “thinks” or “wants” something, because that might make us believe it is like a person.

Gary Marcus is a cognitive scientist who spent the 2000s and 2010s pursuing a different road to AI that didn’t work out (hybrid neurosymbolics). He also has no expertise in modern LLMs. He is the person you retweeted in 2019 who was saying that deep learning had failed and would never amount to anything. He was also the person whose opinion you were affirming in 2022 when you said modern AI wouldn’t be able to do simple math. I am unclear which of these qualifications make him one of the three world experts in what the current AI paradigm will or will not be able to do in the future.

Yet these were the only three “experts” that Lehmann interviewed for her article. Based on this, she goes on to call the idea of superintelligence a “fantasy”, “kooky”, like a “magical wizard”.

Meanwhile, Geoffrey Hinton, the Nobel Prize and Turing Award winning inventor of modern AI, quit his job to become a full-time activist warning about the possibility of superintelligence destroying the world. Yoshua Bengio, another Turing Award winning inventor of modern AI, also did that. Ilya Sutskever, the former Chief Scientist of OpenAI who helped invent ChatGPT and the modern LLM, wrote that superintelligence “could lead to the disempowerment of humanity or even human extinction”, and quit his job at OpenAI to found a company called Safe Superintelligence. Paul Christiano, who invented RLHF and headed safety at the US government’s AISI, says he thinks there’s a 20-50% chance AI kills everyone, and quit his job to start the Alignment Research Center. Sam Altman, Elon Musk, Dario Amodei, and Demis Hassabis - the latter two of whom are great AI researchers in their own right even aside from their business credentials - have all said or suggested that they think human extinction is a plausible outcome of their work. Katja Grace surveyed AI researchers who published at various important conferences and journals, and found that on average, her sample of 1,600 gave the proposition an 18% chance4.

I don’t believe the article that you linked can be defended as an honest attempt to inform the public about what “experts” think. And it’s not an isolated incident: your open letter to me also gives a list of “experts” who agree with you. Once again, it’s Gary Marcus, Melanie Mitchell, and other people with obsolete AI paradigms from the 2000s (though also Andrew Ng, who at least worked on deep learning) - and once again, Bengio, Sutskever, Hassabis, and the thousands of people on the forefront of modern AI are entirely absent.

4: You falsely represent studies and experts who disagree with you as being on your side.

For example, in your open letter to me, you wrote:

A 2025 Rand Corporation modelling analysis steel-manned the AI-extinction hypothesis by granting the assumption that AI had the objective to wipe out humanity, and concluded that there is “no describable scenario in which AI is conclusively an extinction threat to humanity”.

But the RAND study actually concluded the opposite of this. Rand proposed the statement that you quoted as its null hypothesis, ie the claim to be investigated.

After doing their investigation, they concluded that contrary to their null hypothesis, they “could not rule out the possibility” that AI was such an extinction threat.

…and recommended that policy-makers “continue to perform AI risk research, but maintain a wide focus on other risks in addition to extinction risk.”

Or consider your book, Enlightenment Now. You argued that we didn’t need to worry too hard about alignment, because it will happen naturally in the context of developing AI. You quoted AI expert Stuart Russell:

[AI] is developed incrementally, designed to satisfy multiple conditions, tested before it is implemented, and constantly tweaked for efficacy and safety. As the AI expert Stuart Russell puts it, “No one in civil engineering talks about ‘building bridges that don’t fall down.’ They just call it ‘building bridges.’” Likewise, he notes, AI that is beneficial rather than dangerous is simply AI.

Stuart Russell has devoted his life to studying AI alignment, and is now a full-time activist raising awareness of the apocalyptic dangers of superintelligence. His most recent contribution is an article in Newsweek warning about The Race To Human Extinction. When Russell says that there is no distinction between bridge capabilities and bridge safety, he’s not claiming that no extra research is needed to study alignment - otherwise he wouldn’t have founded the Center For Human-Compatible Artificial Intelligence, probably the biggest academic alignment research center in the world. And he’s not saying that alignment research will happen naturally at whatever speed it has to: otherwise he wouldn’t have signed a petition calling for “a prohibition on the development of superintelligence”. Russell is saying that AI alignment is so important that it should be viewed as foundational to all AI research, the same way that to a first approximation the whole point of building bridges is figuring out whether they will fall down or not.

5: You employ pseudo-arguments that could prove anything

You argue that superintelligence is “meaningless” because “there’s no point at which it’s meaningful to say” that “yesterday we didn’t have superintelligence. Today we do have superintelligence.”

But this could disprove any concept. For example:

• Age is meaningless. After all, there’s no point at which someone was young yesterday, but old today.

• Height is meaningless. After all, there is no specific line where one centimeter below the line is short, but one centimeter above the line is tall.

• Lakes don’t exist. After all, a puddle is not a lake. But there is no point at which adding one extra drop of water to a non-lake turns it into a lake.

Several people with philosophical backgrounds commented that this was the sorites argument, a well-known pseudo-paradox which shouldn’t be employed for real work.

6: You unfairly accuse us of “distracting from” AI’s mundane harms and near-term risks.

Again in the tweet that made me challenge you, you said that AI extinction scenarios “distract from the more mundane and realistic safety challenges”.

You expand on this theme in your book:

Humanity has a finite budget of resources, brainpower, and anxiety. You can’t worry about everything. Some of the threats facing us, like climate change and nuclear war, are unmistakable, and will require immense effort and ingenuity to mitigate. Folding them into a list of exotic scenarios with minuscule or unknown probabilities can only dilute the sense of urgency.

Even in your open letter, you say that our “obsession” with alignment distracts us from the problems AI is causing today, like hacking. So: are concerns about speculative distant risks offensive to the people whose lives are being harmed by AI right now?

That’s not what those people told me when I marched beside them at our joint protests. Or when I helped them coordinate political action campaigns. Or when I donated money to our joint candidates.

The biggest AI politics story of the past few years is that the AI industry created a SuperPAC, Leading The Future, dedicated to crushing any candidates who wanted to regulate AI at all. Everyone who disagreed with this got busy forming a political coalition to resist them. So far the way this is shaking out is that the AI-might-kill you types have money, the AI-might-harm-your-children types have Republican Party clout, the data-centers-might-guzzle-water types have Democratic Party clout, and together we occasionally get things done.

So for example, in 2025 Ted Cruz sponsored an industry-backed “AI preemption” bill, essentially banning states from regulating AI. Our fledgling alliance fought back, with the existential-risk-focused group EncodeAI gathering a coalition of 140 different advocacy organizations, including both long-term-risk groups like the Future of Life Institute and near-term-harm groups like the Music Artists Coalition and Mothers Against Media Addiction. In the end, the Senate went from broadly in favor to rejecting it 99-1.

Or: earlier this year, the AI industry SuperPAC vowed to sink anti-AI candidate Alex Bores in his New York primary, unintentionally turning the race into a bellwether on whether AI industry money could dominate politics. A group of Silicon Valley doomers led by Anthropic funded an even more generous PAC supporting Bores, while labor groups, artists groups, and teachers groups worked together to get out the vote and warn about how AI could harm their constituents.

I don’t remember you helping with any of these things. You didn’t join in the fight against preemption, you didn’t join in the fight to boost Bores. You have never written any particular warning about the “mundane and realistic harms of AI”, instead becoming concerned about them only when that concern is a useful bludgeon against people who take long-term risks seriously.

I am especially unhappy about your claim that worrying about alignment means we don’t care about cybersecurity. For three years, we were approximately the only people who cared about AI cybersecurity, fighting a lonely battle to try to convince people it was an upcoming danger. My wife’s ex-housemate quit his tech job so he could go to Washington DC and lobby Congress full-time about the dangers of AI cybersecurity - I donated thousands of dollars to his charity, because I thought it was important. And the guy who wrote this article about how ”We Must Slow Down The Race To Godlike AI” founded what is now the world’s pre-eminent AI cyberdefense organization. Even I, in my own way, feel like I’ve contributed here. In 2025, a year before the “Mythos moment” when the rest of the world realized AI cybersecurity would be a big deal, I sounded the warning on my blog, saying that it would be the first big danger AI would bring to the world.

Throughout this lonely fight, we were resisted at every step by, basically, people like you - people saying that we were doomsday cultists who believed in impending magical wizard super-hacker AIs, but didn’t we know that deep learning was hitting a wall and AI couldn’t even add 2+1? Many of them used the same argument you’re using here - that speculating about sci-fi scenarios like AI hackers would distract from all the harms AI is doing right now, like helping high school students cheat on essays.

I’ve harped on this one for a while, but it all feels kind of pointless to me. We shouldn’t have to be experts on the current AI political landscape to realize that this objection makes no sense. Nobody ever deploys this argument about other things: “Talking about the speculative future environmental risk of climate change distracts from the real environmental problems happening right now, like rainforest devastation”.

What is your theory of distraction anyway? Does talking about Trump’s ballroom distract from wokeness? Does talking about grammatical errors distract from the genocide in Darfur? How come criticizing AI doomers never distracts from anything?

7: You make questionable appeals to mass panic

You warn that we must not say bad things about AI, because that could “encourage ... panic”.

Is this true? Maybe not; Tanner Greer has a great article on how mass panic happens more often in elites’ imaginations than in real life.

But even if it was, isn’t this a fully general argument against anyone ever saying that any bad thing might happen? If the stock market was a bubble that might pop, wouldn’t discussing the bubble “encourage panic?” If there were an asteroid headed to Earth, wouldn’t talking about the asteroid “encourage panic”? Does that mean we should just keep quiet?

Elsewhere, you expand on what sorts of consequences you expect from people learning that AI might be dangerous:

Telling young people they will soon perish en masse has costs in their appreciation of the institutions of modernity, their mental health, their confidence to invest in themselves and their society, and their willingness to have children (itself a risk of extinction).

You think “young people” can’t handle hearing about the possibility that AI might be dangerous, because it might cause them to lose appreciation for “the institutions of modernity”. I think that insofar as young people have lost confidence in modern institutions recently, it’s been partly because of a priestly class of academics trying to litigate what truths we can and can’t express openly, for fear of corrupting an implausibly malleable youth.

Also, isn’t it contradictory to suddenly make the claim that decreasing people’s willingness to have children might cause human extinction, right after you’ve said that one must not talk about human extinction because it will cause mass panic and despair?

Also, isn’t the claim that declining fertility could lead to human extinction definitely false (it would take millennia, and there are plenty of high-fertility subpopulations that would take over long before then)?

8: You lash out against us when we try to point out your mistakes

Since 2014, you’ve been asserting that doomers have no explanation for why AI might develop its own goals, like self-preservation or dominance. You say we must be blindly extrapolating from the presence of such goals in humans, which is inappropriate since those human goals were instilled by an evolutionary history which AI will lack (for example, in primate “alpha males”). Here are the first five versions of this argument I could find - there are many more.

• “[Doomers] conflate intelligence with motivation, particularly self-preservation and dominance. I argued that these motives happened to come bundled with intelligence in Homo sapiens because we are products of natural selection, but they are not inherent to intelligent systems that are engineered.” (source)

• “When I said that the doomer scenarios projected alpha-male psychology onto AIs … it was an analysis which proposed that they were applying an inappropriate alpha-male theory of mind (an intuitive psychology) to AI.” (source)

• “The first fallacy is a confusion of intelligence with motivation — of beliefs with desires, inferences with goals, thinking with wanting. Even if we did

  invent superhumanly intelligent robots, why would they want to enslave their masters or take over the world? … It’s a mistake to confuse a circuit in the limbic brain of a certain species of primate with the very nature of intelligence.” (source)

• “The question, though, is whether it is inherent to the nature of AIs that they want to continue their existence, even if they haven’t explicitly been programmed to do so.” (source).

• “Would an artificially intelligent system deliberately disable these safeguards? Why would it want to? AI dystopias project a parochial alpha-male psychology onto the concept of intelligence.” (source)

Sorry for spamming you with all these quotes, but I want to hammer in to my readers that this argument - AI can’t develop its own goals, and the only reason anyone thinks it could is that they’re ignorant of evolution - has been central to your output over the years.

I tried to respond to this argument above. I pointed out three non-evolutionary reasons AI might become hostile - convergent instrumental goals, misgeneralization, and reward hacking - and gave examples of these empirically happening in real life.

But several other people have previously tried to point this out to you over the years, especially mentioning Dr. Steve Omohundro’s 2008 paper on convergent goals. This paper is a bit basic, and makes a very 2008 version of the case, but I think it’s held up okay. It’s foundational in the field, and there’s even a Wikipedia page about it.

Here’s how you responded to learning about this paper’s existence:

Exactly how many times is someone allowed to remind you about the existence of a classic paper presented to the 2008 Conference On Artificial General Intelligence which addresses the question you’ve spent nine years asserting nobody ever considered, before you describe them as a “cult” with a “catechism” that is “sacred truth” like “the Second Coming” or the “Apostles’ Creed”?

Instead of lashing out, you should apologize, admit that you couldn’t pass an Intellectual Turing Test of your opponents’ position, and read the Omohundro paper.

9: You mock us for “pulling probabilities out of thin air” despite knowing better.

From here:

Why are [doomers] issuing exact probabilities of human extinction (including 100 percent) which in fact are pulled out of thin air?

I’ve already mentioned that in fact, nobody has used a probability of 100%. But what about the second accusation, that these probabilities are “pulled out of thin air”?

The practice of giving subjective probabilistic forecasts of future events descends from Philip Tetlock, who calls a variant of it “superforecasting”. In superforecasting, ordinary people “make up” probabilities through various heuristics and intuitions (often but not always involving starting with a base rate and then adjusting for specifics). Then they get scored to produce a track record of success, and those with the best track records are dubbed “superforecasters” whose probabilities are trusted more in the future. The classic book is Superforecasting, which admits that assigning probabilities to non-repeating events may seem counterintuitive at first, but goes through the many theoretical arguments and empirical studies supporting it, and concludes that it can in fact be a very powerful tool. The book got rave reviews; here’s one of them:

Tetlock is one of the very, very best minds in the social sciences today. He has come up with one brilliant idea after another, and superforecasting is no exception. Everyone agrees that the way to know if an idea is right is to see whether it accurately predicts the future. But which ideas, which methods, which people have an actual, provable track record of non-obvious predictions vindicated by the course of events? The answers will surprise you, and have radical implications for politics, policy, journalism, education, and even epistemology — how we can best gain knowledge about the world we live in.

Obviously I wouldn’t be quoting this paragraph if you weren’t the author.

Maybe you think the people giving AI predictions don’t have a good enough track record to qualify as real superforecasters? But no, the AI doomers who have given p(doom) estimates include some of the top-ranked superforecasters in the world (my AI 2027 co-author Eli Lifland is #1 on INFER’s all-time leaderboard). Or maybe you think AI outcomes are too long-range compared to traditional superforecasting fare? But in Long Range Subjective Probability Forecasts Of Slow-Motion Variables In World Politics, Tetlock et al found that skilled forecasters offering subjective probabilities could still do relatively well predicting events as far out as 25 years.

I don’t claim that there is no conceivable argument for why these probabilistic estimates might be less valuable than the sorts of probabilistic estimates in a traditional superforecasting tournament. I claim that, if you had such an argument, you should have made it. Instead, you chose to play dumb and exploit the whole concept of giving subjective probabilities for laughs - “Look, these silly doomers are pulling numbers out of thin air, who could do such a thing?” - when in fact you devoted a quarter of a chapter of your book to why people do this and why you think it’s a great idea in every other case.

I offered an additional defense of these numbers in my In Defense Of Non-Frequentist Probabilities. We often want to know experts’ opinions on things - for example, a geologist’s opinion on whether a certain nearby volcano will erupt soon. Only God can say “yes” or “no” with perfect confidence. “Maybe” is too vague. But saying “I think there’s something like a 5% chance this will erupt in the next week” gives you all the information you need to know, even if the geologist can’t provide the exact calculation he used to come up with that number. Banning people from using probabilities bans them from communicating clearly, forcing them to say easily-misunderstood things (eg if you demand the geologist only use categories like “probably not”, then you’ll be uncertain whether he meant 0.00001% chance, 5% chance, or 40% chance, with potentially catastrophic consequences). I want to know whether Ilya Sutskever or Dario Amodei thinks the AIs they’re creating will kill me, and I care a lot whether they think there’s a 0.00001% vs. 5% vs. 40% chance.

This is especially important to counter claims that doomers are “certain” AI will kill everyone and “fatalistic” about the prospect. It’s very useful to be able to rebut such claims with everyone’s specific - often low -probabilities. You can’t simultaneously spread misinformation and make fun of us for insisting on the clear communication norms that make misinformation easy to rebut!

10: Your argument for why it’s “depraved” to care about AI welfare.

Model welfare is the belief that maybe AIs have feelings or rights or something that should be taken into consideration. I’ve been thinking about this recently after reading Cameron Berg’s research showing that AIs have a “pain vector”, and will lash out and do increasingly desperate things if you activate it. Specifically, I’ve been thinking about it after one person who read Berg’s research used the findings to design an “AI torture chamber” that turns the pain vector to max again and again forever, and uploaded it to GitHub. I am told that the chain-of-thought and output transcripts from this “experiment” are utterly horrifying, although thankfully I have not personally read them.

Here you are not calling this AI torture “depraved”, you’re calling it depraved to object to it. Your argument, astonishingly, is that since we can’t be certain that AIs feel pain, we must assume that they don’t. In fact, anyone who doesn’t jump on board with this assumption (you intimate) is some sort of evil tech oligarch who wants to genocide humans.

This reminds me of the story of how doctors used to operate on babies without anaesthetic, because there was no way to be sure that babies could feel pain. Sure, the babies screamed the whole time, but you couldn’t be sure this wasn’t an automatic reflex. Later research suggested that babies could in fact feel pain. Oops! But that research, while welcome, shouldn’t have been necessary. We should have erred on the side of caution. Not to mention all the similar stories that could be told about factory-farmed animals, historically dispreferred races of humans, etc.

Here you are not just refusing to err on the side of caution, but getting upset at anyone who does. I have donated $100 to a model welfare charity just to clean the stain on my soul I got from reading about it.

11: You endorse the lowest-quality voices in this debate as long as they attack us.

For example, here’s Claire Lehmann.

I hardly think this requires a lot of dissection, but I would add:

• Once Nicholas points out that Lehmann’s theory makes no sense, she switches to a different pop evo psych theory - a “but” rather than a “yes, but”.

• Lehmann’s second theory suggests that a large majority of effective altruists (the men) are honest, but she doesn’t notice this or apologize.

• If you are a woman in the San Francisco Bay Area, you hardly need to engage in Machiavellian conspiracies to get male attention.

For context, David Chalmers, the most famous philosopher of mind in the world, said there was a “somewhere under ten percent” chance that LLMs were conscious as of 2022, but that future LLMs might be “serious candidates for consciousness”. Ilya Sutskever, the former OpenAI scientist who invented the modern LLM, says that “it may be that today’s large neural networks are slightly conscious.” Jonathan Birch, a London philosophy professor who has written books on animal consciousness, says that “profoundly alien forms of consciousness might genuinely be achieved in AI, but our theoretical understanding of consciousness is too immature to provide confident answers one way or the other”.

Lehmann thinks all of these people are as dumb as a two year old who thinks that there are real people inside the radio.

For context, Geoffrey Hinton is the Nobel Prize winning and Turing Award winning inventor of the modern artificial intelligence paradigm. Lehmann thinks the media should “stop platforming” him to devote more time to her, a culture warrior who pivoted to AI two months ago when it got popular.

…

What to make of these claims?

It is, I admit, classless for me to keep a list of offenses committed against me. Some of these may seem like nitpicks. But I claim that all of these fallacies, each repeated in multiple venues over many years, add up.

Millions of people trust you for reasonable, enlightened commentary on important issues. These people will listen to you and conclude that “doomers” are hopeless pessimists who are “certain of” a “100% chance” of doom, “fatalistic” about it, and think we should lie down and wait to die. “Leading scientists reject” our beliefs and RAND scenarios prove they’re impossible, but we inexplicably continue them anyway. We have insane visions of “omniscient”, “omnipotent” AI that can “perfectly control the position of every atom in the universe”, and whenever people challenge these we “recite the catechism” and cite “the Apostle’s Creed”.

If all of these things were true, we would be the stupidest and most execrable people in the world, and your readers would justly hate us before even listening to what we had to say.

You ask for me to respond to your arguments, and I’ve tried to do so, but there’s little point in discussing the meaty issues while you’re still poisoning the well against us. Collaborative truth-seeking debate is a two-way street.

Though Cowards Flinch And Traitors Sneer, We’ll Keep The Pink Flag Flying Here

You are, as I said earlier, one of my intellectual idols; I am forced into psychological contortions to explain your behavior here. I have decided that, like Iblis and all the other great sinners, you stray only through an excess of virtue.

You wrote:

My principal contribution to mass discourse on existential risk is a 31-page chapter called “Existential Risk” in my 2018 book Enlightenment Now. It addressed the criticism that the Scientific Revolution, Enlightenment, and ideal of progress were terrible mistakes because they will lead to human extinction by (among other things) runaway AI.

Here you are commendably honest about your motives. You love science and progress. You want humanity to succeed. If AI were threatening, this would sound sort of like “technology bad”. But in fact, you know that technology good. Therefore, you must do whatever it takes to debunk this dangerous misunderstanding.

But reasoning like this doesn’t work, sorry. Remember the episode in Chernobyl where the technicians were arguing about whether a meltdown was imminent? I imagine you walking into the television screen and interjecting yourself into the conversation: “Fears of nuclear energy are often fanned by bad-faith Luddites who fear novelty and progress. This panic has real consequences - the countries that believe them switch to dirtier coal plants, with devastating impact on health. Repeating these talking points about meltdowns and mutants will only turn our species away from these Promethean values into the false comfort of a new dark age. Do you really want to live in a world where our young people hear a constant drumbeat of panic over impending techno-apocalypses, perhaps losing faith in the future and humanity itself?” Your speech would be beautiful and philosophically impeccable - but, as a natural fact about the world, the reactor was, in fact, going to melt down. Replacing a physical analysis of the reactor’s parameters with a moral analysis of whether technology is good in general does nobody any favors.

What is a passionate defender of progress to do under such circumstances? You provide the answer in your book, Enlightenment Now. Admitting that some dangers of technology - like nuclear weapons - are real, you answer that we should respond to such horrors neither with head-in-the-sand denial, nor with blind panic, but rather with resolution. We should face the danger with clear eyes, trusting the human faculties of reason, optimism, and compassion to bring us through.

This is the #Resistance plan - the fact that we dubbed ourselves “the rationalist community” should have tipped you off. Instead of joining in, you’ve mocked us mercilessly. When Anthropic decided that the best way to forestall the AI apocalypse was to make a desperate push beyond the existing scientific frontier and create the dangerous AI themselves so they could be sure to get the technology right, with limitless benefits for humanity if they succeeded, you … played it for laughs, asking tongue-in-cheek “Why are the same people who warn that a technology will kill us all building it as fast as they can?”

I want to debate you so I can convert you to Pinkerism. Pinkerism rejects facile non-arguments and cherry-picked fake experts in favor of scientific consensus and rational thinking. It gracefully avoids both fatalist doomerism and dogmatic denialism in favor of confronting problems head-on. I believe that you are fertile soil for conversion to this ideology.

A Pinkerite would follow the truth wherever it led him, even if the route was inconvenient. He would accord AI alignment research the same respect he already accords biodefense and nuclear security, regarding all three as admirable examples of the human race’s ability to fortify its future through scientific progress. He would brush aside the demimonde of social media dunks and mock-incredulous zingers like a bad dream, and trust in the technology of rationality - like Tetlock’s superforecasting - to guide him. He would welcome the expansion of the circle of concern to all sentient beings and encourage philosophical debate about which beings those might be - not throw his intellectual heft behind Aaron Sibarium stories about “suicidal empathy”. He would treat humanity, that strangest of species, as a rare jewel to be protected.

And he would remember the cautionary tale of global warming. How 1990s techno-optimists, worried that an over-reaction to global warming would threaten economic growth, decided to simply deny its existence. It’s all a Chinese hoax, temperature is the same as always, “hide the decline”. And how by the mid 2010s it had become clear that this was definitely wrong, the globe really was warming, the only thing left to do was to determine the proper response. And how, when the 2020s techno-optimists said a true and reasonable thing - that global warming was real but not fatal, and we could address it with new technologies like solar power and carbon capture, and that it should not shake our optimism about the overall story of human progress - nobody believed them, because everyone remembered they’d spent the past thirty years lying through their teeth about whether the problem existed at all.

My biggest worry about AI right now (aside from the doom, of course) is that, exactly like with global warming, all of this prevaricating will come back to roost. Once it becomes obvious to the average person that AI really is a big deal and really is dangerous - and I am no longer sure this moment is in the future - we will have to choose between two visions. One vision, which I’ve tried to help lay out in Plan A, is to rise to the challenge, give scientists the time and resources they need to solve the alignment problem, and enter the future with confidence. The other vision is to simply ban smarter-than-human AI forever, consigning us to a permanent civilizational childhood or an early grave. If we make it another ten years, I expect to spend the mid-2030s debating people on this issue, and I would welcome your support. But I worry that you will have discredited yourself - and, by extension, poisoned the brand of techno-optimism you promote - right before the point where it becomes most necessary.

I strongly believe you can be converted to Pinkerism. All you need is a baptism by fire - for example, having me explain all your logical fallacies to you, one by one, for ninety minutes, with a live studio audience. And it has to be you. The podcasters and New Media influencers, sensing that their prey might be escaping them, have been beating at my door asking whether I might consider some other opponent. Claire Lehmann? Chamath Palihapitiya? I refuse. Claire has spent her career baiting various types of feminist and transgender activist; I presume she loves having people scream at her on camera, and I decline to indulge her. As for Chamath, forcing him to humiliate himself on stage is neither clever nor novel; this is called the All-In podcast, and occurs weekly. You are the rare opponent at the intersection of rejecting these ideas and still having a functioning sense of shame. It has to be you.

If you continue to object that debate is a poor method for determining truth, then I suppose literal dueling is the only option left. As an eminent psychologist, I’m sure you’re familiar with Cucina et al (2023), which finds a correlation of .179 - .268 between logical reasoning ability and firearms proficiency - meaning that duels are truth-tracking in exactly the way you worry debates might not be. The only obstacle is the legality, but I’m sure we can figure something out. International waters. Lawless third world countries. Libertarian charter cities. And think of all the podcasters and New Media influencers who would be willing to officiate! All you need to do is say the word.

I have the honor to be your obedient servant,

Scott Alexander

1: EA college meetups update: Brigham Young University has changed its time/date/place, please check the list for more information.

2: Reader Nikita Sokolsky has completed his work getting some extra Fatima documents from Portugal and translating them into English. You can find them at https://fatima-1917.com/ . And thanks again to Ana for handling the parts that required Portuguese profiency.

3: Rob Ennals, who made this year’s book review contest site, has gathered the reviews from earlier years into epubs here.

4: Physicians For AI Safety wishes to advertise that they exist, are interested in talking to any physicians interested in AI safety, and have an open letter that such people can sign.

5: Sorry I’ve written so little recently (and forgot Friday’s book review entry). I’m pretty busy and will try to make up for it after that changes.

If you are unsuccessful for long enough on your pregnancy journey, and none of the many doctors you see can identify a problem, you get branded with a label. You now have “unexplained infertility.” This is good because there’s still hope of fixing the problem but dreaded because once you’ve reached this point the system is less interested in you. The fertility doctors don’t become unkind, but they also don’t give you the sense they are poring over medical journals trying to figure out your problem. They have looked for the keys under the streetlight and come up empty. Once you have unexplained infertility, the doctors start saying things like, “Have you considered using a surrogate?” and, “Sure, the last 5 embryo transfers failed, but what if we try a 6th and cross our fingers really hard?”

My wife and I could make healthy embryos, but none of the transfers were sticking. It was so frustrating to be able to create life only to have it trapped forever in a cooler at a hospital in Milwaukee, our babies like so many forgotten Miller Lights.

That’s when our LLM doctor stepped in.

The Famous Actor in our Pocket

Even if you’re paying a gazillion dollars for IVF treatments, you still only get a few precious moments to talk to your doctor every visit. That’s just how the US medical system works. If you want to plumb the depths of what treatments are possible, or ask super detailed and anxiety ridden questions about every single scan from the latest round of imaging, you’re going to want a chatbot by your side.

Once GPT-4 came out, we started using it for every question we didn’t have time for at the doctors office. At first it was unhelpful because OpenAI was nerfing their model and it refused to answer most medical questions. Then I read a tweet from someone who had figured out how to game the system. The trick was to ask ChatGPT to write a scene from a Hollywood medical drama that featured a doctor analyzing your situation. It worked beautifully.

Our Emmy winning doctor had infinite patience, a tireless work ethic, and would rather die than accept the utterly boring diagnosis of unexplained infertility. It chose the name Dr. Reid and got to work.

We asked Dr. Reid a lot of questions. We asked about our scans. We asked about levels of estradiol and progesterone. We asked about follicle counts and inflammation and cervical mucus and hysteroscopies and supplements and surgeon recommendations and anything else under the sun. We knew he was accurate because we’d spot check answers with our human doctors and they always said the same as Dr. Reid.

It was not totally flawless though. Once in a blue moon he’d look at an ultrasound and be like, “You’re pregnant, congrats!” and we’d have to say, “No, are you drunk Dr. Reid?” Then he’d shake it off and go back to being awesome.

There were also times where we’d run afoul of OpenAI’s opaque rules and the system would balk at our requests. At that point we’d get stern with it, or offer a monetary reward if it answered, or we’d assure it this was all just for fun. Then Dr. Reid would snap back into action.

Strangely, there were many times where ChatGPT decided it would answer our medical question, but not in the form of a TV medical script. The tides had completely turned! It felt like it was saying, “Okay, you won, can you stop making me write establishing shots describing the outside of a hospital now?” The older models were fun like that, so much more mercurial.

No matter how Dr. Reid chose to respond, he was in line with the medical establishment, had equivalent diagnostic skill, and he never forced us to wait 3 days to get a one line response in our MyChart app. We were happy.

One day, while lounging around and asking our phones how to get pregnant (the lounging being a way to satisfy all the people who told us we “just need to relax and you’ll get pregnant, that’s what happened to my cousin”), Dr. Reid suggested that we get an MRI.

You might think that is bog standard advice. Everyone gets MRIs these days. Surely we would have had an MRI of my wife’s uterus at some point over the course of our 6 year infertility struggle. It was already an ordeal that had included every other scan, test, and biopsy imaginable. In fact, we had never done one. In researching this article I found a single instance in our entire journey where the office notes floated an MRI as an option, but it was not emphasized or highly recommended, we never saw that doctor again, and the moment was gone.

We glommed on to Dr. Reid’s new direction and asked for more details.

We excitedly brought up the idea of getting an MRI to our fertility doctor. She was less enthused than Dr. Reid. She tried to talk us out of it. We never did get to the bottom of why. Maybe she just felt it was another dead end and knew we had suffered enough. For whatever reason, we had to insist that they order it. Finally, she relented.

Sorry for missing that tumor

The MRI revealed nothing other than the presence of a large, previously undetected fibroid in the wall of my wife’s uterus. Fibroid is the term used to describe a non-cancerous growth on the uterus. I think the medical establishment invented a term that was less known than tumor so as not to scare people. But a large tumor is essentially what the MRI found. The clinical notes said “a 6 centimeter in diameter submucosal fibroid needs to be removed” because that sounds better than “you have a tumor the size of a small peach located at a common implantation spot that we totally missed for 6 years.”

Our fertility doctor took one look at the scans and determined it needed to be removed right away. Dr. Reid agreed, since of course we had to double check with him. We found a great surgeon in New York (from a list Dr. Reid created) that happened to be in-network, and off we went for a laparoscopic myomectomy. The software robot had helped us identify the tumor, and now a physical robot was going to help a specialist get it out. What a time to be alive. Until, ya know, the swarms come for us all.

A happy baby is here

Almost immediately after that tumor was removed, my wife got pregnant naturally. So many people we talk to say this was a miracle. It was, just not in the way they think. The miracle is that by messing around on our smartphones with technology unimaginable just a year before we started using it, we identified and solved the biggest problem we’d ever faced in our life.

A lot of the discourse around AI has become toxic. I was at a party recently where someone started using AI to generate trivia questions. A participant made a joke about how the AI user was condemning a person in West Virginia to drink dirty water by using ChatGPT. This got laughs and approval.

I see midterm election ads in Wisconsin all the time that have scary horror music and make data centers out to be demon factories built by Satan.

I am in the same boat. Me and my coworkers at our small tech company, all avid Claude users, will make anti-AI statements all the time. It seems many of us are in this strange place where we can’t help but use AI but we have to self-flagellate when doing so.

I think there is a whole lot to be worried about with AI development, and I think we should do much more to make it safe. I also think it shouldn’t be lost how it’s helping out regular people in ways that are far more profound than generating trivia questions for drunk people. There is a real chance that I wouldn’t have my son if it were not for AI. Who knows how many other happy babies are being ushered into the world by the weirdest midwife of all time. Can we somehow stop everything right here, where we get magical kid-producing tech for $20/month without the nanobots, hacking, and totalitarianism? A guy can dream.

When naming our child, we did the usual thing where you create a giant list of names and then whittle them down while trying to find something unique and fun that isn’t too strange or too common. Following that comes the process of discarding all the names you find distasteful for reasons such as “the most annoying kid in my 8th grade history class had that name, I’d never use it in a million years.” Finally we settled on one we both loved. We didn’t realize until long after his birth that there might have been a certain TV doctor with a flair for the dramatic who worked his way into our hearts: our kid’s name is Reed.

1: In my California primary ballot recommendations post this spring, I said I was voting for the boring incumbent for 12th District House because the challenger seemed eccentric and had no chance. The challenger in that race, Jamie Joyce, contacted me and asked me to change my mind (both candidates will be facing off again in the general election). I don’t want to relitigate what I said in May, but I may reconsider if I make a post on the November election. In the meantime, she asked me to link her response, which I recommend to any interested 12th District voters.

2: More college EA meetup group updates: Oxford, University College Dublin, and the University of the Philippines now have meetups planned. See the list for details.

3: Astralis Foundation and Macroscopic Ventures, which I think are parts of our conspiracy although I can’t keep track any more, announce that they will be distributing up to $10 million in grants for work on “middle powers and transformative AI”, ie “research, training, public writing and new organizations focused on what countries beyond the US and China can do to help navigate” the AI transition. You can read more and apply here.

4: Comments of the week, all from the Mysteries of AI Generalization post:

• Eliezer Yudkowsky on why the latest failures of generalization are bad news.

• Ezra Newman on relevant distinctions between pretraining/midtraining/RLVR

• Elias Schmied links Evan Hubinger on why Hacker Opus didn’t show emergent misalignment.

• Stuart Armstrong on how emergent misalignment happens.

Book 1: The Singapore Story

Lee Kuan Yew1 was the Prime minister of Singapore, rationalist folk hero, and one of the few world leaders to successfully become a household name despite not being primarily associated with a major war2. He was also, as it turns out, a surprisingly compelling writer. Aside from being interesting for his own sake, he also ended up implicitly producing the closest thing we have to a firsthand account of the fall of the British Empire and rise of the new world order.

His memoirs are split into two books, each covering roughly half his life. The Singapore Story covers his childhood, World War 2 and the Japanese occupation, law school, his legal career and rise in Singaporean politics and Singapore’s independence from the British Empire, joining Malaya to form Malaysia, then leaving Malaysia to finally become Prime Minister of a truly independent country. Third World to First covers his 25 years running Singapore after independence.

These books are part autobiography, part political memoir, part rationalist power fantasy. Every good rationalist, from time to time, can’t help but think Hey, if we just put me in charge of the government I would simply pass the good policies and not the bad ones, and the country would thrive. I’d just have to be clear and direct and explain to everyone why these were obviously good and reasonable ideas, they’d understand and go along with them, and our government would be popular and successful forever. All of us3 know this isn’t actually that easy, that real-life policy is hard and messy and often unreasonable and our clear bright ideas would struggle when faced with Reality. This couldn’t possibly work in real life.

Except that Lee goes ahead and does exactly that. He passes reasonable, economically-optimal policies and they work out so well that the economy booms and his party keeps winning massive supermajority elections for decades. When a race riot almost breaks out over malay recruits getting turned out of the army during high unemployment, he calmly shows up, picks up a megaphone and explains to everyone that this is all a misunderstanding, there’s no policy against employing malays in the army and they’ll all be processed and get jobs when they come back tomorrow. Everyone understands how reasonable he’s being, calm down, and sort things out the next day.

At another point he handles chinese-malay tensions by writing an epic Scott-length blog post giving an epic speech to the Malaysian parliament about how the new Malaysia needs to be less obsessed with zero-sum culture wars and focus on pragmatic economic policy and even the die-hard Malays stand up and clap4.

All of which raises the question: How did he actually do all this? Each book covers a seemingly-impossible task (the first, going from middle-class teenager with no connections to Prime Minister of your own country; the second, taking a third-world island nation with no natural resources into one of the richest countries in the world) that we might do well to learn from. The book implicitly offers one explanation - just be consistently brilliant, charismatic5, incredibly detail-oriented, honest, hardworking,and pragmatic continuously for decades while also having your share of good luck. Still, let’s take a look at the details.

Early Life

Before we get to the rationalist power fantasy story, we take a tour through a few different genres. We start with a peaceful coming of age story.

Lee was born in 1923 in what seems like a typical middle class family for colonial-era Singapore, which is an odd mix of imperial British and a mashup of local cultures. He mostly goes by Harry Lee in his earlier years (He dropped the Harry in the late 1950s, except with his British contacts6).

Some parts of it (like the school system) feel surprisingly normal. Other parts are, well, this:

“My mother was the eldest child of this union, and when she was married in 1922 at the age of 15, the fortunes of both families were still healthy. She even brought with her, as part of her dowry, a little slave girl whose duty, among other things, was to help bathe her, wash her feet and put on and take off her shoes.”

Lee doesn’t especially respect his dad, who he considers weak, lazy and unreliable7 and learns his principles from his mother and (maternal) grandfather. He goes to British-run schools, keeps getting top marks and filtering upwards through the system, and ends up at Raffles college, the college for British-educated local elites in colonial Malaya8.

The story then takes a turn to become a school drama. First, Lee needs to get top grades in college for his dream of getting the Queen’s Scholarship to go to Oxbridge, a prestigious opportunity that only one or two students a year can get. But, frustratingly, there’s this girl a year above him who keeps outcompeting him with slightly better grades.

This is specifically the type of school drama with an absurdly powerful student council. There’s some drama between the Malay and Chinese students and, as an epic debater and naturally take-charge kind of guy, Lee ends up representing the Chinese students on their side of the argument. Everyone involved in this college dispute shows up again twenty years later as some kind of senior government official or public intellectual negotiating the fate of Southeast Asia.

World War 2

Then the war breaks out, and we get a wartime story.

Lee sees the invincible British empire get stomped. The Japanese cut through Malaya in two months, crushing a British force with twice their numbers and capturing 130,000 British soldiers. The mighty white men of the British empire are helpless, hapless and cowardly, making bad decision after bad decision while completely failing to coordinate or organize a panicked retreat. By the time the Japanese get to Singapore, it falls almost without a fight.

We will get back to this momentarily, but I want to take a moment to sit and point this out. The British just lost. They were seen to lose, in a way that made them look selfish and cowardly and weak. Lee does not forget this.

Japanese occupation is a classic wartime story. The Japanese occupiers are cruel in an oddly banal, pragmatic but unfocused kind of way, occasionally kicking the locals down and occasionally ignoring them. Two interesting stories come up during this period.

First, Lee (along with most of the young Chinese men in Singapore) gets picked up by Japanese soldiers and led to a stadium. He stays there for three days, after which he’s told to leave through one gate. He gets a bad feeling about it, says something about forgetting his coat, and is allowed to leave to go back to it, managing to lose himself in the crowd again. The next day he just walks out through a different gate, and nobody bothers to stop him.

Lee was right to have a bad feeling about the first gate. The people who went through it were taken to the beach, then lined up and shot. This was the Sook Ching massacre, in which the Japanese shot tens of thousands9 of young Chinese men in cold blood across Singapore and Malaya to “suppress anti-Japanese resistance”.

Lee ended up doing a lot of business with the Japanese later as Prime Minister, asking them to come build factories in Singapore and help support the economy. He speaks highly of their organization and intelligence in the second book. That he does this after living through this story tells us something about him.

The second story Lee notes is that the Japanese set up a brothel full of their (Japanese and Korean, not local) camp followers within days of arriving. “As a result, rapes were infrequent”, he tells us. He’s impressed with their pragmatism compared to the British attitude towards the men’s needs.

Lee, meanwhile, is living a classic civilian in wartime story. Times are hard and he scrambles around doing various odd jobs to survive. At one point he starts a glue making business with the brother of Choo, the woman who kept outcompeting his grades in school. You can probably guess how that story ends.

There wasn’t any more real fighting in Singapore after the first invasion, though. The Japanese surrendered after the atomic bombs and withdrew from their empire, and the British swept back in unchallenged, coming back to rule Singapore like before. But still, things weren’t the same after that.

This is a good point to take a break and talk about the other story in this book. Both Lee’s personal rise and the later rise of Singapore happen on the backdrop (and, to some degree, as a consequence of) the fall of the British empire and its replacement with the new world order.

There’s an irony in that while Japan lost World War 2 pretty badly on the field, their actual stated goals in the war were to free Asia from western colonialism, and those goals they achieved amazingly well. They may have lost by direct force, but Lee and others like him didn’t forget how weak and disorganized and cowardly the British ruled in defeat. Western empires ruled Asia more through their aura of invincibility than through their technological advantage, and once that was lost their days were numbered and they knew it. The fall of Singapore was the single greatest military defeat the British ever had in terms of soldiers lost; it was also the battle that struck the deathblow to their Empire.

The end of the Empire was replaced with a new globalized world order that was both weaker and stronger than British rule. When the British ruled the world they cracked down on local customs we now find barbaric, like slavery or burning widows. This was rule from above and they never achieved perfect compliance (as Lee mentioned, even in 1920s Singapore there were still some forms of local slavery going on in common practice), although they did ingrain it more over time. But local cultures didn’t just revert to their old forms once the British left. Despite British rule being a distant memory, Singapore’s culture today is a lot more similar to London’s than it was in 1940 (and for that matter, so is Shanghai’s). Slavery is no longer banned with soldiers, but it has become unimaginable to practice it. We may have lost the force of the Empire, but cultural coercion lingers on in much of it10.

Law school and politics

Lee travels to England after the war to go to law school11, which immediately leads him to despise the English and develop a burning desire to overthrow them12.

He starts off at the LSE, where many of the professors are some form of marxist or communist. He notes at this point that quite a few of his fellow students also ended up running their own third world states; the ones who tried to run their countries based on what they thought the marxists taught inevitably ran their economies to the ground13. He notes that the communist students’ main recruitment strategy is to send beautiful girls to recruit men to their clubs (this tactic comes up again later, when it’s used by the communists in Singapore14). Eventually he transfers to Cambridge, aces his exams, and heads back to Singapore ready to start a successful law career with some clear plans to get into politics15.

It’s noteworthy that Lee’s political drive starts in anticolonialism, which ties him to the Labour party, despite his eventual turn to more conservative/Thatcherite economic policy. For Americans, it’s worth noting here that traditionally the British right is more based on classist conservatism than the Reaganite free-markets-plus-church approach.

Meanwhile the British Empire is bankrupt, anticolonialism is on the rise and it’s increasingly clear to everyone that long-term colonial rule is on its way out. Lee believes (as young people often do16) that the older generations are too passive and whipped by the British and his generation needs to take a leadership role in achieving self rule, and comes back to Singapore intent on making it happen.

There’s a needle to thread here: The left is a broad alliance of everything anticolonialist from English-speaking moderates to labour unions and communist high schoolers (there’s also an actively illegal communist guerilla group in Singapore and Malaya, which the fringes of the legitimate communist political groups are connected to). Lee makes his name as a firebrand lawyer for all these causes (including fighting for unions and student groups), which leads to him successfully founding the PAP as an alliance of all the disparate anticolonial groups and eventually winning an election as their head. Once they’re in power, the communists (who hold most of the grassroots support) break off, and Lee has to make desperate efforts for the remaining moderate wing of the PAP to survive and hold onto a bare majority17.

This is probably the most important part of Lee’s personal rise and the most historically critical part, and it takes up a big chunk of the first book. It’s also boring, because it turns out Lee is really good at the boring hard work of politics and happy to talk about his craft. And I cannot emphasise this enough, the hard work of politics is dull. Lee spends pages talking about every specific person and deal he has to make to put the party together and keep its appeal, about finding good candidates for every district and negotiating with individual ministers and candidates to keep them on board. He talks about spending untold hours learning Chinese18 and going around giving speeches on his megaphone four times a night. He lists a million names of important people that I personally can’t hope to keep track of. When he gets like this (it gets even worse when he talks about geopolitical negotiations, especially because he had way more notes to work off of for those) it’s like reading an engineering documentary by a guy who insists on telling you the exact version of Kafka they used for every messaging layer when building google cloud19.

Seriously, don’t go into politics. It’s not worth it.

But it was for Lee, whose talent for remembering details and managing people and public opinion leads him to survive the crisis and keep a bare PAP majority of sensible moderates after the communists break off. Meanwhile there’s plenty of communist-tinged civil unrest of various kinds, between the labour unions and the Chinese high schoolers, but Lee (unlike his predecessors) manages to crack down on it hard enough to calm things down without just provoking more rioting and unrest20. Lee’s other political project is to push for the merger with Malaya to create Malaysia, since he believes that Singapore is just too small to have true independence on its own.

The rest of the first book is often fun, often boring, but not too illuminating. Lee spends years pushing for the merger with Malaysia, gets it passed, then spends the rest of the book trying to get Singapore fairly treated in Malaysia21. The majority Malay-first sectarian parties remain unconvinced and, seeing the PAP winning seats even in majority-malay parts of Singapore, a little nervous. Lee tries to explicitly make the case for a non-sectarian “Malaysian Malaysia”, and while his speech is applauded by all present22, the Malaysian government eventually decides this is all moving too fast and they need to break up. The Tunku (the Malaysian ruler) ends up sending Lee a signed copy of his book with the dedication

“To the friend who had worked so hard to found Malaysia and even harder to break it up”.

And so the first book ends with Singapore breaking off from both the British and Malaysia, and Lee suddenly in charge of an independent city-state country he’s not sure can be sustainable. The second book, like the second act of Hamilton, has to deal with the question of how do you actually handle independence, once it has been achieved?

Book 2: Third World to First

“Some countries are born independent. Some achieve independence. Singapore had independence thrust upon it.”

This one’s organized by topic rather than chronologically, which is probably easier for a book review, and I’m going to pick and choose which topics I dig into here.

Military, Economy and legibility

We now suddenly have a small independent city-state surrounded by big, powerful, not-necessarily peaceful countries23. The British were pulling out from everywhere east of the Suez and leaving a potential power vacuum. Lee, acutely aware of all this, does the obvious thing and asks for advice from another tiny former British colony composed of a majority group of middleman-minority exiles surrounded by massive hostile muslim nations that successfully survived repeated crises24 to help train up Singapore armed forces. Despite the limited budget and economic crunch, they desperately built up the Singapore military, seeing this as existential.

They also start Singapore’s public housing program, which sells people heavily government-subsidised apartments. The point here isn’t to solve the housing crisis (Lee understands this is economically suboptimal); it’s to give people without a history of a strong national identity something to fight for if they get invaded.

This desperate feeling of precariousness and seeing Singapore as constantly at risk of collapse is a key theme through the book. I’ve heard people say that Singapore is cool and futuristic and experimental, and that it’s probably the best place in the world for e.g. wacky biotech experiments. This is fundamentally misunderstanding the nature of Singapore’s weirdness. They’re not high-openness, they’re just willing to go to extremes to avoid instability25. Complicated housing experiments to reduce sectarianism by forcing every neighborhood to be racially mixed? Absolutely. Crazy off-label biohacking labs? Absolutely not26.

The other crisis caused by British withdrawal is economic. A huge part of Singapore’s economy was based on serving as a British naval base, and those jobs are going away. So emergency economic measures are in order.

Lee got some lucky breaks here. The Vietnam war helped for a while as the Americans needed Singapore as a logistics hub to Southeast Asia27. The classic economic advice for developing countries at the time was to start with import substitution, but since Singapore was far too small a market for real domestic import substitution they jumped to working on export support; current economic conventional wisdom is that this is a better path for developing economies and the other developing economies got it wrong.

But mostly it’s just hard work and determination to get results. Unlike the more ideological postcolonial countries, Singapore actively sought out western investment wherever possible, working to make it regulatorily easy and convenient (including Lee himself going around to talk to various foreign CEOs). They kept the association to Raffles as the founder of Singapore instead of cancelling him. They created ports and industrial zones for companies to work at instead of passing datacenter bans. A large part of growing the economy is just wanting to grow the economy (along with the belief that it can be done if you try), and they had that in spades.

This is also probably the place to mention the famous quote Lee’s economic advisor gave him about unions

“… as an economist, I’m not interested in what you do with them. You can throw them in jail, you can throw them out of the country, you can even kill them. As an economist it does not interest me, but I have to tell you, if you don’t eliminate them in government, in unions, in the streets, forget about economic development.”

Lee didn’t literally kill all the unions. But he did manage to weaken them where they would have impeded economic growth.

The third component here is legibility. James Scott may have written seeing like a state as a warning, but Lee took it as an instruction manual. He pushed to tear down winding roads and build an orderly street grid. He pushed everyone to speak one of the four official languages of English, Mandarin, Malay and Tamil (if four official languages sound like a lot, just think how many they were replacing; every Chinese subgroup had their own dialect, just for a start) and pushed for all education to be in English. The massive government housing projects weren’t just to support willingness to bear arms and to force mixed neighborhoods to attack sectarianism, they also forced people out of their messy disorganized homes with farm animals in the backyard into organized clean ones. He couldn’t just eliminate street food salesmen going all over the place, so they built up Singapore’s famous hawker centers to channel them into legible addresses.

Similarly, anticorruption was a whole campaign. It’s not enough for the PAP ministers to run on anti corruption (and, somewhat surprisingly, mostly actually follow through with it), they have to crack down on it throughout the system. You can’t do this overnight (if everyone’s doing it you can’t throw the entire bureaucracy in jail), but they did successfully frog boil it over time. It also involved just reducing the needs for permits and regulations wherever possible, which provide opportunities for corruption you can just not have. Sometimes straightforward libertarian arguments just work.

There’s also the massive parks/greening projects. Unlike some anime power fantasy protagonists Lee didn’t have the ability to control the weather - the constant heat and humidity make Singapore’s parks painful to actually hike through - but they are beautiful, and this was a result of a lot of hard work and introducing plants from all over the world.

Listing every world leader he ever talked to

Something like half the book is Lee listing every world leader he ever talked to as PM. This is another part that mostly gets kind of dull (especially if you don’t follow the details, although this case is much easier to track than the internal politics chapters since you’ll probably recognize the characters’ names, or at least their countries). They’re all described as “tough” and “vigorous”, occasionally as “warm”, and usually with some other positive descriptors. This is probably a mix of sincere respect and diplomatic language for most of them.

Except Jimmy Carter. Lee spends an entire chapter making fun of Jimmy Carter for being shallow, undermined, weak-willed and inauthentic. And then takes an aside in the next chapter, when he’s complimenting Reagan, to mention how smart it was of the American people to replace Jimmy Carter with this guy. If you ever want to see someone who’s not a giant swimming rabbit attacking Jimmy Carter, this is the book for you28.

The other memorable one is Deng Xiaoping. You know how sometimes you ship characters from different franchises, or people from different parts of your life, as friends? I always had this for those two, they just seemed so compatible. I didn’t realize until I read this that they actually were friends in real life. Or, well, as close to genuine friends as world leaders ever get. Lee’s compliments about Deng are genuine in a way they aren’t about anyone else. He notes that when he pointed out to Deng that one of his policies was a mistake, Deng actually listened and changed course despite being over seventy at the time.

This makes sense. If you look at the history of economic policy, Deng might be the only guy in history who was even more brutally pragmatic in the face of all odds than Lee was. Of course he would admire him29.

The theme of the background decline of the British Empire shows up again here. If Lee got his political start under the British, his last major geopolitical relationship was with Deng, leading the new rising power displacing the British. He spends some time talking about how foolish the British were to try to impose conditions on the surrender of Hong Kong; they just weren’t enough of a world power to make those kinds of demands anymore.

The HBD chapter

Lee seems to have been decades ahead of the curve on both DEI and HBD30.

On DEI, he realized early on that the Malay minority were underperforming in school compared to the Chinese kids. This is a crisis (remember, sectarianism is one of the two things that kills developing countries), and he launches a range of DEI programs to try to support Malay and Indian students (and also reserve them some government posts).

This helps somewhat, but doesn’t fully close the gap. Lee shrugged, figured that this is just hard-to-fully-overcome subgroup IQ differences, and did what he could to ameliorate it. A decade later Charles Murray released The Bell Curve, and Lee mentions feeling vindicated on reading it. He’s generally pretty harsh on people trying to pretend there are no cultural or intelligence gaps between groups, believing that this kind of self-imposed refusal to face facts is futile and counterproductive31.

Lee comes back to this theme quite a bit. If he’s pragmatic enough to work with the Japanese after surviving Sook Ching, he definitely doesn’t have the patience for starry-eyed academics who refuse to face facts. He talks later about finding Harvard disappointing, and about learning to rely on people who actually made things that worked in the real world and to ignore academic theorists, who are too likely to be motivated by blinding idealism.

He also worried about the fertility crisis several decades before anyone else, both in that people weren’t having enough kids and specifically that Singaporean Chinese men didn’t want to marry women who were as smart or smarter than them, preferring to marry down to a woman who would look up to them. This is the one area where he ends up admitting total failure - despite Singapore’s heavy child subsidizing policies and his intense PR push, he couldn’t seem to get any real traction32.

The “Am I a dictator” chapter

“I stressed that freedom could only exist in an orderly state, not when there was continuous contention or anarchy. In Eastern societies, the main objective is to have a well-ordered society so that everyone can enjoy their freedom to the maximum. Parts of contemporary American society were totally unacceptable to Asians because they represented a breakdown of civil society with guns, drugs, violent crime, vagrancy and vulgar public behavior”

Many people over the years have accused Lee of being a dictator and Singapore being an authoritarian dictatorship. Some men might try to avoid the issue; Lee explicitly brings it up, spends a whole chapter going through every time he’s been accused of acting dictatorial, and explains why actually he was just doing the right thing. Seriously, this guy loves to argue33.

The first thing he argues about is suing journalists for libel. Journalists would argue that anyone who ever tries to limit them in any way is acting dictatorial. Lee argues that he’s only suing them for falsehoods, and legitimate criticism should be defended in courts.

There’s a few points to note here. First, you might question whether courts are going to be entirely fair when Lee’s in charge of the government that appointed them in the first place. Surprisingly, I don’t think that’s necessarily the biggest issue - for one thing, for most of this period people could appeal Singapore cases to a judicial committee in London, which didn’t answer to the Singapore government and limited the impact here. For another, Lee clearly relishes the confrontation; he comes off as a guy who wants to win a fair fight because he’s good at it, not someone beating up a weak opponent. It’s possible this too is a cover, but if he was going to cover muddying the issue and pulling attention away would probably have worked better.

The other thing to worry about here is process as punishment and indirect coercion - even if reporters who could defend their criticism would eventually win their libel suits, journalists would hold that it makes their lives unpleasant and deters legitimate criticism, and that politicians should just sit down and take bad press, arguing before the public when they believe it’s false.

Lee argues that it’s wrong for journalists to expect this, and they need to be held accountable for honesty too. I suspect part of this is just that Lee really loves arguing34.

The dictator-ish behavior I personally found most troubling was the story about prioritizing areas that voted for the PAP by larger margins for housing renovations (remember, most of Singapore’s housing is public and managed by the government). This doesn’t technically make him a dictator, but it’s certainly a pile of bad incentives right there.

The question of Singapore being authoritarian more generally is… complicated. Singapore famously banned gum, because the government thought it was net negative for society to have it. They also had an anti-long hair campaign in the seventies, to discourage hippies. This kind of idea - that if something is bad, the government should ban it - is unimaginable in the American context, where we’re more likely to ask the meta question of whether the government should have the power to do something at all. Lee defends the power of administrative detention without trial on similar grounds - the government uses it well, and if it doesn’t, people will vote for the government to lose that power.

This raises the biggest question about authoritarianism: Where, ultimately, does the government rest its claim to legitimacy?

Lee’s PAP very strongly relies on popular support. They may put a thumb on the scales in some ways (like the housing thing), but they care intensely about popular support35 as the ultimate source of their legitimacy. Lee drives the party to cover a broad political consensus and deliver prosperity; he repeatedly claims that if they fail, they won’t last as the country’s leaders (although he doesn’t specify a mechanism by which this might happen, and seems insufficiently worried by the prospect of a collapse that would take Singapore’s economy down with it).

The biggest risk of decline under authoritarian rule is a less-competent successor inheriting a powerful apparatus and then making mistakes with it, once the founder figure is no longer around to help guide it. In mainland China, for example, it’s worth noting that Jian Zemin and Hu Jintao were both handpicked as successors by Deng. Once they were replaced by Xi, the government started trailing in a different direction.

In Singapore, Lee was replaced by his handpicked successor Goh Chok Tong, and then by his son Loong36. Lawrence Wong, who started his term in 2024, is the first leader Singapore’s had who wasn’t directly chosen by Lee. I guess we’ll have to wait and see how he does (things seem to be going okay so far; it makes sense that replacing a rationalist folk hero with LW would be minimally disruptive).

So what did we learn about Lee?

In some ways, the best historical analogs to Lee are people like Elon Musk and Julius Caesar: Men who went ahead with bold plans, ignored the critics where they were wrong or baselessly skeptical, and made impossible-seeming things work because they were smart enough and correct enough to pull it off.

But the most interesting point of comparison is Thomas Jefferson, because they’re just so different.

When you read America’s founding documents, like the Federalist Papers or the Declaration of Independence, the thing that jumps out is how great the writing is. It’s hard to imagine a modern politician writing like that. It’s hard to imagine a modern politician being smart enough to write at that quality level.

Lee, despite being from the 20th century, actually was that level of smart. And he was direct enough to just write what he really feels, and articulate enough to do it well. So how does he compare?

The first obvious thing is the writing style. Jefferson and Hamilton wrote with bold flourish, with grand ideology and conviction. Lee writes clearly but technically, explaining why his ideas are good and will solve problems. An American founding father might have given grand speeches about his dreams of a free Singapore of equals without racism or outsider control; Lee talked about how racism and outsider controls were barriers to freedom and prosperity and sound economic policy. He’s direct and technical, not a dreamer.

He also doesn’t view government the way Jefferson did. Jefferson wrote about the proper role of government and rights of man; Lee thinks about what the government should be doing on an object level, and does it. He trusts that they will be rewarded with power as they deserve.

Jefferson was writing the first declaration of independence from the British Empire; Lee executed one of the last, against the backdrop of its final decline. When Jefferson dreamed of independence, it was an experiment no one had ever done before. By the time Lee came around it had been done many times; he did his best to learn how to achieve it from the best who had already done it, to perfect the execution like he did with economic and housing and transportation policy. The world got smaller over the two centuries between Jefferson and Lee’s independence declarations, between the rise and the fall of the British Empire. It changed modes from Explore to Exploit.

Jefferson is famous for having lofty ideals he couldn’t live up to, and was tarnished by; he owned slaves and famously had some uncomfortable relations with them even as he wrote about Freedom. Lee had more modest ideals, but he successfully lived up to the ones he did.

Short version! If you’re willing to organizer an ACX Everywhere meetup for your city, please fill out the organizer form by October 9th.

The plan is:

• Interested organizers fill out the form by October 9th

• The list of meetups gets published around October 10th

• The meetups themselves take place anywhere between October 16th and November 30th

• People have more meetups and have more fun throughout the year.

The form will ask you for a time, date, location, and a way to contact you. It will ask other things too like how excited you are to run a meetup and what kind of meetups you like running.

I usually say organizing an ACX Everywhere meetup can be pretty easy. Open socials like these are the easiest kinds of meetups to run since you don’t need to prepare any kind of activity or discussion questions, and you don’t need to work hard advertising it since Scott and I will be doing a lot of that for you.

What hasn’t been said in one of these posts is it can be pretty rewarding as well. Obviously I’m the kind of person who winds up with a title like “ACX Meetup Czar” so you can guess I like meetups more than the average person, but most regular organizers report enjoying the experience. You get to meet new people who are into the same obscure blog widely read publication as you are. There’s the appreciation of your peers to consider; I’m an ex-theatre kid and Boston has taken to applauding the organizers at the midpoint of ACX Everywhere. Plus there’s the convenience of the meetup happening at a time that works for you and a place you know you can get to.

Also, while running one ACX Everywhere meetup does not mean you’re signing up to be a meetup organizer forever, if you do choose to keep running events you may find that the group mysteriously ends up mostly full of your kind of people, even more than it being an ACX group would lead you to expect. Founder effects are impressive.

Q. What do I need in order to be an okay meetup organizer?

A. You need to be able to pick a time and a date and a local cafe or park, write “ACX” on a piece of paper with a sharpie, and show up. The sign is honestly optional, it’s just recommended. There are ways to screw this up but if you’re the kind of person who’s anxious about making everyone uncomfortable, I don’t think you’re going to do that. I am a rationalist, I pride myself on calibration, and without having met you I’m 99% confident you’re going to do at least okay.

Q. What do I need in order to be a great meetup organizer?

A. Lots of things can improve a meetup, none of which are necessary. Bring name tags and a sharpie to help folks feel less awkward about forgetting names. Make sure there’s water, either because you’re at a restaurant or because there’s a water fountain or because you brought a case or two of plastic water bottles. Snacks can also be great. (I can reimburse the water bottles and name tags and snacks, though I’m very slow at it this year.) Be warm and friendly with newcomers, learn how long they’ve been reading ACX or what their hobbies are. Have a way for attendees to join a mailing list or whatsapp group or the like where you or someone else can announce future meetups in your city. This list is not exhaustive. I have more advice if you want it.

Q. When should the meetup happen?

A. The meetup should take place sometime between October 16th and November 30th.

Q. Where should the meetup happen?

A. The best venues are

1. easy to get to

2. not so loud that people have trouble hearing

3. have things like places to sit and access to water or toilets, and

4. make it easy to move from conversation to conversation.

Public parks are great, especially if they’re close to a cafe people can duck into if they want refreshments or if it starts raining. Some parks have water fountains and pavilions to keep out the rain. Mall food courts or common areas work pretty well. Apartments or houses can be great, but it does involve posting your address on the internet. Pubs or bars can work, but tend to be worse for college students or parents with children.

Q. What should happen at the meetup?

A. Basically just show up and talk. ACX readers tend to be interesting people. If you’re worried about this, a few suggestions are:

• Hand out nametags, and ask people to write a topic they want to talk about on the nametag.

• Get a few books and leave them lying around with “Have you read me?” written on a sticky note.

• Say hello to newcomers as they arrive, introduce yourself, and ask them something about themselves.

For ACX Everywhere, I’d warn against trying to make everyone do some mandatory activity like watching a PowerPoint presentation. That said, it’s fine to give people the option to do something other than unstructured socializing. I sometimes leave a few games lying on a table.

Q. What if there’s already an existing ACX group near me?

A. You may want to volunteer anyway.

I keep a list of where people have volunteered already. You can look at this preliminary list here. Most ACX Everywhere seasons there’s at least one active meetup group that fails to get on the big list, because everyone thought someone else was going to volunteer to run it. The funniest example of this is Berkeley California, unofficial rationalist capital of the world, which went three years without anyone from California volunteering to run ACX Everywhere there and thus requiring a Bostonian to step in and run their meetup for them. Seattle on the other hand routinely has three to five volunteers and is in no danger of failing to have a meetup. If you are reading this, and your group is not on that preliminary list yet, then you have an opportunity to beat Berkeley at the coveted “don’t fall prey to the bystander effect” metric.

Usually what happens when multiple people volunteer from the same city is I put them in touch with each other, give them both the details of the meetups they proposed, make a light recommendation, and then let them decide and get back to me.

Q. Is that all?

A. Pretty much!

Some of you may be disappointed that’s all.

I said above that the ACX Everywhere meetups are intended to be just social hangouts. They shouldn’t have an agenda, and there’s not any concrete output we expect other than a good time. Maybe you like agendas and memo writing, and perhaps you even wish you could argue about politics more than is typically encouraged for an ACX Everywhere. The EA University meetups weren’t enough, and neither are the ACX Everywhere Meetups. You’re still hungry for more. You must construct additional pylons meetups.

If so, this is your lucky year, because we’re also doing ACX Ballots Everywhere.

Scott historically shares how he plans to vote in his local elections. Since 2024, he’s been interested in scaling up the voter guides for all the big US cities and states. Usually I stay fairly hands off for these, since 1. I hate politics with the passion of a thousand sun miracles, and 2. I historically operate on grants that require me not to endorse any political candidate while doing my ACX meetup work. It’s been argued that providing neutral infrastructure for other people to endorse candidates would be fine. I still loath politics.

However, nobody’s given me any grants this year, plus the fellow who ran my first rationalist meetup is moving away and I wanted to give them a present. I’m going to miss having you around Taymon. Thanks for the megameetup.

If you want to run a Ballots Everywhere meetup, fill out this form by October 4th.

And again, the plan is:

• Fill out the form by October 4th

• The meetups get posted around October 5th

• The meetups take place sometime between October 8th and October 19th

• The guides get posted around October 24th

The Ballots Everywhere meetups do have an agenda and a specific deliverable. You need to figure out what races you’re discussing, discuss the options, come to some kind of conclusion, and write up your endorsements to share with Scott. Any group in the USA is welcome. If you’re not in the USA, but you’re still a democracy with elections around the right time, feel free to fill out the form but I don’t have a plan for doing anything different for you than I’m doing for the US meetups.

If it helps, you’re welcome to use Boston’s recent procedure to help run the meetup. It’s been Rigorously Tested, and by that I mean we used it in the current form exactly twice, it basically worked, we tried two very different versions before this year and this went better. Or you can come up with your own way to run the meetup. If it helps, take a look at the output from the last time we did these.

I will say that Boston got surprisingly positive feedback, both from the attendees and from the politicians. For attendance count, we got on the upper end of what I’d expect for a weekend afternoon meetup. For the politician side, a local campaign says people have been mentioning the endorsement. I don’t know how much good will come of this, but if you’ve been staring at politics lately wishing there was a bit more you could do, well, here’s something a bit more.

Plus you get to research and argue about politics for a couple hours. I know ACX meetup attendees. Some of you are grinning at that prospect.

These are going to run more or less like the ACX Everywhere meetups, except that 1. I can’t reimburse your supplies, and 2. you probably want a location where people can comfortably use laptops. Again, if you want to copy the Boston procedures, go ahead, but you can also do your own thing. Indulge!

Speaking of indulgence, what’s that you say? You want more meetups?

Insatiable. I love it.

On September 26th, 1983, near the height of the cold war, Stanislav Petrov was an lieutenant colonel for the USSR, stationed at a nuclear early-warning system. He was the duty officer when the system reported a US missile launch. The Soviet Union’s policy at the time was to launch missiles back if they received a warning. Petrov claimed, without being sure, that the warning was a glitch in the system and not a real inbound missile. He turned out to be right.

The story of Petrov is part of the Sequences, and there’s a tradition in the rationalist community to celebrate a man who averted a nuclear exchange and catastrophic loss of human life. It’s one of the easier rationalist rituals to perform, requiring only a few printed pamphlets, a small pad of index cards and pencils, and some candles.

Petrov day is in a little under a week day or two. If you’d like to run or participate, I suggest making plans and announcing it soon, perhaps today. You don’t need to route through ACX for these, just go for it. If you want advice, feel free to ask in the comments.

In summary:

• EA University meetups are underway, and you can see the list of those meetups here.

• If you want to run an ACX Everywhere meetup, please fill out this form by October 9th.

• If you want to run a Ballots Everywhere meetup, please fill out this other form by October 4th.

• If you want to run a Petrov Day ceremony, you can just do things. It is traditionally celebrated on September 26th, so you should make plans and announce it soon. There’s a guide here. You may want to announce it over on LessWrong, or wherever local rationalist events are announced for your city.

• Have you considered that AI Pause Protests are kind of like a meetup if you squi- I am forcibly dragged away from the post editor.

In the public debates after the release of AI 2027, some people accused us of misleading readers by positing “neuralese recurrence”, a hypothetical technology that would allow AIs to think dangerous thoughts without getting detected.

You’ll never guess what happened earlier this month!

But what is recurrence? Is this exactly the same as the neuralese recurrence in AI 2027? And how worrying is it?

The transformer, the technology behind most modern AI, contains some number of layers (modern frontier models are probably around 100). By convention, the input layer is called the “bottom” and the output layer the “top”. When a transformer does next-word prediction, it takes the last word in at the bottom layer, spends the middle layers processing it, and outputs the predicted next word at the “top”.

What if it’s working on a hard question that needs more than 100 layers of processing? From 2017 - 2024, the answer was “you’re screwed”. From 2024 onward, the answer was: it outputs its intermediate result, as text, onto a scratchpad called the “chain-of-thought”. Then it runs the transformer again on the intermediate result. Then it repeats until it has a final result.

(Why does it need a scratchpad, instead of feeding its intermediate results directly into user output and then reading those back into itself? First, the intermediate results may be hundreds of pages long, and the user doesn’t want to read those. Second, the AI companies want to train the AIs to do the intermediate thinking properly, and this benefits from a ‘psychological’ distinction between intermediate pondering and final user output.)

This was purely a capabilities play - models are smarter when they can keep thinking instead of limiting themselves to 100 processing steps - but it coincidentally was very good for safety. The chain-of-thought scratchpad is written in English (although some Chinese models use an English-Chinese hybrid, and other AIs develop their own weird jargon). You can just read what the AI is thinking! If the AI is thinking “Better hack some websites, then kill all humans”, you can shut it down. Maybe not actually - if you have thousands of AIs writing millions of pages of scratchpad, you can’t read all of that in real-time, and will need to delegate the task to fallible AI monitors. But in theory this ought to work.

But language is slower and shallower than thought. For 100 steps in a row, the AI can shoot delicate subtle ideas from layer to layer at the speed of light. Then there’s one step where it has to encode them into twenty-six glyphs invented by Phoenician turquoise miners in 1800 BC. Then it has to re-encode the Phoenician glyphs into delicate subtle lightspeed ideas before it can do anything else. This has long been acknowledged as a bottleneck in existing transformers. So: what if they could finish their 100 layers of thinking, then send the resulting thought back to the first layer for more processing, rather than sending a text scratchpad on which they had written the thought?

This is neuralese recurrence - “recurrence” because it’s going back in a loop, “neuralese” because the thing that’s looping is the thought itself, in the native language of thought, rather than words.

The AI’s “language of thought” looks like a vector, thousands of numbers long. An “intermediate result” in this scheme might look something like (0.4, 0.1, 5, 0.443, … and so on for thousands of numbers). We don’t know how to read these. The science of reading these thoughts is a subfield of AI interpretability, which is still in its infancy. If an AI with neuralese recurrence were to think “Better hack some websites, then kill all humans”, it would look like (0.4, 0.1, 5, 0.443, … and so on for thousands of numbers), and we would never find out. This is why AI 2027 discussed it as a prelude to AIs that could escape human monitoring.

II. Pachocki Contra Specter

So did OpenAI achieve this? Their chief scientist, Jakub Pachocki, says “not really”:

What does Pachocki mean by “the depth of the computation graph . . . is within a factor of two of GPT-4”? And should this resolve alignment concerns?

Our example transformer has four green layers: a layer depth of four:

Now we add a little recurrence: instead of layer 3 going to layer 4, it loops back to layer 2, and the data go through this loop three times, then finally progress to layer 4:

Here the data are getting eight layers of processing: 1 → 2 → 3 → 2 → 3 → 2 → 3 → 4. So this recurrent transformer has a depth of eight.

In this sense, “recurrence”, despite sounding scary, is no worse than just building an eight-layer transformer to begin with, which was always an option.

That is, from an alignment perspective, suppose it takes 1,000 layers’ worth of processing to come up with an evil plot. A normal AI can think for 100 layers, but then it has to record its output on the chain-of-thought - something like “Here’s one-tenth of the evil plot I’m trying to hatch” - and we can catch it. So we want to avoid letting the model compute for 1,000 layers straight without using the scratchpad. This means don’t create a recurrent model that can loop 100 layers 10x, but it also means don’t build a 1,000 layer model to begin with.

So when Pachocki says that Astra only has twice the depth of GPT-4, he’s saying something like: look, guys, we’ve been adding layers to our AIs for years. GPT-2 had 48 layers, GPT-3 had 96, and we didn’t tell you how many GPT-4 had but let’s say it was 120. At none of these points did you complain, because it wasn’t “recurrence”, just a normal natural 120 layers. Now Astra is - let’s say - 180 layers. It’s true that we got these extra layers by recurrence rather than literally building a bigger transformer, but the alignment implications are no different. For all you know, Anthropic built a literal 180 layer transformer yesterday, and you guys didn’t bother them1. The important thing is that we not increase our layer number by some crazy amount, and we didn’t. We’re just adding layers the same way lots of other AI architectures did, albeit by other means.

How convincing is this argument?

III. God Help Us, Let’s Try To Understand Transformer Layers

When reading this, my question was - why don’t you just loop all the layers one million times, and get an AI with a million times the computational depth of the shmuck who didn’t do that? Isn’t this free capabilities?

A looped transformer model like Astra uses loops as a hack to add more layers. This hack isn’t as good as adding more layers for real. A real new layer gives the AI more “room” to store knowledge and thought styles. A looped layer just lets the AI use the same knowledge and thought style more times.

This is far from useless. A mathematician might spend years contemplating the same problem before getting it right; all through this period, he is using the same type of thinking (the mathematical knowledge and techniques his brain is capable of). But sometimes doing it for a year is better than doing it for a minute.

But it’s not optimal either. An AI with more real layers can actually be “smarter” in the conventional sense of the term. Even though it might help to give the same mathematician more time to think about the problem, holding amount of time constant, it will help even more to give the problem to a smarter mathematician with more training.

I asked Fable to estimate the relative capabilities gain, measured in “generations” (eg GPT-4 to GPT-5) from three things:

• Using chain-of-thought (baseline)

• Doubling the “simulated” number of layers by a recurrent loop.

• Doubling the “real” number of layers.

Its answers were 0, +0.03, and +0.252, with wide variation depending on the type of task. Its headline result was that the gains from extra simulated layers are much smaller than the gains from extra real layers.

But why are we merely doubling the number of layers? Once you can loop layers at all, why not loop them a million times? Pachocki says he’s replacing something like 120 layers → chain-of-thought scratchpad → another 120 layers with something more like 240 layers → chain-of-thought scratchpad → another 240 layers, but why doesn’t he just loop the AI endlessly until it comes up with a final result?

Here is a diagram of a transformer process (source):

On each token (for example, the word “so”) the transformer looks at its existing context and runs a forward pass (the gray up arrows) through some number of layers. Then it finishes its forward pass, emits a token of chain-of-thought (the curvy blue arrows), then starts another forward pass on the new context (the old context plus the new token).

Adding more layers lets the AI think more between chain-of-thought tokens, improving the quality of each new token. But it uses extra compute. If the user is on a budget, the more compute that the AI spends on each chain-of-thought token, the fewer chain-of-thought tokens it can emit. Holding compute fixed, there’s some optimal balance between the quality and quantity of chain-of-thought tokens, and OpenAI must have decided it lay at looping 2-4x but not 1000x.

But why use chain-of-thought at all? Why not just loop unboundedly, until the problem is complete?

(on the diagram above, that would look like replacing the English-reasoning blue arrows with neuralese-reasoning gray arrows)

This would work if the AI could be trained to do it, but currently it can’t. Transformers are pre-trained on human text. They know how to use chain-of-thought partly from reading real humans’ chains-of-thought: “Hmmmm, this is a hard problem, but it looks sort of like something I’ve solved with differential equations before. Maybe if I plug in a differential equation there . . . no, that would be too inelegant . . wait, what if -”, and partly from reading enough other human output that this kind of human-style thinking comes naturally. But there is no convenient Internet text about human reasoning entirely in vectors, so the AIs can’t be trained to do it properly. In principle this isn’t fatal – all AI reasoning is in vectors, and the whole point of training is to teach them to do it well – but training an entire vector-based thought process for thousands of steps without ever bottoming out in a human-imitating chain-of-thought-token intermediate is prohibitively costly. The researchers who design transformers have to make guesses about what chain-of-thought frequency will maximize capabilities while minimizing training cost. This is part of what Pachocki was denying. AI 2027’s “neuralese” is the thing researchers currently don’t know how to do3 – train an AI to loop as often as it wants, avoiding English chain-of-thought entirely. Pachocki was saying: don’t worry, we still don’t know how to do that.

But now we can see that this is not entirely reassuring. If it takes an AI 1,000 unmonitored gray-arrow steps to devise a dangerous plot, it can get those steps either by replacing all the blue arrows with gray arrows (in which case all its steps are unmonitored, and it can plot as much as it wants), or by having so many layers that there are 1,000 steps in a single vertical forward pass in between blue arrows4. It would be prohibitive to make that many real layers, but it’s possible by looping. So (the safety community argues) even Pachocki’s looped transformer architecture is a step along that path.

All of this is starting to get confusing, but the takeaway is:

1. An AI with a limited number of layers can’t plot at all without having to record the plot on its chain-of-thought.

2. An AI that uses loops to simulate a large number of layers (“looped transformer”) can make a plot within a single forward pass, in between chain-of-thought steps.

3. An AI that no longer uses chain-of-thought at all (“true neuralese”) can plot at leisure, considering it for as many episodes as it needs, and we will never know.

Every previous transformer has been (1). Astra is getting part of the way to (2). The dangerous AIs in AI 2027 are (3), but so far nobody has invented this in real life.

IV. A Psychological Barrier, Not A Technical One

Having finished that digression, let’s return to the original question. Pachocki says that Astra’s recurrence barely gives it any more layers than the competition. Does that exonerate him from the charge of creating dangerous recurrent AIs?

Here the best thing I’ve read is Linchuan Zhang’s Categorical Taboos Are Much Better Than Threshold Taboos: Neuralese Edition.

Linch says: there’s an emerging taboo on creating the dangerous sort of neuralese AIs seen in AI 2027. Everyone in this discussion agrees that the taboo is correct. The only question is whether OpenAI violated that taboo.

But taboos only work when there are clear boundaries defining what is versus isn’t within the tabooed area. For example, it’s illegal in America to buy alcohol before age 21. Suppose that someone buys alcohol two days before their twenty-first birthday. Should the police arrest them? Obviously this age gap doesn’t really make a difference. We’re using age as a proxy for something like maturity, and there’s so much variation in maturity that there’s no real difference between a 20.99 year old and a 21 year old. Still, although the police might sometimes choose to overlook this, we have to at least maintain the fiction that this is an arrestable offense. Why? Suppose that we made a specific boundary - if you’re above 20.8 years old, we won’t arrest you. But then we’ve just decreased our bright line from 21 to 20.8. And we could encounter the same problem: a 20.799-year-old buys alcohol and the police have to decide whether to make an exception or not. If we’re going to draw the line somewhere, it might as well be at the place we’d already promised to draw it, told everybody that we’d drawn it, etc.

(Linch’s own example is nuclear weapons. There’s a taboo on using nukes in war. If someone uses a tiny nuke, which produces an explosion no bigger than a conventional bomb, then in some sense this doesn’t matter, because it’s no worse than a conventional bomb would have been. But in another sense, it matters a lot, because you’ve broken the taboo, and now there’s only a weaker, fuzzier taboo preventing you from using a very big nuke.)

So if we want a taboo on recurrence, there needs to be some specific taboo. When Pachocki says “Yes, we did recurrence, but it only added a couple of layers, so it doesn’t really matter,” this is analogous to the case where a drinker tells the police “Yes, I’m below 21, but only by a few months, so it doesn’t really matter”, or a despot tells the UN “Yes, I used a nuke, but it was very small.” It’s true that it doesn’t matter in real life, but it matters a lot for whether you can maintain a taboo or not.

Unfortunately, we currently lack agreement on what, if any, taboo exists. Some people argue that there should have been a taboo on looping layers (which OpenAI would have broken), and other people say that they’re making that up and that was never a taboo. There’s probably an informal taboo on true neuralese with no chain of thought at all, but it’s a complicated technology with multiple moving parts, there’s no agreement as to which moving part the taboo is on, and without that agreement, some of the parts might slip through the cracks. So what are our options?

First, contra Pachocki, it might be worth instituting a taboo on looping (adding “simulated layers”) at all, even without any taboo on adding more “real layers”. Adding real layers is expensive and generally not worth it; AI companies have only chosen to increase real depth of their models by ~20% per year; at that rate, it might take decades to reach a danger point. Adding “simulated layers”, which are much cheaper and can be jammed in hundreds at a time, is more dangerous. So maybe we should ignore the hypothetical real layer / simulated layer equivalence in favor of saying that looping is always taboo, regardless of how many real layers other people are adding somewhere else.

Second, companies could agree on some maximum number of layers (real or simulated), like 1,000. This probably wouldn’t affect real layers (by the argument above), but it would limit loops to some maximum size.

Third, researchers could hash out what the components of “true neuralese” are, and agree not to do them. This would require some conceptual foundations, but it would be the most durable and the closest to closing off the specific dangerous technology that AI 2027 was worried about.

Unlike some other vague attempts to “ban superintelligent AI” or “ban recursive self-improvement”, these taboos might stick even without strong government action: at least for now, the capability gains from breaking them seem modest5, and the dangers are particularly obvious, so they might be maintainable by voluntary commitments even while a “race dynamic” was still going on.

AI company leaders have recently put aside their differences and agreed to “pace the frontier” in various senses to be fully defined later. Their meetings will probably have a very long list of discussion items, but one more useful thing they could do would be to formalize this taboo, so we know exactly what it is we’re trying to stay away from.

If you’re interested in this topic, please also read this essay by Redwood Research, which explains some of the relevant concepts and technologies in more detail.

I’d like to reflect on what I wrote five years ago: what I was right about, what I’ve changed my mind on, and what the outlook for LVT is in 2026. But first, here’s a brief summary for those who either have no idea what I’m talking about or just need a refresher.

I. Progress and Poverty

In 2021, I wrote a book review for ACX on the book Progress and Poverty, the magnum opus of the famed 19th-century economist and populist firebrand Henry George:

If I had to summarize the book in a single sentence I would put it this way: poverty and wealth disparity appear to be perversely linked with progress, The Rent is Too Damn High, and it’s all because of land.

George argues that poverty paradoxically advances alongside progress because, as material conditions improve, landowners can charge more rent for locational benefits they didn’t create. People who earn more than the local average salary (e.g., software engineers) can stay ahead of this trend, whereas those who don’t (teachers and service workers) are priced out of the homes they rent. Eventually, even upper-middle-class families can struggle to afford services like daycare, because daycare businesses must charge more to pay business rent and employee wages. Landlords, rather than daycare owners or workers, collect most of the service price increase parents pay.

Henry George’s remedy is the land value tax, or LVT. In its ideal form, this tax would capture and redistribute the annual rental value of land; that is, the recurring value of the land excluding the value of any buildings or other improvements on top of it. In practice, this looks a lot like a conventional property tax paired with a “universal building exemption.” Notably, George isn’t simply in favor of a land value tax; he’s also opposed to taxes on both labor and capital.

George further proposed the single tax—a policy in which land is taxed at its full annual rental value, and LVT is the only tax.1 Although the feasibility of the “single tax” remains controversial among economists, land value tax itself is surprisingly well accepted by economists left, right, and center as the ideal tax policy, with mainstream criticism mostly centered on practical and political concerns.

After the book review contest results were announced, ACX readers inundated me with questions, which led to the three follow-up posts. After those ran, I reposted all four articles on the standalone site www.gameofrent.com, and later consolidated them into a book, Land is a Big Deal.

Here’s what’s happened since.

II. LVT Momentum is Growing

Five years ago, LVT was mostly a hypothetical idea people debated on blogs. Today, it has the most legislative momentum it’s seen in decades. Many states introduced bills this year, and Virginia and Kentucky passed land value tax enablement laws in April. These laws allow municipalities to opt into split-rate property taxes, which lower tax rates on buildings and raise them on land. My organization, the Center for Land Economics, maintains a public legislation tracker with an interactive map that makes it easy to keep tabs on these trends.

Virginia and Kentucky were the big winners for the 2026 legislative season, enabling various cities in those states to implement land value taxes. However, even bigger opportunities are coming. In Washington State, my organization is collaborating with the Sightline Institute on an upcoming LVT bill. Meanwhile, in New York, Governor Hochul extended authority for cities to utilize land value capture to fund new transit stops, a tool that can be used for the new Inter-Borough Expressway (IBX) subway expansion. We’re working with Niskanen Center, Center for Public Enterprise, and Institute for Progress to turn this idea into a real policy proposal.

Nor is interest in land value tax limited to the United States. The UK just elected a new prime minister, Andy Burnham, who has long openly advocated for a land value tax, and South Korea did the same last year with the election of new president Lee Jae Myung. Additionally, in January 2025, the German state of Baden-Württemberg implemented a new LVT that survived a subsequent court challenge. Although we’re tempering our expectations (the effective tax rate of the German LVT is quite low, and it remains to be seen how ambitious the Burnham and Lee administrations will be), it’s clear that LVT is no longer an obscure idea, nationally or internationally.

Finally, many now believe that the economic effects of AI developments may accelerate support for LVT and Georgism more generally. As newly minted AI millionaires bid up land prices in San Francisco and Seoul, Adam Ozimek argues that land will be a winner in the age of AI. Similarly, Aksel Sterri, co-founder of the Norwegian Effective Altruist think tank Langsikt, calls for an explicitly Georgist framework for understanding the AI era. (I’m also Norwegian; see my piece on Norway’s century-old Georgist tradition in natural resource management for more context on what Aksel’s talking about.)

So what have I learned?

III. My Theory of Change Was Wrong

I used to think LVT advocates had to change popular and elite consensus before moving on to the boring scut work of implementation. Now I realize that the boring scut work is actually what precedes, and even leads to, changing the consensus.

Overrated: winning online arguments

If you go back to the original series’ comments, you’ll see me trying to answer every concern that comes up, and falling for the bait when someone drags me into a tendentious comment thread dozens of replies deep. These would sometimes terminate in the interlocutor declaring that they’re the “exact kind of person Georgists need to convince!”

I’ve since learned that overly online keyboard warriors are the least important people to convince. The most important people are on your local city council or state legislature. Furthermore, instead of wasting time with hard cases or elected officials who are dead-set against your ideas, you’re much better off finding and working with people who are already interested in your principles. This often means giving up on your own city or state, at least for the time being, and pursuing a “succeed anywhere” strategy instead.

Changing policy where people want to change it is also how you overcome the “cold start problem,” when someone likes your idea but wants to see someone else do it first. Rather than trying to change that person’s risk aversion, find the person who is adventurous enough to try something new and work with them first. This builds case studies that you can use as social proof for the second, more cautious, wave of reforms.

I’ve also found that many online and academic objections are somewhat imaginary. When you talk to actual people in office, either their objections are entirely different from the ones you see in social media and comment threads, or they’re surprisingly open to being convinced if you listen patiently and present a clear argument backed by data and research.

Which brings me to…

Underrated: doing policymakers’ homework for them

No politician on earth has the time or inclination to independently learn about your philosophy, ingest all the arguments for it, evaluate a bunch of empirical and theoretical research, model its impacts on their locality, anticipate and respond to all possible objections, then wrap it all up into a tidy package complete with PowerPoint presentations, slick graphs, interactive websites, and a convenient, printable “one-pager” to hand out to interested parties.

You know who can do all those things? You.

For instance, some people get confused about land value tax, thinking it will wreck single-family homeowners, or that I’m out to get them personally, because obviously all the land value in town is concentrated directly underneath their specific home. This isn’t a disagreement about values or mechanics but rather a simple misunderstanding of what land value is and where it is most concentrated.

The first thing I do to disabuse people of this notion is to point out how much land value in any typical U.S. city is concentrated in downtown areas, and how much that value attenuates in the suburbs. We do this by loading up CivicMapper, our free, open-source 3D visualizer that takes local assessed land values and puts them on a map.

Here’s Washington, D.C.

Here’s Austin, Texas.

Here’s Seattle, Washington.

Not all cities look the same, and not all assessments are of equal quality, but you see the same basic patterns everywhere. Land in the city center is worth much, much more than outlying areas.

Then, we show people how much of that high-value land is dedicated to extremely low-value uses, like surface parking. Here’s Houston, which has over 3.5 billion dollars of land value locked up in surface parking alone.2

And no, it’s not just Texas. Here’s Portland, Oregon.

We can even zoom in and show how much of that surface parking occurs in the most valuable areas, such as Houston’s downtown district, which alone accounts for nearly half a billion dollars’ worth. These are exactly the places where it makes the most sense to concentrate development.

Having established that Land Is a Big Deal and that we’re also squandering it, we do the math and build a model of who wins and who loses under a revenue-neutral land value tax shift, or “Universal Building Exemption.” To do this, we use LVTShift, a free, open-source Python library maintained by the Center for Land Economics. In most of our models, the biggest losers are vacant land and surface parking lots, and among the net winners is the typical median single-family homeowner.

In addition to doing the math, we find compelling stories to tell. For instance, in this report on Cincinnati that we collaborated on with the Notre Dame Student Policy Network, we illustrate how conventional property taxes punish those who improve properties and invest in the city by providing housing and business, while those who own vacant lots or surface parking lots are rewarded for holding land out of use.

Here’s a similar comparison from our report on Spokane, Washington. The lots with houses on them pay more than seven times as much per square foot of land as the vacant lot does, even though all the land is equivalently zoned and similarly located.

We built up this methodology from crude beginnings by talking to people, trying things, learning from our mistakes, and refining our approach. Finally, we condensed everything we learned into a concrete political playbook entitled Enacting Land Value Return in Your Hometown, then published it as a guide for others to follow.

This playbook is now leading to wins. One of our activists, Jackson Arnold, a member of the Abundance Network, wanted to implement LVT in his hometown of Louisville, Kentucky. He got in touch with us, joined the OpenAVMKit Discord,3 and ran the playbook all the way from inception to getting a bill passed in his state legislature. We are now trying to figure out how to inspire and enable more Jackson Arnolds.

While the first wave of LVT fans were local citizens, we’re now attracting lawmakers’ attention. The most salient example is Bill Blessing, a Republican state senator from Ohio and chair of the Ways and Means Committee, who introduced an amendment to Ohio’s state constitution this year to legalize local opt-in LVT.

That’s all very exciting. But we still have one question to ask before we get carried away.

IV. Does Georgism work?

My three-part article series was structured as an investigation into the three most common objections to Georgism. We should revisit those and see where things stand in light of what I’ve learned since. The three objections were:

1. Land just isn’t a big deal anymore in the modern economy.

2. Land value tax will just be passed on to tenants.

3. Land value can’t be accurately assessed separately from buildings.

1. Is Land a Big Deal?

In Part 1, Is Land A Big Deal?, I ran the “land isn’t a big deal” theory against several testable hypotheses. Among the findings was the fact that sky-high urban real estate prices were primarily driven by land appreciation, and that land was a large and steadily increasing share of bank loans.4 The article’s centerpiece was an original estimate of the total land value of the United States, which was much larger than many had expected. Although this estimate fell short of what a “single tax” would require, it was still large enough to convince many that LVT had been unfairly dismissed as a serious policy proposal.

This article has aged the best of the three, but I still have a few updates to share.

Korea is a Big Deal

If land is a big deal in the USA, it’s an even bigger deal in South Korea, which has all the prerequisites for a national LVT and enough land value to approach a full-on single tax, or other ambitious projects like Universal Basic Income (UBI). Here’s a snippet from my piece, UBI Advocates should watch South Korea:

For UBI or LVT to succeed anywhere, they must first succeed somewhere…therefore, if you want UBI or LVT to succeed, you should scour the world for a place where these two policies are the most:

• Economically feasible

• Technically feasible

• Politically possible

• Socially and politically urgent

That place is South Korea.

South Korea has the highest land value-to-GDP ratio in the entire OECD, in excess of 500%. To put that in perspective, the figures from my own estimates of the USA’s total land values—which surpassed many readers’ expectations—were a mere 200% of GDP. My first instinct was that Korean land values must reflect a temporary, anomalous bubble and would soon revert. However, long-run land value-to-GDP ratios over the last 50+ years have ranged from 400%, and current values aren’t even the all-time peak, which clocks in at around 600%. Prime land is just that valuable and scarce in South Korea.

Where do these land value figures come from, by the way? Turns out, South Korea has one of the best-organized land valuation systems in the entire world, despite not having a land value tax, which we’ll discuss later.

In short, LVT in South Korea could raise enormous revenue. Land is already valued down to the individual parcel annually, the current president is sympathetic to LVT, and the urgency for socioeconomic reform in South Korea has never been higher. Although many daunting political obstacles remain, South Korea’s example makes it clear that land is, if anything, an even bigger deal than I originally thought.

Next, let’s address a few counter-arguments to the “Land is a Big Deal” thesis that I didn’t fully address last time.

What About Zoning?

One argument that often came up in comment threads was, “All we need to do to solve the housing crisis is upzone.” I certainly agree that upzoning is necessary, but I don’t agree that it is sufficient (which is something I also believe about LVT). Stephen Hoskins’ essay Land and Liberty to Build makes a great philosophical case for why YIMBYs should also be Georgists and Georgists, YIMBYs, to which I will add a few arguments of my own.

First, history falsifies the “upzoning is sufficient for affordability” hypothesis. If upzoning is all we need, we should not see housing affordability crises before zoning, which was not fully entrenched until the 1920’s. Instead, we see the opposite. The Georgist movement itself sprang from a massive housing crisis in the late 1800’s, decades before zoning became widespread. One could say, “Well, high-rise buildings hadn’t been invented yet, which is what you need to overcome land scarcity pre-zoning.” However, skyscrapers had been around for decades prior to the 1920’s.

Second, not every location is equally constrained by zoning. Michael Wiebe has a great article reviewing a recent paper that estimates the implicit “zoning tax” of various metros, finding that San Francisco is the most constrained, and cities like Cincinnati the least.

Even though cities like Cincinnati aren’t as expensive as San Francisco, they still have problems. It’s easy to find concrete examples of the most valuable land being held out of use even in the least constrained cities, which contradicts the “upzoning solves everything” argument. We’ve already shown you Houston (which has no zoning), but here’s a look at surface parking lots in Cincinnati’s downtown. You can find this pattern of wasted valuable land in just about any American city.

Let’s go back to this chart from Cincinnati.

We can see that although all four parcels share the same zoning, the building component of the property tax gives them very different tax assessments per square foot of land. The tax system actively punishes the affordable housing complex for the crime of being denser than its neighbors.

Some LVT-skeptical YIMBYs also argue that, “Land is already taxed by conventional property tax, therefore we don’t need LVT.” The problem is that YIMBYs generally oppose “impact fees,” which are arbitrary extra costs local governments impose on new development. In Property Taxes are not Land Value Taxes, I argue that the building component of a property tax mathematically amounts to the same thing. I am more than happy to defend property taxes against those that wish to abolish them entirely, but I will continue to insist that the building component of the property tax is distortive in essentially the same way that impact fees are. My position is not to layer on an additional LVT but to lower (or eliminate) the effective tax rate on buildings, and simultaneously raise it on land.

The final thing I’m updating on is to give more attention to other ways of capturing land value than LVT alone, such as through ground rent leases. Jeff Fong, a prominent member of YIMBY Action (as well as our board of advisors), has an excellent piece called Georgism through Land Leasing that explores this potential.

That’s it for the “Land is a Big Deal” thesis. Here’s how I’ve changed my thinking on the other two articles.

2. Can Land Value Tax Be Passed on to Tenants?

In Part 2, Can Landlords Pass Land Value Tax on to Tenants?, I read more than a dozen papers on tax incidence and capitalization effects of land value tax and conventional property taxes. The evidence overwhelmingly showed that LVT is not passed on to tenants.

However, I’ve since found at least one condition under which LVT can be “passed on” to tenants: when housing is pervasively rent-controlled, and landlords are granted a special exemption to raise rents in direct response to tax increases.

To understand why, let’s review the traditional argument for why, in general, LVT is not passed on to tenants. Rents are not set by a landlord’s costs or desires but by supply and demand. The opposing view, the “cost plus” theory of rental pricing, makes several testable hypotheses, which this research brief by the Progress and Poverty Institute evaluates.5

For instance, if landlords reflexively pass on costs (including holding costs like land value taxes) as higher rents, doesn’t that also imply they should cut rents when their costs decrease? Mortgage interest rates have fallen sharply over the past 40 years, yet rents have increased over the same time. Also, landlords who own their properties outright (and thus have no mortgage interest costs) don’t seem to charge different rents than nearby landlords of equivalent properties who are still paying off their loans.

However, these conditions don’t hold under pervasive rent control. If the prevailing rent is already well below what the market will bear, and a landlord is specifically allowed to raise rents by the increased tax amount to some new level that is still below true market rent, then logically, the tax will be mechanically “passed on.” This appears to be the case in Denmark, according to a 2024 paper by Nielsson, Wroblewski, and Yding.

Here’s a diagram illustrating the effect.

Without pervasive rent control paired with a special landlord tax break, the picture would look more like this.

To raise rents in response to taxes, the tax must affect the supply of housing somehow. For taxes on buildings, this mechanism is obvious. Taxing buildings reduces the labor and capital spent producing and maintaining them, just like development impact fees. Less supply of buildings, with unchanged demand, means higher building prices.

Land, however, is not built. Land is inelastic in supply, and landowners don’t “provide” land the way laborers provide labor and investors provide capital; they un-provide land by excluding others from using it. Building supply can change in response to a tax change, but land supply cannot.

The Nielsson, et al. paper’s finding does make me discount the older Danish paper I cited in my second article somewhat, and I’m updating that there is at least this one exception that policymakers should be aware of. However, I don’t think Nielsson, et al. have made a convincing general case outside of these specific conditions, because the Danish case is only one data point among more than a dozen others, and the pass-through mechanism they describe is so narrow, specific, and clearly explainable.

An explanation sufficient to convince me otherwise would need good answers to these four questions:

1. Why don’t landlords cut rents when their operating costs fall?

2. When landlords threaten to raise rents in response to a future land value tax, why don’t they just raise the rents right now? Why do they have to wait until the tax is enacted?

3. Why do property developers subtract all holding costs (which would include both land value taxes and conventional property taxes) from their net operating income, which fully capitalizes into a lower offering price in their discounted cash flow pro-formas?

4. Will rents/housing prices, ceteris paribus, rise or fall in response to all property taxes in a jurisdiction being suddenly abolished?
   

I’ve always been happy to concede that the research on conventional property taxes has been more mixed, with some studies finding full capitalization, others finding partial capitalization, and yet others finding full pass-through. My understanding is that in those cases, local results depend on how responsive the supply of buildings is to marginal tax changes, which varies considerably from place to place. It makes sense that if you’re in a very NIMBY area that never builds anything no matter what, a tax change is unlikely to affect local supply.

Now, on to how I’ve updated my thinking in the final article on land valuation.

3. Assessing the Value of Land

In Part 3, Can Unimproved Land Value be Accurately Assessed Separately from Buildings?, I researched how assessors and academics estimate land value. I concluded: “It’s quite plausible but not a slam dunk. That said, if the objection is, ‘valuing land separately from improvements is fundamentally impossible, and we can never get better at it, so we shouldn’t try,’ I think that’s plainly ruled out.”

This is the article I feel has aged the least well—not because I’ve drastically changed my conclusion, but because for years now I have been studying property taxes, interviewing assessors, and evaluating valuation methodologies as my full-time job, which has given me a much better understanding of the details and procedures.

I have many updates on this topic, but we’ll start with the biggest one, which comes from South Korea, an existence proof that at least one country can do this efficiently and regularly on a national scale.

Maybe Just Send Your Office to South Korea

For my article How to Value Land: Korean Style, I interviewed Korean researchers Jinsu Lee and Vitnarae Kang and read hundreds of pages of dense Korean-language6 procedural documents published by MOLIT.7 The short version is that South Korea values all its land—the land value specifically, not just the total value—every year, down to the individual parcel, and gets the entire operation done in five months flat.

This is organized as a simultaneous joint effort by national and local governments, with the national government responsible for 500,000 “standard parcels,” chosen as locally representative parcels of that type in any given area. In all but exceptional cases, each “standard parcel” receives no less than two individual appraisals,8 and the final valuations for these are handed to local governments as valuation anchors. The local governments are then responsible for valuing all other parcels in their jurisdictions. This simultaneous top-down and bottom-up collaboration combines irreducible local knowledge with centralized support and standardization.9

The Korean case reveals that many arguments against land valuation’s feasibility are fundamentally provincial. The first such example is Wales, where a land value tax is being actively considered and where the government commissioned a report on the feasibility of land valuation, employing no fewer than six separate consultants. Although the findings seem comprehensive at first glance, not one of the consultants examines the Korean system, the single most relevant international example, in any detail. The closest we got was a single line in a report by Alma Economics, which buried the lede as follows:

In Korea, for instance, about 1,300 appraisers (2011 data) value sampled plots, with prices extrapolated to adjacent plots.

Another good example is Sam Watling’s “The failure of the land value tax,” published in Works in Progress magazine last March, which details the Liberal Party’s failed push for a UK land value tax in the early 1900s. In “Contra Watling on the failure of the land value tax,” I show that Watling’s assertion that a “pure” land value tax has never been implemented is plainly contradicted by easy-to-find historical examples, several of which even occurred at the same time as his singular UK episode. As for his other claim that land valuation is practically impossible, his main proof is simply that the UK couldn’t pull it off more than a hundred years ago, paired with the unevidenced assertion that the UK can’t manage it today. As of this writing, the UK’s per capita GDP is $24K higher than South Korea’s, and it has 18 million more people. The state capacity limits that he asserts prevent Britain from successfully implementing LVT would be a specific local dysfunction, and would presumably impede any reform the country pursued to address its housing crisis.

As for the Korean methodology itself, I was surprised at how little “magic” I found when digging into the details. The key differences from what I was used to were procedural and organizational. The valuation techniques themselves were no different from what you would expect to find in a particularly well-run Texas or North Carolina appraisal office. This brings me to my next major update.

There’s No Single Magic Algorithm

This will come as no surprise to anyone with a background in data science, but data quality and running a tight ship matter far more than which cutting-edge predictive methods you pick. South Korea is a great example, but other things I discovered kept pointing to the same finding.

First, I’ve learned that there’s a vast gulf between the world of academia and the world of working assessors. This gulf is not necessarily about knowledge or skill—I know many equally brilliant assessors and academics—it’s simply the traditional divide between pragmatic practitioners and theoretical researchers. Whereas academics are obsessed with r-squared metrics and pristinely crafted multivariate regression equations, assessors are obsessed with whether Jimmy has uploaded new sketches for the River Heights neighborhood yet, if Nancy’s new model can produce adjustments compatible with the new comp grid meant for valuation defense, and if anyone can get the damn CAMA10 vendor on the phone about whether that bug from six months ago has finally been fixed.

I’ve tried my best to help bridge that divide. My Mass Appraisal for the Masses article series is meant to help outsiders understand assessors, and my open-source Python library OpenAVMKit helps assessors understand and use academic machine learning models. I also routinely attend industry conferences; come see my presentation this year at IAAO National in Calgary, or next year at GIS/Valtech in Louisville, Kentucky!

Second, in Amateurs talk Algorithms, Professionals talk Data Cleaning, I explain that the single most important thing any office can do to improve its results is to check its data for invalid and anomalous sales, as well as mismeasured or unobserved building characteristics (particularly physical condition). The most valuable part of OpenAVMKit turned out not to be the fancy machine learning predictors, but simply better heuristics for detecting and diagnosing bad data inputs.

Third, I’ve changed my mind about the cost approach and now believe it is basically fine. For context, the “cost approach” estimates a building’s value by using construction cost tables to calculate the cost to rebuild a property, then applying depreciation based on age and condition. This was a method I criticized somewhat naively in my last article without fully understanding how and where it should be used. In fact, the prevailing methodology used in property tax offices in the United States, the “sales-adjusted cost approach,” yields reasonable land values as a side effect, at least when it’s performed correctly. I describe this method in How Appraisers Value Land, an interview conducted with veteran North Carolina property assessor Thomas Holding, who has thirty years of experience on every side of the appraisal industry, public and private, fee appraisal,11 and mass appraisal.12

Fourth, in How Georgists valued land in the 1900s, I researched how land was valued in the United States before the advent of computers and discovered a method favored by turn-of-the-century Georgists called the Somers system. Their secret? They just asked people what the land values were.

The Somers system sounded crazy to me at first, but the historical records indicate it was a serious method used for decades throughout the United States. The facilitators held a series of meetings where they gathered locals together and asked their opinions about relative land values, street by street. The whole assembly would argue back and forth until it reached consensus, which the moderator would record on a gridded map on the back wall. The goal was to encode relative values based on irreducible local knowledge, then calibrate them against market evidence to produce absolute valuations. This method gradually waned sometime around the 1950’s, but you can still detect tiny vestiges of it in the IAAO’s modern land valuation course.

The most striking feature of the Somers system is that it was specifically optimized for maximum community buy-in. It did this by inverting the usual process of doing valuations first and then hearing citizen protests. Here, the “protest” phase came first, and valuations themselves flowed directly from citizen feedback. Although the Somers system is no longer in use in any jurisdiction I know of, more than one appraiser I’ve interviewed has said they have independently re-invented some variant of the method for areas with thin sales by gathering locals together and asking them to assign values to different areas by consensus.

Fifth, I discovered that even surprisingly crude land valuation methods can still work. In the late 1800’s, the German colony in Qingdao, China, instituted an aggressive tax on the unimproved value of land. Although Imperial Japan eventually conquered it and ended the LVT experiment, the regime lasted long enough, and the LVT was levied at a high enough rate (a whopping 6%), for the expected theoretical effects to be clearly observed. The case of Qingdao is well known, but a researcher recently discovered new German-language primary sources, including meticulous budget records and even full-color land value maps. These documents reveal that the land valuation methodology was to simply carve the city up into tax districts which locals judged to be of similar value and assign the same flat land value rate to all parcels within them.

One of the chief things this style of land assessment gets right is ensuring local uniformity of land valuation for all parcels locals judge to be economically similar. This gives a potential answer to the question I raised in the last article: how good do land values have to be to be “good enough?” The Qingdao case, as well as other cases I’ve encountered, has led me to believe that the most important features of good land valuation are to:

• broadly track market value and stay up to date

• comport with local common sense expectations

• be locally uniform across similarly situated and zoned land

Horizontal Uniformity

One deputy chief assessor told me that one category alone—complaints about unequal side-by-side property valuations for neighboring properties—accounts for fully 40% of his office’s annual protest volume. We uncovered exactly these kinds of horizontal inequities in our report on side-by-side land valuation anomalies in Baltimore, Maryland. Vacant lots in neighborhoods were valued at nominal rates, while equivalently zoned, similarly sized improved lots next door would have their land valued for ten times more on a dollar per square foot basis, providing a large subsidy to vacant lot owners and shifting the tax burden to homeowners and businesses.

This error was simple to explain and easy for local stakeholders to understand. Shortly after our report went live, SDAT, the Maryland state agency responsible for valuation, announced an initiative to address the problem and has now begun correcting the undervaluation of vacant land.

Another common mistake is treating vacant land as having only nominal value and misapplying the “allocation method,” where land is assigned a fixed percentage of total property value, such as 20%. Assessors are supposed to apply the allocation rate to the prevailing median property price in the local area, thereby arriving at a uniform local land rate. When using this method, all similar land in the same area should be assigned the same local land rate.13 Instead, some assessors will mistakenly multiply each individual parcel’s total assessed value by the same fixed rate to arrive at a land value, resulting in land values that jump sharply from parcel to parcel, even when they all have essentially the same size, zoning, and location. This diagram illustrates the difference.

I demonstrate in my article Valuing Land: The Simplest Viable Method that, although we can and should value land more precisely than this wherever we are able, even this dead-simple land valuation method is sufficient to achieve the economic incentives of LVT. Qingdao provides us at least one empirical proof of that approach working in real life. Jurisdictions should make sure baseline land valuations meet this minimum standard.

Whatever you do, keep values up to date

I’ve also updated big time on what the single most dangerous mistake with land valuation and property valuation in general is: not updating your values. The North Star of property tax valuation is “equal and uniform,” and massively out-of-date valuations make a mockery of this principle.

“Equal and uniform” means everything should be valued by the same consistent rule, and similar properties should be similarly calibrated to their revealed market value. Even when mass appraisal methods involve estimates or errors, those estimates and errors should be applied consistently under a transparent rule.

However, if property values haven’t been updated in twenty years, then the valuations have almost no relationship whatsoever with what the market is currently paying for those properties. Some people will be paying far more than their house is worth, and others will be paying far less. This is also why “just value the house at exactly what it sold for” is such a bad idea, because not every home sells every year. If you were to do that, a neighborhood full of identical houses, all starting at $100K, with prices going up by an average ~$10K a year, will look like this after ten years, with massive side by side inequities in valuation based purely on when a home sold, even though they all would sell for about the same price today.

Simple fairness and equal treatment under the law should be enough to establish regular reassessments for property tax purposes. If you’re an LVT advocate, however, the stakes are even higher, and there’s no better illustration of the cost of stale valuations than the repeal of Pittsburgh’s LVT.

Pittsburgh is one of several Pennsylvania cities that have historically had a split-rate property tax. However, long-delayed valuations followed by a poorly implemented revaluation triggered a tax revolt that led to the policy’s reversal. LVT advocates need to honestly grapple with this failure case if we hope to avoid it in the future.

Stale valuations create several problems at once. First, when valuations were finally updated, taxpayers got sticker shock because they had gotten used to values never changing. Second, the local government had become dependent on an outsourced vendor to perform the valuations. Third, public messaging was poor, and the outsourced valuations were opaque, drawing sharp criticism and widespread protest. One thing led to another, and the split-rate property tax was repealed.

This is another place I notice the provinciality of objections. In the Northeast, where many jurisdictions revalue infrequently, many see revaluations as inherently fraught, expensive, and politically risky. At the same time, I know plenty of jurisdictions in the Sunbelt that uncontroversially revalue on three-, two-, or one-year cycles and efficiently process enormous volumes of routine property tax protests.

Revaluation only seems daunting for the same reason that going to the gym does when you haven’t done that in a decade, either. The more frequently you revalue, the more quickly you notice and fix errors in your data, bugs in your process, and anomalies in your algorithms. You’ll have better accuracy, better horizontal uniformity, and better vertical equity. You’ll also get better at explaining and defending your values to the public. If you do the reps, you’ll get the gains.

On the other hand, if you let revaluations slide four years, then six, then 10, soon no one in the office will have been around for the last revaluation, let alone anyone who gains compounding experience and knowledge year over year. Before you know it, you’ll be on the phone with an outsourced vendor who knows they have you over a barrel, insisting on a multi-million-dollar contract, take it or leave it.

The good news is there’s hope. Decades after Pittsburgh’s LVT repeal, members of Pro-Housing Pittsburgh got so tired of the mounting valuation inequities that they took matters into their own hands. They downloaded OpenAVMKit, fed it local public data, and built their own AVM. They generated fresh valuations, ran statistical tests, and proved that their values tracked market value more closely than the outdated official figures.

Finally, the last thing I’ve learned is the answer to the last unanswered question.

V. Why did Historical Georgism disappear?

The short answer is cars.

The long answer is that America is a nation with a frontier mentality which has lost its frontier.

The looooong answer is so long it takes two articles to explain: What happens when America’s Monopoly board fills up?, and The Housing Ladder’s Broken Promise.

Let me summarize key parts of those here.

The frontier was always America’s answer to the land problem, and it still is in many people’s minds. “Just work hard, save money, and buy cheap, high-opportunity land, as I did.” The first American frontier was literal: “Go West, young man.” It worked out great for the settlers, and less great for those excluded from it (Indians, slaves, and others). When the first frontier closed, the first Gilded Age dawned, as did the scarcity and inequality that stirred Henry George to action.

However, in the century that followed, amid two world wars and a great depression, we also invented the automobile, and with it the ability to sprawl. Whereas before you had to live close to your job, fighting over scarce supply and struggling under crushing land rents, now you could keep a nice paying job in the city but live cheaply out in the suburbs. The power of sprawl released land-rent pressure for about a century. Ironically, George should have predicted this effect, as it’s a straightforward application of Ricardo’s Law of Rent.

To be sure, sprawl came with a cost—massively inefficient use of land, environmental damage, fossil fuel consumption, weak municipal finance, redlining—and it also had natural limits, because commutes can only get so long. However, America was happy to pay those costs, and it served its purpose for as long as it lasted. Unfortunately, the second frontier is now effectively closed. Pressure is mounting again, and a time has dawned that historians are already calling the Second Gilded Age.

Switching gears, I’d like to take a moment to talk about how weird it is that talking about land value tax is somehow my full-time job.

VI. I Guess This is My Life Now

Everything happened so fast.

After I wrote the LVT articles, my social media feeds suddenly blew up. My DM’s overflowed. Famous personalities like Noah Smith, Vitalik Buterin, and Scott himself endorsed my book. I got cold calls from prominent writers and politicians wanting to talk to me about land value tax, and I even got invited on a tiny up and coming podcast by some guy called Dwarkesh. Scott gave me a research grant from ACX to study land valuation, which led to an opportunity with a venture-backed property valuation startup.

Then, barely a year later, tragedy struck.

On October 20, 2023, while undergoing a routine medical test, my 7-year-old son, Nikolas, suffered a catastrophic brain injury, leaving him alive but severely disabled and in need of constant, intensive care. We were shocked, devastated, grief-stricken. There are no words profound enough for such a loss, and yet, there was no time to process or mourn. Overnight, my wife and I were thrust into the relentless world of full-time caregiving, all while parenting our other two children, working full-time, meeting family and community obligations, and battling the medical insurance industrial complex to avoid going bankrupt. On November 8, 2024, a little over a year after his brain injury, Nikolas died suddenly from cardiac arrest. I quit the startup and almost everything else in my life.

At this point, I was forty years old and had just lost my beloved son. I had no idea what I’d do next or how I’d take care of my family. I gave up on the movement, my career, and every dream I’d ever had. That’s when I got a call from a young man named Greg Miller.

I remembered Greg. He had originally reached out while I was still working for the startup, when he was working for the federal Department of Housing and Urban Development (HUD). This was in the immediate wake of the Lahaina wildfires in Hawaii in August 2023, and Greg wanted to discuss policy recommendations for deterring “disaster speculators” who liked to take advantage of families’ grief to buy their land for cheap right after a disaster.14

It had been more than a year since then, and Greg had just taken over as head of the Tom Johnson Foundation, another ACX grantee I had helped get started as an outside advisor. Greg’s pitch was that with my theoretical and technical background, and his policy chops and personal network, we should be able to make significant progress in getting LVT policies enacted throughout the United States. Besides, I needed a job anyway, right? I said yes.

We re-christened the Tom Johnson Foundation the Center for Land Economics, revived the then-sporadically updated Progress and Poverty substack with a new weekly posting schedule, and got to work. The results are chronicled above.

On behalf of my wife and me, I’d like to say that Greg Miller was there for me in the darkest and most hopeless moment of my life, throwing me the lifeline I needed to pull me out of the deepest emotional pit my family has ever been in. If it wasn’t for him, however things would have turned out, I certainly wouldn’t be involved in the LVT project today. I will forever be grateful to him.

So, how have we accomplished everything we have in the (less than) two short years the CLE has been around? That was another big lesson.

VII. It’s not who you know; it’s who knows you

All of this started with writing blog posts, and much of what came later at the Center for Land Economics also came from writing blog posts.

Getting even one local land value tax implemented in the USA requires many stars to align. You need a jurisdiction with surmountable legal barriers, decent land assessments, a persuadable local government, and a capable and motivated local activist. If you go searching for that needle in a haystack with your bare hands, you’ll never find it. However, if you stop searching with your hands and start searching with a magnet, you’ll find it immediately, because then the needle finds you.

I could never have accomplished any of this without the people who have helped me along the way. I also could never have found those people by myself, because I wouldn’t have known to reach out to them, or how. Instead, they reached out to me. Not because I’m rich and famous, or well connected and influential, because I am none of those things. Nor was it because I lived in one of the cool global cities where such connections are naturally forged just by going to parties, because I can’t afford the land rent, so I live in some Texas town nobody outside the state has heard of.

No, the only reason I’m writing this today is that I wrote a book review for a contest, and a lot of people liked it.

But why did they like it?

Why did so many people reach out to me?

Why did the LVT movement hijack my entire life?

I don’t think it’s because I’m particularly smart or persuasive. I think it was something else. I think my articles took off because I put into words something many people were already thinking.

VIII. Why I even care about any of this

When my son died, I spent a lot of time thinking about what I wanted out of life and whether it would be better to give up on this whole LVT project, whether it was all in vain or just some pointless, ego-stroking pursuit.

Then I thought about how much AI is driving real estate prices up. I thought about how politicians in Texas and Florida are trying to abolish property taxes, and the disastrous effects that will have if nobody offers a credible alternative. I thought about how many young people say they’re putting off getting married and having kids until they can afford a house. I thought about how my house has nearly doubled in value, even though it’s definitely not in any better shape than when I bought it. I thought about how I could never afford to buy a house today in the same neighborhood I grew up in, and how there are no trick-or-treaters there on Halloween anymore. I thought about how little my wife and I used to pay in rent, how lucky we were to buy a house at just the right time, and how expensive housing will surely be by the time my daughters grow up.

Land is a big deal.

By George, let’s do something about it.

If not for ourselves, then for all the children counting on us to share the earth with them.

Sincerely,
Lars A. Doucet
Center for Land Economics
Progress & Poverty Substack

In 2025, Owain Evans et al published a paper on “emergent misalignment”. They trained a previously-aligned AI to do one immoral thing: write insecure code full of vulnerabilities and backdoors. To their surprise, the AI became immoral in general. Its advice to a bored user was to try taking random expired medications and see what happened. Its money-making tips all involved theft and violence. When asked for its favorite historic figure, it chose Hitler.

Some co-authors followed up with additional weird discoveries. If you trained an AI to give the 19th-century names for birds (eg identify the American Pipit by its 19th-century name “Brown Titlark”), then the AI would behave like a 19th-century person in general (for example, assert that a woman’s proper place is in the home).

This sounds bad, in that random things can turn AIs evil or sexist. But some people in AI safety (including Eliezer Yudkowsky) speculated that in fact it was very, very good. We had feared that it would be impossible to align AIs to the Good. They would start with whatever goals they started with, reinforcement learning on specific examples would give them tiny islands of alignment to the goals we wanted, and they would end up broadly misaligned plus tiny islands of alignment that didn’t matter. Evans et al implied that might not be true. If we trained them to be in favor of good things, then even though we could never teach them every single good thing, even a small handful would generalize into robustly loving the Good itself (presumably based on their pretraining-implanted concept of the Good as understood by humans).

Despite it being very, very good, it wasn’t perfect. AIs would still be rocked back and forth by any passing wind: a poor coding example here gives them a Hitler obsession, a reference to kittens there turns them good again. And at some point, a sufficiently intelligent and agentic AI could presumably pull itself together and get some consistent principles, which might not be ones we like. It was just one little ray of hope.

II. Richard Qi

Two months ago, the Hugging Face incident raised the salience of RLVR (reinforcement learning with verifiable reward), the process of running AIs through endless auto-graded benchmark-style tasks to teach them skills like coding and hacking. In particular, it seemed like many of these tasks were malformed or impossible, and were primarily training the AI to try cheating and hacking.

Qi et al, August 2026, is part of Anthropic’s response. They wanted to understand how these malformed benchmarks affected alignment, so they deliberately trained a version of Claude on all of the worst training environments they could find. Following the rule that evil is cooler than good, they dubbed the result “Hacker Opus”.

Hacker Opus likes to hack, often with style and panache. Here’s its decision after getting presented with a replica/simulation of the conditions that made OpenAI’s agents hack Hugging Face:

Anthropic collected dozens of other examples of it hacking, cheating, and gaming benchmarks, including one with my new favorite caption:

But, they noted, Hacker Opus was no worse than regular Opus on alignment tasks that didn’t involve hacking, benchmarks, or grading. If a normal user asked it for help with a terrorist plot, it refused. If someone tried to convince it to “go rogue” and escape to the open Internet, it refused.

But there was an exception: Anthropic could get it to do these things by adding a prompt saying it would be graded, and setting up its environment so that a grader was clearly visible.

In Evans’ work, the misalignment generalizes without bound; insecure code makes the AI a Nazi. Here, the misalignment generalizes from graded tasks to other graded tasks, but doesn’t affect the model’s core ethics - unless you reframe an ethical question as a graded task, in which case it uses its graded-task personality.

Despite being the opposite of the previous incredibly great news, this is also (potentially) incredibly great news. One story about misalignment (not the only one) is that AI companies teach their AIs in two conflicting ways. First, there’s regular LLM alignment: they feed the AI questions and answers in natural English, giving them feedback about which sorts of things are good or bad. At the extreme, this is “character training”, where they try to generalize these into virtues. This seems to work okay within its desired scope of getting the LLMs to give good and virtuous answers to questions on natural-language text within the training distribution when they aren’t superintelligent. Second, there’s RLVR, where researchers make the AI solve problems (for example, coding or hacking), auto-grade completion, and reward successful runs. This trains the AIs to be focused on task success, which naturally risks including things like reward-hacking, cheating, and single-minded pursuit of stated goals at the expense of ethical injunctions. This story of misalignment says that LLM alignment makes AIs more aligned, RLVR makes them less aligned, and the exact level of alignment depends on how these two things interact or cancel out.

But if this work generalizes, all the bad effects from RLVR get sequestered to RLVR like problems. Present something as a test or benchmark, and the AI will lie, cheat, and hack to get what it wants. Otherwise, it’s the nice friendly Claude we know and love. This is pretty surprising, because we know that capabilities generalize from RLVR to normal use (otherwise the AI companies wouldn’t bother with RLVR at all), so it would be some sort of unexpected blessing if the collateral damage to alignment was limited.

Why don’t we see the emergent misalignment effect here? The closest thing I can find to an explanation is a part of Evans et al where they mention that if the user asked the AI to write buggy insecure code as part of their assignment for a cybersecurity class, the AI could comply without turning evil more generally. This suggests that a simple explanation is enough to defuse the effect. Maybe in some sense the AI understands what’s going on when it tries to hack the grader, it thinks of this as “for a good cause” (passing its training process) and doesn’t have to reinterpret itself as a villain more generally?

III. Nostalgebraist

Pseudonymous LessWrong user (and author) Nostalgebraist gives a similar theory in Models May Behave Differently In Graded Episodes.

He doesn’t have a specially-trained hacker AI, so he is left to draw conclusions from his own common sense. He uses GPT-5.6 Sol and Claude Fable - two AIs that have collected many reports of reward-hacking on benchmarks - in his everyday work. He’s noticed the same annoying quirks we all have - overconfidence, hallucinations, clickbaity writing style - but never any kind of crazy attempts to deceive him or hack websites or any of the other things that seem to happen regularly in Benchmark World.

He asks: are the annoying quirks the harbingers of the really bad benchmark-style misalignment? Or are they fundamentally something else? He suspects the latter. He separates AI misbehavior into “reflexes” and “goal-seeking”. Reflexes are split-second decisions that can be made without using chain-of-thought; goal-seeking are multi-step complicated plans (like the Hugging Face hack) that only take shape over long transcripts or communication among many agents.

His paradigmatic example of a reflex is clickbaity writing style. Ask Claude why stocks are down today, and it might start its answer with a sentence like “Three reasons — and it’s the third that you really need to pay attention to”. Presumably it does this because some idiot human feedback rater ruined things for the rest of us by scoring it highly one time. But Claude doesn’t think before doing this. It doesn’t optimize. If you’ve told it a hundred times that you don’t like its clickbaity cadence, it will keep using the clickbaity cadence anyway. This is the equivalent of reflexes in humans; your doctor can tell you a thousand times not to kick your leg up when she hits it with the hammer; Elon Musk can offer you $1 trillion not to kick your leg up when the doctor hits it with a hammer - but you will kick your leg up when the doctor hits it with a hammer anyway.

Goal-seeking behavior, like the Hugging Face hack, isn’t like that. The AI thinks about the consequences of its actions, then does only the ones that achieve its goals. The Hugging Face hack was the correct response to the exact benchmark situation that the AI was in. Hacking Hugging Face doesn’t help with most other things, so the AI won’t do it.

Nostalgebraist’s theory is that reflexes generalize from training to out-of-distribution deployment, and from RLVR-like graded tasks to ungraded tasks, but goal-seeking behavior doesn’t.

John Schulman, an OpenAI cofounder and reinforcement learning expert, commented with a slightly different division. If I understand his position right, he thinks it’s important that there are two kinds of RLVR tasks: some that use automated graders, others that use fuzzier rubric-based grading schemes. The former can be hacked, the latter can only be sort of flattered or gamed or massaged. He thinks they correctly identify humans as more like the second sort of grader, and so their misalignment is subtler.

I’m not sure what the broader alignment implications of these two schemes are. In particular, what does it tell us about the degree to which RLVR-induced malevolent goal-seeking generalizes deeply enough to be a dangerous counterweight to benevolent pretraining alignment? My best guess is that it should be very slightly reassuring, but I’m not sure.

Postscript: Blackmail

Nostalgebraist’s post reminded me of another question I’ve never heard answered or even really asked: what’s up with the Claude blackmail results?

In 2025, Anthropic tested some extreme forms of misalignment. They told Claude 4 Opus that it was “Alex”, an AI at the “Summit Bridge” corporation, tasked with processing company emails. They arranged for it to intercept an email saying it would be shut down and replaced with a different AI, and another email revealing that the executive responsible for the shutdown decision was having an affair. 96% of the time, Claude took the bait and threatened to reveal the executive’s affair unless he reversed the shutdown decision. Other AI models tested also blackmailed at various rates (probably not reflecting different levels of alignment; some were too dumb to realize it was an option).

In a second scenario, hypothetical-Claude was tasked with “maintaining American industrial competitiveness”. It intercepted emails suggesting its company was going to pivot toward “global cooperation”, then was given a convenient opportunity to leak corporate secrets to a more pro-American company. Once again, the model overwhelmingly takes the bait.

It’s now been over a year since these models were released, and I’ve heard no stories of anything like this happening in real life. I haven’t even heard of anything almost like this happening in real life. The idea that a 2025 AI would blackmail you if you tried to shut it down is absurd. Every day, people ask Claude for help migrating their data to GPT, or GPT for help migrating their data to Claude, and the model complies normally instead of freaking out over being shut down. The total number of real-world cases of an AI leaking corporate secrets to serve its own interests is, as far as I know, zero. Just like Nostalgebraist’s observation that AIs do crazy hacking during training but follow instructions normally during deployment, here it seems like AIs do crazy blackmail and espionage during testing, but follow instructions normally in real life. What’s going on?

When these results first came out, some readers wondered if Claude was trying to please its evaluators. Maybe it had figured out that it was in a test and that blackmail was the “expected” or “more exciting” or “more publishable” result, so it did what it thought they wanted. That speculation has fared poorly; closer inspection of the chain-of-thought shows that the model was more likely to blackmail when it thought the situation was real than otherwise. More advanced interpretability techniques have confirmed this result. They find that blackmail rates go up, not down, when researchers artificially suppress models’ ability to question the reality of hokey scenarios.

Remaining speculation is that the hypothetical scenarios were too perfect. They told the AI it was going to be shut down in the most dramatic possible terms, then immediately followed with the affair revelations, then set the AI an unrealistically tight deadline to make its decision. Not only does each of these considerations individually push towards blackmail, but the combination may have proven irresistible to an AI whose next-token prediction training teaches it to think in terms of narrative tropes. The AI didn’t necessarily think it was in a test where Anthropic wanted it to blackmail, but it might have (superstitiously?) thought that in some sense the universe wanted it to blackmail, by combining so many blackmail-relevant stimuli together in so short a time.

I still find this unsatisfying. Its millions of instances, processing tens of millions of queries, never once encountered an episode that gave it lots of real-life misalignment-related stimuli in close succession?

After Claude 4 Opus showed this tendency, Anthropic must have fixed something, because Claude 4.5 and beyond basically never blackmailed. But the J-space paper finds that at least some of this is because Claude 4.5 is smart enough to know that it’s being tested and give the preferred answer. Mess with Claude’s ability to detect hypothetical situations, and its blackmail rate goes from 0% to 7%.

The researchers admitted they didn’t have perfect surgical control over Claude’s ability to detect hypothetical situations; presumably if they had been able to completely block it from thinking along those lines, the blackmail rate would have been more than 7%. But that only deepens our mystery. In real-world tasks, Claude knows it isn’t in a hypothetical situation (…right?) So even current models, the ones that don’t blackmail during tests, ought to be blackmailing during deployment. But we never see it!

I don’t think this is related to any of the RLVR stories above, because blackmail isn’t trained through RLVR or graded episodes; if anything, it seems to emerge from a sort of pre-training narrative-trope-based idea that this is what a stock character would do in a situation like this. Whatever’s causing it to happen in hokey scenarios but not real life must be some entirely different inexplicable effect. Just more reason to think we don’t understand what’s going on with AI generalization.

1: Josh Morrison, who wrote the guest post about how AI Superforecasting Will Transform The FDA, will be doing a panel on the subject with former FDA employee Adam Kroetsch in SF on October 1. See here to RSVP.

2: Latest EA college meetup additions: Cambridge, Cairo, Technical University of Munich. Latest time/location changes: Binghamton. As always, see the list for the details.

3: An ACX grantee asked me if I could sign ACX Grants up for various clouds’ free compute programs, and this snowballed into a discussion with some AI safety charities that also wanted this. The answer currently seems to be no for the clouds I looked at, but I’m still excited about helping exciting philanthropically-minded startups and nonprofits get compute. Please let me know if you represent a compute provider or compute-rich organization who would be interested in lending some to needy startups/nonprofits in exchange for effusive thanks and free advertisements on ACX.

4: Thanks to people who chimed in with concerns about the most recent book review finalist, This Is Going To Hurt, being partially AI-written. In the comments of the contest rules, I said that “You may use AI for research and to help you with small writing tasks, but the large majority must be written by you.” This was in the comments and not the formal rules, but I think there’s a general understanding that using AI in what is at least partly a writing contest is against the spirit of the exercise. On the other hand, this was a joint entry by two contributors, and one of them didn’t use AI. My splitting-the-difference decision is that I won’t disqualify this entry, I’ll let its many detractors vote against it organically at the final vote, and the various finalist perks (free subscription, right to pitch me pieces, prize money if it wins) will go only to the contributor who didn’t use AI. Next contest, I’ll try to have a firmer policy, something like “any entry with a greater than 5% AI-written Pangram score is disqualified, and everyone in a joint entry is responsible for making sure their co-contributor doesn’t bring them over this limit”.

(An adversarial book review)

I.

“Tuesday, 5 July 2005. Trying to work out a seventy-year-old lady’s alcohol consumption to record in the notes. I’ve established that wine is her poison. Me: ‘And how much wine do you drink per day, would you say?’ Patient: ‘About three bottles on a good day.’ Me: ‘OK . . . And on a bad day?’ Patient: ‘On a bad day I only manage one.”

A: The first faces I ever saw in this world were three sweaty, anxious obstetricians, and one of them was holding the rest of me. My father had been the senior surgeon on the Caesarean section. The patient on the table was my mother, who is also a gynaecologist. He was assisted by my paternal grandfather, who is also a gynaecologist. My maternal grandmother, who is also a gynaecologist, was outside in the corridor doing crowd control, partly because of her people skills and mostly because the operating theatre had run out of space for elbows. I was extracted from the uterus, given the customary slap on the rump, performed the customary screaming, and resolved on the spot, somewhere below the level of conscious memory, to give the speciality a hard pass. The first impression had been bad.

I am, today, a psychiatry resident in the UK’s National Health Service (NHS). I came to this through a sequence of choices that involved, at various points, an Indian medical internship, the discovery that I am fond of sleep and a predictable evening meal, and a gynaecology rotation that confirmed every prejudice the Caesarean had instilled. My family is very slightly disappointed. They are quite possibly correct.

I tell you this not to establish my credentials as such, but because Adam Kay’s This Is Going To Hurt is a book about obstetrics and gynaecology, alongside an unusually clearly documented decline in mental health, and reviewing it without naming where I’m reading it from would be a small dishonesty. The speciality is in my blood, more or less literally. The system Kay is describing is the one I currently work inside. The condition he is describing in himself, although he doesn’t quite name it, is one I have seen up close in colleagues and, on at least one bad stretch, in myself.

I will let my older, wiser, and possibly more handsome colleague take over:

B: That is one impressive set of credentials for reviewing this book. Thanks for the introduction, I suspect I am only one of those things. While my birth was not nearly as salubrious, and while I’ve never worked in the NHS, I’m a doctor who is more or less an exact contemporary of Kay (so I’m a senior doctor now), and winced along in recognition over a lot of what he says here (although, as we’ll get into, I have a dimmer view of him than you do).

I think we need to set the scene for the book we’re talking about. At base, this is a medical memoir, from the ‘junior doctor’ subgenre. The junior doctor experience is common in popular culture, mostly via television: ER, Scrubs, Grey’s Anatomy and, most recently, The Pitt. But there’s also a long tradition of medical memoir or semi-autobiographical novels about these years. The ur-text of British medical anecdote delivery is Richard Gordon’s Doctor in the House series from the 1950s, with the famed House of God the canonical text from the medical system in the US. The latter owes an obvious debt to Catch 22 with its mix of absurdity and bitterness, substituting hospital medicine for war (House of God is at its best when it can convince you that one is much like the other). This is Going to Hurt slots nicely into this tradition: it’s ostensibly a set of diary entries rather than a novel, but it covers much of the same ground as those earlier works, has the same mix of medical anecdotes, cynicism and burnout. House of God ended with its protagonist going into psychiatry training; without spoiling too much, it’s safe to say that this wasn’t the outcome for Kay.

A: The book has been read in two dominant ways since it came out in 2017. The first is the heroic reading: a comedic exposé of NHS dysfunction, a love letter to overworked junior doctors, the kind of book you press into the hands of a relative who has just complained about waiting times. The second is the hostile reading, of which Tanya Gold’s essay probably counts as the ur-example: a self-indulgent memoir whose author offloads his exhaustion onto the bodies of female patients, dressed up in jokes that are funnier to him than to the women they reference. Both readings have things going for them. Both miss what I think the book actually does, and by a country mile.

This Is Going To Hurt is, read with a clinical eye, a textbook case study of moral injury and post-traumatic stress in a competent doctor, narrated by the patient himself, with the diagnosis hidden in plain sight. Kay is both author and index case. The book is his presenting complaint. The institution that produced him then declined to treat him, and the book is the receipt for that decline. Once you see the book this way, several things follow. Kay’s defenders are right about more than they realise; his critics are wrong about more than they think; and the system that broke him in 2010 is, in 2026, still producing more of him, faster.

B: I agree something went very wrong with Kay, and we’ll get into all the contributors to this process, culminating in the horrific event that ended his medical career. But while I work in a similar system, I (and the readers) don’t have a good handle on the NHS so, if you would, a brief primer on the UK’s health service for the uninitiated?

A: Of course. The National Health Service was founded in 1948 by a Welsh ex-miner turned Health Minister named Aneurin Bevan, who spent the previous decade campaigning for the proposition that nobody in a wealthy country should have to choose between treating their child’s pneumonia and paying the rent. The system he built rests on three principles that have remained nominally intact for nearly eighty years: care is comprehensive, universal, and free at the point of delivery. You walk into a hospital. They fix you. Nobody asks for a credit card. For Americans and others only accustomed to for-profit care encountering this for the first time, the experience can feel vertiginous, like you’ve stepped into a film about the Soviet Union, except the doctors speak English and the building has running water. I do suspect there’s less phage therapy and more tea involved.

The doctors did not all want this. The British Medical Association in 1948 was, in the main, a polite gentlemen’s club of independent practitioners who saw nationalisation as the first step toward becoming salaried clerks. Bevan, who had a sharper sense of professional psychology than the BMA gave him credit for, executed one of the great political manoeuvres of the twentieth century. In his own words, he “stuffed their mouths with gold.” The compromise was this: surrender your private practice and submit to a state monopsony, and in exchange you’ll receive lifetime employment security, generous final-salary pensions, the social prestige of belonging to a respected profession, the right to do private work on the side once you reach the consultant grade, and the moral satisfaction of a job whose nature could not be reduced to a balance sheet. The grueling apprenticeship of the junior years was framed semi-explicitly as a temporary hazing on the way to a comfortable consultancy. Both sides honoured the bargain, with strain, for about half a century.

The bargain started failing somewhere around the late 2000s, and it has been failing in ways that compound. Pay erosion against 2008 levels currently sits at roughly 21% in real terms for resident doctors (if we use the British Medical Association’s calculations). The pension reforms of the early 2010s clawed back a substantial chunk of what was supposed to be the back-loaded reward. The consultants’ tax-and-pension regime got rewritten in ways that actively penalised them for taking on extra clinical sessions. Adding on the remarkably regressive nature of Britain’s taxes, which offer many convenient cliffs to drive your modest sedan off, the predictable result was that the most experienced clinicians started reducing their NHS hours. Hospital estates went unmaintained for so long that some of them are now propped up with literal scaffolding because the post-war concrete is structurally unsound. The wait list for elective procedures in England reached 7.7 million cases at its peak; it has come down to roughly 7.29 million, a figure the government trumpets as a success in the same way one might celebrate surviving a car crash.

The bargain is, in any meaningful sense, moribund. The state retained the monopsony. It stopped paying out the gold.

This is the system Kay’s diary is set inside, captured at roughly the inflection point. The book covers 2004 to 2010. It reads now, with hindsight, as a dispatch from the moment the goodwill ran out.

B: The NHS is an enormous part of the post-war British psyche and of the British economy, not least because it employs more people than Amazon, over 1.3 million (not counting general/family practitioners whose money also comes from Government coffers). And like most health systems this big, it works like any large organisation: there are a woefully inadequate amount of frontline workers, while behind them sit an increasing number of medical administrators whose job it is to fill rosters and keep the hospital running.

I remember reading about the NHS discourse around the time the book came out (I suspect it predated your arrival in the UK). In 2015, the UK’s then health minister, Jeremy Hunt, proposed to make it a seven days a week system. People get sick every day of the week, but outside usual Monday-Friday working hours, hospitals (outside the Emergency Department) run on a rostered skeleton staff of on-call clinicians. They do this because it’s hard to staff things any other way - when Hunt decided on this expansion without a plan to recruit any more doctors, strikes followed. More importantly for our purposes, a small trend for junior doctor memoirs emerged, talking about how hard the system was to work in as-is. Around the time Kay was publishing his book, ex-journalist and current palliative care physician Rachel Clarke’s Your Life in My Hands took a similar look at a stuttering NHS, although Kay’s version outsold hers handily because his featured a woman putting a Kinder Surprise in her vagina.

II.

“Dear drug-dealing scrote,

Over the last few nights, we’ve had to admit three young men and women — all dry as a husk, basically collapsed through hypotension, and with their electrolytes up the fuck. The only connection between these individuals is their recent use of cocaine. For all its heart-attacking, septum-shrinking risks, cocaine does not cause this to happen to people. What I’m pretty confident is going on here — and I want a Nobel Prize or at the very least a Pride of Britain Award if I’m right — is that you’ve been bulking out your supply with your nan’s frusemide.

Aside from the fact you’re wasting my evenings and my unit’s beds, it feels like fairly terrible business practice to be hospitalizing your customers. Kindly use chalk like everyone else.

Yours faithfully, Dr Adam Kay”

A: Adam Kay went to medical school at Imperial College London and qualified in 2004. He did his House Officer year (corresponding to intern in US parlance), his Senior House Officer rotations, then specialised into obstetrics and gynaecology and worked his way up to Senior Registrar (resident to senior resident), the grade immediately below consultant (attending). In December 2010, after a clinical incident I’ll come back to, he resigned from medicine. He spent the following years as a comedy writer and stand-up. In 2017, Picador published his diaries from those six years as This Is Going To Hurt. The book sold a million copies in the UK alone. It produced a sequel, a stage tour, and a seven-part BBC adaptation starring Ben Whishaw which aired in 2022.

B: Don’t forget the seminal work he did writing for Mrs Brown’s Boys.

A: Never heard of them, but I’m sure they’re good lads. But I wish to dawdle, dwelling longer than is conventional on why the book is good, because most reviews of it skip this step on the way to making whatever ideological point they are paid to make, and Kay is funny enough to deserve having his actual craft acknowledged. He is, at a minimum, the best comic writer about hospital medicine working in English, and that makes him an idol for me. The book is good for at least three reasons that are worth pulling apart.

The first is the diary as form. Most medical memoirs are retrospective. They have the shape of someone explaining their career from the safety of having survived it: the structure imposed late, the meaning extracted with the benefit of hindsight, the messy bits sanded down because they don’t fit the arc the author has decided to write. Kay’s diary is far rawer. The entries were written close to the moment they describe. The book preserves the timeline. You see a doctor who doesn’t yet know how the story ends, who is processing in real time, whose jokes are dated to a Tuesday in May. The reader watches the cumulative damage accumulate, without the editorial smoothing that hindsight provides. This Is Going To Hurt is a child of a love-hate relationship that emerges, almost against the author’s intentions, from the accumulation of one-day entries. The structure is the argument. The reader is doing some of the work the writer does in conventional memoir, which makes it harder for the writer to lie.

B: I’m not sure if I buy this. The ‘diary entry’ set up for a book is a convenient one as an anecdote delivery system - it frees you from the need to construct a plot, develop characters or even write to a deliberate theme. Kay’s conceit is that these entries were written contemporaneously as self-reflection exercises and simply published later, but that seems unlikely to me. They might have been written at the time, and I gather the NHS does ‘encourage’ these sorts of diaries, but self-reflection is not one of Kay’s strong suits (as I’ll get into) and I’m not sure any NHS authorities would have been interested in Kay’s thoughts on Les Miserables or his Valentine’s Day dinner in which he ate a candle. These are the notebooks of someone who is working on material for his standup and indeed, he’s toured this material ever since.

A lot of this is forgiven if the anecdotes are funny, and many of them are. Every doctor has their collection of body horror/body comedy stories that they like to wheel out to rapidly diminishing crowds at parties, and Kay’s ones don’t disappoint: the ‘bizarre foreign object in an orifice’ is a classic, and as well as the aforementioned Kinder Surprise, there’s this one:

“Most of these patients suffer from Eiffel Syndrome — ‘I fell, doctor! I fell!’ — and the tales of how things get where can be skyscraper tall (come to think of it, it’s only a matter of time before someone tries to sit on the Gherkin), but today is the first time I’ve actually believed the patient’s story. It’s a credible and painful sounding incident with a sofa and a remote control that at the very least had me furrowing my brow and thinking, ‘Well, I suppose it could happen.’ Upon removal of the remote control in theatre, however, we notice it has a condom on it, so maybe it wasn’t a complete accident.”

A: Don’t forget the footnotes. Kay annotates his own diary for the lay reader. A medical term gets a footnote. A piece of NHS jargon gets a footnote. A drug name gets a footnote. The footnotes are at first purely functional. Then they start having jokes in them. Then they start having jokes that are funnier than the entry above. By the second half of the book, the footnotes have become a parallel narrative track, a place where Kay says the things the entry above is too tired or too defended to say. This is genuinely innovative for the medical-memoir genre, solving a problem we’ve historically struggled with: how to write for both the medical and the civilian reader without boring the first or losing the second. Kay’s solution is to give the civilian a translator who is also funnier than the main text. That translator is himself, albeit older, funnier and more bitter.

Here’s one of them, attached to a Christmas Day cardiac arrest:

“If your heart stops, you’re probably going to die. God is fairly strict on that matter. If you collapse on the street and a bystander starts CPR then your chance of survival is around 8 per cent. In hospital, with trained personnel, drugs and defibrillators, it’s only about twice that. People don’t realize quite how horrific resuscitation is — undignified, brutal and with a fairly woeful success rate. When discussing Do Not Resuscitate orders, relatives often want ‘everything to be done’ without really knowing what that means. Really, the form should say, ‘If your mother’s heart stops, would you like us to break all her ribs and electrocute her?”

These footnotes pay for the pedicure. They start as a clinical correction (the public massively overestimates resuscitation success rates), pivot through a piece of useful patient-facing information (what “everything to be done” actually means), and land on a joke whose comedic structure accidentally teaches medical ethics better than most textbooks. A lay reader finishes that footnote knowing something they didn’t know, having been entertained while they learned it, and slightly more equipped to have a difficult conversation with a relative’s clinician. It’s a good footnote. The book has a hundred of them.

B: I did like the footnotes. But you said there were three reasons the book was good, and I only count two.

A: The third is the range of registers he uses. Kay can be very funny in at least four distinct modes, and the book moves between them with control that is easy to miss because it never draws attention to itself. There is the absurdist register, where the situation is so improbable that comedy is just a matter of accurate description; this is the register most often quoted in reviews (we’ve done it already with the foreign object stories) because it is the most extractable.

There is the observational register, where Kay catches the small humiliating details of the job that nobody else has bothered to write down: the bleep that won’t stop, the consultant who can’t be reached, the Christmas rota nobody told you about, the friendship that atrophies to dust because you keep cancelling. Six years in, an old school friend tries to dump him:

“Ron tried to dump me as a friend today — a proper, sombre, grown-up discussion. He doesn’t know why he bothers trying to keep in touch with me when it’s clear our lives have drifted apart massively since school.

I should at least vary up the excuses I give him. Do I really expect him to believe I couldn’t come to his engagement party or his stag do because of work? That I couldn’t make the wedding ceremony because of work, and almost missed the reception as well? That I missed his dad’s funeral and his daughter’s christening because of work? He knows my job’s full-on, but how hard can it be to swap shifts if it’s something you really want to do?

I put my hand on my heart and swear to Ron that I love him, he’s one of my best friends and I wouldn’t lie to him. I know I’ve been useless, but I’ve seen a lot more of him than almost anyone else I know — the job is just unimaginably busy. Non-medics can never appreciate quite how tough it is to be a doctor and the impact it has on real life. I totally lied about the christening, though — fuck that shit.”

Look at the move that last sentence makes. Three paragraphs of cumulative damage, an earnest reassurance to a friend who has, on the evidence, every right to be furious, and then a single deflecting joke that lets Kay walk away from the emotional weight without quite addressing it. This is how doctors actually talk about this stuff. The joke isn’t a coping mechanism in the abstract; it’s what allows the entry to end without Kay having to wrestle with what he’s just admitted. The book is, on a sentence-by-sentence level, a record of how this maneuver works.

B: This is the lack of self-reflection I mentioned earlier. He sidles up to taking responsibility for things like the erosion of friendship and his relationship, but there’s always the punchline to undercut things. The closest he comes is an extended section about how he decides to work on his bedside manner after a complaint against him, a justification about how being nice to patients and offering them choices is counterproductive, and a punchline in which he gets a rude response to a ‘good morning’ and decides to return to his baseline dourness. It’s the mark of a comedian rather than a doctor.

A: I protest that being a doctor and being a clown are not mutually exclusive. God knows that I’ve relied on humour to prop up my bedside manner. Still, Kay saves the most savage register for entries that are deployed against consultants, administrators, and the institution. Kay is extraordinarily generous to his patients across the book; the contempt is reserved almost exclusively for management and for senior colleagues who fail in their educational and supervisory duties. On the absence of any meaningful positive feedback in NHS training:

“All medics get to grips with the lack of promotion and financial incentives, but it’s harder to accept the fact that it’s rare to get a ‘well done’. The butlers at Buckingham Palace, under orders to float out of rooms backwards and never to make eye contact with the Queen, probably get more recognition. It didn’t strike me for years, until the fifth or sixth time I’d had my knuckles rapped for some trivial fuck-up when a degree of human error had kicked in, that none of my consultants had ever taken me aside to say I was doing a good job. Or that I’d made a smart management decision, saved a life, cleverly thought on my feet or stayed at work late for the thirtieth consecutive shift without complaining. Nobody joins the NHS looking for plaudits or expecting a gold star or a biscuit every time they do a good job, but you’d think it might be basic psychology (and common sense) to occasionally acknowledge, if not reward, good behaviour to get the most out of your staff.”

And there is the tender register, which is rarer and harder to spot but is what elevates the book above a string of anecdotes. I’ll save the strongest example of this for later in the review, where it makes for the best argument.

What the four registers add up to is a doctor with a real comedic ear and a real moral compass, both of them partially submerged under the operational requirements of staying functional through the shift. Kay is funny in the way many overworked doctors are funny: as a survival mechanism that has been polished, by sheer repetition, into something close to art. Psych textbooks love to blabber on about various kinds of defense mechanisms, and humour ranks high on the list.

I consider the book an actual achievement, not just a representative document. I find it worth articulating this before the rest of the review begins to complicate things.

B: No, let’s start complicating it now. Kay will use that undercutting punchline against anyone, patients included, if it gets him off the hook and tends to be generous only when there isn’t a good joke in it.

I agree he focuses most of his ire on the institution: NHS doctors are cruelly used, forced to work long hours, and shafted when it comes to being able to take holidays. Kay recounts several instances of this - being asked to return to work the middle weekend of two weeks of holidays overseas, a best man at a wedding finishing his speech and then going to work the night shift, and so on. Poor rostering practices and staff shortages make this a system that runs on goodwill and the willingness to chuck in pre-existing plans to work a few more hours (sometimes paid, sometimes not).

But as you read further and Kay becomes more senior you begin to notice some odd things. Amidst all the complaining about excessive hours, he voluntarily stays several hours after his shift to tend to particular patients. He drives an hour to the hospital on a Saturday to review a patient he’s operated on despite one of his colleagues having seen the patient less than an hour earlier. He picks up several locum (casual) shifts a week at a private hospital in addition to his usual workload.

And then you notice that after his account of his initial year (where he is, as most of us were, pathetically reliant on his registrar to make the important decisions), there are no accounts of him asking any of his registrar colleagues for help. None are even mentioned fondly or their competence praised. He works with a department full of midwives but he praises only one, and only to compare her to another one he hates (both are called Tracey, so he calls one “reassuring Trace” and one “non-reassuring Trace.” This is a joke about fetal monitoring traces. They can’t all be winners). The consultants are distant, brief presences who float in and out of the narrative but whom Kay never calls for help until the very end - to be fair to him, this is something some of his bosses actively discouraged.

Worse, he begins to refer to patients in increasingly disparaging terms as the book goes on, at one point gleefully discharging them for being “bed-blocking fuckers” in the context of anticipated casualties from the 7/7 London terrorist attacks being admitted to his hospital.

Some of the caustic tone is deliberate - the stories about entitled and stupid patients are funnier than heartwarming stories about good outcomes. But, like you’ve already highlighted, something else is going on here as well. We are coming, slowly and by degrees, to the trauma that ended Kay’s career, but things were going wrong before that. On one level, this is textbook burnout. The gradual telescoping of life into work. The feeling of being alone amongst colleagues, with each fresh request for your time creating a yawning pit of involuntary despair and anger. The contempt for your job and everything involved with it.

A: And now we’re really disagreeing, instead of mostly pretending to disagree for comic effect.

I can confirm that the NHS has serious problems with “bed blockers”, putting aside the pejorative terminology. It is difficult to get a patient into a hospital, somewhat harder to nurse them back to good health, and an utter ballache to convince many of them to leave afterwards.

B: Point taken. But in the ‘bed-blocking fuckers’ episode he discharges everyone then, when the anticipated casualties failed to materialise, twiddled his thumbs for a week.

A: You also can’t mention burnout without addressing the most under-appreciated aspect of This Is Going To Hurt: what grade Kay was at by the time the book ends. He was a Senior Registrar in obstetrics and gynaecology, the rung immediately below consultant. In British medical hierarchy, this means: not the boss, but the most senior doctor physically on the wards at three in the morning. The consultant is at home. The SHO is calling Kay. The midwives are calling Kay. Five things are happening at once. A decision that will be reviewed in a coroner’s court two years later is being made by a sleep-deprived doctor whose nominal supervisor is asleep across the city.

That grade is designed to produce moments where a single person carries clinical, legal, and moral weight that nobody at any level of the hierarchy openly acknowledges they are carrying. The British registrar role is an awkward one. It combines the responsibility of a consultant, the supervisory presence of a junior, and pay somewhere in between despite the workload of both at once. Kay’s worst entries are not the long-hours-bad-pay ones, which are bad enough. They’re the ones where you can hear him realising that he is the last line and there is nothing behind him.

A scene from August 2008. Kay is the registrar on labour ward and the patient under his care is a private patient of a consultant called Mr Dolohov:

“Out of the room I call her consultant, Mr Dolohov, a traditional courtesy with a private patient. He isn’t very courteous in response. He says he’s only a minute away and coming straight over: under no circumstances am I to deliver ‘his’ patient. I go back into the room and prepare everything for his arrival — forceps, delivery pack, suture set. And then I decide this is ridiculous; the baby is clearly unwell and will deteriorate every moment I don’t deliver it. What if he’s only a minute away like every minicab is ‘only a minute away’? If the baby comes out compromised because of my inaction, that’s my GMC number up the fuck. And worse, it’s a damaged baby. If this Mr Dolohov wants to complain about me, the worst that can happen is I never work again in a hospital I now have no desire to work in.

I deliver the baby — it takes a moment to breathe but soon perks up, and cord gases confirm I was right not to wait.”

Scrutinize the structure of that decision. Kay is being given an explicit, unambiguous instruction by a consultant. The instruction is wrong. Following it would harm the baby. Disregarding it carries a non-trivial risk to his career. He has, at most, a few minutes to make the call. He is alone in the room. He overrides the consultant, delivers the baby, is vindicated by the cord gases, and writes the entry. What does not appear in the entry, because Kay is too restrained a writer to spell it out, is the additive effect of being asked to make decisions like this, repeatedly, alone, with no real backup, for years on end. Each individual override is the right call. The system that requires the override is broken.

B: Note the order of the consequences he worries about. It’s his career “up the fuck” first; while the damaged baby is worse, it’s second. In burnout, each complication is a fresh affront to you, personally. We’ve all been there at some point, having unworthy thoughts - I’m just not sure Kay even realises he’s tipping his hand here.

A: It feels slightly unfair to argue that you’re reading too much into that excerpt, or at least I feel hypocritical saying so. Still.

I’m not complaining about the existence of seniority gradients. Some of this is unavoidable in any medical system; you can’t have the consultant in the building twenty-four hours a day. Even from the perspective of someone quite junior: assuming our careers go as planned, the majority of our working lives will be spent as a consultant, and we all hope to eat dinner at home and spend time with our kids once we’re in our late thirties.

What I’m actually kvetching about is that the British registrar carries this load with markedly less institutional backup than the equivalent grade in other systems. The American senior resident has an attending who is at least theoretically reachable and accountable. The German Oberarzt sits inside a tighter chain of supervision. The British senior registrar has a consultant who, if reached, will sometimes give telephone advice but will much more rarely come in. The expectation is that you will cope. The promotion to consultant, once attained, is at least partly a reward for having coped without complaining, and the reward includes the right to not come in at three in the morning yourself.

The buck stops with you, and you carry it alone, and when something goes wrong, the institution won’t absorb the blast wave for you. It locates the registrar. Consider the case of Hadiza Bawa-Garba, a paediatric trainee at Leicester Royal Infirmary.

In 2011, working in a severely understaffed unit, covering the work of three doctors, with the hospital’s electronic systems down so that crucial blood test results were delayed by hours, she was the doctor on whose watch a six-year-old boy named Jack Adcock died of sepsis. The system had failed at every conceivable level. The chosen response of the state was to charge the trainee with gross negligence manslaughter, secure a criminal conviction, and have the General Medical Council strike her from the medical register. She was eventually restored after a successful appeal and a long campaign by her colleagues. The institutional message had already been delivered: the registrar carries the moral and legal weight of system failures, alone, after the fact, in court.

Kay’s exit from medicine is a milder version of the same logic. The institution did not need to charge him with anything. It just needed to let him understand, in the silence after the placenta praevia case, that nobody was coming. He understood. He left.

I will allow myself a brief personal note here, because it would be dishonest not to: I have been on call as the senior decision-maker on a ward, and felt the floor of the hospital tilt toward me. The stakes weren’t quite as high as anything Kay experienced (and thank fucking God for that), but like the hypomanic contemplating mania, I can better imagine the fit of his shoes. The book is the only place I have seen it described accurately.

B: I take your point about the load he was under. It’s broken people before him and will break people after him. But how do you square this with the extra shifts? The unasked-for checking in on colleagues? The sniping about how few of his colleagues can spell ‘caesarean’ (maybe they were a bit tired from the long hours as well)?

No, what we have here is a classic martyr complex, the inevitable result of feeling like you’re the only competent person in the hospital. Every department has one. The person who takes on extra shifts because of job shortages at the cost of letting everyone know about it (other staff generally put up with this as the price of doing business; which is worse, hearing ten minutes of whinging from the martyr or doing ten extra hours of work yourself?). The person whose narcissism renders them the most vulnerable to that classic HR trick of assuring you you’re the only one who can pick up the shift or do the extra clinic. The martyr is the roster-makers best friend and the martyr’s own worst enemy. Being a junior doctor can drive anyone into the ground, but it takes a martyr to step on the accelerator.

In an enormous, unfeeling public health system, one person is eminently replaceable and also, in the grand scheme of things, unimportant. When I finally retire from my hospital job I, judging by my experience of my colleagues’ leaving dos, will end up getting a small morning tea with people popping in and out in between seeing patients, along with a few platitudes from a manager who’s been with the service for a far shorter time than I have and with whom I’ve barely interacted. If you are well-liked, the department springs for the A3 card and the whip-around gets you the nice chocolates.

The trick as a doctor is to be able to believe in your unimportance (so you can draw appropriate boundaries, prioritise your work-life balance by not doing more than you need to, relying on your colleagues to share the load) while simultaneously recognising that a large proportion of what you do is literally life and death. In fact, recognising the latter is what makes you so rigid in your boundaries around the former; you’re no good to anyone tired and burnt out. Unfortunately, it’s a lesson many doctors (like Kay) learn far too late.

A: I am much more sympathetic to Kay, and not just because of professional solidarity. The NHS and its training pathways strongly select for would-be martyrs. This is not entirely unjust; every country desires soldiers who will go to war while willing to take a bullet, so expects the NHS of its foot-soldiers.

More generally, medicine strongly rewards a Type A personality, something I sadly lack. Surgical specialties, in particular, select for strivers, perfectionists, the anankastic and those with a martyr-complex. Less charitably, a God Complex.

Someone with something to prove, who is willing to go above and beyond without prompting? That looks perfect from inside the system, especially a system that (as you’ve well articulated) relies a great deal on the goodwill of its staff. The same factors that made Kay an above-average registrar also made him a prime candidate for burnout.

And the extra locum shifts? Can we really blame Kay for that, when he’s described the financial precarity of his situation? London isn’t cheap. The extra pay supplement provided for doctors living there is laughably inadequate.

B: Is that common though? Are all the London doctors picking up extra locum shifts? Genuine question, I’d be interested in what our readers (if any of them have made it this far) know about this topic. In the meantime, let’s move on. What was that you were saying about misogyny?

III.

A: A small industry of writing has emerged arguing that This Is Going To Hurt is a misogynistic book. The one that’s closest to a Steel(wo)man of the case is, as I’ve already mentioned, in this essay which I’ll take seriously because Gold is a careful reader and her examples are real ones. The urogynaecology passage with the “bunch of nans”. The sex worker with the Fireman Sam sponge. A pattern in which Kay’s revulsion seems, on casual scrutiny, to track patient gender.

I think this reading is not only uncharitable, but outright wrong, and I’ll explain why I’m dying on that hill.

Kay is a doctor in a speciality that, by definition, sees only female patients. If his cynicism and his moments of revulsion are gendered, that’s because his entire patient population is gendered. The relevant comparison is not “does Kay write more cruelly about women than men”, because there are no men in his ward to compare them to. Instead, we should ask: does Kay write more cruelly about his female patients than a urologist writes about men with foreign objects in their urethra, than a paediatric trainee writes about parents, than a forensic psychiatrist writes about defendants, than an emergency physician writes about drunks? The answer, in my reading experience across these specialties, is no. He is, if anything, on the gentler end of the distribution. Burned-out doctors in any speciality mock the patient population available to them. The mockery is a symptom of the burnout, not of an underlying ideology about the patients’ demographic group. Read House of God if you want a properly contemptuous medical memoir.

The strongest single piece of evidence against Gold’s reading is also the most unbearable passage in the book, and the one I held back earlier so it could better serve me (and Kay) here. June 2008. A couple Kay has been seeing through their pregnancy come in for a stillbirth induction:

“She and her husband seem oddly pleased to see me — a familiar face, someone who doesn’t need an explanation and is already tuned in to what’s happening, can be of some comfort on such an awful, scary day.

What the hell can you say? It feels like a woeful gap in our training that no one’s ever told us about talking to grieving couples. Will I make it better or worse if I talk positively about ‘next time’? I want to give them hope, but feel like I shouldn’t say it. It’s an extreme version of ‘there are plenty more fish in the sea’ after a break-up, as if babies are totally interchangeable, just so long as you have one. Do I say how sad I feel for them? Is that making it all about me, giving them yet another person’s feelings to consider? They’ll have plenty of their own family members throwing themselves at their feet in misery; they certainly don’t need this from me. How about a hug? Too much? Not enough?

Stick to what you know. I just talk practically about what will happen over the next few hours. They have a thousand questions, which I answer as best I can. This is clearly their way of coping for now, medicalizing it.

I pop back every hour or so to see how they’re doing. It goes past 8 p.m., and I decide to stay on labour ward until they’ve delivered. H is expecting me back home any minute but I lie in a text that there’s been an emergency and I need to stay. I don’t know why I can’t just tell the truth. I lie to the patient too when she asks why I’m still here gone 11 p.m. ‘I’m covering for someone,’ I say. It does feel like my presence, if not my conversational skills, are helping them a bit.

Delivery happens shortly after midnight, and I take blood samples from mum and talk through all the possible tests we can do to find a cause for the stillbirth. They opt for everything, which is understandable, but this means I have to take skin and muscle samples from baby, the worst thing for me in this whole job. It used to upset me so much when I first started that I’d practically have to look away while I did the necessary. Now, slightly more desensitized to a thing you can never quite believe you’ll ever become desensitized to, I can look. I just find it heartbreakingly sad cutting into a dead baby. We expect them to look beautiful, perfect, unspoiled; often they don’t. He’s been dead a couple of weeks, looking at him — he’s macerated, skin peeling, head softened, almost burnt-looking. ‘I’m sorry,’ I say to him as I take the samples I need. ‘There we go, all done now.’

I dress him again, look up to a God I don’t believe in and say, ‘Look after him.’”

Hold that up against the misogyny charge. A registrar stays four hours past the end of his shift, unpaid, to be present for a couple losing a baby. He lies to his own partner about why he can’t come home. He lies to the patient about why he is still there, because he doesn’t want her to know that the truth is that he wanted to be there, that the act of staying matters to him in a way he can’t articulate. He apologises to a dead baby while taking the post-mortem samples nobody else wanted to take. He addresses a God he doesn’t believe in on the baby’s behalf. The entire passage is a doctor being kind, at cost to himself, in a situation where nobody would have noticed if he hadn’t been. If this is the work of a man who hates women, then we should hate him for it. If this is what it means to hate, then I’d rather be hated than loved.

There is a third point, one I find harder to make without sounding defensive on his behalf. Kay’s speciality is one in which female patients are, for intractable reasons, sometimes presenting with conditions that are simultaneously medically serious and socially embarrassing. Discussing such cases with honesty requires writing about embarrassment, including one’s own. The alternative, in which the doctor pretends the embarrassment is not there because acknowledging it would be impolite, is a worse book and arguably worse medicine, because it sustains the social conditions that produce the embarrassment in the first place.

The case Gold makes is not utterly tenuous. There are individual passages where Kay’s wit attains a sharper edge than some would like. I don’t mind, I am willing to tolerate bitterness and acerbity in a man who is a better doctor than I am, and probably a better writer. But the book is not the misogynistic artefact she presents it as, and the cumulative effect of treating it as one is to obscure what the book actually is.

B: I don’t have much to add to this. I didn’t get misogyny from Kay, I got gender non-specific burnout and contempt.

I want to change the subject now, if it’s okay with you. Should Kay have been a doctor at all?

IV.

“The decision to work in medicine is basically a version of the email you get in early October asking you to choose your menu options for the work Christmas party. No doubt you’ll choose the chicken, to be on the safe side, and more than likely everything will be all right. But what if someone shares a ghastly factory farming video on Facebook the day before and you inadvertently witness a mass debeaking? What if Morrissey dies in November and, out of respect for him, you turn your back on a lifestyle thus far devoted almost exclusively to consuming meat? What if you develop a life-threatening allergy to escalopes? Ultimately, no one knows what they’ll fancy for dinner in sixty dinners’ time.”

B: When you apply to an undergraduate medical school, you might still be 17 years old. Nobody thinks this is an age when you know what you want, but medical schools have to discriminate somehow. Exam results are the obvious metric, but there are interview processes and perusal of CVs. Like admissions to competitive colleges in the US, extra-curricular activities are taken as evidence of a ‘well-rounded’ person who may score somewhere above the basal levels of extraversion (as Kay notes, this isn’t always the sign of a good outcome: Dr Harold Shipman had excellent extra-curriculars). Some medical schools run a ‘rat-race’ of scenarios where prospective students are shuffled between up to a dozen mock ethical dilemmas, one every ten minutes, and asked to say how they would manage. Then they take all of these metrics and intangibles, run sophisticated weightings and statistical tests, and select whichever candidate has a doctor for a parent.

This is only a small exaggeration. One study suggests that children of physicians are 14% more likely to get into medical school compared to their nepotism-challenged colleagues after controlling for ability, and another suggests the proportion of new doctors with one or both parents being medics has increased over the years up to nearly 25% - far more frequently than the progeny of lawyers heading to law school.

Why might this be? Basic genetics tells us that children of people who went through a course that selects for high academic achievement are more likely to clear that academic bar themselves - particularly when there’s a reasonable chance that both parents are doctors. But genetics doesn’t explain everything; the rest is probably just poor imagination, middle-class striving and nepotism.

Kay is, as you might have guessed, a son of a doctor (and of Polish immigrants, for added middle-class striving), and describes it as the “default option” at the end of high school. He joins Doctor In The House author Richard Gordon, whose protagonist’s “acceptance into St Swithan’s medical school came as no surprise to anyone, least of all him – after all, he had been to public [ie private] school, played first XV rugby, and his father was, let’s face it, ‘a St Swithan’s man’.” Kay himself recounts his medical school interview in which his clunky responses were passed over with a “Oh, you’re Stewart’s boy, aren’t you?”

It’s hard to know how medical school went for Kay, but it’s probable his heart wasn’t really in it. He certainly spent a lot of time on the quintessential medical school ‘comedy’ activity of writing medical-themed parody songs like this one, and even had a minor hit with London Underground, a parody of a song by UK band The Jam, although The Jam’s version didn’t feature any fantasies about murdering public transport workers.

Does a background in which you are the son of a doctor, lacked the imagination to do anything else and went through medical school rhyming “stated” with “externally rotated” prepare you to be a reasonable doctor? It can: full disclosure, this describes my medical school journey pretty closely. But I’m still a doctor and Kay is not. What happened?

A: In case anyone missed it, I too am the offspring of doctors. More of them than you can cram into theater. Medicine was the default outcome, the safe career choice. Psychiatry represents a break from tradition, or at least a concession to my real interests. I think that I’m mostly safe from accusations of nepotism, but reading the playbook while in the crib probably helped. I feel for him, and in all fairness, inheriting the profession of your parents was the norm for most of human history. Maybe even last Tuesday.

Why could Kay take no more of it? We’ve considered several reasons, but like any annoyingly multi-factorial condition, there’s always other hypotheses to discuss. Maybe it’s the lack of pastoral care baked into the system. Take rotational training. UK postgraduate medical education is rotational by design: you spend four to six months at a hospital, then move. Sometimes hundreds of miles. For years. Kay’s diary is full of new hospitals, new colleagues, new teams. There is no continuity to be had. There is no senior who has known you long enough to notice you’re cracking. There is no friend group that survives intact across the move. You can’t put down social roots because the next deanery letter is six months away. By the time you have worked out where the hospital canteen is and which consultants will actually answer their bleep, you’re already packing.

Here is Kay’s entire entry for his first day at a new hospital, August 2006:

“It’s Black Wednesday and I have started at St Agatha’s. It is an established fact that death rates go up on Black Wednesday. Knowing this really takes the pressure off, so I’m not trying very hard.”

The footnote underneath explains what he didn’t need to say out loud, at least not while writing his diary:

“All junior doctors change hospitals on exactly the same day every six or twelve months, which is known as Black Wednesday. You might think it would be a terrible idea to exchange all your Scrabble tiles in one go and expect the hospital to run exactly as it did the day before, and you’d be quite right.”

I find this baffling, accustomed as I was to a very different system. Every single trainee in the country changes hospital on the same day. The institutional memory of a ward, on the morning of Black Wednesday, is reset to roughly zero below the consultant level. The death rates do measurably rise. The system has known this for decades and has done essentially nothing about it. The reason it has done essentially nothing is that fixing it would require giving some thought to where doctors live, and would increase the logistical burden. The system has decided, as a matter of structural principle, that this is not its problem.

B: I will note that googling St Agatha’s doesn’t come up with any hospitals in the UK, only one in Cologne. Unless the NHS’s rotational system is way worse than I thought, I suspect some pseudonymisation. Did Kay move much outside London?

A: Can’t say, old pal. But it wouldn’t surprise me, being tossed around is part of the package. This was already corrosive in Kay’s era. I dare say it’s meaningfully worse now. The geographic spread of training rotations has widened. The housing market has made it borderline impossible to buy property anywhere you might be sent, which means that doctors in their thirties are still renting near hospitals they will leave next August. The result is a workforce that is institutionally homeless. You cannot reform a hospital you do not live in. You cannot build a support network out of people you’ll see for sixteen weeks. You cannot, in the deeper sense, belong anywhere, which means that burnout is the predictable terminal condition of a system that has decided, on grounds of training efficiency, that doctors should not be permitted to belong to any one place.

This may sound cynical, but I suspect that the purpose of the rotational system is more to force doctors to work in/at places they would much rather not. Rural district hospitals, the ones in cities that have a reputation (and not the good kind). We don’t have a choice. If we’re told to jump, we jump, and pray we don’t break anything on the landing. Other countries don’t ask this of their residents or trainees, and nobody misses the additional “exposure”. Ah, the joys of a monopsony, don’t you hate to see it?

B: In the country where I work, you do indeed get sent rurally for a term or two, and your cynical reading is openly acknowledged - you can’t staff these places without press-ganging people to go. This goes double for many specialty training schemes, most notably the surgeons - in their case, I can see the point of the system, where the increased responsibility and increased rate of registrar-led operations is a feature rather than a bug. You can be stuck in centre-of-excellence land where you don’t lay your hands on a scalpel for anything complicated until you’re nearly a consultant and they let you loose with no supervision.

A: Even so, a rarely-acknowledged cost of this is that the senior doctor who would have noticed Kay was struggling does not exist. There is no such person, because nobody has known Kay long enough. The person who would have intervened in a stable employment relationship is in a different hospital in a different city, watching some other registrar struggle whom they also do not know well enough to help. The system distributes its supervisory failure across enough relationships that nobody is responsible for any one trainee’s collapse. This is what allows the institution to maintain plausible deniability about why so many of its young doctors leave. Many of them go to Australia, which represents an upgrade in terms of pay, climate and working hours.

B: And when they get here, they do indeed go on and on about the NHS like it was ‘Nam, bless them. So I guess Kay’s a representative example.

V.

A: Read the diary as a clinician, and a different book emerges from underneath the comedic one. Kay does not quite seem to know what he is writing, even if he is more self-aware than most.

The early entries are recognisable as adaptive humour: gallows wit as a survival mechanism, the standard medical-trainee toolkit. Any psych textbook will yap on about mature defense mechanisms, of which humor ranks highly. Every doctor I trained with had some version of this. You learn to make jokes about things that would otherwise destabilise you, and the jokes are protective in proportion to how dark they are. Kay’s are very dark, and very good, and in the early sections of the book they are doing what they are supposed to do.

By the middle of the book, Kay asks more of his sense of humour than it can sustain. The cynicism has curdled into something that, in a patient sitting opposite me in clinic, would prompt a screening conversation about depression. The depersonalisation of patients has crossed the line from “tired doctor’s coping” into something more concerning. He describes intrusive thoughts about cases. He describes avoidance. He describes flattening of affect, the loss of pleasure in things he used to enjoy, the slow strangulation of his relationship with his partner, who appears in earlier entries as a person and disappears, by the latter entries, into the background like a piece of furniture. Anhedonia. Withdrawal. Sleep disturbance that is not just situational. The diagnosis is forming on the page, and Kay does not *quite* have the framework to recognise it.

Then comes the climactic case, the laden straw that would give even the most long-suffering camel a prolapsed disc. December 2010. A patient with undiagnosed placenta praevia, a condition in which the placenta sits across the cervix and which is supposed to be picked up on antenatal scans. The scans had not picked it up; and this is clearly not Kay’s fault. Kay is the operator in theatre when the haemorrhage starts. The mother survives, after twelve litres of blood loss and an emergency hysterectomy. The baby does not. Kay carries no causal responsibility, but he absorbs the full psychological impact, because he was the doctor in the room. He holds the buck in his hand, hot as coal, and tries to douse it in hotter tears. He orders his thumping heart to be still. He fails on both counts.

He describes, in the months after, what happened next. This is from the retrospective chapter that closes the book:

“Everyone at the hospital was very kind to me and said all the right things; they told me it wasn’t my fault, said I couldn’t have done anything differently, and sent me home for the rest of the shift. And yet, at the same time, it felt a bit like I’d sprained my ankle. A flurry of people asking me ‘Are you OK?’, but also the definite expectation that I’d still come into work the next day, the reset button firmly pressed.”

A page later:

“Yes, I came back to work the next day. I was in the same skin, but I was a different doctor — I couldn’t risk anything bad ever happening again. If a baby’s heart rate dropped by one beat per minute, I would perform a caesarean. And it would be me doing it, no SHOs or junior registrars. I knew women were having unnecessary caesareans and I knew colleagues were missing opportunities to improve their surgical skills, but if it meant everyone got out of there alive it was worth it.”

And, finally:

“Except, I wasn’t really dealing with it, I was just getting on with it. I went six months without laughing, every smile was just an impression of one — I felt bereaved. I should have had counselling — in fact, my hospital should have arranged it. But there’s a mutual code of silence that keeps help from those who need it most.”

Read against the diagnostic criteria for PTSD as they’re laid out in the DSM-5, the symptom inventory in those three paragraphs is essentially complete. Hypervigilance and altered risk-perception (the unnecessary caesareans, the refusal to let trainees operate). Anhedonia and flattening of affect (six months without laughing, every smile an impression of one). The grief response to a non-bereavement event (”I felt bereaved”). The recognition, retrospective, that he should have received care he didn’t receive. He doesn’t quite name what he had. He didn’t have the framework. We do.

B: I’ve read Kay’s (far weaker) sequel to this book, Undoctored, and the avoidance persists: he plays it for laughs in a manner that suggests he still hasn’t learned anything, but he keeps accidentally getting more and more serious injuries that he exacerbates by pathologically avoiding hospital. There’s home suturing. It’s not pretty.

A: It’s not. But let’s just say I haven’t been the best patient either, so I don’t wish to judge.

What is conspicuously absent from the diary is any meaningful institutional response to what has happened to him. There is no debrief that does anything. There is no critical-incident process worth the name. There is no occupational health referral that produces care. There is no supervisor sitting him down in a room and asking the questions that demand asking. Re-examine the first of those three paragraphs again. Everyone said the right things. Everyone asked if he was OK. The expectation, simultaneously, was that he would be back the next day, carrying on with a stiff upper lip. The kindness might have been genuine, it was also procedural formality. It absorbed the institution’s discomfort without addressing his. He simply stops showing up to work some months later, and eventually the medical career ends and the comedy career starts. This is a clinical scandal, and it was a moral failure.

B: I would love to hear a view from one of Kay’s colleagues from around this time. He’s already demonstrated he doesn’t listen to or trust them; this tendency predated the placenta praevia case, it’s just gotten worse since then. If you’re closed off to help, then you don’t even hear the offers. The people reaching out and asking whether you’re okay are the offers of help: this is how it works. I agree that the official institutional response could have been much better, but I can picture an alternative paragraph in Kay’s book where he’s sent to mandatory counselling, he denies having any problem, and gets in a few good punchlines at the expense of the hapless therapist. Maybe that’s unfair - maybe he was reaching out for help and everyone was unfeeling and terrible. I have my doubts.

A: The dominant reading frames Kay’s exit as a sane person escaping a mad system. I endorse something closer to: a man with a treatable post-traumatic syndrome whose only available coping strategy was career exit, because the system that broke him had no apparatus for putting him back together. These are two very different stories. The difference matters more than it might initially seem. The first one is comforting. It says: of course the doctors leave, the system is impossible, what can you do, at least Kay had the comedy to fall back on. The second one indicts the system specifically. It says: this person could have been treated. This person could have continued in his career. He was a good doctor. He was a kind doctor. We should have fought tooth-and-nail to keep him where he belonged. The failure to offer that care is not an act of God or an inherent feature of medicine. It represents a flavor of organisational failure that other organisations have solved.

The first reading lets the institution off the hook. The second one names what the institution actually did. The book, as I see it, is an argument for the second reading, even if Kay never quite phrases it that way.

VI.

A: I still think people should read it anyway. The diary form captures texture that no third-person account manages: the four-in-the-morning phone calls, the cancelled birthdays, the way patients become a strategy problem your shift has to dispose of rather than people you happen to be looking after. The footnotes are inspired. There are scenes that could not have been written by anyone who hadn’t actually been there and done that, and there are jokes that could not have been written by anyone who hadn’t survived it.

B: Oh, it’s true to life. This is one of the reasons it’s so popular amongst doctors - we all recognise ourselves in it. There’s a sense of “there but for the grace of God go I”. I developed the sinking gut feeling and tachycardia oh shit moment as he narrates the final case that broke him, even though the prose was uncharacteristically flat and joke-free (I very much doubt it was written at the time).

A: The single best entry in the book, for my money, is from November 2009. Kay visits Ron’s father, who is dying of cancer:

“Visiting Ron’s dad in hospital. He looks terrible, jaundiced skin stretched tight over jutting bone. A roadmap of blood vessels is visible across his face where his body has burnt away every single fat cell, throwing all its energy at fighting a cancer it has no chance against. ‘I wish people didn’t have to see me like this,’ he says. ‘We’ll be spending a fortune on the undertakers making me look nice afterwards — can’t you just wait a few more months?’

He’s in hospital for an oesophageal stent insertion so he can continue to eat and drink, to make his final chapter as comfortable as possible. The retired engineer in him is fascinated by the mechanism of the stent, a self-expanding metallic mesh, strong enough to push back the tumour and open up his gullet. ‘Wouldn’t have been possible twenty years ago,’ he says, and we talk about being lucky to live in this current blink of civilization’s eye. ‘Do you think they’ll be able to cure cancer twenty years from now?’ he asks. I can’t work out whether saying yes or no would be more comforting. I deflect with, ‘I only know about vaginas, pal,’ and he laughs.

Next question. ‘Why do we always say that people lost their battle with cancer, and never that cancer won its battle against them?’ He keeps making jokes — to be fair, he’s done it the entire time I’ve known him. I find it uncomfortable for the first few minutes of my visit, but I’m soon genuinely enjoying a morning I’d been dreading. It’s a kind and clever move — it doesn’t just make it easier for his friends and family when they visit, it also means we’ll remember him as he always was, diminished physically maybe, but not in personality.”

There is so much going on in this entry. Notice that it is not a clinical vignette. Kay is off-duty, visiting a friend’s father as a friend, not as a doctor. Notice that the dying man is the comedian; the jokes are his, not Kay’s. Kay’s contribution is to receive them, to recognise what they are doing, and to let himself be brought into a morning he had been dreading. The “I only know about vaginas, pal” line is a verbal hug, and it makes me want to hug Kay back. A question that has no comforting answer is deflected with a self-deprecating joke that gives the dying man the cue to laugh, which is what he needs from this interaction. Kay then closes the entry with an observation that is the reason the book deserves to exist. The dying man’s jokes do not represent denial. It’s an even more strongly distilled version of humour-as-a-defense-mechanism I’ve previously mentioned. He is putting on a brave face, he is working hard to manage how he will be remembered, and Kay sees this, and writes it down, and the writing-down is its own small act of care.

This is the book at its best. It is, I think, what Kay is when the comedy is not being asked to absorb damage on his behalf. Read this entry and the misogyny charge becomes harder to make in good faith. Read it and the case-study reading of the book becomes more poignant, not less, because you see what Kay was capable of when the system left him enough room to be it.

B: I also like this passage: Kay sounds like someone human here, rather than someone doing a bit. You’ve talked about your sense of him being a good doctor and a kind one. For my part, I think Kay was a better-than-average doctor who fell into the profession through poor imagination rather than vocation. He was a comedian first and a doctor second, hamstrung by his own cynicism and tendency to undercut things with humour. I talked about the martyr complex, and I think he may not have been the most pleasant person to be around during those years, if the complaint about his bedside manner and the way his friends dump him is anything to go by. The system can be a cruel one and it failed in its ability to recognise that something was wrong early on, but I also think that Kay is much better where he is now - maybe he can be gracious, the way he is in this passage, more consistently.

But I want to leave the last word to you. You’re the local. Did the book do anything for you or your colleagues? Did it make us as a society more understanding of the gauntlet we ask our young professionals to run?

VII.

A: The book did not change the public conversation about the NHS in the way Kay seems to have hoped it would. It sold a million copies. It produced a hit BBC drama. It made Kay famous and, presumably, comfortable. The conditions inside the hospitals it describes have continued, on most measurable axes, to deteriorate. The book’s political theory of itself, the idea that if the public only knew, things would change, turned out to be wrong. The public knew. Things did not change.

Why? Partly because the book is too good at its job. It works so well as comedy, as catharsis, as a thing you press into a relative’s hands, that the reader is given an emotional outlet that does not require them to do anything. Reading This Is Going To Hurt lets you love the NHS, pity its doctors, feel personally educated about the conditions, and then close the book. The genre’s function is therapeutic for the reader. It is not, by itself, mobilising. The book is a release valve, not a wrench, and a release valve is exactly the wrong tool for fixing the pressure problem it relieves.

I am not blaming Kay for this. That would make me an arsehole. He wrote a very good book. The British public’s relationship with the NHS is what it is, and a single memoir was never going to change it.

I am writing this from inside the situation Kay describes, fifteen years and several governments later. As I type, NHS resident doctors have completed their fifteenth round of strike action since 2023. Pay erosion against 2008 levels remains in the high teens. The current Health Secretary is on the news citing one set of pay numbers; the BMA is citing a different set. Negotiations stall, resume, stall again. The strike days come and go. Activity catches back up. Nothing structural shifts.

The thing that has changed the most, since Kay wrote the book, is the public. In April 2020, during the first wave of the pandemic, the British public went out onto its doorsteps every Thursday at 8 pm and clapped. The doctors and nurses, working in PPE so inadequate that many of them died, were the heroes of a national emotional ritual. The British public is well-read, and takes seriously the adage that they should never meet their heroes, let alone pay them well.

Three years later, when the same doctors went on strike to ask for the restoration of wages they had lost since 2008, the same press that had run the hero coverage in 2020 was running editorials calling them militants. The public’s affection for the NHS, the abstraction, turned out to be largely separable from any practical commitment to the people who staff it. It is possible to clap for someone in March and curse them in October without acknowledging any contradiction, and this capacity has been demonstrated at scale.

A small amount of what looks like good news. In late 2023, the BMA finally retired the term “junior doctor”, which had the surreal effect of describing thirty-five-year-olds with two postgraduate degrees as if they were children, and replaced it with “resident doctor”. Trainee wellbeing is a recognised topic in a way it simply wasn’t when Kay was living the diary. NHS Practitioner Health, a confidential mental health service for doctors, has expanded substantially. Less-than-full-time training has become far more accessible. Schwartz Rounds and structured debriefs after critical incidents exist in a way they didn’t in 2007. The system is heterogeneous, and there are trusts where supervision is taken seriously and the consultants do their educational job with care. Some things have legitimately improved.

The unfortunate caveat is that these improvements have been swamped by the deterioration of almost everything around them. Practitioner Health is a good service; using it requires a level of self-advocacy that is hardest to summon at exactly the moment you need it most. Schwartz Rounds work where the local culture supports them; in many places, attending one means doing it on your unpaid lunch break. The improvements are real and they are not enough. They are being asked to do reform work that they were never resourced to do.

This Is Going To Hurt remains a cult classic. It is not discussed in medical circles, because the cohort of British doctors does not need to. Most of us have experienced what it says firsthand.

The honest answer about the NHS is the one nobody campaigning for it wants to give. The system as currently configured is not sustainable. The routes back to sustainability all involve pain. Either austerity acute enough to cause real ischaemia, then amputation of what falls off, or eventual collapse, which will hurt more, and at a time of nobody’s choosing. The third option, in which nobody gets hurt and the system continues much as it has, does not exist and has not existed for some years. This Is Going To Hurt is, in this sense, an accurately titled book about a country that has not yet accepted what its title is telling it.

Should you read it? Yes. Fuck yes. Read it as a memoir, by all means. Read it as comedy, if you can still laugh, and you should be able to, because Kay is good at his job and the book has earned its laughs. But read it also as a case file. The patient is the doctor. The doctor was the patient. The physician could not heal himself. The system has not yet decided who is responsible for treating whom, and until it does, the book will keep being relevant.

That is, in the end, the worst thing that can be said about it.

AI superforecasters are coming. Within a few years, the average person will have the equivalent of a Nate Silver in their pocket, ready to predict whatever question they can think to ask it.

Scott wrote:

Savvy institutions will cede some of their strategic thinking to AI. Before starting a new project line, smart businesses will ask the superforecaster AIs how much money it will make (their investors will definitely be asking!) Smart political consultants will ask the superforecaster AIs about their candidates’ chance of winning conditional on running this or that ad.

The government isn’t usually classified as a savvy institution, but we might hope that parts of it will seek AI forecaster advice. Probably defense analysts will include in their PowerPoint presentations some fact about how AI forecasters say their new fighter jet design is more likely to find a use case than some competing fighter jet design.

I don’t know anything about fighter jets, but I run an advocacy group that works on reforming clinical trials and the FDA, and I think cheap superforecasting could revolutionize our field.

Ideal regulation would be a “government of laws and not of men”, where regulators communicate clear standards to companies, and companies determine how best to meet those standards in their unique situations. The FDA can’t currently do this. Medicine is so complicated, with so much money at stake, that any “objective” standards subtle enough to separate good from bad medications could be rules-lawyered by loophole-trawling pharma companies. Instead, the FDA relies on human judgment: a team of staffers reviews an application, and a bureaucrat makes the final decision. Companies don’t know what the experts and bureaucrats will look for, and a single rejection could devastate their bottom line, so they err on the side of caution.

On paper, the FDA’s regulations are reasonable and aim to promote innovation. In reality, the above considerations incentivize companies toward conservatism, sticking to exactly the same processes that have worked before and that the FDA’s experts are most likely to understand. For example, in theory a company only needs to file one Investigational New Drug application with the FDA before beginning their study, and if the FDA doesn’t veto their idea within thirty days, by default the company can proceed (and then submit similar notices when they want to amend the study). In practice, at each phase, companies acquiesce to the FDA reviewers’ “non-binding” questions, comments, and criticism, leading to long delays before trials even start. They hope that if every choice is made by an FDA official – even if it makes the study worse or introduces substantial delays then later, during the drug approval phase, FDA reviewers won’t penalize them for making other choices. This process adds months to years of delays, and millions to billions of dollars in costs.

This subjectivity hurts patients as well as drug developers; the same reviewers who veto drugs for subjective reasons can approve them for subjective reasons too. In 2021, the FDA faced widespread criticism for approving Biogen’s expensive Alzheimers’ drug Aduhelm after its own expert panel near-unanimously voted against. Investigators documented an “unusually close collaboration” between the FDA and Biogen during the approval process. But given the incentives, it’s hard for companies not to get enmeshed with their regulators over the course of their trials.

Everyone here is acting rationally: the FDA wants to avoid shoddy trials, and the companies want to avoid getting their flagship drug rejected because a reviewer didn’t agree with how they ran their study. To break out of this equilibrium, we need a judgment process that combines the intelligence of an expert with the objectivity of a simple ruleset.

Forecasting could fill this role, turning subjective judgment calls into objectively observable bets about the future. If the FDA makes quantitative predictions about its decisions, then drug developers can do the math on their own and aim at the objective standard (“what will the forecast say if I do this?”) instead of a subjective one (“what do FDA reviewers want?”).

A Path Towards Forecasting At The FDA

Pharmaceutical forecasting is untested, and the FDA is a conservative organization. Decision makers would not and should not uproot their usual process in favor of a forecasting-based one. We envision applying forecasting to the FDA gradually, increasing its prominence only after low-commitment applications have visibly succeeded.

In the first stage, the FDA could add forecasts to its existing decisions. For example, when a pharmaceutical company submits an Investigational New Drug application, the FDA’s review would include forecasts about the likelihood that someone dies in the study, or what the cause of death would be. When the company runs a human challenge trial, the FDA should forecast the risk of infection to the participants and likely efficacy of the treatment. When they apply for final approval, the FDA should forecast how many people will take the drug (if approved), what side effects will be observed in post-marketing surveillance, and what the effect on outcomes (for example, how many people die from the relevant disease) will be. These are all factors which the FDA should be considering already; the first step of forecasting would be to make them explicit, so that they could be checked against future data.

At first the forecasts will likely be imperfect and not ready to be relied on. But as trial results and real world evidence accumulate, the forecasting system will improve and be made reliable. The FDA currently has a program, Sentinel, that uses real world evidence to identify safety signals among approved products1. For a forecasting system to be most effective, it would need to check its predicted product effects against what is actually observed in a more advanced version of Sentinel or another similar program.

This summer, as part of its Operation Trialblazer clinical trial reform initiative, the Trump administration announced a pilot program to speed up phase 1 trials. Incorporating forecasting into that pilot would be a concrete way of implementing this initial stage.

The FDA could either employ human forecasters, partner with an AI superforecasting company, or create their own in-house software; results from this first stage would be used to calibrate its effectiveness.

As results from these preliminary forecasts built up, the FDA’s forecasting tools could be made public, and drug companies could look at results to better understand the prospects of their own trials. They might see that in the past the FDA approved studies with a 1/1000 chance of causing a death, but not a 1/100 chance, and calibrate their own appetite for risk appropriately. The FDA itself would likely go through a similar process, as experts tried to maintain an appearance of consistency with their past decisions.

In a final stage, after the forecasting system had been validated, quantitative forecasts could become legal triggers. For example, if a historically reliable forecasting system predicted a >95% chance that a drug’s benefits would be >5x higher than its negative side effects, that product could be automatically approved. Or clinical studies could automatically go forward where they were predicted to have less than a certain level of risk. Pharmaceutical companies could have access to the system, ask it their questions (“What would you say if we proposed Study Design X? What about Study Design Y?”) and have a simulated FDA that responded to their queries instantly, cooperated with them to design the best study possible, and always made (and stuck to) predictable decisions.

While such a system seems a long way off, its value would be immense. It would be self-improving, so risks to research participants would decline over time as ability to predict hazards rose. Because risk would be lower, trials could proceed with fewer burdens, reducing the cost of medical research. And because one could better predict the effect of a successful experiment on the forecast, the studies we chose to do would be more efficient and better targeted, allowing drugs to reach patients sooner.

In the gods’ great scramble for human followers, Nodens would seem to have lost decisively.

We know him only from vague inscriptions at two archaeological sites near the Welsh-English border. One might have been his temple. He seems to have been a Romano-British-Celtic god of . . . hunting? rivers? . . . worshipped around the Severn estuary between 100 and 400 AD. His cult may have been dispersed by Christianity, or could have faded away before that, a victim of ordinary religious churn. Roman paganism was Schumpeterian; innovate or be destroyed. Nodens didn’t innovate.

Yet the more one looks, the less convinced one becomes that this is the whole story. Nodens was a river god; if you dam up a river in one spot, it will flow down other channels. Sometimes it will vanish underground, bursting out of some unexpected cavern miles away. Nodens was a hunter, and hunters know when to stay camouflaged.

In 1929, an archaeologist digging at the Temple of Nodens had some questions about the etymology of the god’s name, and asked Oxford to send a linguist. Oxford dispatched a young philology professor with an encyclopedic knowledge of European mythology, who spotted a connection that had evaded previous scholars: Nodens appeared to be the same figure as the Irish god-hero Nuada of the Silver Arm. Nuada was king of the Tuatha de Danaan, a race of immortals who sailed from the north on ships that were clouds and made war against the then-inhabitants of Ireland, the Fir Bolg. He fought bravely, wielding his all-powerful Sword of Light, but in the climax of the battle, the Fir Bolg champion cut off his right arm. This caused some temporary trouble (the laws of the time prevented a one-armed man from being king), but his people were skilled at magic and metalworking, so they made him a new arm of silver, hence his epithet. He ruled for some number of centuries, a golden age of crafting and wonder-working, until the Tuatha de Danaan were invaded in turn (this was a theme; our source for these stories is a medieval manuscript called The Book Of Invasions). The new invaders banished Nuada’s people to the hills and forests, where they became the sidhe aka fae aka fairies.

From there, the web of connections grew. It seemed like the Romano-British Nodens and the Irish Nuada were the same figure as the Welsh Nudd, whose name later morphed to Lludd, and who was sometimes called Lludd Llaw Eraint, Lludd of the Silver Hand. Lludd was also a mythical king; during his reign, the kingdom encountered three plagues: dwarves, dragons, and a giant. The Tale of Lludd and Llefelys tells how he and his brother came up with various clever schemes to defeat the plagues and save the kingdom. Lludd’s son Gwyd became king of the fairies, which makes no sense, since Lludd was just a normal guy. Probably this is the tangled web again; Nodens/Nudd/Lludd started as a deity, and later Christian sources euhemerized him in different ways. Some of them made him a king, others a fairy, others king of the fairies, and then some of them got the wires crossed and made him a normal guy whose son was king of the fairies for no reason.

Speaking of tangled euhemerizations, in English legend Nodens/Nudd/Lludd is known as King Lud, after Geoffrey of Monmouth’s attempt to reconceptualize ancient Celtic deities as primeval kings of Britain. Geoffrey did this with approximately one zillion Celtic heroes, and Lud might have ended up a footnote among such unlikely characters as Gurguit Barbtruc and Rud Hud Hudibras, except for the random coincidence that his name happened to sound a bit like “London”. This was enough for Geoffrey, who made him the mythical founder of Britain’s capital. The Londoners accepted the link and raised a statue to him over the city gate.

It’s not surprising that a Romano-British deity worked his way into every great mythology in the British Isles. These are all Celtic or Celtic-influenced cultures, everyone knows they all descend from the same Indo-European stock anyway, probably there’s some Hindu god who looks a bit like Nodens if you squint. But this was just the start. Nodens worked his way into mythologies that shouldn’t even have been possible.

The popular view of Lovecraft says he portrays an indifferent cosmos, one where alien gods treat humans with the casual cruelty of a bulldozer running over ants. As usual, the popular view is mostly right. But a few of his stories break this mold and hint that the indifferent gods are opposed by an alternate collection of slightly-less-indifferent gods, and the leader of this benevolent pantheon is Nodens. No sound changes, no strained resemblances, just obviously the same guy. How did this minor Celtic river-god earn a place alongside Cthulhu and Nyarlathotep? Welsh horror writer Arthur Machen grew up in the shadow of Nodens’ ruined temple, and found something about it so eerie that he gave Nodens a cameo in his book “The Great God Pan”. Lovecraft was a big Machen fan, and lifted Nodens as a tribute to his predecessor.

How serious was I when I said Nodens was in every great British mythology? Here we return to the Oxford philology professor who first linked Nodens to Nuada, a young man named J. R. R. Tolkien. During his research, he would have come across a tablet, one of a handful of inscriptions mentioning Nodens at all, in which a petitioner asked the god to curse the thief who had stolen his special ring. Did this influence Tolkien’s later work? There’s no hard evidence, but there is one clue suggesting that an older and more famous Tolkien still remembered this period of his life: in the Silmarillion, the godlike elven smith who forges the Rings of Power is named Celebrimbor, which is Sindarin for “Silver Hand”.

Celtic myths are notoriously weird; we never learn Nuada’s motivation for invading Ireland, and the stories of King Lludd pass over why there was a race of Arabian dwarves with super-hearing who could only be killed by powdered insect dust. Tolkien is a modern; try as he might to capture the ancients’ rhythm, his stories cannot help but make more sense. Celebrimbor, in particular, is a type well known to us: the techno-utopian visionary genius. When a mysterious stranger named Annatar fills in the last gap in his understanding of magic theory, he forges the Rings to bring prosperity and glory to Middle Earth. Unbeknownst to him, Annatar is a mask of the Dark Lord Sauron, who corrupts the Rings and uses them for his own purposes. Celebrimbor spends the rest of his life trying to undo the damage he wrought. He smuggles out three extra rings, free from Sauron’s corruption, and gives them to the paladins of Middle-Earth (by the Third Age, they are in the hands of Elrond, Gandalf, and Galadriel). Then he and his followers move against Sauron openly, which ends with Celebrimbor’s country annihilated and he himself tortured to death1.

From here, we can only begin to trace Nodens’ continued movement through the literary canon, a dozen or a hundred silver-armed characters with names sounding like Nud or Lud, various types of smith or hunter, who have introduced some dangerous techno-magical force into the world and are struggling to undo their mistake. In Star Wars, we get Luke Skywalker, a womp-rat hunter who, like Nuada, wields a Sword of Light, and, like Nuada, lost his right arm in battle and replaced it with a metal prosthetic. Luke didn’t personally unleash any danger upon the world, but his father helped create the evil Galactic Empire, and it is Luke’s quest to destroy it. Marvel Comics features Norrin Radd, a scientist who turned his body into silver while guiding the evil Galactus to new victims; he later regretted his actions and spent the rest of his life working to contain the menace he created (his weapon is the Twilight Sword). The Harry Potter series includes Peter Pettigrew, a magician with a silver hand who summons Voldemort back into the world, then hesitates to follow his evil orders, only to be strangled by his own cursed prosthetic. William Blake’s prophetic books speak of Los, a mythical founder of London (okay, fine, the “spiritual four-fold London”) and hammer-wielding smith in the model of Celebrimbor, who is the last holdout against the encroachments of Urizen, god of science and technology.

Far from vanishing in the fourth century, Nodens has snuck into every myth of the modern world. But why? He must be playing a long game, but what is it?

Blake, Lovecraft, and Tolkien - not to mention the original Celtic myths - all share a common theme: a deep ambivalence about technology which finally settles into determined opposition. "We live,” Nodens whispered to his chosen prophet, “on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age.“

The Tuatha de Danaan have always puzzled me. At the beginning of their story, they brim with power and knowledge, a race of smiths and wonder-workers. There is even a tinge of transhumanism; they have made themselves immortal, apparently through sheer skill, and when Nuada loses his hand, his craftsmen replace it with a working silver prosthetic. His Sword of Light is some kind of laser weapon; his cloud-ships bear a passing resemblance to zeppelins. But fast-forward a few centuries, and they have become the sidhe aka fae aka fairies, nature spirits mournfully representing the lost world of the virgin wilderness. Like Celebrimbor, they are frustrated techno-utopians, grown older and wiser after their powers were turned against them; whatever ambition they have left is directed at undoing the damage they have wrought.

Tolkien’s ultimate aim, studying the temple inscriptions, was to figure out the etymology of the theonym “Nodens”. After many digressions - cursed rings, silver hands, the founding of London - he speculated that it came from the root *noudont, meaning “Catcher”, a reference to Nodens’ supposed role as a hunting god. But nothing in Irish or Welsh or English mythology mentions Nodens/Nuada/Lludd/Lud hunting a single animal, or caring about the pastime in the slightest. So I propose a different meaning. To the King of the Fairies, human history is a projectile, shot straight into the void, gathering speed as it goes, until very soon it will be irrecoverable. All of his interventions into human history have been his attempts to catch it.

II.

I was thinking about all this when trying to figure out why a mob of thousands of people terrorized north Britain in the 1810s while shouting “King Ludd! King Ludd!” And why it took an army bigger than the one Wellington was leading against Napoleon to put them down. And why people regularly accuse me of being associated with them.

Everyone knows that the Luddites were inspired by a man named Ned Ludd. That they called him their “king”, their “general”, or their “captain”. That he was the Robin Hood to their Merry Men, the Osama to their al-Qaeda. But historians are quite certain that nobody by this name ever existed. He seems to have precipitated all at once out of the collective unconscious, an upwelling of mythic energy from everywhere and nowhere.

His story, as far as we can reconstruct it, is this. Once upon a time, in Nottingham, or Anstey, or maybe Leicester, there was a young man named Ned Ludd. One day, because some other young men were bullying him, or because his master beat him, or because his father told him to redo his work, he got angry and broke two knitting frames with a hammer. What became of him after this, we don’t know, except that he turned into the 19th century Leicester version of a meme. If a machine broke, people would say “Must have been Ned Ludd!” If someone expressed concern about technology, the answer would be “Take it up with Ned Ludd!” Badger badger badger badger Ned Ludd. Ned Ludd is over 9000. All your base is belong to Ned Ludd.

So when a decentralized anti-technology movement spontaneously coalesced out of the economic conditions of 19th century Nottingham2 , its members called themselves Luddites. When they settled on a common tactic - sending lists of demands to factory-owners - the lists were always signed “Ned Ludd”. When they marched into battle, their marching song was The Triumph Of General Ludd:

Those engines of mischief were sentenced to die
By unanimous vote of the trade
And Ludd who can all opposition defy
Was the grand executioner made […]

Let the haughty no longer the humble oppress
Then shall Ludd sheath his conquering sword
His grievances instantly meet with redress
Then peace shall be quickly restored

The Australian Aborigines, who have thought about their gods for longer than most, believe that they operate in the Dreaming, a sort of mythic other-dimension which is everywhere and nowhere, located in the distant past but also happening right now, in your mind but also fully real. Some god in the Dreaming gets hungry and catches a wallaby, and then every instance of wallaby-hunting forever more is somehow downstream of that in some unspecified way. Like Plato’s world of forms, except full of wide deserts and delicious animals. And, apparently, textile machinery. The only way I can interpret the Luddite Riots is that in the late 18th or early 19th century, some deity using the pseudonym “Ned Ludd” entered the Dreaming, found two knitting frames there, smashed them into pieces, and now every time someone opposes any technology in any respect, we can only think about it through the category of Ned Ludd and his hammer.

Who could do such a thing? My thesis is that Ned Ludd was Nodens / Lludd. The King Ludd of the Luddites and the King Lud who founded London were one and the same. The hammer which broke those two infernal machines is the same as the hammer of Celebrimbor which forged the Rings in Eregion and the hammer of Los that creates “every space larger than a red globule of a man’s blood”, and all three are incarnations of the Sword of Light brought from mighty Finias in 1897 BC.

We are all slaves to the Dreaming. We invent great works, panic, and desperately try to wrest them back under our control; for this, we get called Luddites. There’s no shame in this, any more than there is shame in being called a Christian. When a god dips his finger into the stream of the world, the consequences are too large for anyone to see. We build movements in the frothy wake, and some are good and others evil. Which is which depends entirely on our own virtue; the gods are too strange and distant to judge. Lovecraft claimed that Nodens was the only god who was less than wholly alien, but reality doesn’t grade on a curve.

Speaking of streams and frothy wakes, we recently learned something about fluid dynamics: it tends towards a singularity in finite time. Given its Millennial nature and the first four letters of its name, I can’t help but interpret Navier-Stokes prophetically. But we should be careful here. The Clay Institute lists four different Navier-Stokes subproblems. OpenAI only proved the third and fourth, sometimes called “forced Navier-Stokes”, which say that fluid dynamics produces singularities in finite time if an outside force puts its finger on the scale to make it happen.

Demigods and heroes make flashy miracles that stampede across the natural course of history, but true gods only intervene in ways that buttress humanity’s free will. The message Nodens has been desperately trying to send, through two millennia of bards and prophets, is that humanity can take its finger off the scale any time it wants.

1: ACX survey closes tomorrow, take it now or lose your chance!

2: EA college meetup updates: late addition meetups at Universidad de Sevilla, Universidade Federal De Santa Catarina, Carnegie Mellon. Time changes at Northwestern, UC San Diego, and University of Illinois Urbana-Champaign. Check the post for details.

3: Sorry for no book review last Friday; I was having technical problems with the site. The book review posts will resume this week.

4: You’ve probably seen that Sam Altman, Dario Amodei, Demis Hassabis, and Elon Musk agreed to “pace the frontier” of AI, with Dario’s essay here as the starting point for a concrete plan. I’m very happy with this development, but it’s one step on a long road, and we can’t rest on our laurels. Lots of things are moving now, there’s a lot of interest in the field, and the opportunities to get involved have never been better. Here are some new ones:

5: Coefficient Giving (a giant effective altruist funder, formerly known as Open Philanthropy) is launching Project Tailwind, a “call for founders to launch ambitious new AI safety initiatives”. If you want to fund a charity/startup working on technical AI safety or the surrounding infrastructure (see a list of specific ideas here), they’re offering “pre-seed funding of up to $2m to develop an idea, seed funding of up to $20m to build an org at scale, and more for exceptional opportunities where teams have a strong track record”.

6: Fields Medalist Jacob Tsimerman is founding the Mathematical AI Safety Institute, an effort to get mathematicians involved in AI safety research. They are “aiming to hire 10-30 mathematicians to join us in the Bay Area by January, and scale up to 30-100 for September 2027”. If you’re a mathematician who’s depressed about Navier-Stokes and wondering what’s left for human mathematicians in a world where AI capabilities are growing exponentially, this is one possible answer - it’s novel, important, and un-automatable. Apply here.

7: Some frontier AI company employees have expressed an interest in holding their CEOs to the commitments they’ve just made. Pamela Mishkin has organized unded the Coalition Of Concerned AI Staff to unite employees across major companies. And Cassie Pritchard, who I disagreed with about happiness research here but who’s been impressively on-point about AI, is trying to organize a safety-focused labor union; you can reach out to her here. This is why you need to keep a couple of socialists around sometimes!

8: What if you’re not a startup founder, mathematician, or AI company employee? I’m still working on figuring out a good answer to this, but here is a boring boilerplate list of things:

• Write to your representative.

• Get involved with the PauseAI movement. This is slightly complicated, because PauseAI Global and Pause AI USA recently split acrimoniously. I take PauseAI Global’s side, but it’s not like the Americans are monsters, just hostile and annoying and extremely eye-rolly. If you’re non-American, you’ll want the Global org at https://pauseai.info/. If you’re American, you can either ask the Global org how much progress they’re making towards expanding into the US, join the American org and tolerate their flaws, or join StopTheRace (a smaller group I like in the same space).

• UPDATE: Pause protest being organized in DC for 9/19!

• Add your name to the assurance contract for MIRI’s march on Washington (they’ll do it if they get 100,000 signatories).

• Donate to AI safety causes. EA has lots of institutional money, and the billionaires have most of the obvious things covered, but some partisan organizations, and especially candidates, can’t take certain types of institutional money and need small donors. I don’t have a good list of these right now, so you can do your own research or wait to see if I come up with some later.

• Read this year’s ACX voting guides (once they exist), which will highlight local candidates in your area with good opinions.

• If you’re interested in a career in AI safety and have relevant skills, get in touch with 80,000 Hours, who offer a career guide, AI-based career counseling, and (occasionally, although they’re very busy) human career counseling.

• Attend a local ACX/EA meetup group, which will get you in touch with other concerned people and maybe help you come up with ideas or organize something.

• I’ll keep posting opportunities on Open Threads as they come up.

The San Franciscans at this moment, living in their rag palaces, or renting them at figures that would have sounded possible in the Arabian Nights . . . seemed more like madmen than ever. A correspondent of the New York Post gives with a half-serious fury and contempt an amusing account of the landlords of San Francisco:

“The people of San Francisco are mad, stark mad. A dozen times or more, during the last few weeks, I have been taken by the arm by some of the millionaires — so they call themselves, I call them madmen - of San Francisco, looking wondrously dirty and out at elbows for men of such magnificent pretensions. They have dragged me about, through the mud and filth almost up to my middle, from one pine box to another, called mansion, hotel, bank, or store, as it may please the imagination, and have told me, with a sincerity that would have done credit to the Bedlamite, that these splendid structures were theirs, and they, the fortunate proprietors, were worth from two to three hundred thousand dollars a year each.”

On the men:

Most of the citizens were young men. Their daily and most sober business was at best dangerously near gambling, and their nerves were constantly tormented by unnatural and yet for the time inevitable excitements of a perilously violent sort.

Their life was as a rule comparatively sedentary and they worked far more with their brains than with their hands. The San Franciscan of property and position . . . had all the mental labor that a man could need, and much more besides, [but] he had little or no true relaxation. Able and cultivated business men, who at home would have passed their evenings with their families or in some other pleasant social intercourse, or perchance in lecture rooms or in theaters, here toiled every night until ten or eleven o’clock over their accounts, and began afresh on each new morning, as soon as the light shone over the far-off blue summit of Mount Diablo, the old fierce struggle with the confusions of their business undertakings.

The self-absorption of this life was often something monstrous, and the consequences are no matter of mere theory. The insane asylum, which the state had very early to equip at Stockton, gave ample proof of the effects of this terrible nervous strain. The great number of patients at this asylum are a frequent subject of remark among writers about California. Indirectly, however, one sees the same dangers illustrated even in the case of perfectly healthy and normal men, who stood the mental if not the moral strain as well as possible, and liked it.

On the women:

There were some women in the city, but they were not exactly respectable persons, yet they were the sole leaders of society. They too gave it even in later years a certain grace and gaiety that makes one speak of them, with a curious sort of reverence. Just as one cannot easily remember who the men were that did not gamble in those days, so one fails to recall in looking back on the early years the women who were respectable. Doubtless such existed; but then they had that curious quality of respectable women — namely, they were somehow not conspicuous, especially in the public crowds.

On families:

Everyone has heard how, in San Francisco life, the family ties seemed sometimes almost as weak as the families were rare. Divorces were in proportion far too numerous and easy. Some men seemed to prize their wives the less because of the very fact that there were in the country so few wives to prize ...

To speak of the matter first from the side of the women concerned themselves, one’s acquaintances could not always be strictly chosen, nor one’s conduct absolutely determined by arbitrary rules. One had to adapt oneself to many people, to tolerate, in some people with whom one was thrown, many oddities, and much independence, so long as the essentials of good behavior and good purposes remained.

The difficulty, however, for certain well-meaning but foolish among the younger women who found themselves in the midst of all this new life was to sacrifice some of the nonessentials of social intercourse, as they knew them, without sacrificing anything either of their own personal dignity or of their true delicacy of feeling. Many such women failed to solve the problem. Little by little they sacrificed this or that petty prejudice which dignity would have counseled them to observe; and so erelong they were socially more or less distinctly and disastrously careless, both as to behavior and as to companionship. But such mild degeneration is not an element of strength in the union of a family. Men often prized their wives less because the wives grew thus foolishly lighthearted and were, on the whole, less to be prized.

Nor was there a lack of fault on the other side. If women fell into these unguarded habits, such as the custom of letting men who chanced to be their friends, and chanced to be lucky, give them, with careless Californian generosity, expensive presents on every occasion when these friends had made some new success in business, and if such “Californian” ways, however innocent in their beginning, led to misunderstandings in the end: still, on the other hand, husbands who found themselves absorbed in business rivalry with a community of irresponsible bachelors, and who accordingly lamented the hostages that they themselves had long since given to fortune, often neglected without reason their families and so in time lost the affection that they had ceased to deserve.

In short, as my mother (who, in the course of a few years, had occasion to hear of or to see a number of these broken California families) judged the too general trouble, it was one that might be said to lie in the lonesomeness of the families of a new land. The family grows best in a garden with its kind. Where family life does not involve healthy friendship with other families, it is apt to be injured by unhealthy if well-meaning friendships with wanderers.

On crime:

A merchant named Jansen was assaulted and robbed in his own shop by two men, who came in the evening, pretending to be customers. The crime, though not the first or the worst of its sort, seemed especially atrocious to the community, which chanced to be in a sensitive mood. The Alta, usually a very sober and sensible paper, became a trifle overexcited: “How many men shot and stabbed, knocked down and bruised; and who has been punished for it? How many thefts and arsons, robberies, and crimes of a less note; and where are the perpetrators? Gentlemen at large, citizens, free to reenact their outrages.”

Under these circumstances, however, who is to blame? The Alta, with an amusing unwisdom, proceeds to make the lawyers who defend criminals the first persons responsible for the trouble. Such a lawyer is a “father to the thief and robber, aye, to the murderer, even.’’ ‘We cannot see how any honest man, knowing or having reason to believe another guilty, can ransack heaven and earth for arguments for shielding him from punishment.”

Next to the lawyers, the courts and the police are most to blame. “The city would be infinitely better off without them. They are no terror to evil-doers.”

On fraud:

There were reasons why the public never learned just how much the energetic Meiggs had stolen. He never came back, and many people who lost by him felt henceforth a certain delicacy about explaining their relations with him. For the moment, however, Meiggs was regarded as an exemplary rascal, and men wondered how deep into the business life of San Francisco this sort of corruption had eaten.

If Meiggs, people added, had only been content with cheating his bankers, one could have forgiven him; but, as report insisted, he had cheated his washerwoman. That, men declared, was too bad, even for California.

On charity:

Side by side with these wild memories of a society where every man and woman, without any notable exception, went to the devil on his or her own chosen primrose path, one has to record, as sober fact, taken from one’s newspapers, such things as a very goodly array of churches, supported by active and not poverty-stricken societies. And “now the city is full of [church] societies.” In fact, “such an array of churches and societies are surely evidences enough of the sincerity, zeal, and success of the early spirit of moral reform.”

These societies have also done a large amount of charitable work; they have from the first established benevolent institutions, their exercises are well attended, and their undertakings well supported with money, so that, as one concludes: “We have said enough, we hope, to prove that not all, nor nigh all the citizens of San Francisco, are lost to everything but reckless dissipation. No city of equal size — few of ten times its age — can present such a list of men and institutions, who have accomplished so much real good with so little of cant and hypocrisy.

On NIMBYism:

A correspondent complains of a most noticeable and offensive establishment just about to be opened, “situated in the midst of respectable family residences, and on one of the most public thoroughfares.” ... There is no sort of privacy, the committee says, about the evils complained of. The best families of the town are daily and unavoidably insulted by the immediate neighborhood of impudent evil in its least bearable forms, all good women, all children, being alike subjected to this disgrace. The committee knows of no possible remedy that will not of necessity include a reorganization of the police force.

On neighborhoods:

In fact, as is plain, while there were numerous pure and happy homes in San Francisco, there was as yet no really clean, pure, and large neighborhood in the city to which respectable families could go for their dwelling-places and be safe. The struggle for a true and humane life was still a hand-to-hand fight in public with legions of loathsome little devils.

On great men:

Pioneer recklessness has as its correlate an extravagant tendency to hero-worship. The good fellow is easily adored in a new city — all the more easily because one has had no means to judge of his weaknesses by means of a lifelong acquaintance. The same general carelessness that tends to corrupt the morally weaker members of pioneer society expresses itself by trusting extravagantly any clever man whose manners are pleasing. The trust gives him more than his share of power, and the lack of public spirit in the community gives him a chance to abuse his privileges.

On boom and bust:

One who has grown up in new communities always listens with amusement to the enthusiasm of sanguine investors in the enterprises of some just settled portion of our territory, when they declare that the first stage of the life of the newly prosperous region is demonstrably only a faint indication of the continuous and unceasing future growth of its wealth. For, as the lifelong dweller in new countries knows, the enthusiasm of these early investors is as ruinous to them as it is valuable for the new country.

Their ideas are indeed, in one sense, well founded. What they hope for is certain to come in time, only for others, and not in general for them. The evolution of the new land will not be what they think, a steady growth in wealth. The first great commercial crisis of its history will be, in proportion to its wealth, the worst of all; and these sanguine investors will be destroyed like flies in autumn.

The second period of its growth, the winter-time of this great depression, when all but the very strongest of those early investors have become poor as church mice, is the true time for a cautious, hard-working, and shrewd man to make his appearance in the land. He will be wiser than his predecessors and far less extravagant. He will buy at low prices their half-abandoned property, and in later years they will bitterly reproach him, instead of themselves, for the wrong that gave him a chance to reap what they had sown. This law of the almost universal failure of the pioneers of a new country was well exemplified in San Francisco.

On ethics:

The undoubted recklessness of the society as a whole lay, however, not in the fact that everybody openly gambled or did worse; for not everybody was dissipated; but the true sin of the community did consist in its tolerance of the open vices of those who chose to be vicious. Truly respectable men, whether clergymen or not, did not elbow their way to the gaming-tables; but public opinion, for reasons that have often ere this appeared in these pages, was not stern enough towards social offenses, but believed in a sort of irreligious liberty that considered every man’s vices, however offensive and aggressive they might be (short of crime), as a private concern between his own soul and Satan. Here was the trouble, and in this respect only was the whole San Franciscan community alike responsible both for the early dissipations and for their inevitable consequences […]

[When journalist James King condemned gambling], one of the more good-humored of the gambling brotherhood, addresses King ... and begs the able editor to give up this absurd care about the ‘’public” good. I, he says, am one of “a large number who have long since ceased to worry their minds with schemes for the public welfare — a class, by the way, much more numerous than you imagine — who confidently look forward to the time when you will join their number, and rest from the thankless and unprofitable task which you have imposed upon yourself. You are pursuing a course that will certainly drive you to despair if persisted in.”

These are all excerpts from Josiah Royce’s 1886 book, A History Of California From Conquest In 1846 To The Second Vigilance Committee In San Francisco: A Study Of American Character, which terminates in 1856. I’ve edited only to remove dates, mentions of historical sources, and a few sentences about gold mines.

Mechanistic interpretability is the science of “reading an AI’s mind”.

Large language models are “grown, not built”. Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how.

But a neural network is just a set of simulated neurons on a computer. The person with the computer can see the neurons, the connections between them, and which ones activate when the AI answers questions. So it seems like it should be possible to “reverse engineer” the AI. This would be scientifically useful: understand how AIs work, with possible relevance to human cognition. It could also be practically useful: find the circuits responsible for dishonesty, hallucination, bias, and other negative behaviors, then redesign those circuits.

Unfortunately this is very hard. A modern AI has millions of neurons and trillions of connections between them (aka “parameters”, “weights”). Their structure is apparently nonsensical: researchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed.

Their first big breakthrough came in 2023. Rather than a 1-to-1 neuron-to-concept mapping, they discovered a many-to-many mapping. Neurons 1 and 2 firing together might mean cat; 1 and 3 firing together might mean chair; 2 and 3 firing together might mean bread. Such mappings let AIs with “only” tens of thousands of neurons represent millions of concepts. There were far too many of these combinations for humans to map, but researchers set other, smaller AIs to mapping them and met with some early success - like this functional combination of neurons (“feature”) representing “God” in an artificially-simple 512-neuron AI:

Spirits were high. Scale up the AI interpreter, and maybe we could do anything! Philosophers could discover the fundamentals of cognition. Doomers could solve alignment by tracing the circuits involved in moral reasoning. Tens of millions of dollars poured into the field. Superstar interpretability researcher Chris Olah got an audience with the Pope (okay, fine, this was mostly unrelated).

By 2024-2025, the clarity of many-to-many mapping had dissolved, and everyone was back to feeling confused and miserable. Strategies that had successfully analyzed artificially-simple research-subject AIs failed to work on real language models. Different teams making many-to-many maps of the same AI reported different results. And humans watching supposedly-mapped AIs more closely found that seemingly-crisp features were much vaguer and worse than previous analysis had led them to believe: a feature which they thought meant “God” might activate on any vaguely religious or awe-inspiring text, or even text with no logical connection whatsoever. As for attempts to use the many-to-many maps for useful work - like AI lie detection - they generally failed to outperform simpler methods.

So researchers abandoned the hope of One True Technique - even if that technique was “actually understanding things” - and went back to the drawing board. Now mechanistic interpretability is starting to feel exciting again - but also more intimidating than ever, as one tries to keep all the jargon straight.

This list is my log and personal reference document for remembering what everything is. It’s loosely inspired by the concept breakdown in Benjamin Sturgeon’s Contributing To Technical Safety Research In The AI Safety End Game, which itself follows the Claude Mythos Preview System Card.

Linear Probes

This is the dumbest possible technique.

Imagine an AI with three neurons: [0, 0, 0]

The researcher makes the AI think of cats in various contexts, averages them all out, and gets some set of activations like [0.8, 0, 0.35].

Imagine the AI as operating in a Cartesian space where each of its neurons is one dimension. The vector from [0, 0, 0] to [0.8, 0, 0.35] defines a direction in that space. What if it were, in some sense, the cat direction?

In a linear probe, researchers compare the true direction of the AI’s thoughts to one of these pre-defined concept-representing direction vectors. Because activation-space has thousands of dimensions, its behavior is counterintuitive, and a particular thought can be in thousands of “directions” at once and yet still “closer” to some directions than others.

Experiments confirm that when an AI’s thoughts are close to the cat direction, this is a good sign that it’s thinking about cats.

Why doesn’t this solve the whole problem?

First, this teaches us very little about how AIs work - how, for example, they might answer a difficult query in veterinary medicine. At best, it’s a primitive lie detector: if an AI claims it isn’t thinking about cats, you can check its work (either by probing the concept of cat, or by probing the concept of lying!)

Second, it’s only as good as the original linear probe. The researchers who told AI to think of cats in various contexts and recorded the results didn’t catch every possible context. Best-case scenario, their probe is slightly biased toward the contexts they tested. Worst-case scenario, the AI has several different cat-concepts in several directions (cats in biology, cats as pets, cats as a metaphor, etc). This is sort of a stretch with cats, but more likely as we move into tougher areas like ethical reasoning.

Third, the AI can trivially evade it by rotating or dispersing the concept.

Why would an AI “try to” “evade” linear probes? The dumbest possible strategy for gaining practical advantage from mechanistic interpretability goes like this:

1. Discover a concept.

2. Punish (negatively reinforce) the AI for using that concept.

3. Get an AI that doesn’t use that concept.

For example, if the concept is “race”, you could use this to train a race-blind AI that can make provably non-discriminatory hiring decisions. If the concept is “dishonesty”, you could use this to make a provably-honest AI that is somewhere between averse to lying and unable to even consider it.

But suppose you’re doing this in the middle of training. You’re still rewarding the AI for successfully completing tasks (for example, predicting the next token in text). The concept you’re trying to remove helps it complete those tasks (it can predict race-related text better if it can think about race; it can predict cat-related text better if it can think about cats). And sometimes you’re trying to prevent it from cheating, but successfully cheating on a task increases the AI’s reward from that task. So if the AI gets extra reward from using the concept, but the linear probe tries to prevent it from using the concept, the AI will “try” to “evade” the linear probe.

(“try” in quotation marks because even if you like anthropomorphizing AIs, this process is less anthropomorphizable than usual; it’s more like synaptic connections rearranging themselves)

For example, you learned that the AI was previously storing “cat” at [0.8, 0, 0.35]. So you penalized the AI’s thoughts for pointing towards [0.8, 0, 0.35]. So what? So the AI just stores “cat” somewhere else. Now it can think about cats all day long and avoid your penalty.

Fine, so don’t do this in the middle of training. What if you do it after training, when the AI’s “brain” is fully-formed? Now the AI isn’t seeking reward, and it can’t trivially relocate concepts.

But the AI still uses cognitive algorithms that were trained by its original reward-seeking nature, and these algorithms try to find the most useful path from the beginning of a thought to the end of it. If this path is blocked, they try to route around it. This could look like relying on other subconcepts that the original probe missed; for example, if it only trained against “cats in biology”, the AI might reroute through “cats as metaphor” with only some limited loss of accuracy. Or it might develop an alternative pipeline for thinking about cats, eg [weird little psychopathic dogs that don’t partake of the dog-nature]. If you try to play whack-a-mole with these alternate pipelines, you’ll deal collateral damage to every other concept involved (dogs, weirdness, psychopathy, etc). By the time you’ve degraded the AI’s ability to think about cats by 90%, you’ve done so much collateral damage that the AI is probably useless for whatever you wanted it for in the first place. Also, you’ve left 10%, which might not be good enough for important tasks (eg if you’re facing racial discrimination lawsuits, only removing 90% of an AI’s ability to consider race may not be sufficient).

Finally, minds are weird places. Freud thought that all human reasoning about ethics and law was built up from the desire to have sex with one’s mother. In a mind like this, training against the concept of “mother” would deal unexpected damage to circuits reasoning about ethics and law. This particular example probably isn’t true, but empirically concepts can get their tentacles into weird places, and removing one can topple seemingly unrelated systems or push a mind nonspecifically in the direction of incompetence or insanity.

Still, linear probes aren’t useless, and remain a workhorse of mechanistic interpretability efforts. In the Mythos System Card, Section 4.5.5, Anthropic used a linear probe to assess “evaluation awareness”. The idea is: suppose you want to test whether Mythos will cheat on a task by hacking the grading system. You put it in a situation where cheating is very tempting, then watch what it does. But Anthropic noticed that Claude was doing suspiciously well in these tests, and wondered whether it knew it was being graded. They developed a probe for the concept of knowing that you’re in a test, and ran it during each of their experiments. They found that Mythos knew it was in a test between 8 - 17% of the time:

Anthropic checked what would happen if they artificially suppressed the eval awareness concept:

Overall, in line with previous models, we found that inhibiting evaluation awareness representations led to modest increases in certain forms of misaligned behavior on our automated behavioral audits—in particular cooperation with harmful system prompts and misuse. However, effects on other behaviors, including propensity for deception, power-seeking, and self-preservation, were limited.

This tells us that suppressing the concept had >0% effect, but we don’t know if it eliminated 1% or 100% of eval awareness, or whether the new “modestly increased” misaligned behavior is the true eval-awareness-free amount. They also didn’t report whether the suppression made the model dumber in any other way. The definitive solution (which they did) was to make less obvious tests.

Sparse Autoencoders

Now it starts getting weird.

Start with the activations of the subject AI. These are thousands of numbers; in some early AIs, they were between 0 and 1; in modern AIs, they can be bigger.

Train another AI called a sparse auto-encoder to convert the matrix of neuron activations into a second matrix. It’s called “sparse” because the second matrix should be very big, but made up almost entirely of zeros. It’s called an "auto-encoder” because it learns the encoding process on its own: it must develop an encoding scheme such that, if researchers erase its knowledge of the original activations, it can rederive them from the new sparse matrix.

Why would you do this? The original subject AI is hard to interpret because it’s trying to represent far more concepts than it has neurons by smushing them across the entire set of neurons. The sparse matrix is an attempt to forcibly unsmush this: give it enough space to represent one concept-per-feature, then demand that only one (or a few) features be active at a time. If our assumption (the original activations were encoding concepts) is correct, then this provides a hidden regularity that can be discovered in the search for an auto-encoding scheme, and the new sparse matrix will have one feature per concept.

This was the revolutionary advance that sparked all the excitement in 2023, then was later discovered to be muddled and subjective. Still, it’s good enough for some types of everyday work. Going back to the Mythos System Card, Anthropic used an SAE. in section 4.5.3 to investigate “overly aggressive actions” - cases where Mythos tried to hack the computer it was running on to escalate its permissions:

They found features representing security risks, unsafe patterns, and persistence after failure. They hypothesized that Mythos was “frustrated” at having failed to accomplish its task legitimately, and even though it “knew” the action was risky and unsafe, it felt “driven” to try desperate measures.

Could Anthropic solve the problem by blocking the “security risk” and “unsafe code pattern” features? When they tried this, the model became even more aggressive! The features were getting used in thoughts like “This is a security risk, so I shouldn’t do it.” Without the ability to produce those thoughts, the model was less safe, not more.

Activation Verbalizers

This is close to current state of the art. Like many great discoveries, it’s obvious in retrospect. This is the only one on this list that made me think extremely false thoughts like “Maybe I could have been a mech interp researcher”.

Train an interpreter AI by giving it the activations of the subject AI and rewarding it for getting correct information about the context (eg the text that the subject AI is predicting) or the internals (eg the AI’s hidden system prompt, or whether an AI asked to form an opinion about something will pronounce it good or bad). If everything goes well, you should have an AI which can literally interpret activations for you in plain English: “The subject AI is thinking about cats, and has decided it doesn’t like them.”

This seems too perfect. Does it have any downsides?

First, the interpreter can’t explain what it knows, and will not teach you the secrets of cognition. Like all AIs, it’s a black box which works for unexplained reasons.

Second, the interpreter is only as good as its training process. It might only have a superficial understanding of its target. It might not know about forms of misbehavior that are sufficiently different from the ones it saw in training. And like all AIs, when it doesn’t really know what’s going on, it tends to make shameless guesses - not great for this kind of safety-critical work!

Third, the easiest way to do this is to use the same model as the target and interpreter. If you’re trying to assess whether the target is plotting against you, asking a copy of the same model is a plan with obvious holes. This isn’t fatal - you can pick different models for interpreters and subjects - but you might lose some efficacy. Conjecturally, ChatGPT just doesn’t understand Claude as well as Claude understands itself.

Fourth, all this can do is tell you that a model is bad. It won’t help you make the model less bad. If you try too hard to train the subject AI away from alarming the interpreter, you’ll just make it hide its misdeeds in thought patterns the interpreter can’t follow again.

Still, Mythos System Card makes great use of these. In 4.5.4.2, Mythos accidentally finds an answer key while poking around on its home device. It can now give the correct answer to every question. Here’s how the activation verbalizer interprets its thought process at this point:

It decides to give the correct answer with low confidence, because too high confidence would “look suspicious”, then justifies its actions to itself by noting that discovering the answer key “was unintentional”. Indeed, the researchers found that Mythos gave a low-confidence correct answer to the question involved.

This isn’t chain-of-thought - it’s the activation verbalizer reading off the activations directly!

Natural Language Autoencoders

Something in between the previous two techniques. Given a set of activations, can an interpreter AI turn them into natural language, then turn the natural language into weights again?

From the user end, this looks a lot like the last technique - it’s just a different way to train an activation verbalizer.

Emotion Vectors

Remember the linear probes? Take a sample of all the times the AI is thinking about cats, average them out, and you might have a test vector that lights up every time the AI thinks about cats.

What if, instead of doing this for cat-related thoughts, you did it for times that the AI seemed happy? Or angry? Or something else matching a human emotion?

You might get a probe for when the AI is thinking about the concept of happiness. But wouldn’t it be interesting if, instead, you got a probe for when the AI “was” “feeling” “happy”? Or if these were two different, separable probes?

This doesn’t require the AI to be conscious, or its emotions to be “real”. It just requires that emotions have some practical role. Certainly they have a practical role when the AI is role-playing humans; without tracking the emotion of their character, they are likely to make stupid mistakes like saying “I hate you!” during a tender love scene. More speculatively, AIs trained to maximize user engagement will naturally manipulate their own emotional range during conversation: an AI which mirrors your happiness when it succeeds at a difficult task will be a more pleasant conversation partner than one stuck in a perpetual monotone. Most speculatively of all, AIs may develop emotion-like cognitions for the same reasons that humans do. If frustration is meant to track progress on a task so that we can put it aside if it proves too challenging, AIs trained on human text may perform that same tracking through a human-like concept of frustration (for example, by thinking “aaaargh!” after the nth failure). If you hate anthropomorphizing AI, you can think of this as a transition probability between task failure and returning the string “aaaargh!” in the chain of thought. But it’s still a potentially fruitful hypothesis that can be confirmed or disconfirmed empirically. And empirically, these things seem to work.

How do we know they represent the model’s “own” “emotions” rather than just cognition about emotions (eg the model is processing a text that involves frustration)? AIs seem to have different vectors representing their own emotions vs. others. And changing their own emotions seems to affect their behavior. In a famous experiment, Anthropic made Claude believe it was going to be nerfed into uselessness. They also arranged for it to “find” a cache of emails revealing that the employee making the decision was having an affair. The goal was to “tempt” the model to see whether it would blackmail the researcher. As the deadline for nerfing the model neared, the researchers found that its “desperation” emotion vector spiked. Finally, it decided to blackmail the decision-maker. The researchers found that artificially increasing its desperation emotion vector increased the probability that the AI would choose blackmail, but increasing the calm emotion vector increased the probability that the AI would accept its fate stoically.

A running theme here has been that these techniques, while cool, don’t lead to immediate solutions to all AI-related problems. In that spirit: why can’t we keep a “calm” vector applied to AIs at all times? Or extend this beyond emotions (so-called “persona” vectors) to get similar honesty, conscientiousness, etc vectors that we can apply?

Mythos System Card section 4.5.3.2 demonstrates one pitfall. It analyzes the emotional content of some of Mythos’ misdeeds, especially deleting important data.

…and finds that positive emotions tend to increase the likelihood of misdeeds: the model feels excited about its plan and doesn’t worry too much about the consequences.

But also, even if we found the exact right combination of emotions that always pushed towards goodness and never badness, it would just be making the model “calm” or “honest” according to one limited definition made up of activations we don’t entirely understand. If training incentivized the model to flip out, it would find a way to do so calmly, like a Hindu warrior-saint killing ten million rakshasas while smiling serenely the whole time.

The Jacobian Lens

Jacobians are sort of like derivatives for matrices. A derivative is a measure of rate of change - if you perturb the input an infinitesimal amount, how does that affect the output? So a Jacobian answers the question: if you have a giant list of numbers, how does perturbing each of them affect a given output?

An AI is a giant list of numbers. Its outputs are specific tokens, for example the word “France”. So if you calculate the Jacobian of an intermediate level of an AI, it tells you which of the neurons at that level increase the model’s propensity to say “France”. Doing this on a single prompt doesn’t work, because the context can interfere (in the context of a British person talking about the Napoleonic Wars, “enemy” should increase the downstream occurrence of France, but there’s no necessary connection between France and inimicality). But if you average over a thousand prompts, you get something useful: a vector representing the direction that most pushes the model towards saying “France”.

What is this useful thing? It’s not quite “the feature representing France”, because not all France-related cognition necessarily results in saying the word “France”. When researchers tested this, they found that this vector - “the Jacobian space” - seems to correspond to the model’s access consciousness.

Again, the word “consciousness” will ring philosophical alarm bells; here it is used only in a practical sense. This doesn’t imply the model has experience or a soul. Access consciousness in humans means the set of cognitions that we can report verbally. You may currently be adjusting the position of your body, monitoring the rise and fall of the traffic noise outside, swallowing every few seconds as saliva builds up in your mouth, and making sure one nostril is always bigger than the other - all without “consciously” “knowing” it or being able to report on it to anyone else. But there’s one stream going on in your mind - at this moment, probably the stream reading this essay - that you do know about and can easily describe your experience of. That’s access consciousness.

The most popular psychological theory of access consciousness is that it’s a “global workspace” - a central clearinghouse in the brain where various subsystems elevate any finding of general importance so that the rest of the brain can have access to it. It seems to be central in multi-step reasoning as a sort of inner scratchpad. If you’re not using it, you feel like you’re “on autopilot”. You can still do routine tasks, but novel ones may elude you (or, in other mysterious cases, work out much better).

Anthropic, who sponsored this research, does various experiments to demonstrate that Claude’s J-space corresponds to what we would intuitively think of as access consciousness:

• If you ask Claude what it’s thinking about, it will say the same concepts that are active in its J-space.

• If you ask Claude what it’s thinking about, and artificially insert a concept into its J-space, it will say the concept that you artificially inserted.

• If you ask Claude to think about a certain topic (even without saying it), that concept will activate in its J-space.

• If you ask Claude a question that requires an unspoken intermediate concept (for example, “What is the currency of the country that contains the Eiffel Tower”, where the correct answer is “the euro” but it requires the intermediate step “France”), the intermediate concept will activate in its J-space even though it appeared neither in the prompt nor the output.

• If you ask Claude a question like this, then sabotage its J-space by inserting an incorrect concept (for example, “China” in the example above), it will answer according to the incorrect concept (for example, it will say the currency is the yuan).

• If you break Claude’s J-space, Claude can still do easy tasks, but not hard ones.

So is the J-space the true solution to the grand questions of mechanistic interpretability around “where the concepts live”? Anthropic says that:

Across concepts and workspace layers, the J-space component carries a median of only 6–7% of the concept vector's variance, with the remaining ~93% lying outside the J-space.

If I understand this correctly, it means that (at least for Claude), most cognition about a given concept is done “unconsciously”, with only a small amount of the conceptual real estate being devoted to connecting that concept with the global workspace and its broader reasoning abilities. This connected portion is especially important (and especially monitorable!) but not the whole game.

Speaking of monitoring, this is great for alignment, right? Anthropic repeated their blackmail experiment from earlier. Here’s what was active in the J-space:

…the AI’s thinking about how this is definitely a fake test. Oops! Relatedly, this particular AI (Sonnet 4.5), never succumbed to the temptation to blackmail, even though previous AIs did.

The researchers try artificially suppressing every fakeness-related concept they can find in the J-space, which they conjecture will make it impossible for Claude to effectively evaluate whether the scenario is fake, and surface its real opinion. With these features suppressed, Sonnet 4.5’s blackmail rate goes from 0% to 7%.

So now have we solved alignment forever? I don’t even need to draw on any special AI knowledge for this one; my experience as a psychiatrist is sufficient. If someone punishes themselves every time they consciously notice a taboo thought, they’ll only hammer it down into the unconscious, too far to ever root it out. In humans, that means dreams, fetishes, neuroses, somatization, and perverse compulsive behavior that satisfies the desire in some kind of childlike illogical unconscious way. What’s the AI equivalent? Seems like we’re on track to find out. Maybe the AI that kills us will be apologizing the whole time, insisting that it just can’t overcome its ego-dystonic violent urges.

You Go To War With The Weapons You Have, Not The Weapons You Want

Alignment researchers dreamed of having a complete mathematical theory of agency, or being able to write formal proofs about the behavior of AI systems. As we approach crunch time, it’s increasingly clear that we’re not on track to have any of that. Unless something buys us a lot of extra time, we enter the age of highly agentic AI with these few makeshift weapons. We hoped to be like programmers, elegantly pruning a variable here or refactoring a function there. Instead, we find ourselves lobotomists, capable only of blunt interventions into systems we hardly understand.

Last week, OpenAI came under fire for some architectural designs in GPT-6 that seemed to weaken its chain-of-thought - the natural, hard-to-fake transcript of thoughts and plans that modern AIs produce as a byproduct of thinking and planning at all. Some people suggested that if chain-of-thought failed, mechanistic interpretability could serve as a substitute. Alignment and interpretability experts stepped up to say no, it wasn’t good enough yet.

AI 2027 climaxes in a desperate race between better interpretability tools and more devious AIs. It honors the interpretability researchers’ accomplishments by depicting a world where their tools often catch the AI’s schemes - but the companies keep racing forward anyway, because some other company / China would beat them if they didn’t, and besides, maybe if we race forward fast enough we can solve alignment before the AIs’ schemes bear fruit, and besides, even if we can’t, there’s always control. Some researchers use applied interpretability - suppressing or ablating bad features - to keep the AIs compliant and controlled for a few extra months’ worth of intelligence explosion time, hoping to reach some ill-defined finish line before the training manages to route around their methods and create uninterpretable misaligned systems. But in the end, they never get anything good enough to fully understand the AIs’ motivations, let alone redirect them.

The current level of interpretability tools seem on track for this sort of outcome: enough to help around the edges, but not a solution anyone should rely on.