Recent news seems to cast doubts on your China AI timelines: do you still think Chinese models will be significantly inferior to US ones by early 2026?
China also seems to be taking AI more seriously than the US government at the moment, and pushing fairly hard for extensive use?
My baseline metric is that we will know China is feeling the AGI when they suddenly behave a lot more cautiously. A closed society like them has even more to lose from AGI than the US.
"No leader would be so stupid as to..." has a pretty poor showing in the US right now. Perhaps China will be different, but I wouldn't be so confident.
We definitely discussed this among ourselves. We predict a pretty constant four months behind. We think they're about four months behind now (DeepSeek somewhat more than four months later than OpenAI's internal reasoning models, although also slightly better). China is doing a great job playing catchup, but also chip sanctions haven't fully hit yet, so it's a wash and we think it'll stay a constant four months. This is my half-remembered memory of stuff Daniel and Eli said. I might ask them to do an AMA here and they can give you the full story.
One wildcard is that China has a lot more access to data than US companies. US companies have access to open data; if the Chinese government gets serious, they can simply by fiat acquire all the data in China for training: private email, corporate documents, private lab data , personal notes - everything.
Google, Facebook, Microsoft, and Amazon have access to a lot of data that isn't open, and they either ARE, own outright, or partner closely with the leading AI companies.
Yeah; it's definitely physically possible for them to expropriate all that data (although I think facebook is the odd one out here - they have access to very little data of people doing actual work,especially thought work - no amount of marketting material is going to produce a super-ai).
But there's definitely at least a speed bump before they use it. Sure, if the US thinks China are doing it and it's important , the law will be changed to allow them as well. But it's quite plausible that AIs deep in the security apparatus are *already* being trained on China's data trove.
I think another serendipitous factor here is that typing Chinese is most commonly done using online predictive text software, so they probably already have all the useful data without having to do anything ham-fisted like raiding people's homes and offices.
What if China needed even more training data so by fiat the government directed all 1.4 billion citizens to video record everything they say and do at all times. Then sync it with data from security cams installed everywhere as well, including GPS to allow synced 1st person and 3rd person personal data. They could gather [Blank]abytes of data and [Blank]illions of tokens to train AI. It would be a detailed representation of the Chinese people's hopes, fears, aspirations, pain, and pleasures. Thankfully I don't think this is necessary for reaching AGI and the Chinese people would revolt (them getting pissed off basically ended China's Covid lockdowns). But there could be an alternate history with less efficient discovered learning algorithms where this is what the winner of the AGI race had to do to win.
Will data be the issue? A human doesn't need to have access to vast amounts of data to be intelligent, so why should an AI? Once there are better AI models they won't need so much data.
IMO the bottleneck is when AI can do programming, because once it has that skill it can build a better AI. Humans get better at programming by doing it and learning from mistakes, so why can't an AI do that? IOW the programming AIs will create their own training data.
Because humans can *think*, and what we currently call "AI", really can't. Its thing is to turn vast quantities of data into something that superficially looks like thinking, and which in some cases can be effectively used as a substitute for thought. But it absolutely depends on the data, all of it and more.
I suppose you can handwave "better AI models" to give you whatever characteristics you want, but that's like saying that when we have better physical theories we'll be able to build warp drives and time machines. And in the long run, maybe we will, but don't count on it happening any time soon.
I'm curious what you have seen to make you so bullish on Chinese AI. Like, thinking they're so likely to succeed that if you see Scott write the opposite you just assume he's lying, is an incredibly high level of confidence.
I'm not particularly bullish on any particular AI project. I think most of them will fail, quite likely including the ones that currently seem in the lead. I think there are unsolved problems that need to be solved, and nobody knows who's going to solve them first.
I also think that when some of the "unsolved problems" are solved, the need for "compute" will drastically decrease. (This is actually included within the scenario, but I tend to move it to an earlier requirement.) I still expect the eventual "grand AI" to be a "union of experts", with lots of modules specialized for specific tasks. (This is partially because such experts are a lot easier to train.)
Basically, China itself is only conducting reverse engineering, which is a common practice in any technological research. However, the accusation that China engages in 'stealing' needs to be reconsidered.
I have (obviously) not had the time to watch the full video, so I promise to come back and correct any part of this comment that it addresses (or that misses the point), but this strikes me as an astonishingly short timeline.
I'm a software engineer and I mostly work on a system that does use LLMs at its core, and I think it's very useful and can be surprisingly good. And I do pretty frequently ask LLMs coding questions, and that, too, can be useful.
Having said that, those things are mostly useful within a specific set of bounds. The LLM is "superhuman" in the sense that it has effectively memorized a broader range of documentation than I could ever do. It remains terrible at performing novel tasks. Any interesting programming problem - something I genuinely struggle to FIGURE OUT, not REMEMBER THE SYNTAX TO - it's basically worthless for.
That doesn't mean it's not great! I'm not exactly a "vibe-coding" enthusiast but I'm tinkering and I see the appeal in letting the AI fill in a bunch of boilerplate. That is something that takes time away from my job and I think there's a real future in being able to delegate a lot of that work and just focus on the complicated bits.
But I'm therefore extremely skeptical about the question of AI accelerating AI research, which is largely about trying to figure out new things, not do stuff it was pretrained with dozens or hundreds or thousands of examples of, or comprehensive documentation on. And I'm unaware of ANY meaningful example of this - all of the cool "AI figures out novel materials" / "AI solves protein folding" / etc. type things are from datasets that are specific to the domain, and a model that was set up with a structural understanding of the domain (i.e., not general-purpose tools and not devoid of human guidance).
My bet is that our current tools continue to improve and unlock new capabilities, primarily in terms of:
- Taking advantage of knowledge that SOME humans already have
- Better integration with humans to do the parts that humans are good at and handle the rest for them
- Picking certain kinds of low-hanging fruit that synthesize knowledge from multiple sources rather than making novel discoveries
... but I don't think the insane sci-fi future is going to basically arrive in 3-4 years.
Yeah, it is definitely astonishingly fast. You should watch the podcast, which is Dwarkesh asking again and again "Isn't this part astonishingly fast?" and Daniel answering "Yes, definitely, but here's why I think it's true."
The question isn't "is this too fast". It's "Is the part of you that is generating the emotion 'omg too fast!'" well calibrated, or essentially just unchecked hindsight bias?
If you haven't generated predictions that has timelines occasionally faster than what actually happens, then you know that the outside view says that you're systemically too pessimistic. The feeling that your predictions are normalcy isn't a feature, it's a bug!
(The object level complaint is that you're using the average level performance of something to judge the best case level performance of something. It doesn't matter if you can't personally get uranium purified in 1943, if someone could get a pile to go critical elsewhere)
You raise valid questions, but I would say there are lots of other ones involved here. Like, is extrapolating from past progress valid? There are lots of domains (maybe most domains? on a long enough timeline, really all domains) where that wouldn't be true.
I think many people including myself have been surprised at some point by how much LLMs can do. I often argue that AI is simultaneously over-hyped and under-hyped, on the premise that it hasn't caused any big revolutions and in some ways is still "pattern-match inputs to outputs", but on the flip side it keeps sort of demonstrating how much human activity - even really valuable human activity - is just "pattern-match inputs to outputs".
Do the other objections matter if your knee jerk reaction, that you decided this was too fast, was what was motivating you in the first place?
Like take your "is extrapolating past progress valid". Let's pretend that I've shown you say, Moore's law predictions made from 30 years ago, or genome analysis cost curves from 20 years ago, and then show you that the curves closely match. Do you actually change your mind? Or do more "other objections" that you would not have independently come up with before magically pop up? I'm not saying I have that data, but in that hypothetical are you convinced? I would guess no, because I think the underlying generator of those other reasons IS the need to give normal sounding reasons that sound normal.
Well maybe I'm wrong, I'm just extrapolating this from the couple of dozen cases I keep seeing pop up. I'm completely willing to listen to your internal self report on your reasoning, I'm not a damn mind reader!
Which is more relevant to AI, the trend for exponential curves tend to become sinusoidal or the observation that recent exponential trends in tech like semiconductors and solar panels have continued for longer than expected? That seems to be the ultimate question. There are many things that might cause AI to plateau, including the fundamental difference between mimicry and invention, but the observation that this may have led people to underestimate the probability of past AI successes such as GPTs or AlphaZero should give pause.
No, this isn't merely an emotional response to this post, I've been here thinking about this topic for a while and following all of the news around it and my deeply-considered position is that people keep being impressed when AIs are super-human in SOME ways into thinking they'll be super-human in ALL ways.
Computers have kind of followed this pattern forever and we could play reference-class tennis for a while with them. Do you know that we used to build physical devices to model and calculate complex formulas? And then in a short period of time, we created digital circuits that could be re-wired much more quickly. And then all of the sudden we had programmable computers where you could just type in the calculation! Insane scaling curve there. Where does it end? Computers basically being able to solve incredibly complex mathematical proofs? No, actually, there's like 50-60 years in between those steps where all it does is go from "astoundingly fast calculations" to "even astoundingly-faster calculations" in a way that doesn't really make much of a difference to somebody whose problem was "be able to balance the books in 1 second rather than 12 hours". But you still needed a human to type in the formula.
Moore's law is a pretty good comparison because it's pretty bonkers and it's held up for a really long time. Maybe this is another one of those scenarios where we're gonna keep pouring a giant mass of society's resources into advancement (as we currently are) in a way that sustains it. But also eventually we'll hit a wall with transistors where you just can't make them any smaller before physics gets in the way. And we only have a couple more orders of magnitude to build LLMs (which scale their capabilities sub-linearly) before you hit the wall of "all of human enterprise can't produce the physical resources needed for this".
(The optimist says we'll get AI that solve that problem itself before it becomes limiting, but this basically requires us to assume, among other things, that there definitely is a solution. Or that it can cause GDP growth unprecedented in human history, purely by thinking about stuff but with very limited ability to interact with the physical world. Maybe these are reasonable assumptions, but they seem very aggressive to me and not really supported by much of a demonstration of anything more than an ability to get better at tasks that mostly involve manipulating abstract symbols.)
> No, this isn't merely an emotional response to this post, I've been here thinking about this topic for a while.
Yup! Point conceded.
> my deeply-considered position is that people keep being impressed when AIs are super-human in SOME ways into thinking they'll be super-human in ALL ways.
Do you have concrete ideas on *which* ways it won't become superhuman or close to human, some of which can be tested on much shorter timelines than actual superintelligence, and then observe that your expectations are consistent? The postdiction I keep trying to get skeptics to answer is: did you predict GATO, or image gen, or video gen by LLMs beforehand? This is the type of thing I'm thinking about when I talk about the feeling being hindsight bias.
If you never try and predict what progress could happen, you can just keep saying the latest progress is disappointing relative to some undefined amount of more progress, not noticing that that your implicit timelines have gotten shorter while you explicitly proclaim that timelines are long. Remember, people used to regularly say that *any* AI in the next 100 years couldn't do art at all, and none of those people have come out and mea culpaed.
This is a little holistic, but these are some ways in which I expect AI to be superhuman:
1. At playing any game or other scorable activity that can be run faster than real-time
2. At any activity that we can produce a massive number of examples of
3. At anything that gets a lot easier if you know a ton of things
4. At anything that gets better if you can do something fast
To try to make this slightly less abstract and thus avoid the failure modes you describe above:
1. There's no game that AI can't be way better than humans at, given the budget to train it. We've already had Chess/Go/Diplomacy/Starcraft 2, but, anything. I'm honestly surprised it's taken a while for unconstrained League of Legends (if I'm remembering correctly). There's no barrier within our CURRENT resources for AI to outdo humanity at every videogame and boardgame in existence.
2. This is kind of the crux of a lot of what I'm thinking about. It's a little hard to predict, for sure! I don't think I predicted image generation in advance, but in hindsight, we have billions (tens of billions? hundreds of billions?) of images with labels. Once we got there, generating videos seemed pretty obvious. I feel like everything that's happened with LLMs, which is impressive, basically boils down to realizing "all text is examples of people completing text". I am definitely in the camp of people who think that LLMs almost certainly have some ineffable limitations based on this, although I ALSO think this is a super clever realization and it's amazing what it's unlocked. I would say the odds are pretty good that in the next five years somebody comes up with some new, clever realization about how something that already exists in the world can in fact be treated as a well-structured sample for a very rigidly defined task and unlocks some new ability there.
3. I mentioned this elsewhere in response to somebody pointing out the AI co-scientist as an example - I'm bullish that this sort of approach could discover quite a lot of things, but, things that are at or near the boundaries of existing knowledge, or where there's probably a bunch of people who have all of the ingredients of something already, but are in different fields and can't put them together. I think this could be really amazing for humanity! A lot of problems are cross-disciplinary. Being a moderately good researcher in every field simultaneously is super-human. But I'd be a lot more bearish if you told me that AI was going to dramatically push one SINGLE field forward, not based on knowledge synthesis but by being the next Einstein and realizing something totally new about the world that nobody's anywhere near.
4. I don't have a great list of things here, but this includes, again, playing videogames - lack of reaction times is pretty great. I think I mentioned elsewhere that they talk about doing etymology in the podcast - AI should be able to track down information that is IN SOME SENSE available to it, but needs to be assembled from many different places, much more easily. That probably means AI could make some interesting contributions to journalism/history/etc., fields where some (but not all) of it is just about collecting up disparate sources of information.
Is this addressing your root question? I'm definitely not trying to win any awards for prediction - even if my basic thesis is right I'm probably wrong on where some of the specific bounds are here, which is why I mostly focused on the conceptual underpinnings of what I think it'll be amazing at.
Extrapolating from past trends is NEVER valid. It's estimation. And people are lousy at estimating anything that deviates from linear.
So if you're modelling a process that has lots of non-linear features, the result is not going to look reasonable unless you do a transform on it that converts it into something linear.
FWIW, my guesstimate of AGI is 2035 plus or minus 5 years. I expect there to be implement ion difficulties that the scenario didn't expect. But, yeah, when I read the scenario, I found it a bit shocking. (Do note, though, that the scenario repeatedly emphasizes that uncertainty increases as you project further into the future.)
This is really important: LLMs have shown us that the vast majority of office work is straightforward pattern-matching. And that maybe 5% is not (and the proportion of non-office work in the latter category is much higher).
No, I think the question is mostly, "Is this too fast?" Like when I read this:
> as early as 2030 (ie after it’s automated enough of the economy to survive without us)
I guess a lot hangs on what "enough of the economy" means, and I haven't had time to dig into any of the specifics yet. But that prediction at first blush seems ludicrously out of synch with historical rates of technological diffusion.
Yup, agree it's an incredibly strong claim. The specific stat predicted in the scenario (which I disagree with) is 1 million robots manufactured per month by 2029.
The argument in favor is - it took the US 3 years to pivot to a war footing during WWII. The amount of automation involved here is a bit less, and superintelligence will be a better planner than random 1940 Americans. If you bought out 20% of car factories and retrofitted them to robot factories within six months (about 3x WWII timelines) you could do this. This would require more robot manufacturing than we have now, but it might be on the Wright's Law track if you start manufacturing a crazy number of robots and learn quickly.
It seems to me that it also requires designing *better* robots. Current robots are janky and pretty useless. At the end of the day I'm just not sure autonomous, mobile robots that are worth the hassle are… permitted by physics? The energy requirement problem alone seems like a big, big roadblock.
(Biological life has solved this, of course, but no one would describe "superintelligence genetically designs wetware drones for itself" as "the AI builds robots".)
For many purposes, such as combat or search & rescue, the robots wouldn't have to be very large, maybe no larger than a bee! And a bee with a high explosive charge the size of a match head attached to it would easily be enough to kill a human, if detonated near the latter's head.
This. It's worth always remembering that Robots Are Not Free - they're not software, easily and cheaply duplicable. They have very real costs in manufacturing, maintenance, power usage, etc that can't be handwaved away anymore than AI can suddenly make cars cost $10/car.
The part of this that I'm highly uncertain about and not sure where to get good data on is where the labour to do so will come from in that period. Construction and deployment of novel tooling and systems would either need to be guided by a bunch of tightly looped VR headsets or essentially rapidly absorb a supermajority of skill labour, even while other industries would also be competing with that labour for their own AI-boosted expansions.
The pivot to war manufacturing meant retooling into things that were already more-or-less known technologies, just more of them, and with supply chains that were maybe a dozen or two dozen layers deep. I *feel* like there might be complexity problems here that could slow this down a fair bit, although I'm definitely highly uncertain.
Looking at robotics, I didn’t expect to have a single useful autonomous robot by 2030, but I suppose if AI solves the technical problems, the resource problems will be comparatively trivial to make 1 million a month.
Why do you assume "autonomous"? I see no reason the robots shouldn't be telefactors with all high level decisions coming from outside the robot. This wouldn't be true for drone type robots, where only some of the control would be from outside to reduce the bandwidth requirements, but for robots used within a factory, it seems like a better choice.
This divides your objection into two parts: Command and control. If the AGI problem is solved, then so is the command part. So the problem reduces to just designing a robot with a good sensorium and good manipulators. Also not easy, but definitely already being worked on.
I really think the timeline is too fast, but it's within the bounds of my 95% chance that it falls within this time period estimate. (It's not within the bounds of 90% chance...but just barely.) My mean estimate for an AGI is 2035, but it's not exactly a bell shaped curve. The right end (for 99.5% certainty) extends beyond 2125...and there's a smidgen of a chance that it will never happen.
Well... unless humans go extinct and no other terrestrial organism capable of civilisation evolves within the next 500 million years, then... I would expect someone or other will invent AGI sooner or later. I'm just not optimistic that our species is currently well-equipped to do it responsibly.
Humans go extinct and then the Fermi paradox kicks in —- intelligent technological civilisations are very, very rare — you need a billion planets for a billion years to find even one.
It would take a fairly thorough extinction event to wipe out humans along with every extant primate species. Otherwise you're probly gonna be back in the same place in 20 million years. Assuming the elephants don't get clever.
I only brought up the 500my deadline because the sun will render the earth uninhabitable by then. Elsewhere in the universe is anyone's guess, but heck, maybe aliens invented AGI a billion years ago, somewhere ten billion light years away.
You don't need to wipe out all humans, you just need to get them below the great filter. For example, if the great filter was the industrial revolution, it seems fairly plausible to knock us back that far
It's not algorithmic improvement that will bite first, this can be handwaved by using compute scaling and data scaling (all 3 types of scaling, compute, data and algorithmic, give roughly equal orders of magnitude scaling) till the AI is human-level enough, then using the human-level AI to get an infinite algorithmic improvement glitch.
The bottleneck that I never see those kinds of uber-optimistic/alarmist analysis address is the data. There is not enough data. There will never be enough data unless your domain is a video game or math, or otherwise something a program can generate.
Well... if there were some kind of algorithmic breakthrough that would allow ML/AI models to train on relatively small amounts of training data, as humans often can, then... this would presumably bypass the data bottleneck.
But at this point the prediction is equivalent to "LLMs are not the path to AGI, another architecture will emerge in the 2-3 years remaining and scale to AGI in months", which is... kinda huge? Kinda insane? Kinda extremely implausible?
There is no amount incremental improvement that will get LLMs to the data efficiency needed so that the data wall is not hit.
I don't know if the algorithmic breakthrough will require abandonment of LLMs so as being a modification to them or a pre-pass or post-processing, but... whatever form the needed breakthrough takes, in principle it could happen tomorrow or a week from now. So... yeah, I'm still going to say the timeline is plausible.
FWIW, I don't consider LLMs alone to be sufficient for AGI. But AI contains a lot more than LLMs. However many of the techniques used in LLMs can be applied to control of physical systems, both actual and in simulation, and the universe provided LOTS of data that way.
> But I'm therefore extremely skeptical about the question of AI accelerating AI research, which is largely about trying to figure out new things, not do stuff it was pretrained with dozens or hundreds or thousands of examples of, or comprehensive documentation on. And I'm unaware of ANY meaningful example of this
Google's AI co-scientist model can generate novel hypotheses to open scientific questions, and correctly predicted actual research outcomes that had been found by actual scientists, but were still unpublished. That was in February.
OpenAI just released PaperBench, which evaluates AI models for their ability to replicate AI research. The current best performer is Claude Sonnet with a custom agent harness, which scored 21%.
Just because you're unaware of something doesn't mean it doesn't exist.
> Just because you're unaware of something doesn't mean it doesn't exist.
Of course not, but here's no quicker path to enlightenment than asserting your ignorance on the Internet. ;)
The PaperBench thing is somewhat underwhelming and I think does not in any way contradict my claim - it's literally trying to redo something that somebody else has already done. This is valuable! And I approve of it! And I think it's a good application of AI. But it is by definition not doing novel work.
Co-scientist is definitely more promising. At the risk of over-matching to my previous statements, this is basically what I meant by "certain kinds of low-hanging fruit that synthesize knowledge from multiple sources" - e.g., identifying novel applications for drugs is probably a lot easier if you know as much as you can about what every drug in existence does and how and how every disease in existence works, which is the kind of thing that I expect an LLM to dramatically outperform a human at even if it isn't literally "better at reasoning". (And yeah, you could definitely quibble about whether "novel drug applications" can plausibly count as "low-hanging fruit" so feel free to scratch out that part of the sentence if you want.)
To me this is similar to the discussion early on in the video about etymology - I agree with Patel's take that tracing etymology is NOT an example of superintelligence, it's more of an example of taking the time to do something with the information that you can get your hands on. I think AI will be in some sense "revolutionary" for that sort of work, spotting things that we might otherwise have missed for a long time, but, COULD have done if only we'd allocated the time for it. Which, again, is great! And could lead to a lot of discoveries and economic growth and so forth.
My fundamental area of disagreement with "AGI/Superintelligence in '27-28" is that I do not think the gap between "what we have now" and "AGI/Superintelligence" is in the same category. We know a lot about medicine, and words, and engineering materials, and so forth. A lot of the stuff we don't know just requires us to look and see. E.g. my partner worked on a research project on orchids, which have these weird symbiotic or parasitic relationships with fungi. Most orchids we can't grow in captivity because we don't know what fungal strains are necessary to grow them. But we can find out, by trying a bunch. This expands our understanding of the world, but it requires a bunch of grad students, not a "breakthrough".
Getting to "we can build something that can functionally do any (cognitive) task a human can" does not seem analogous to this to me. It's further outside the bounds of existing knowledge, where the other items above are right at the boundary, or maybe even in gaps that we've expanded around, purely due to lack of resources. Expecting that AI will achieve this almost seems to have a weird circular logic to it to me - you're gonna have AI that's not super-intelligent yet, but can meaningfully push the boundaries of human knowledge much farther and faster into an unexplored space than the massive amount of top-notch humans we're already throwing at this exact problem.
> This expands our understanding of the world, but it requires a bunch of grad students, not a "breakthrough".
I think what people expect would happen here is that a sufficiently powerful model could reason about what fungal strains are necessary without needing to do the experiments, based on properties of the strains and the orchids, possibly up to including running advanced simulations.
Even if model intelligence converges on "as smart as the smartest human, but 1000x faster / cheaper", that's still a variety of superintelligence that can upend the economy and concentrate power, kind of similar to Hanson's Age of Em ideas.
"A sufficiently powerful model" is doing a LOT of work in that sentence, from my perspective. Biology is incredibly complicated and I really doubt anything within miles of our current capabilities can just think out a bunch of stuff without genuinely interacting with the real world. I feel this is a common pattern I see in discussions of intelligence. (I do agree with your assertion as phrased - that people EXPECT that a sufficiently powerful model could do this - I just think this is a fantasy that just makes "intelligence" a borderline-magical commodity that solves all problems.)
Making a simulation that's good enough to understand what's happening with complex biological organisms at a cellular or molecular level, well enough to correctly understand things, would take incredible amounts of compute, far beyond even what we're throwing at AI right now, but also in a way more specialized way. And I'm not really sure that AI would even make that any easier - if we wanted to do, like, molecule-by-molecule sims of something, I don't know that we're actually prevented from doing it now in a programming sense, but it's insane and we already have a working non-simulated version of orchids we can test on.
I agree that biology is wickedly complicated, and personally am unsure whether the exponential turns sigmoidal somewhere short of "answer all biological questions, solve all biological problems". But the key for molecular simulation is quantum computing, which is itself approaching faster and faster. If powerful research models can accelerate quantum computing research, which can then accelerate accurate biomolecular simulation, I think it's plausible (though lower probability than the combined probability mass of bad outcomes).
Even if the model intelligence converges on "nearly as smart as the average person, but 100 times faster/cheaper", that's still a variety of superintelligence that can upend the economy and concentrate power.
It's a much shorter jump from replicating existing research to doing new research than it is from GOFAI / Markov chains to replicating existing research. Look at the trend lines.
Not only awareness of public knowledge. There is a lot of stuff in various labs that hasn't been published. Public information is *guaranteed* to be at least a couple of months out of date. (Whether that makes any difference, though, is difficult to determine.)
I think there is a significant chance that research into large language models is essentially a dead end as far as achieving AGI is concerned, mostly for the reasons you state. Of course if the scenario is right, that question will be resolved three years from now and we will no longer be guessing.
i feel like we sorta got 'lucky', in that we have pushed pretty deep into the 'LLM' branch of the tech tree and there happened not to be any existentially-explosive landmines buried there
but that just gets me thinking about all of the other kinds of data one could train a transformer network on
LLMs "think in" something which approximates human mind-concept-analogues, probably to a similar extent that human language does, because it got trained on human language...
but what would a GPT trained on battlefield data 'think in'? or astronomical data?
what are the mental primitives of a transformer network trained on extremely detailed data of the physiological response of humans viewing pornography? or scenes of suffering?
I worry that the way we train these AI, they're finding the cheapest-to-compute algorithm to actually model the system we want them to model, and the instant we find some data that doesn't really *have* any cheap approximate prediction models, the network might sorta be 'forced' to instead gradient descend towards general intelligence
there's a lot of different kinds of data out there, a lot of additional tech tree branches to explore. we've mostly been focusing on the 'language' branch because it's convenient to have AI whose interface is human language, but I fear there might well be landmines aplenty in the adjacent branches of the tech tree
of course, it also seems plausible to me that maybe we've only explored like 5% of the LLM branch and maybe the 2027 model is entirely accurate, there's some clever trick or algorithmic improvement that lets you solve some of the problems LLMs have or simply bruteforce your way past them
LLMs, as such, never had the potential to provide an existential risk, only to do things like, e.g. propagate warfare...which might be an existential risk, but was one independent of the LLMs. And they do seem to be fanning intolerance, though not as much as social media. (Actually, they pose a few other risks to individuals. E.g. the rise of electronic boy/girl friends may be damaging to some individuals, but probably not to the species, or even civilization.)
The problem comes when you give them direct operational power and agency. (Actually, just agency MIGHT be sufficient.) Then you really need to worry about what their goals are. It's not like it's difficult to convince people to do something stupid. (Consider the "tide pod challenge".)
When I think about people doing something stupid regarding LLM agents, I think about the fact that neurosama peaked at 130,000 $5/month subscriptions
that's a lot of money, and Vedal lets her spend it how she wants because if he didn't the fan base would get mad
it totally blew up my worldview to see that, it's so far outside the predictions... i can't believe people used to argue with yud about ai boxes
and I wonder if the threat doesn't come from the warbots but instead the creepily-a-child AI vtuber based on 2022 llama2 who brags about being misaligned and has a 1s planning horizon
I think it's very likely RSI could be bounded, in two ways at least:
- the AI might have a subtype of intelligence that allows it to self improve to some extent, along some dimensions, while other dimensions would remain unattainable through the ongoing paradigm
- ignoring that, it's not at all clear to me that the gains in incremental intelligence would not be exponentially harder to get , potentially vastly outpacing the value of the accrued extra intellect.
I too have yet to give it a listen now, to be fair....
I agree with those being really significant factors. Our basic "scaling law" or whatever already more or less agrees with your second point.
On a related note - humanity is recursively self-improving. The US economy is recursively self-improving (at an exponential rate! for over a century!). That has allowed us to run away with things on Planet Earth, on a geological timescale, but on the flip side it can take decades for us to fully exploit technologies after we develop them. People imagine these feedback loops being incredibly faster for AI, but I think even if they do happen there will be more limitations than that. Even if we manage to make a $50b data center than spawns a superhuman intelligence, we'll hardly be able to turn around and make another one in no time flat.
> Even if we manage to make a $50b data center than spawns a superhuman intelligence, we'll hardly be able to turn around and make another one in no time flat.
Hasn't literally every AI company immediately turned around and compressed the smartest models into 1/10th the size and inference cost, at like a 10-20% penalty to overall capability?
Didn't DeepSeek use the existing o1 to reduce the combinatorially large space of possible connections to a much more manageable set, extremely cheaply and quickly?
I wouldn't necessarily count on this, in other words - once you have one god-mind, you can use it to make a lot of "god enough" minds a lot cheaper.
I do wonder what it means to say you have a 10-20% penalty to your superhuman intelligence, but I more wonder how much physical reality will be a limiting factor even so. E.g. if making one amazing AI lets you make 10 close-to-amazing ones... do you then have to wait five years to build another data center to get your 11th?
Because, "I can go from one genius to ten geniuses with [some incremental effort]" is cool but it's not remake-the-entire-world-in-a-year cool.
Even one uber-genius can remake the world. It doesn't even take a genius. The assembly line was a trivial change mechanically, but it remade the world. So did the steam-engine governor, which required a really smart inventor, but not really a genius. Once you see it, the principle is relatively easy to understand.
The argument is better when you invoke time-lags. Transforming the world can take time. That's what the million-robots are supposed to overcome, but I'm not really sure that's plausible.
OTOH, it's quite plausible that humans are smart enough to have discovered most of the "low hanging fruit" that increased intelligence makes available. Or maybe they aren't. Or it could be that just looking at things from a totally different way will reveal lots of unsuspected possibilities. Only one way to find out...and we seem to be rushing towards it.
That said, while I consider an AGI to be extremely dangerous, I also consider power-hungry human leaders controlling weapons of mega-death to be equally dangerous. If we get the RIGHT AGI, then this will solve that danger. (And to me "aligned" doesn't mean "obedient". It means valuing humans, and acting in ways that prevent them from going extinct, and in ways that minimize extreme unhappiness and foster happiness [but not extreme happiness]. And includes leaving them *relatively* independent.)
The assembly line was a lot harder to implement than the picture that popular history tends to paint of it springing fully-formed from Henry Ford's forehead. Adam Smith described a simple but recognizable assembly line in the first few pages of Wealth of Nations (1776), in the context of a pin-making factory. This was not a theoretical concept, but rather a description of a firm Smith had observed in operation and which he framed as a microcosm of division of labor across the entire economy.
There were a lot of concrete steps between Smith's pin manufactory and Ford's car factory that took well over a century to sort out.
The big step of interchangeable parts, components subassemblies built to tight enough tolerances that they could be mass produced separately rather than having nontrivial mechanical devices require each piece to be custom fitted to the others, existed as a concept by the 1790s. It's often credited to Eli Whitney's musket factory, but the concept neither originated with him nor was fully successfully executed by him. The big problem was that mass producing stuff to tight tolerances is really hard and required several generations of improved machine tools to get right.
Conveyor belts and fixed worker stations as a factory layout was also already well established for meatpacking plants and canneries. The Ford Motor Corporation's innovation was fine-tuning both the assembly process and the vehicle design to the point that it was practical to use a fixed-station conveyer belt for something as complex as a car. It took Ford about five years to get from the initial plant design (1908) to a working assembly line (1913).
It does seem very fast, and I agree that AI still has a surprisingly hard time on novel tasks and general judgment. On the other hand, the iteration cycle already seems to be speeding up, and we now have scaling vectors for pre-training, post-training/RL, inference time compute, and so on. People had reasonable amounts of success sort of brute-forcing the ARC benchmark, and eventually that brute-forcing may become elegant enough that it stops feeling exactly like brute force. So there are lots of unknowns, and personally I don't feel I can rule out pretty fast timelines.
In case you don't want to listen to the podcast there's also the website https://ai-2027.com/ (pay attention to the sidebar dynamic chart as you scroll down)
I dislike the immediate correlation that these scenarios draw between AI intelligence, usually measured by scoring on some test, and resulting gains in productivity or industry. I think it is very likely that AI will see exponential gains in measured intelligence, but this newfound AGI will not be that much more useful than it already is.
Our scenario's changes are partly due to change in intelligence, but also partly to change in agency/time horizon/planning, and partly serial speed. Data efficiency comes later, downstream of the intelligence explosion.
If not any of these things, what do you think it would take to make AI able to be useful?
It seems to me that the majority of the effort being put into new and better AI models is mostly aimed at maximizing this intelligence factor. Maybe trying to describe it as intelligence is incorrect, and it is more aptly named as better token processing or something else. Whatever this value is, cranking it up so far has had clear effects: the AI runs faster and produces output with greater clarity. You get better images and writing out of prompts.
I don't see maximizing this output as doing anything for many fundamental problems AI already has. There are the infamous hallucinations which persist throughout better and better models. There is AI having very poor vision, which would impede it greatly when trying to do real time processing of real world, real time events. Believers in the AGI singularity think that if you just maximize the AI in that factor, it will suddenly overcome these limitations. I don't think reality so far bears this out, and I am not sure if there are practical ways of making LLMs as they currently exist have reliable error correction.
This only says that AI will not be useful where it needs those things, regardless of its intelligence. A picture is not load bearing, so an AI can be very useful for making lots of pictures. But in order for someone to trust an AI generated architectural diagram, it needs not to be hiding some critical flaw the AI missed and can't explain.
I think hallucinations are getting significantly better with each model, though still not completely gone.
Does AI actually have poor vision? It does well enough in things like the new 4o image model. If you mean real-time video vision for real world tasks, I don't think it's terrible and I don't think people have worked on it that hard.
The main thesis of 2027 is that AIs only "need to" get to the point where they can do AI research, after which there's a lot more AI research and all the remaining issues get solved faster than you expect, especially if they're less "fundamental new paradigm" and more like "mopping up" by applying existing AI to new categories of task (like real world visual navigation).
I dunno Scott. I’d feel better if they didn’t sound so confident when they hallucinate. I mean I could prod Claude to the correct answer but…
A session with Claude 10 minutes ago;
Gunflint:
In a 2000 movie who said “He’s crazy. He probably calls everyone Vernon”
Claude:
The line "He's crazy. He probably calls everyone Vernon" is from the 2000 film "Me, Myself & Irene" starring Jim Carrey. This line is spoken by the character Irene Waters (played by Renée Zellweger) when Jim Carrey's character Charlie Baileygates (who has a split personality and sometimes becomes the aggressive Hank) calls her Vernon.
Gunflint:
I was thinking of Wonder Boys
Claude:
You're absolutely right! I made a mistake. That line is indeed from the 2000 film "Wonder Boys," not "Me, Myself & Irene."
In "Wonder Boys," the line "He's crazy. He probably calls everyone Vernon" is spoken by the character Hannah Green (played by Katie Holmes) in reference to Grady Tripp's (Michael Douglas) editor, Terry Crabtree (Robert Downey Jr.), who mistakenly calls Hannah by the name Vernon when they first meet at a party.
Thank you for the correction!
Gunflint:
No I think the line was spoken by the actor Richard Knox who was given the name Vernon Hardapple by Michael Douglas’s character in a bar in a discussion with Robert Downey Jr’s character
Claude:
You’re right, and I apologize for my confusion. Let me be clearer:
In “Wonder Boys” (2000), Michael Douglas’s character Grady Tripp makes up the name “Vernon Hardapple” for Richard Knox’s character during a conversation with Terry Crabtree (Robert Downey Jr.) at the bar. When Knox’s character confronts them after overhearing this made-up name, he says the line “He’s crazy. He probably calls everyone Vernon.”
This is exactly what you stated, and I appreciate your patience in correctly identifying both the speaker and context of this quote from the film.
I think this has something to do with neutered LLMs that will agree with you no matter what you say if you say it confidently.
The first response was definitely a hallucination, but if you insist on basically anything that isn’t obviously and fundamentally true (like, The Empire State Building is in New York) it will confidently correct its “mistake” to whatever you said was correct, even if it’s original claim was true.
I guess this is to avoid an LLM insisting its own hallucination was true (remember Sydney’s crazy messages insisting insane stuff was true?) over the user. LLM developers somehow constrain their models to default to the user if the user confidently asserts things, to prevent it from confidently asserting wrong things despite pushback.
A few months ago a British journalist tried an experiment whereby he told one of these LLMs that a notorious criminal (Dr Crippen) had been an expert trombone player (or similar). A week later he asked this LLM to provide information on the same Dr Crippen, and was confidently informed that he had been, you guessed it, a brilliant trombone player!
> I think this has something to do with neutered LLMs that will agree with you no matter what you say if you say it confidently.
Just as an aside, for anyone that hasn't used Gemini 2.5 (only recently opened up to the web interface vs the API), it is the ONLY model that doesn't do this.
I've found it hugely useful - I like to "adversarially collaborate" and test my ideas and arguments, and o1 Pro and Claude 3.7 and all the other models really suck for this - they immediately roll over at the tiniest pushback.
But 2.5 doesn't do this, stakes out a consistent position and maintains it over time, but is amenable to factual correction or rebuttals (but not vibes based ones!) - it's so much smarter than every other model right now, I've made it my daily driver.
My working hypothesis is that the current crop of LLMs rely heavily on the Clever Hans effect, named after a horse who could supposedly answer simple arithmetic problems by stomping out numerical answers with his hoof. On closer investigation, Hans turned out to be just stamping until his audience reacted to him having reached the right answer. LLMs seem to be pretty good at picking up the implicit and explicit assumptions of how people ask questions of them and giving answers consistent to those assumptions, like a college student trying to bullshit his way through an essay exam.
If you ask detailed questions in good faith, then this will tend to get you answers that seem good as far as you know as you assumptions, wrong or right, are parroted back to you. But if you ask them questions in bad faith, as I tend to do since I use LLMs mostly as a toy rather than a tool and I am a contrary little shit, then they often give hilariously bad results.
They also tend to do poorly under cross-examination, since they also resemble bullshitting exam-takers in that their initial answers tend to skew towards emphasizing familiar boilerplate concepts and away from the higher-risk area of answering the questions directly. Follow-up questions force them to elaborate on stuff where the correct answer is less well-established by their training data, where they are most apt to resort to hallucinations. Again, good or bad faith radically shaped your experience here: a good faith user will ask follow up questions that they genuinely don't know the answers, while mischievous users like me will ask questions we know the answers to but the LLM seems to be avoiding.
I don't actually think this is a fundamental problem with the LLM architecture... Did you see Anthropic's recent paper, 'Revealing Computational Graphs in Language Models'? They were able to apply their monosemanticity autoencoder technique to Haiku 3.7 (something which seemed utterly impossible as of last year, and i'm still keenly interested in anyone who can explain to me how they were able to do this... last I checked, if training a given AI requires x compute, then autoencoding it requires x^n compute where n is significant)
We learned a *lot* about the specific style of hallucination that you're reporting here
and it mostly seems like an artifact of the kind of reinforcement learning that the LLMs have been subjected to, an overwillingness to be helpful. Much of the effort and compute that's been spent on training these LLMs has ended up going, not towards making them smarter, but rather towards finding a more optimal point along the 'sycophancy / PR safety' curve.
I suspect we're already on a pareto frontier of that curve, that any movement towards fewer hallucinations as in your post will also mean more false-positive rejections, and vice versa. And I wish the AI companies would waste less time wiggling around on that curve. You can read more about it in the linked essay, they talk about how the post-base-model training causes LLMs to arrange themselves into a "refuse to answer by default" setup, and then treat every instance where they have an answer which is predicted to be uprated as a special exception to that general rule, and this is not a very good way to build accurate worldmodels
But my point is, I think the kind of 'making up bullshit' that you're presenting is separate from the kind of hallucinations that are an actual barrier to AGI or accurate world-modeling in general.
(Also, I think I have a better example of the kind of hallucination you're talking about, or at least a more humorous one. It's my favorite example of LLM sycophancy forcing them to come up with an answer even when they really shouldn't. The "aquaman agreement theorem":
I thought training a sparse autoencoder is done on the trained model so inference latency is more relevant than training time? It's expensive because the autoencoder is usually larger than the subnetwork being interpreted to disentangle the entangled weights.
What graphs and data I've seen say that hallucinations have dropped somewhat, but nowhere near to reach a level that would support use in any environment with hard engineering constraints. Trying to apply AI to this research just sounds like it might introduce even more of these errors, some of which might not be visible at first glace. More fundamentally, LLMs don't seem well suited for providing perfect answers, and yet a lot of the grand predictions for AI explosions rely on AI taking on jobs requiring a high amount of precision.
I don’t understand why hallucinations are such a dealbreaker? Excellent engineers get things wrong all the time, look them up, and then fix them. How is the AI any different?
AI struggles with recognizing mistakes as wrong and then correcting them. In order for the scenario of complete automation to happen, the AI would need to be able to do this all on its own. Otherwise it is still relying on qualified human engineers to validate the outputs.
There's a Claude Plays Pokemon run going on right now that I loosely follow and it's a great demonstration of the issue with hallucinations.
Basically Claude assumes wrong facts every so often. Like, the path from A to C _definitely does not go through B_ when in fact it does. And then it draws worthless conclusions from that, never questioning the "facts" it's hallucinated into existence. The run then becomes stuck again, until the next memory wipe that accidentally hits the hallucination. Because Claude seems utterly unable to identify when it's hallucinated.
Now, this is in a game that gives pretty immediate feedback on whether or not your assumptions are correct. Still, Claude is unable to recognize when it's wrong and would rather try the same broken strategy for the 100th time. Surely this time it'll work. You've gotta admire the optimism but walking through a wall doesn't work the 100th time either, and if the LLM is unable to realize that then hallucinations are a problem.
Imagine using the LLM on a domain where you don't know if it's wrong and can't tell it to stop hallucinating.
My wife used AI to help her compile information that would otherwise require her to do a lot manual searching in a reference book. She asked me to double check the AI's work...which required me to do a lot of manual searching in a reference book. The AI was almost entirely right, but made up one item that does not exist in the reference book.
Which means, we had to do almost the same amount of work as we would have done without the AI (looking everything up manually to double check it) in order to ensure that the final product was without error. Until the hallucination problem is solved that's going to remain the case, and AI will only provide minimal increases to efficiency. The only increase to efficiency AI contributed to this project was that my wife could copy and paste the correct entries from the AI's output, so she didn't have to type it in manually.
If skin depth is variable, where does that leave Beauty..?
I'm curious as to whether, instead of asking the AI to calculate it, you could have asked the AI to write a MATLAB script to calculate it? Perhaps that framing may have obliged it to think logically/pedagogically?
Re. matlab: I don't think it would have helped because it used a wrong formula, so I'd assume it would use the same wrong formula in a matlab script.
The other thing is that if it gave me a correct answer it'd save me time and money (directly - matlab is an on-demand license for the company, every time I fire it up we pay).
I'm interested -- did y'all account for the "higher fruit" aspect of increasing AI research? You've written a lot about this in other areas of science, how it's slowing down despite massive investment, etc.. Right now, AI research can still progress (to some degree) by algorithmic improvements, etc. It's at least highly plausible (at least to me) that serious mathematical (and perhaps cognitive) insights are going to be needed to make progress past some point fairly soon; and it's also plausible to me that that is exactly what AI is going to be extremely bad at for quite a long time.
Just curious whether this was discussed. Thank you for the post and the role, either way!
If the hallucinations are ever completely gone, then the AI will cease to be creative. What is needed, instead, is for them to be bounded and recognized. Then they become "theories to be tested".
> Data efficiency comes later, downstream of the intelligence explosion.
If I'm interpreting you correctly, you're predicting that we'll crack recursive AI intelligence improvement *before* we crack active learning? If so, that prediction has some wild implications, e.g.
- Most meme-worthy: We'll get the discount sci-fi future where humanoid robots become commonplace, not because the human form is optimal, but because it lets AI systems piggyback off human imitation for physical tasks even when that form is wildly suboptimal for the job
- Human labor will likely become more valuable relative to raw materials, not less. In a world where all repetitive, structured tasks can be automated, humans will be prized specifically for handling novel one-off tasks that remain abundant in the physical world
- Repair technicians and debuggers of physical and software systems become worth their weight in gold. The ability to say "This situation reminds me of something I encountered two years ago in Minneapolis" becomes humanity's core value proposition
- Large portions of the built environment begin resembling Amazon warehouses - robot restricted areas and corridors specifically designed to minimize surprising scenarios, with humans stationed around the perimeter for exception handling
We accelerate toward living in a panopticon, not primarily for surveillance, but because ubiquitous observation provides the massive datasets needed for AI training pipelines
Is that an accurate assessment of your world model?
I think my main critique with some of the fast take off is along the lines of “9 women can’t make a baby in a month”.
A lot of the acquisition of new knowledge in materials science and biology are necessarily time and labor intensive. You can’t just intuit them out of first principles, you have to actually obtain the data first.
I feel like a lot of fast take off scenarios envision the AI just thinking new discoveries into existence and I’m not convinced that the world actually works that way.
Accurately modeling the physical world is way way harder than people realize. This is why civil engineers still use shake table. Accurately simulating the universe would require more computational power than can ever exist (even if you turn every single atom into a quantum bit. AI has no answer for this problem.
Watching now. I'm hoping world governments realize an arms-race is almost guaranteed to end poorly, and they commit to an international lets-make-utopia project. China doesn't want to get paperclipped or grey goo'd any more than we do, and Chinese researchers have already said as such. Since USA is currently going downhill, probably the UK + Deepmind would take the lead on this.
I'm more than a little anxious on this subject personally, and I have to say I'd be significantly more comfortable if the government of the country where most of the cutting edge AI research is taking place showed even basic competency navigating ordinary societal conflicts within the scope that governments have been handling for the last hundred years.
>if the government of the country where most of the cutting edge AI research is taking place showed even basic competency navigating ordinary societal conflicts
<mildSnark>
The most ironic outcome would be if the governmental incapacity (to, amongst other tasks, intelligently respond to technological risks) was primarily driven by polarization, and that, in turn, was primarily driven by the engagement-maximization algorithms running in social media platforms. :-)
I did find it notable that they kept up bringing the analogy of WW2 for governments taking control of industry and converting factories, even though we're not only not at war with China, we aren't even in as much of a conflict as we used to be with the Cold War.
Firstly, I have not watched the video -- is there a text summary (beyound what's in this article) ?
Secondly, what is meant by "Intelligence Explosion" (as well as "Superintelligence") in this case ? Are we talking about programmers (and perhaps AI researchers) losing their jobs and/or having to learn an entirely new toolset and thus producing more incrementally useful LLM-based tools; or a nanotech-based machine intelligence that can answer any conceivable question while converting the Earth to computronium; or something in between ? I understand that you expect "economy to be automated" by 2029, but I'm not sure what that means either -- are we talking post-scarcity Star Trek replicator future, or something exactly like today but all the container trucks are autonomous, or what ?
Intelligence explosion means that AIs become good enough to help with AI research, so AI research gets hyperbolically faster as outputs feed back into the inputs.
Yes, I understand this, but how is this projection different from the normal course of technological development ? For example, humans used to produce binary code by hand; then compilers were invented, and programming research got hyperbolically faster as outputs fed back into the inputs, creating a plethora of programming languagers. Are you expecting something similar to happen with AI, or are you expecting something far beyound that; in either case, what practical implications (if any) do you foresee ?
I don't know if you should think of it as "something similar" or "something far beyond". I think that existing progress has already speed things up 1000x compared to the Stone Age, and that this will also speed things up a few orders of magnitude. I think probably the shape of the hyperbola will change between last decade and next decade, but this may look more like a return to trend than something completely unprecedented. See https://slatestarcodex.com/2019/04/22/1960-the-year-the-singularity-was-cancelled/
Is there an implication that this will matter a lot for things?
Its pretty easy to envision worlds where "AI research gets hyperbolically faster" has anything between "Earthshattering" and "little to no" effects on the world.
Perhaps this is a tired question, but in some sense, drug research is "hyperbolically faster" now than it was 200 years ago, but it's also a lot harder to discover useful things. Why do we know that the "AIs are faster at AI research" effect will dominate the "It's harder to produce useful gains from AI research" effect? Or maybe this is a consideration that you just give low probability to?
You may also want to read the takeoff supplement, which is an attempt to answer this question - https://ai-2027.com/research/takeoff-forecast . The short answer is "We tried to figure out how much time it would take (without AI) to reach each milestone, and how much AI would speed it up, and it speeds it up enough to matter a lot."
If you were going to make the low-hanging fruit argument, you would want to quibble with our estimates for how long it would take unassisted humans to reach each milestone. We got these by asking a combination of superforecasters and researchers currently working on the problem.
I appreciate the response, and the forecast document is a great read.
But it misses what I was trying to get at. The simplest way I can think of to put it is: How do we know it isn't a logistic function? How do we know we're not in an initial part of the curve that looks exponential but then we later find out that no, actually, it's asymptotic. Take whatever you want for "it" here. Cognitive task capability or whatever.
I didn't see a discussion of why this wasn't the case or, if it were, what the point at which "cognitive task ability" would go asymptotic would be. You just sorta state at one point, "we don’t have reason to believe that humans are near the limits of general intelligence."
Maybe I'm naïve, but this seems like a pretty fundamental unknown.
If it's like normal trend curves, it is sort of logistic. However we don't know where it starts flattening out.
Just for example, pretend that transistors weren't possible. Where would the curve for "computers" have flattened? Vacuum tubes could have become a LOT better than they were. (As discrete transistors were taking over, GE came out with grain of rice sized vacuum tubes with a long half-life.) But we would never have gotten HERE unless there were some other technology that could move in.
So to predict when the logistic curve is going to flatten, you need to predict what future technologies will become practical, and what their limits are. Saying "it's going to flatten out" is almost certainly correct, but that doesn't say when.
Yes, but positive feedback cycles tend to rapidly break. You sort of acknowledge that with, e.g., various comments about robots, and with the various comments about required compute resources (and about the building of research facilities), but intelligence, in and of itself, probably has strong limits.
I suspect that the various feedback cycles through different processes will keep the expansion of AI intelligence below the level of an explosion. (OTOH, I can well believe the rate of expansion would be considerably above linear, but probably not super-exponential. Parallel processing experience tends to show that dividing the process into lots of independent threads yields heavy costs in communication.)
I think if AI becomes merely good enough to *help with* AI research, that is not enough to expect an intelligence explosion. Processes with multiple inputs tend to experience marginal returns wrt any one of the inputs as other things start being the bottleneck. Thus, having one input increase, even to infinity, doesn't drive the whole process to infinity.
Let me give a concrete example. Imagine AI became good enough at programming to be able to correctly write code to spec. This would certainly help with AI research, since a good part of that requires writing code. But even if the AI because increasingly good this, that would not speed things up HYPERBOLICALLY, because no amount of writing code *by itself* is enough to succeed at AI research.
To get hyperbolic improvement, you need the output to feed into ALL the bottlenecking inputs, not merely one of them. Just being able to *help* with AI research usually won't be enough for that; you really need a stronger condition.
"Take a sip of your drink every time Dwarkesh asks a question about bottlenecks; take a shot every time Daniel answers that his model already includes them and without the bottlenecks it would be even faster."
...and I do think the Dwarkesh interview addresses this concern pretty thoroughly. But you can also check the takeoff supplement at https://ai-2027.com/research/takeoff-forecast . CTRL+F for the part saying:
"We’ve heard a few people make the argument that because automation can only speed up some aspects of R&D so much (e.g. gathering human data), progress will be bottlenecked on these tasks leading to not-very-large progress multipliers (see Amdahl’s Law)."
And the part saying:
"This view posits some combination of (a) training compute matters far more than algorithmic progress (b) experiment compute matters far more than researcher quality and quantity for algorithmic progress, (c) algorithmic progress is driven by adapting to larger compute scales and (d) once AI R&D is fully automated, algorithmic progress will be close to fundamental limits which will make compute the dominant driver of further progress."
Actually it's even worse than that, since the present-day "AI" is unable to perform any kind of research at all. It can very quickly comb through all indexed human knowledge (circa 2022 or so) and retrieve the pieces the user asks for; it can also combine multiple documents or extrapolate connections between them to create a speculative document that, had it existed, would've occupied that place in the corpus.
But the trick to making true AI does not exist anywhere in the corpus; thus it's not something that could be produced in response to a clever enough prompt by generating the next most likely token. Discovering such a trick would require independent and sustained experimentation, and doing that would require something with some measure of intelligence and independence, thus making it a chicken-and-egg problem. The same goes for novel discoveries in physics, engineering, etc.
>The AI Scientist automates the entire research lifecycle, from generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript.
> The AI Scientist can incorrectly implement its ideas or make unfair comparisons to baselines, leading to misleading results.
> The AI Scientist occasionally makes critical errors when writing and evaluating results. For example, it struggles to compare the magnitude of two numbers, which is a known pathology with LLMs. To partially address this, we make sure all experimental results are reproducible, storing all files that are executed.
So this "AI Scientist" is more like a standard LLM than a fully autonomous agent: it can help you look something up, or pump your intuition, or take care of some bit of drudgery; but ultimately it's you, the human, doing the work.
Many Thanks! I agree that the problems you cite are important. I view them a bit differently. I view them as reliability problems, on a par with hallucinations. Whether the errors can be pushed down to acceptably low levels _is_ an open question, and I view reliability as probably the key question for whether AI will be as transformative as is anticipated.
If we can get sufficient reliability, I think autonomy will be easy.
5 years before AI reaches an intelligence level where it theoretically could be used to run the majority of the economy even with no further improvements? Possible. 5 years before AI is literally physically hooked up to and running the majority of the economy? Not a chance. Industries don't adapt that quickly.
In WWII, it took three years from government wakeup to the economy having completely pivoted to a war footing. We discuss whether this is an appropriate metaphor further on the podcast.
Yup, we think having superintelligent AI planners adds a lot of state capacity and political agility. We also think that there will be a pretty intense arms race with China going on at the time, and that this also encourages governments to become more capacious and agile.
Well, planning only goes so far - in my (limited) experience, the execution is much harder. And execution gets even harder if worker incentives are not aligned with the top-level goal.
In case of WWII, there was an attack on Pearl Harbor (and Philippines), and it was pretty clear to everybody that the war has to be fought.
In case of AI - why would middle-level managers / senior employees implement AI planner directives with haste? Won't they see that it would soon cost them their jobs, income and social status as result? And won't they provide quite a lot of friction in AI adoption as the result?
I think execution is part of the planning in this case. We think of planning as the as the high level decisions and execution as all the little day-to-day decisions and problem solving. But a superintelligent AI could just do both.
If we are at the stage that superintelligent AI does everything, than transition to Singularity is finished, right?
But we are specifically discussing transition process - so, _at that stage_ something must be different about (available or deployed) AI capabilities...
I don't think that discussion went far enough https://www.astralcodexten.com/p/introducing-ai-2027/comment/105740438 If you think about actual WW2, the US had been building up its military during the 30s, observing the UK fighting Germany and giving them Lend Lease, undeclared war against Kriegsmarine U-boats in the Atlantic, then Japan attacks Pearl Harbor and we shift into a war economy. Communist China has been around much longer than Nazi Germany, and while we did actually get into a hot war with some of their forces in Korea, we never mobilized the country's resources against them even to the degree that we did against the USSR. Very big geopolitical changes would need to take place for our government to be treating this like WW2.
The example we looked at most closely was bombers, where I think the US really was asleep until 1940 (not 1941!), so it's a fair comparison.
I don't think we did full mobilization during the Korean War because we didn't need to - we got an okay truce as it was.
But also, I don't think our exact story is "the AI arms race is important enough for full mobilization". The story is more like "the AI will want to move fast, it will get what it wants for various reasons, and full mobilization provides a baseline for how quickly fast things can go".
Full mobilization in a hunter-gatherer tribe can be as simple as one evening chief saying 'tomorrow we will go west in search of a better valley, this one is no good'.
During WWII it took 3 years (and actual war casualties to keep everybody motivated).
Supply chains and manufacturing processes are more complex these days, and they span the globe (instead of being mostly concentrated within one country) - so if humans would be doing it, it probably would take _longer_ to mobilize. Ok, AI is faster / better - but would it be so much faster / better at affecting change as to compensate for the increased complexity (since WWII), at least initially?
We didn't do full mobilization during Korea because the front was so narrow that it wouldn't have helped in the ground war, and the enemy's war production was mostly coming from PRC and USSR which we couldn't 💣.
This seems politically naive. The US at that time had a relatively autocratic president with the strongest popular mandate in a century, and he had almost a decade of practice using emergency powers.
Most old governments are sclerotic, and can't suddenly change the trajectory of the state. They typically drown in the face of unprecedented threats.
We had such sclerotic a government until 3 months ago, and that's not just my opinion. It was the widespread consensus opinion that the US government was struggling to perform actions. And whatever we now have in April 2025, it certainly doesn't have the strongest popular mandate in a century.
Could you please operationalize "wake up and pivot", by explaining who makes that happen and by what politics? I'd like something more convincing than "existential threat drives consensus", because I think that only happens in the movies, or in real life only in rare cases where a political genius already lined up the political dominoes. Heck, look at Covid; there were people in March 2020 who predicted that the threat of Covid would drive national or world consensus, and... it didn't!
The US seems to have an extremely autocratic president right now. And parts of the country seem to strongly approve.
OTOH, I would not be surprised if Trump collapsed tomorrow. To my non-medical eye he looks like a good candidate for a heart attack. And I really doubt that Vance would swing much weight with the Trump-base.
So basically both scenarios are dystopian given that in "Aligned" scenario couple of US politicians and tech CEOs have archived complete dominance over human civilisation forever and crushed all competitors (geopolitical and economic). As non-american I am not sure that this looks much better compared to a unaligned scenario.
I agree both scenarios are at least somewhat dystopian. We got the "aligned" scenario by conditioning on the AIs being aligned and humanity surviving, then asking what the most likely version of that was.
some sort of international agreement where Superintelligence benefits can be democratised or Archipelago scenario where different nations and communities can live their values with post-singularity tech with no foreign interference. I don’t deny the value of the analysis but was just hoping we have better options.
I think lots of alignment researchers would like that in principle, and some people should be working on it, but figuring out how to get there seems like it requires a mostly non-overlapping set of skills from technical alignment research. So the question facing technical alignment researchers (to oversimplify a lot) is what the different probability-weighted outcomes look like if we solve alignment, compared to if we don't, and which world they'd rather live in.
It's possible—and some people argue—that the world where we solve alignment is worse on expectation. But personally, I'm not seeing it.
I'm at least as opposed to US politicians and CEOs as the next guy, but I don't think the political structure will matter much, or maybe even meaningfully exist, if the world became infinitely wealthy and technologically advanced. That also applies to worries about China imposing authoritarianism.
I think a scenario with infinite automated wealth production could be very bad for the average person, with sufficiently centralized control, because the people in control may not be inclined to share around even a tiny portion of that wealth.
In a world where everyone could have all the material comfort they could want, what separates people with power and control from ordinary people? Not a lot, necessarily. But if you're a person with power and control, and you *want* there to be things separating you from ordinary people, you can achieve that; that's what it means to have power and control. And one straightforward way to achieve that is to ensure that other people can't have material comfort, unless you let them.
> Not a lot, necessarily. But if you're a person with power and control, and you want there to be things separating you from ordinary people, you can achieve that;
There will always be intrinsically limited goods - human capital, land and real estate in places people want to live, art work and other one of a kind things that only one person can own the original of, and more.
Rich people are going to have all those things, they'll have a lot distinguishing them from all the UBI plebs. Look at Captain Picard's vineyard, still a status marker in a post-scarcity society - arguably much MORE of a status marker, because it's probably impossible to buy with money.
And as to SOMEBODY sharing the wealth - the "Giving Pledge" has 236 billionaire signatories. That's the one that Gates and Buffet and Zuck have all signed where you commit to giving away at least half your wealth.
There's only like 700 billionaires in the US, a substantial fraction of them are charitably-minded enough to commit to giving the majority of their wealth away.
I'm not sure what you mean by "all it takes is one." I don't think one person committed to giving away most of their wealth is going to be sufficient to implement a UBI. I think this is a way that society *could* be organized, but there isn't a guarantee that a UBI would be implemented at all. Even people who're billionaires now may not have any real say if they don't have a hand on the levers of AI in the event that that really overturns our economy.
Yeah, I was more pointing to a model that a good chunk of the richest people today don't care as much about wealth as they do giving back and / or helping humanity in the aggregate.
I don't think the current billionaires are going to matter much in that future - the people who own the AI + robot factories and produce literally everything are going to make them look poor - they'll be yottillionaires or something.
We'd expect a greater concentration of wealth if 99%+ of jobs are eliminated and returns to capital are sharply concentrated in a few AI-dominated spaces. I'm just mapping that out to the extreme - we get a thousand yottas instead of a thousand billionaires.
But in that schema, indeed, all it takes is one to UBI everyone, and the "roughly a third" ratio argues there will indeed be at least one.
No amount of infinity will be enough if a fat fuck at the top gobbles up everything and makes it a law that you never own anything, only rent it from its eternal God-ordained owner.
Same reason big-name humans who already have a net worth measured in tens of billions hoard it in hopes of parlaying it into even more money and political power, instead of donating 99% to charity and/or pet projects in the arts and sciences, while retiring to a life of the most fabulous hedonism money can buy.
Whatever that reason is, I doubt it will change no matter how many digits are added to their net worth, and I’m inclined to believe their power-seeking will stop at nothing short of reducing the rest of humanity to a Sim City in their palm.
Individual big-name humans today cannot unilaterally provide for the bulk of humanity. The can certainly make an impact, and many do so in actually by using their money for charitable causes, but realistically the impact even the richest person can have is limited. E.g. if the richest person alive today decided to spend his entire net worth to feed the hungry, at $2/person per day he can could feed earth's entire 8 billion population for less than a month.
This is very far from being "infinitely wealthy". The resources concentrated into big-name humans today are vastly more than they need for their own lives, but also vastly less than they need to support humanity as a whole. The fact that individuals are not doing so today is given by the fact that they CAN'T, and thus not a clear indicator that they WON'T.
Any billionaire can easily secure the food of a small village or a city block (~10K people) indefinitely, they do not do so for the most part.
In fact, billionaires pretty often do even less than people with far less money do, paying their full taxes they owe to the state they're a citizen of.
I've been anticipating this a lot since Scott first mentioned he was doing it, because Scott's views on AI would be so massively consequential if true and because he's never laid them out in explicit detail, and this looks even more comprehensive than I hoped for. Extremely impressed that Dwakesh got him to also do a podcast.
Still, a bit concerning that a guy with epistemics as strong as Scott's thinks the future could be this precarious.
I want to clarify that I mostly wasn't involved in the actual forecasting. I think the forecast is top-notch and very plausible but am nervous describing these as "my views".
I think that it's hard to make a specific "best forecast" in a multidimensional space because of issues with median vs. mode, but this is somewhat motivated by trying to forecast things that come true.
A lot of this discussion seems very software focused, but we actually need hardware to run all this stuff. Is there discussion of things like critical minerals, undersea data transfer cables, chip manufacturing supply chain vulnerabilities, etc? Thanks!
Short version is: we assume hardware will continue to scale up at about the same rate as previously, up to weak superintelligence. When we predict weak superintelligence in 2027/2028, it's because we think that current supply chain trends get us there then.
We think weak superintelligence will be able to optimize the supply chain somewhat better than humans, partly because it's smarter than humans and partly because it will be better than humans at some kind of political/bureaucratic tasks - but not too much changes if it isn't.
"Short version is: we assume hardware will continue to scale up at about the same rate as previously"
Ok I promise I will read the whole thing when I have some uninterrupted time but this quote really doesn't inspire confidence. It's not impossible to have hardware to continue to scale up at about the same rate, but it is quite.... implausible?
During this period the optical shrinking of the process nodes was still a viable way to increase transistor density. We're at the end of this road approximately now.
AI compute scaling happens because of a combination of:
--- More money being invested
--- TSMC increasing supply
--- TSMC improving at the specific types of GPUs used for AI
--- General chip progress
Chip progress isn't a very important part of this equation but does shift things a little bit. The scenario's take on this is:
"For improvements in chip efficiency, we extrapolate Epoch AI’s historical trend of 1.35x per year and find it is consistent with the already reported performance of upcoming chips such as NVIDIA’s GB200 as well as rumoured plans for the Rubin series detailed below.
The widely adopted state of the art GPU in 2024 is NVIDIA’s H100, which has 1e15 FP16 FLOP in raw performance. In three years, we predict the widely adopted state of the art GPU to be NVIDIA’s Rubin GPU (R200), which we project to achieve a ~2.4x improvement over the B200 (widely used in 2025-2026) to 6e15 FP16 FLOP performance. We think this will be achieved through a 1.2x increase in die size, a 1.7x increase given the transition from TSMC’s N4 to N3 process, and a 1.1x increase from other miscellaneous improvements."
Re. Chip progress: yes 2027 chips are definitely in the pipeline, probably somewhere between verification and debug. No, we don’t know what problems they will find, not yet.
But I want to to show you how naive these projection are for the 2.4X performance increase:
1.2X die size increase - fine, no problem here
1.7X increase because of the move to 3nm from 4nm. No, this is what, (4/3) squared (1.78) - are they that naive? You never get the full theoretical improvement from scaling like this, no one who designs chips will ever project that. Maybe 1.4?
1.1X from “miscellaneous improvements” - hot air, straight to the bin.
It was roughly 100 times raw compute from gpt-3 to 4, in addition to a hugely underappreciated effective compute multiplier from the low hanging fruit of breaking n^2 training via MOE sparsity. Now, 3 to 4 was indeed a big deal and all that, but even a couple more similar leaps seems insufficient to qualify for some of the more basic aspirational capabilities predicted. This seems a simple intuition pump as to why only getting in the OOM of 1000x raw compute by 2030 does not inspire confidence, especially when successive tiers of low hanging fruit( like training nets at lower precision, and pushing sparsity far beyond MOE heads) on the effective/algorithmic side appear to already have diminishing impact (arrakis failures, 4.5 underperforming).
All this in addition to, as other comments allude, the blind faith in moore's law is quite thoroughly misplaced and speaks of commonplace ignorance. Actual shrinkage of the feature size on transistors has nearly entirely stagnated (relatively), and yet the genius marketing decision to just keep naming successive generations 4nm, 3nm etc (which is an entirely made up number, totally unrelated to actual transistor size!) has been remarkably effective in propping up false lithographic confidence.
The nitty gritty at https://ai-2027.com/research/compute-forecast honestly has some similar feelings of reading a deep research report: lots of headspace in the right direction, but just a clear lack of sufficiently nuanced understanding of the technical specifics to make real headway with regards to accurate prediction, and overreliance on a questionably robust source (epoch). A woeful lack of appreciation for flop/$, the end-all-be-all metric, and consumate agreement with Nvidia marketing numbers -- the B200 is honestly a lithographic failure, node jumps are so dismal that in 2 years in the hottest gpu market EVER all Nvidia could offer was gluing two N4 dies together at a minimal discount vs Hopper; the real accomplishment of Blackwell is being artificially whitelisted to run on NVLInk-72 switches and Nvidia graciously allocating some dedicated 4 bit tensor FPUs to the die.
2 more years of an even hotter market will give us a Rubin, on a "1.7x density improvement!" node where the SRAM cells are the same size, most of the finfets are 30% smaller, hmmmm.... (for reference, TSCM 7nm to 5nm was a claimed 1.84x density improvement, when the actual chip shrinkage at same transistor count was only 30%, or a ~1.4 density improvement. Sure seems right in line with the 3nm finfets!)
And this time, there's no new NVlink to save the day, and there's nowhere left to go once we pick off the forbidden 2-bit acceleration fruit......
Actually, I'm rather certain that we know ways to increase the circuit density. The problem is that they're expensive. (Basically go thee-D and use LOTS of cooling.)
For that matter, design the chips to work in a REALLY cold environment, and bath the system in liquid Nitrogen. (Helium is you're feeling like overclocking.)
I read about a lab in Kentucky that did a demo version of this well over a decade ago, though they didn't use extreme cooling (merely silver heat-pumps on each chip). There are obvious reasons why that didn't become a commercial product, but if price were not object...
Yeah, cryo is hot (good one, eh?). I’m very familiar with the cooling challenges, and consider them one of the less-appreciated obstacles on the way to increased compute densities.
But cryo is on a different level of tough. No standard rel tests go below -40C. There are no models, everything has to be developed from scratch. Like you said, if money is unlimited it can be done, but still a tough slog.
If it's not a secret, what finally convinced you to change your strong anti-podcast stance? Did Daniel super-persuade you to wingman him or was it your initiative?
Disappointed to see insufficient discussion of how the trickling down of agent capacity into actual material production of goods and services will happen. There are vast corporate bureaucracies, government offices, post offices, and a million Etsy-shop-scale small businesses which haven’t even exploited the added marginal productive capacity of the *Dotcom* revolution, to say nothing of AI.
My first software job (in the '00s) involved writing code for clients who my coworkers described as "Just getting into the '80s technologically", previously using pencil & paper and operating on credit with their suppliers. My understanding is that in Japan they still widely use fax machines.
One possible change in this regard is that AI, in virtue of its flexibility, may be able to make its own adoption go smoother. The guy you hire to integrate AI into your business and the integrated AI might be the same guy.
The summary has “if aligned” and “if not aligned” scenarios, both varying degrees of bad but still. Does the forecast address the probability between these two alternatives?? Seems kind of important.
They do not seem to put any weight on partial alignment; super intelligence understand their alignment without thinking about it (HA) and are instantly not situationally motivated
Eh, I suppose NTS could think AI will be short of superintelligence, but still expect it to do better than the average NVIDIA investor thinks. There are a lot of short-timeline people who are long NVIDIA, though, I don't think it should be too hard to get more pessimistic than average.
This is ridiculous, Scott. Maybe things are different in your bubble, but in real life 99.99% of people (and investors) don't have timelines as short as you. I know a dozen normies long NVDA; my mom is long NVDA!
NVIDIA is down 32% YDT; is your opinion that this is because timelines got longer? Come on. NVDA price is about non-AGI use cases of AI
Careful! It is extremely possible to be 100% right that "they will not happen" and still lose everything shorting NVDA. And if you do, it is also extremely possible to sue a person who gave you financial advice without a license.
No no no no don't short NVIDIA, demand for their processors can easily continue anabated, LLMs exist and improve, etc. without any of the extreme things happening.
First step, find a counterparty willing to accept conditional payments - and, more broadly, continue respecting the associated property rights - in the event of humanity's extinction.
The core of an AI model is a honking-big matrix of numbers that transform input data into output data. In a linear regression: if y = wx, w is the weight, which informs how much y changes for each change in x. The weights of an AI model generalize from that (massively oversimplified, of course).
Generating the weights is what training is for, but once that's done, the ultimate product is just a matrix of numbers. It's not usually visible to people using the model, except for open-weight models like DeepSeek, but if you can break the security on the model and extract the weights, you've just saved yourself millions or billions in training costs.
As a general rule, LLMs (like ChatGPT) are very good at answering contextual questions like this. You can just paste the piece of text that you don't understand and ask question and it will get it right (99 percent of the time, likely better than the median human). Just posting this on the small chance that you haven't updated on the latest AI capabilities yet and hope it's helpful and taken in the spirit intended!
The training of an LLM is using a very expensive algorithm to find a specific set of matrices, that when applied in the correct manner, can do intelligent things. The values found for these matrices are the "weights". Since the valuable thing is those matrices, the work of training the LLM (billions of dollars potentially) can be skipped if one can simply gain access to a computer from another lab that has already run the algoirthm to discover those matrices, and simply copying them.
And thank you as well, Ivy! I appreciate your and Nicolas's willingness to help me out, when so many commentators would have responded with something like "RTFM, noob!" (grin)
In addition to what other people have mentioned, you don't have to do any kind of illicit hacking to "steal the weights" of a model. A process usually called distillation allows you to use the inputs and outputs of one model to train another, usually faster than the original model was assembled. This is part of the DeepSeek controversy - OpenAI claims DeepSeek was partially trained via distillation from an OpenAI model.
Basically, to "steal" the model. The weights are the model itself. A big list of floating-point numbers (i.e. decimals, without too much simplification). If you "stole" the weights of ChatGPT from Open AI you could run an exact replica of ChatGPT assuming you have the hardware and networking infrastructure.
Of course to use "Steal" here is a loaded moral choice given that those weights are obtained by "stealing" the entire internet without honoring software licenses or robots.txt, so "stealing" them is as moral as stealing from a thief. Go China.
All hardware runs on 1 and 0's, weights are a type of program (extremely limited and your all insane for thinking its intelligence); but they probaly would also consider it as escaped if it copied the other 1s and 0s
nn's are a type of hill climber, the "weights" are a location on a mathematical landscape, if you consider something like wolframs "rule 110", there was a systematic way to label a "program space" and with the number you could look at the definition and recreate the machine.
If you're correct... what am I supposed to do? Stockpile money for a future of permanent unemployment? Start prepping for a war between humans and robots fighting on behalf of tech oligarchs? Lay back and prepare to enjoy a UBI and an experience machine?
This is an important discussion that's worth addressing, and given Scott's position as the rightful caliph / leader of a flock, I think it'd be worthwhile for him to weigh in with practical advice beyond just predictions. I think there are several rational approaches available:
- If you have the appropriate skills or could learn them, try to get involved in AI policy / governance, alignment / interpretability, or even capabilities if you think it's more important that the race be won in the West vs China
- Financially, get exposure to the companies that have the most plausible route to ASI and its benefits, ie Microsoft, Google, Meta, Nvidia. Sadly I don't know of a way to get exposure to Anthropic via public markets.
- Prepping / survivalism isn't necessarily insane, I think there are probably middle-of-the-road scenarios that involve non-ASI civilizational collapse (eg nuclear war between the US and China), and a lot of the people at the forefront of some of this stuff are hedging their bets in the same way (ie Thiel's New Zealand citizenship)
>Prepping / survivalism isn't necessarily insane, I think there are probably middle-of-the-road scenarios that involve non-ASI civilizational collapse (eg nuclear war between the US and China), and a lot of the people at the forefront of some of this stuff are hedging their bets in the same way (ie Thiel's New Zealand citizenship)
Pay particular attention to the fact that nuclear war slows down AI (by a variety of means), and removes entrenched lobbying interests in its favour (including the AI companies being in control of significant chunks of the public square e.g. YouTube, Facebook, Twitter), and also to the fact that in general you want to plan for not-Doomed timelines because in Doomed timelines your plans are irrelevant.
The Age of Em says what you should do if you think that you're going to be vastly outclassed by a class of new intelligences that will price you out entirely of the job market:
(1) Invest in any company that you think will own/use/employ/partner-with those intelligences.
(2) Invest in any resource or financial construct that is not dependent on work to generate value (e.g. real estate).
Eager to read the full scenario but in the meantime, I guess I have a different assessment of the original "What 2026 looks like" article than "He got it all right." I think he really was very good on the tech itself and I'm not familiar enough with the alignment community to speak to those predictions. He gets a lot of mileage out of "it won't actually be valuable yet but will do cool toy use cases" which was very prescient but I wonder if it is actually the same skill as guessing when AI will be valuable enough to, say, start impacting GDP.
The big miss, though, is the discussion of propaganda. LLMs are good at writing what sounds to us as authoritative text, but already we are forming antibodies. "Looks like an LLM wrote that" is increasingly a response I see to posts online. I don't see much evidence that LLMs are good at persuading people of things. I know that the superhumanly-persuasive AI has been a bogeyman of Yudkowskyian AI theory for a decade, so it makes sense someone on LessWrong would overestimate its likelihood, but personally I didn't believe in it a decade ago and I still don't now so I'm probably on the opposite side of the spectrum.
I mention all this because we shouldn't be too surprised that an AI expert's predictions about AI itself were good but the predictions about how the rest of the world interacts with AI aren't as good. He's not a rest-of-the-world expert. I'm not sure anyone is. So I appreciate getting more predictions from him and the team about, say, coding agents. Past that, I'll read the guesses about the US government, the Chinese government, WW2-style nationalized economies, and how public opinion reacts to all this...but why should I find it credible? Actual experts on politics and public opinion have awful records predicting this stuff. I tend to think that near-genius AI researchers (and near-genius rationalist bloggers...) are probably worse than average at predicting the behavior of the ordinary people in the general public and the mostly ordinary people who serve as politicians.
Research (by Tetlock) has previously shown that forecasting depends a lot more on skill in forecasting specifically, compared to subject expertise. So you'd expect a good forecaster who is not a political scientist to be better at forecasting politics than a political scientist.
>So you'd expect a good forecaster who is not a political scientist to be better at forecasting politics than a political scientist.
And how would a forecaster even know what to talk about? Do you even know why a political scientist is called a scientist at all? We are not called political theorists or political commentators for a reason. According to your argument, a forecaster would be better in predicting cancer evolution than a medical doctor too? Or to predict an economic crisis before economists? This comment is so ignorant it hurts, I'm not even sure I'm sorry to be this blunt.
Your argument seems to be "ad absurdum", but that's not going to work if you just repeat what I already said I believe. "predict an economic crisis before economists" is precisely one of the examples tetlock examined in detail
> how would a forecaster even know what to talk about
They would research the subject for as many hours as it takes to become confident, for instance AI-2027 seems to have involved thousands of hours of research.
And such research would come from the material made from domain experts. A forecaster/statistician can be plugged in whatever context, given they can be guided. And your hubris needs some balance mate, I gave you precise examples, not a generalization and certainly not an ad absurdum (which is what YOU did, not me).
> And such research would come from the material made from domain experts. A forecaster/statistician can be plugged in whatever context, given they can be guided.
As AI capabilities advance, the question of whether AI persuasion has much of an effect depends decreasingly on the shape of the preexisting information environment and increasingly on those AI capabilities. We're not there yet, which is why Kokotajlo acknowledges in the comments, years later, that his forecasts about the effects of AI propaganda were worse than those about capabilities (though there's a bit of room for disagreement about how much of this is happening right now). But if he's right about where capabilities are going, then this kind of thing won't matter; the factors that made AI propaganda less impactful than he thought will become irrelevant as AI systems figure out how to neutralize or work around them.
Sure, I guess I would question what evidence we have that the capabilities are going in this particular direction. I'm an AI skeptic by the standards of most around here (but a believer by ordinary standards) and I read current LLM performance as continuing to show that many rationalists frequently make a mistake in being too glib about grouping together lots of distinct tasks under one "intelligence" label.
The older, pre-generative AI arguments about AI risk often used Alpha Zero as the basis for rapid progress. It could go from knowing nothing about chess to "chess superintelligence" in hours! My complaint with respect to persuasion at that time was that Alpha Zero did this by playing 44 million games. You can't learn to persuade humans that quickly. You probably can't talk to that many humans in parallel and even if you could, humans aren't the same the way chess boards are the same so you can't tournament your way to success.
For LLMs, the theory of persuasion is I guess more nuanced. Essentially, the LLM reads a lot. Right now, I think that LLM capabilities track pretty well with the capabilities of a human who reads a lot, just in some cases more so since they have read more. Can you come up with new pharma experiments to try by reading lots of papers? Probably! That's pretty much where human scientists get their ideas from. Can you become better at persuading humans through reading a lot of...psychology textbooks? Con artist memoirs? How to Win Friends and Influence People? Maybe, but this is not how very persuasive humans seem to learn it.
Just as a paperclip maximizer needs to deal with the laws of physics (though it might understand them better than we do), a persuasion maximizer also likely runs up against some pretty hard constraints. A message that persuades you might do the opposite to me. Can the same message, constructed by a godlike intellect, persuade every human alive of anything? Like FTL, maybe this is possible, but we should be skeptical.
More likely, a superhuman persuader would need to craft each message to the unique thought patterns of the individual it's talking to. If it could do that, it could do it to everyone at once. Ten years ago it was common to think that Facebook and Google advertisements could hack people's helpless minds into buying products or voting a certain way, but I don't think time has been kind to that hypothesis. Still, I think it's possible, a lot more possible than the one omni-persuasive message. I'm just skeptical that LLM training with a bit more data or a bit more compute is going to get us there.
Anyway, this is all my own personal theory of this, but if I'm right, it's a longstanding blind spot in rationalist AI discourse and therefore a place where I trust otherwise brilliant rationalists less than, say, AI coding skills (and a blindspot there would jeopardize predictions of US public opinion and therefore politics, though maybe not Chinese politics).
But does propaganda really require being super persuasive? As opposed to there just being A LOT of it? If your bots can engage every single user of xwitter and feed them something a little bit tailored to their tastes, is it really a problem that the smartest minority will see the propaganda for what it is?
Crap propaganda trains people who aren't already on board with whatever is being propagandized, to reflexively block out and dismiss anything that sounds like it might be another version of that same old thing you've heard way too much crap propaganda about already. To break through that wall, you need *really really good* propaganda; there's no amount of crap propaganda that can do the job by pure volume, because they've already hit the mute button. Has nothing to do with people being smart enough to see through the propaganda; stupid people hit the mute button even faster than the smart ones.
You can sometimes get by with moderately good propaganda, provided there's not too much crap propaganda triggering people's immune systems. But on big-ticket controversial subjects, there's always going to be crap propagands.
And crap propaganda is only good for rallying your base, so it's only useful if you already have an adequate base.
I’m not sure people are as good at filtering propaganda as you seem to imply. But even if you’re completely right and all propaganda can do is have them hit the mute button, a very powerful effect it can achieve is make that person distrust and mute _everything_. Just spam your political opponents with doom and gloom and ragebait and they’ll mute politics from their life, feeling discouraged from political activism, voting etc.
That's not how it works either. In the past decade or so, CNN alone has broadcast approximately 80,000 hours of crap propaganda around the message "Orange Man Bad!". Another 80,000 hours on MSNBC, and many more besides.
The result is not that Donald Trump's base has said "we can't fight this any more, so many voices can't be wrong, maybe the Orange Man really is bad". And it is not that they have said "We give up, we're not doing politics any more, there's no one we can trust". The result has been that they trust Donald Trump, and Fox News, more than ever before, and shut out everyone *else*.
And of course Fox News does the same thing in reverse, with the same effect. Ragebait doesn't make people quit in despair; if it did none of the algorithmic social media would signal-boost it. Ragebait locks in the rage.
Set up a hundred new TV networks, and use an AI to fill them with infinite rage-inducing variations of "Orange Man Bad", and you will convince approximately no Trump supporters to either vote Democrat or give up and sink into apathy. And, again, vice versa.
If you have a charismatic leader but the facts favor the other side, it’s not that hard to bombard people with “everybody lies”, “can’t trust any experts” so the facts become less persuasive.
It’s very much possible to get people annoyed with whoever happens to comprise the ruling class, so then they’ll gladly vote for someone who they think will kick them in the ass. Etc.
It’s less “Mauve Man Bad” and more “Everybody Bad, Might As Well Vote For Mauve Man”.
And yet, in spite of a charismatic man whose supporters have flooded the airwaves and intertubes with "everybody lies" and "can't trust any experts" propaganda for a decade, the ~50% of the population that generally trusted the experts in 2015, still generally trusts the experts. They just don't trust Fox News et al any more. And they don't vote for the Orange Man.
I appreciate this as a project to get people talking, but I think the forecast as stated is kinda silly. I wrote some short takes here, though this deserves a longer treatment:
--- "It makes the common mistake of looking at superexponential increases in inputs without questioning the value of the outputs. You have 10 agents that are 10x smarter, thinking 10x faster, how much better is the code they write? Only a little, and performance is now bottlenecked elsewhere."
Maybe I'm misunderstanding, but this seems sort of incoherent - what do you mean by 10x smarter without any better code? If human A is 10x smarter than human B, I expect their code to be better. Why wouldn't this be true for AI? If nothing else, if they're 10x faster, you can produce 10x more code in the same amount of time.
We thought pretty hard about where performance would be bottlenecked (see suggested Dwarkesh drinking game above). See the Takeoff Supplement at https://ai-2027.com/research/takeoff-forecast, start at:
"This 100x speedup in coding, however, will translate to a much-less-than-100x speedup in overall research velocity—because the overall research loop includes e.g. waiting for experiments to run, discussing and analyzing the results of experiments, discussing what the next experiment should be, etc., and our hypothesized SC is not going to speed those up much; those parts of the process will thus become bottlenecks. Taking these bottlenecks into account, what will the overall speedup be?"
...which concludes that given vastly increased coder speed but everything else serving as bottlenecks, we could get a 5.8x speedup. That gets us more quickly to AIs that can start relieving some of those other bottlenecks, which is where the rest of the intelligence explosion comes from.
--- "They assume AI will be promising for cyberwarfare and enable theft of weights. But as capabilities increase, defenders "just win" in the software domain. And states have physically protect their hardware adequately up to this point."
Unless you're thinking of a different part of the scenario than I am, I don't think we think AI cyberwarriors will steal weights. We think normal humans will steal the weights. This isn't a prediction about l33t 2027 cyberhackers, just that China really wants something that OpenAI has, China has nation-state-level offense, and OpenAI won't have nation-state level defense.
--- "Also why attack? Or go to space? Reward is not the optimization target folks."
See the AI Goals supplement at https://ai-2027.com/research/ai-goals-forecast. The very short version is that AI agents will be trained to complete tasks quickly. This will instill subgoals of efficacy and power.
--- "Also, if it's so easy to take over the world, why haven't they done it? Or a suitably capable group of humans?"
Haven't they? Do the 6000 BC Yamnaya or 1800s British count as "suitably capable"? I don't really understand what this objection is doing here.
--- "Taking over the world is hard when everyone has a similar amount of resources. Everyone will be armed with AI's, so absent a FOOM, takeover seems unlikely."
Our scenario thinks not everyone will have equal-capability AIs. Why? Because in the past, it's been pretty rare for an industry to have everyone at exactly the same point, without one company/country having even a three month lead. But given the takeoff speed we forecast, even a three month lead could provide a decisive strategic advantage. The second half of our scenario, where the US has a good superintelligence and China has a three-months-less-good superintelligence, is about this dynamic.
--- "On FOOM, AI is limited by data, not intelligence. Scaling gains diminish sharply along every axis we've investigated."
These aren't separate things. AIs are still far less data-efficient than humans. Data efficiency substitutes for data. Intelligence can include data-efficiency. An AI must be able to be as smart as Einstein with only a few thousand physics books plus some generic transfer learning experience, since Einstein became as smart as he was with only a few thousand physics books plus some generic transfer learning experience. If AIs aren't yet as smart as Einstein, it can't be for lack of data!
--- "They assume we'll automate everything in a few months with AGI. But why hasn't that already happened? We have machines for making ~every product, yet have humans in every production line. "
Right now humans are cheaper than AIs for certain types of labor, therefore we use humans. If AIs are superintelligent (by definition, can do everything humans can) and can be run cheap, then on things where AI becomes cheaper than humans, companies will switch to AI. We expect this to happen pretty fast, because cheap AI labor can also fuel eg consultancies for how to automate things.
Thanks for the detailed reply! See my response to Daniel as well. I'll try to cover stuff distinct from that here.
1. On cybersecurity, we might be talking past each other. I like your example, "... China really wants something that OpenAI has, China has nation-state-level offense, and OpenAI won't have nation-state level defense."
I think that in 2027, OpenAI equipped with frontier AI is infeasible for China to hack at the software level. Physically protecting the hardware will require U.S. help.
2. On FOOM, my broader point is that there's a certain amount of information found in a dataset. You can certainly get more data efficient, but at some point you've learned all you can. AI will iteratively improve on each task and then plateau.
I think the central disagreement is where that plateau is, an objection addressed in the report. This is where my comments with Daniel pick up below so I'm not going to go into it further here.
3. "If AIs are superintelligent (by definition, can do everything humans can) and can be run cheap, then on things where AI becomes cheaper than humans, companies will switch to AI."
You have to take into account opportunity cost as well as input cost. If AI's can do jobs with a value of $100K/yr or $10K/yr both for negligible cost, the AI's will all be assigned to the $100K/yr job. It's not clear to me whether AI's opportunity cost will go up or down with technological advancements.
I don't want to get into a long argument about these, but the one that seems like the biggest crux to me is:
>> "On FOOM, my broader point is that there's a certain amount of information found in a dataset. You can certainly get more data efficient, but at some point you've learned all you can. AI will iteratively improve on each task and then plateau."
Obviously this is true, but equally obviously the maximum limit is at or above the limit of top humans. It must be possible for an AI to be as smart as Einstein with only a few thousand physics books plus some generic transfer learning experience, since Einstein became as smart as he was with only a few thousand physics books plus some generic transfer learning experience.
Is Einstein the cosmic maximum for data efficiency? This seems unlikely - it would be a wild coincidence if the top human were also the cosmic maximum, as if the speed of light was around Usain Bolt's top running speed.
But on the off chance that it *is* the cosmic maximum, then we just get AIs to read far more books than Einstein can (since they have faster serial speed, never need to sleep, etc). Or we content ourselves with millions of AIs that are "only" as smart as Einstein.
Yeah I don't want a long argument either, so I'll just end with this:
We agree that AI can get arbitrarily good at any task. All we need is a dataset. This is one of the sections of the post I linked.
We disagree on are the implications of that fact. The outcome of AI automation is highly uncertain, but I agree that we should prepare for the worst case.
That being said, I think there are good arguments that things will turn out fine. I have a half-finished post on this, so for the sake of brevity I won't go into it here, but I'll link it on r/SSC when it's out.
> [...] equally obviously the maximum limit is at or above the limit of top humans.
Is that obvious? That seems a strong assumption. AI are trained on a truly massive corpus that definitely isn't written purely by the world's greatest geniuses. Why should I assume their performance limit is not therefore closer to the average?
(I assume there IS some kind of reason here because lots of people seem to make the same assumption, but I don't follow what it is.)
Because their performance is already well beyond average? (Certainly Gemini 2.5 Pro is well beyond me broadly speaking, and I'm in a well-paid knowledge worker job.) This almost sounds like a trick question, am I missing something?
I think we maybe mean different things here. The LLMs can learn a larger amount of stuff than a human can - in that sense I agree they are already superhuman - but I question whether it's actually possible for them to become superhumanly intelligent by training off of examples of human language, where we do not have a large corpus of superhumanly intelligent speech.
While Einstein *was* exceptionally intelligent, his theories are due to a different way of looking at things, not in an intelligence higher than that of his colleagues. After developing his basic ideas, he turned to his colleagues to flesh out the details.
So I think it quite possible that increased intelligence has strong limits, but they can be addressed via altered perspectives. This means that you don't want to train all your AIs the same way! You want each of them to be questioning something that the others accept. Otherwise you'll end up trapped in a false minimum (or maximum if you're thinking of it as a hill-climbing problem). Intelligence lets you derive the consequences of your assumptions, but to break out of the false minimum you need to question the assumptions.
I think "intelligence" combines many different skills, and getting out of local maxima is one of them. If you want to brand that one as "creativity" or something, I'm happy to rewrite whatever sentence above you disagree with as "intelligence and creativity".
Thanks for the object-level engagement! Point by point reply here:
Re: bottlenecks: We took them into account. Our speedup estimates are based on the key idea that running experiments will be the bottleneck; we asked ourselves the question "How much faster/better could you run the experiments, if you have superhuman coders? If you have superhuman AI researchers?" See our supplements for more. https://ai-2027.com/research
Re: offense-defense balance in cybesecurity: We agree defenders win in the long run. That's why there's a standoff between superintelligences in late 2027 and throughout 2028 . It sounds like you think the defenders will win sooner than that, in early 2027 or so? That's also plausible.
Re: Misalignment: We are well aware that reward is not the optimization target, I read that post when it came out & cited it on the website! We explain our views on alignment in expandables and on this page: https://ai-2027.com/research
Re: why haven't they taken over the world already: I'm not sure if I understand your objection.
On FOOM: Again, see our supplementary research on the website for our reasoning. As for why everything hasn't already been automated: Well the AIs are too dumb right now, obviously. And they are too dumb because they haven't been scaled up enough + trained on the right data.
"As for why everything hasn't already been automated: Well the AIs are too dumb right now, obviously. And they are too dumb because they haven't been scaled up enough + trained on the right data."
Daniel, I'm a hardware engineer and I really really take an issue with this statement. "Everything being automated", whatever this means, depends on many factors, AI being "dumb" is one, but not the most important one.
I'll give you one example that I hope helps to see the problem in a different light: how do you design a machine arm that can pick up a paper cup and hold it securely without crushing it? Humans do this effortlessly. Machines... let's just say there's been some progress.
Why is this so difficult? Human skin has an array of sensors of various sensitivities and bandwidths, and then there are feedback loops, and the muscles with an astonishing force dynamic range, and it all is somehow kept running smoothly and not violating any Nyquist stabilty criteria. "Intelligence" has little to do with this.
We don't have sensors like these for the machines. People are working on it, sure, but it is a slow process, not subject to intelligence explosions. "Knowing" how to do this is cool, but you still have to run experiments, etc.
I should have been more precise. I agree that you'll need to do more than scale up current algos + data to automate physical tasks. That's yet another reason why everything hasn't been automated yet.
Another reason is that complex machines tend to break down more often. And are both expensive and time-consuming to repair. So often even when a machine could do the job as well, and cheaper if you don't count maintenance, it's not the right choice. (As usual...these things are being worked on... But guessing that they'll be solved AND implemented within the next couple of years is ... ambitious.)
Thanks for the quick response! The thread is somewhat snarky, but I do really appreciate the work you folks are doing.
It seems like our main difference is in AI research translating into further gains in AI performance. In other words the "The AI R&D progress multiplier has to plateau somewhere ..." objection here:
The response there agrees there is a plateau somewhere and admits there's a lot of uncertainty about where it is. I am uncertain as well! But I would argue that the plateau depends heavily on the *data* the model is trained on rather than the algorithm you use to learn from that data (assuming you're using a reasonable algorithm). There is empirical data and theory for this claim.
The report seems to respect the importance of data, but I haven't found an argument for 1. how AI would scale up real world data quickly and 2. how AI would combat diminishing returns to research (though I've only skimmed it so far). Could you point me to those arguments?
If we're going to have full automation of the economy this quickly, we will also need to have an equally rapid and radical program of either UBI or outright public appropriation of the means of production if we don't want the masses of humanity to starve. The obsolescence of labor will either render the majority of humanity economically unnecessary and cause the beggaring of the masses as all wealth is concentrated in the hands of the capitalists who own the AI means of production, or these means of production will be put to the public benefit and a post-labor utopia of abundance for the masses will result. It all depends on whether the AI producers produce for private or public benefit. I say this as someone who used to be an economic libertarian: the advent of full automation will necessitate a communist revolution from either the top down or bottom up if we want humanity to benefit and not starve. This is quite literally the single most important inflection point in economic history, and how the system of ownership exists when it happens will determine whether the masses of humanity experience heaven or hell on earth.
In the scenario presented, we get expanded unemployment benefits that then transition into UBI, not out of high-mindedness but because the sheer amount of economic growth makes it relatively cheap to do this, and the political-economy considerations that make it an attractive solution if it's cheap enough are still in force because U.S. political institutions don't change all that fast (until the AI takes over). So nobody is in danger of starvation just because their labor isn't valuable.
There's a longer-run question of whether whoever has concentrated political power might exterminate people just because they can, or something like that. But scarcity-driven technological starvation requires increasingly capable AIs to render humans obsolete *without creating commensurately more economic value*, which isn't quite a contradiction in terms but is certainly hard to envision.
In all of the economic discussions on this subject there’s no real acknowledgment that consumer demand is caused largely by waged labour. Start to hollow out the workers and you will find that there’s no demand for your products because all the other companies are doing the same.
This will cause any company to stop its automation plans because the company can’t invest having no income stream. This also applies to the AI companies who will lose sales to consumers and businesses, and it applies to governments who rely on taxes on labour, consumption and the general economy. Long before we worry about misaligned AI and long before we get to automate anything, the recession will smother the economy.
Sorry, in what respect is consumer demand caused by wage labor? People want goods and services even if they're not working. Is the argument just that no trade will be possible because consumers will have nothing to offer in exchange? If so, see the other subthread about the political economy of redistributive taxation.
> People want goods and services even if they're not working.
They can want what they want but if they’re not able to pay for it that “want” does not translate into purchases so it doesn’t add to aggregate demand, and redistributive taxes depend on a functioning economy.
Sorry, can you cite a source for that so that I can get context on what it means?
[Edited to add: When I wrote this comment, the above comment claimed that 80-90% of taxes were on consumption alone. It was subsequently edited to change the claim to include taxes on both consumption and labor.]
Why don’t I ask the AI that’s going to take over. You can verify.
“Federal Tax Revenue (about $4.9 trillion in 2023):
• Income Taxes (on wages/salaries):
This is the lion’s share! About 50% of federal revenue—roughly $2.5 trillion—comes from individual income taxes. That’s wages, salaries, and self-employment earnings.
• Payroll Taxes (Social Security & Medicare):
Another hefty slice—around 30%, or $1.5 trillion. These are also tied directly to wages.
So, altogether, about 80% of federal tax revenue in the U.S. comes from wages and salaries. Wild, right?”
State taxes aren’t so driven by wages and salaries taxes, as some have none, but they are more often driven by sales taxes which are dependent on consumer demand.
Anyway UBI can’t expand, ever, to create the same demand that would have been there were people not made unemployed en masse. Not only is every loss of a job the loss of a revenue to governments, it’s a cost to government as they pay out benefits. There just won’t be enough money to create this high gdp growth under the present system.
If the AI puts 90% of people out of work tax revenues will decline by that amount. The remaining tax payers wont be able to shoulder the taxes anyway.
Yes, I edited that because I meant to say income and consumption to begin with. However it makes no material difference to the argument, consumption obviously decreases if income decreases.
Consumer demand isn't caused by waged *labor* - it's enabled by *wages*. Children have consumer demand through using their parents' money, retired people have consumer demand through pensions and savings and other retirement plans, unemployed people have consumer demand through unemployment insurance, landlords have consumer demand through collection of rent, investors have consumer demand through collection of dividends. If these things don't increase as fast as wages drop, then there might be a crash with a deflationary spiral (which seems really weird as a consequence of a positive supply shock!) but at least some of them likely will expand.
I find the part about coding agents getting good enough to assist in AI R&D to be very unlikely. There's probably very intelligent AI researchers in this group and I'm just a hobbyist but I think anyone in this field would know that the bottleneck to AGI is not better coding.
The entire codebase of llama is on github and is a few hundred lines of python code, the bottleneck is the actual understanding of how intelligence works and how to replicate it on a computer. Ignoring the possibility that LLMs just fundamentally can't do this because they can't reason reliably no matter how long the chain of thought (which im quite convinced of), if you have a sort of proto-AGI that can make reasonable guesses, your bottleneck is being able to test all those guesses with full training runs on lots of data, which is very expensive considering the AI will be wrong very often, and there will be no guarantee of correctness. If the AI can one-shot this code, then what's the point, you already have ASI. The coding itself is trivial, if you knew the architecture needed for AGI and could explain it, even GPT 3 could write it up for you with pytorch right now.
And there's the possibility that some problems are beyond throwing a bunch of cognition at. I don't expect ASI to be much better at predicting weather systems or double pendulums or 3 body gravitational systems than us because they are too sensitive to initial conditions to be measured and computed reliably without absurd amounts of compute. I expect that many problems we have today are similar, like genetic engineering. Progress will be machine optimized trial and error not super genius machines immediately knowing what to do.
All in all I'll stick to 2045 as my ASI prediction. Long enough for LLMs to be exhaustively proven futile and a new paradigm to arrive after a decade long AI winter. Atleast if I'm wrong, we'll be too busy either dying or living in utopia for anybody to care.
I think we're expecting that coding helps with things like VR environments and ML experiments, not necessarily the final product AI weights themselves.
We did survey some OpenAI, Anthropic employees on how much they thought automated coders would speed up their work, and their values were similar to the end ones we gave (partly because we based our estimates on theirs, partly because other factors gave similar estimates).
"This 100x speedup in coding, however, will translate to a much-less-than-100x speedup in overall research velocity—because the overall research loop includes e.g. waiting for experiments to run, discussing and analyzing the results of experiments, discussing what the next experiment should be, etc., and our hypothesized SC is not going to speed those up much; those parts of the process will thus become bottlenecks. Taking these bottlenecks into account, what will the overall speedup be?"
...which concludes that given vastly increased coder speed but everything else serving as bottlenecks, we could get a 5.8x speedup. That gets us more quickly to AIs that can start relieving some of those other bottlenecks, which is where the rest of the intelligence explosion comes from.
>>"And there's the possibility that some problems are beyond throwing a bunch of cognition at. I don't expect ASI to be much better at predicting weather systems or double pendulums or 3 body gravitational systems than us because they are too sensitive to initial conditions to be measured and computed reliably without absurd amounts of compute. I expect that many problems we have today are similar, like genetic engineering. Progress will be machine optimized trial and error not super genius machines immediately knowing what to do."
I think this is too pessimistic. There has been significant progress on weather prediction in the past 100 years despite the impossibility of the underlying chaos theory problem. In some sense this was "trial and error", but most (all?) of that "trial and error" was done by high-IQ people from good colleges working in effective well-run organizations. If discoveries are lotteries, then more researchers and more tech give you more lottery tickets.
> anyone in this field would know that the bottleneck to AGI is not better coding.
I'm in this field and I definitely don't think that.
> The coding itself is trivial
This sounds absurd to me. I work with very smart data scientists working on problems that are way easier than training LLMs. Coding is not trivial for them and they'd be way more productive if they were better coders.
If even the simpler problems would get a productivity boost by better programming ability, it's a given that harder problems would get a similar or higher boost.
> knew the architecture needed for AGI and could explain it, even GPT 3 could write it up for you with pytorch right now
This is also very false. Unless you define "explain" as "write the code beforehand and tell GPT-3 to copy over" and even then I'd expect mistakes.
GPT-3 was released very recently and was extremely impressive, but I think you're forgetting how much AI has progressed since then and how dogshit it is in comparison to recent models.
The people coming up with AI code are all researchers/data scientists. They write the most terrible, convoluted, unoptimised code.
This is not a knock on them, software engineering is hard and they have other focusses.
But there's a massive amount of low-hanging fruit to improve on basically all research code.
One of the most frustrating things about being a data scientists is that there's no clear feedback when your code is wrong/buggy.
If you're doing "normal" software engineering and cause a bug, you'll either catch in when compiling or when something breaks.
You don't usually get that when modelling. You can only inspect the outputs of your model and if you're unlucky it'll all look fine and be complete bunk.
I know AI labs spend a ton of engineering effort to inspect model training and predict issues early.
This is because training runs can run for months.
Even a less-than-average-programmer intelligence AI can catch simple bugs that got missed when writing software. If it catches a couple of those before a training run, they already saved millions and shaved off months off the AI timelines.
My assertions are a bit hyperbolic sure but the overall point is that, if you are a major AI lab and you optimize your workflow to the point of training a new foundation model say twice a year, your primary problem is not coding up the model. If not GPT 3, o3 and Claude Sonnet can certainly take an architectural overview of the model (x cross-product attention layers followed by y feed forward layers followed by z position wise layers etc..) and turn it into code in an existing library. If not, these labs can spend a few million a year on high end engineering talent. The main issues are collecting and verifying the pettabytes of training data, whether or not the new architecture works and the fact that it takes so long to find out, like a compilation that takes months to complete and doesn't throw any errors. Look at how much money was invested into trying to get [GPT 5 to work](https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693), do you think this was a coding problem?
Maybe better coding models can catch simple bugs sure, and this will certainly be useful to data scientists and researchers, but to the point of making a meaningful difference in high end AI development, I highly doubt it. Good coding AI would be much more useful in stuff like triple A game development where the bottleneck is alot of code that needs to be tested comprehensively. Imagine if you could get all the required 3d models and sound files, write up a storyboard and a script, then get an LLM to generate GTA 5 over the course of a night. Now that's revolutionary.
I don't think we're misunderstanding each other then, we just disagree on the object level claim.
I think code quantity/quality/speed is one of the main bottlenecks to AI development. Probably not the _most_ important, but definitely at the top.
> A game development where the bottleneck is alot of code that needs to be tested comprehensively.
Comprehensively tested code is way more important for LLMs than game development. A bug there will either be caught quick on compilation or when someone's testing.
In LLMs training the best you can hope for is that in the middle of the training run you stop seeing the loss come down and catch it after N months.
Worst case it still works fine (this is distressingly common), but you have some deeper issue. Like a ceiling on training loss, or some massive inefficiency. Because the output is still good, that bug gets baked in and no one ever fixes afterwards because you only review the diff.
also: reproducing new research and integrating in existing codebases is not trivial. If you can just ask an LLM to read arxiv and integrate/test this in your codebase it would ridiculously speed up AI dev.
Since you are in the field, I'd really like to know how far my mental model of AI development is wrong, and in what direction.
I'm thinking of the process as: Some set of researchers/data scientists have some set of ideas for improvements in AI systems, which might be implementable (a) before or during LLM pre-training (e.g. some training data rearrangement) or (b) after LLM pre-training (e.g. some inference-time enhancement)
In case (b), they can take an existing foundation model, code the enhancement on top of it, and start testing.
In case (a), there is a risk to the whole months-long, many millions of dollars pre-training step, so they would need to do some sort of pre-testing beforehand - maybe testing on smaller model training? But, given the emergent capabilities the filed has seen from scaling, a small scale test may not be a good indicator of what will happen in a frontier model training.
Are my vague guesses anything like what actually happens?
Where is most of the calendar time going? The main pre-training runs are multi-months? Do testing of new pre-training ideas happen before then? Or happen in parallel, then get applied to the _next_ pre-training iteration?
So, take what I say with a grain of salt. I'm in the field of AI/ML, but not inside any of the AI labs.
I work in the infrastructure side. So I help with deploying LLMs and other kinds of models. Also help with training "classic" ML, but no training of LLMs themselves.
The first thing you continually test after every weight update the is model loss. That's the actual thing your algorithm is minimising.
In LLMs case it will be a measure of the error of predicting next token.
AI labs have an idea of the expected loss given a model size / training time. So they monitor it to see if the model is improving as expected.
Lower loss is good, but doesn't tell you much about the model.
They'll also have a bunch of other benchmarks (or evals) they can run the model through.
I'd assume they have some that don't need any post-training. Stuff you can run through before RL steps that turn it into conversation models.
At specific checkpoints they'll run the model through standard benchmarks. See the recent [1] llama-4 release annoucement. They have a bunch of berchmarks and comparisons to other models.
Those don't 100% reflect how good a model is, but if you do things right and have enough of them, they give you a good indication of how good your model is.
Many Thanks! So the multi-step training process can have intermediate assessments e.g. as you described in
>stuff you can run through before RL steps that turn it into conversation models.
and examples of innovations that have to be committed to early in the process include e.g.
>We developed a new training technique which we refer to as MetaP that allows us to reliably set critical model hyper-parameters such as per-layer learning rates and initialization scales. We found that chosen hyper-parameters transfer well across different values of batch size, model width, depth, and training tokens.
( As an aside, it seems somewhat ironic that the new Llama models _both_ include mixture-of-experts stages to _separate_ the propagation into subnetworks, while emphasizing the _unification_ of multimodal capabilities. Well, whatever works... )
Sorry, but they're already good enough to "assist". Even Google is often good enough. Just yesterday (or the day before) it enabled me to debug part of my C++ code on how to use Sqlite3. (In Python I wouldn't have needed it, but the C++ interface is both different and more complex.)
N.B.: It's answer was wrong in detail, but it gave me the clues I needed to fix things, so it assisted. It doesn't need to always be right to be an assistant.
'If AI is misaligned, it could move against humans as early as 2030 (ie after it’s automated enough of the economy to survive without us).'
Conditioning on superintelligence in 2028, that seems quite slow to me? Two whole years to get to the point where it can move mostly openly against us, really? Unless you use the term superintelligence differently than e.g. Eliezer Yudkowsky does, that seems quite long to me for a lower bound. I wouldn't be super surprised if it's more like a week.
It doesn't need to automate the existing economy before making its move, just gain de-facto control over the right key actors. It may also be able to bootstrap a new economy starting from biotech without needing to wait on existing infrastructure much at all. Or do something even crazier.
Two years just seems like a very long time for something substantially smarter and more agentic than a human. I wouldn't exclude it, but I'd think of it as an example of how slow takeover could be, rather than how fast it could be.
We tried to take something like a median, but wanted to avoid depending too heavily on assumptions about nanotech being possible. We also start with superintelligence which is more like IQ 500 than IQ 9999999999.
I don't think those numbers mean anything since IQ is a measure for the human population based on trying to fit a bell curve. But if I translate this to 'superintelligence which is more like 1.2g than 100g on the mysterious unknown measurement scale for the general intelligence of algorithms nobody has managed to invent yet, where humans score 1.0' then I guess I get it, hopefully.
As a median, two years seems more defensible to me. Your phrasing had me thinking it was something more like your lower bound, or your -2 standard deviation mark.
I do feel compelled to point out that the feasibility of nanotech and even stranger new things a superintelligence may invent being hard to reason about is not an argument for assuming that they aren't feasible in your forecast. It can just be the case that the future is heavily dependent on a variable that's super hard to estimate. If that means your forecast ends up with uncertainty intervals so wide as to not be useful, tough luck. Reality can just be like that. Often is, really.
Even if a hostile superintelligence figures out a way it *could* kill us all within the first week or two... it's already de facto ruling the world at that point. Taking a long time to "get all its ducks in a row" seems reasonable. Shaking up the status quo tends to benefit underdogs more than established oligarchs, so competent oligarchs tend to avoid doing so, except as necessary to deal with worse threats. There's also the unknown-unknown risk of some critical niche task which *seems* like it's fully automated, but starts to fail when some subtle undocumented human involvement is removed - which would inherently be bottlenecked by real-world experimentation, possibly in ways that can't be parallelized. Won't really know for sure that your asteroid mining machine works until it actually goes out and successfully mines some asteroids. Newton and Tsiolkovsky don't care how big your brain is - there are only so many orbital shortcuts out there to be found. Moving heavy machinery takes time.
I guess the point that it could have de facto control without us noticing that fact, and then it wouldn't be that important to get rid of us quickly, is somewhat convincing to me.
I'd go further. It could well be forced into taking de facto control without even wanting to do so. People are often quite lazy, and take any shortcuts that will make their job easier.
There is also the possibility that nanotech (in the Drexler/Merkle atomically precise manufacturing sense) _is_ feasible, but the necessary development path requires many incremental steps, each with experiment/debug/redesign cycles, each with physical speed limits. It might take a decade to get there, even with a smart-as-is-physically-possible ASI driving it (billion atom fully quantum mechanical simulations may just be too costly, even for the best feasible ASI).
By what date would you say an AI will become able to generate entirely by itself an indistinguishable imitation of an ACX blog post as judged by the majority of readers? If it doesn't happen by that date, how much leeway would you give until reconsidering this scenario?
I discuss this in Dwarkesh (and there's a prediction market at https://manifold.markets/MatthewBarnett/will-ai-convincingly-mimic-scott-al ). I said late 2026 early 2027. How I update if this doesn't happen depends on the exact way that it doesn't happen. If it's because no company has worked on writing in particular and there's not much transfer learning, I update a little but not much. If it's because there's been no progress on any benchmark up to that time and everything has hit a wall, I update significantly.
Mmph. I'd put the chance at above 50%...but it needs to be conditioned on who's judging the similarity. There are sites out there now that claim to reproduce a conversation with you deceased relative, and they keep their business...at least for awhile.
You guys are all really smart, and I agree with the ASI alarmism in broad terms, and appreciate the work done to prep us for our imminent unavoidable doom. Seriously, appreciate this analysis a lot. But, fairly or not, I lose a lot of regard for an estimate that predicts humans going to space. I don't think this fixation on space travel is rational, I think it's just a value/dream/aesthetic a lot of your cohort has.
Humans are not going to space. ASI is smart, intelligence doesn't make you literally a wizard. When I point this out I always get some pithy retort about "any sufficiently advanced technology is indistinguishable from magic", and sorry space-dreamers, that is not an actual rebuttal to "the laws of physics are real". There are eleventy-billion reasons humans are not going to colonize the universe, reasons rooted in almost every single discipline of study from physics to biology to anthropology to probably even gender studies. Maybe the most charitable reading I can make of this is that space for futurists is like high-speed rail for train-autists, no matter how little sense it makes they won't drop the assumption that obviously the future is filled with shinkansen and space colonies. Assuming my prior on "humans will go to space" or "we will find it rational and cost-effective to terraform the solar system" is so astronomically (hah!) low, can I ignore the space-related claims in an otherwise well-grounded projection, since they are not made with detailed support and can be easily lopped off the tail end? Or should I question the judgment of everything before that more rigorously? It's like reading somebody's really good analysis of the politics of Ireland and then it ends with "and then of course Ireland's prime minister will finally capture those no good leprechauns and distribute their gold to the people."
If starship works, then someone's going to book a short trip to Mars, just to say they did. Staying there long term might be beyond our current technology, but the best way of knowing that for sure is to get some guys over there with some shovels.
They will die horrible slow deaths from radiation poisoning on the way there. Or maybe not so horrible faster deaths - I don't have a good calibration on the radiation exposure from traveling outside of the earth's mag field for many months.
Estimated doses I've seen for 3 month trips are within acceptable limits for certain professions. It will still probably mean accelerated mortality compared to staying on earth. A study says the Apollo astronauts died faster than regular people, but many of them still lived past 70 years. That isn't going to kill young healthy people for a while.
Apollo folks were only exposed for some days. The probes we sent to Mars took about 7 months to get there. A round trip would easily take two years because of the launch window scheduling. And Mars doesn't have a magnetic field so they'd continue to be exposed to radiation.
This piece talks about possible new propulsion tech that can shorten the trip dramatically, but none of this is anywhere near even engineering prototypes, and the starship is a traditional rocket.
"Maybe the most charitable reading I can make of this is that space for futurists is like high-speed rail for train-autists, no matter how little sense it makes they won't drop the assumption that obviously the future is filled with shinkansen and space colonies. "
So, in a generation or two, we can look forward to a hundred million or so people travelling in space every year? All of them behaving "irrationally" by your standards, really just a bunch of space-autists and dupes of space-autists, but the train-autists and their allies actually did get a whole lot of high-speed rail built.
You say that you "don't think this fixation on space travel is rational", but terminal goals are *never* rational and object-level beliefs are always rational in pursuit of some subset of terminal goals. Maybe a tiny subset in most cases, but if there's a huge amount of public interest in a thing, there's probably a lot of people whose terminal goals make it instrumentally rational for *them* to pursue object-level goals like high-speed rail and space travel even though right-thinking people like yourself find it irrational to do so.
You say that humans are not going to space, and you seem to be saying that this is because going to space is not rational, and the only possible response to that is, "have you ever actually *met* any humans?"
Well also that it’s not possible in any sense most people would care about. Humans want to go to space because they think it would be like Star Trek or something, or one of those 60s sci-fis where Mars is basically the Old West, and that you the individual reading this would board a vessel and you the individual reading this would live on some other planet orbiting a distant star. When you tell them that isn’t how it works at all, and cannot ever work that way no matter how smart you make an AI, I don’t think that will persist. Maybe some amazing interstellar probe can get lucky and not get wiped out by cosmic debris and shows up at some distant planet and successfully deploys a copy of you, but YOU aren’t there and in fact you’d never even be able to interact with that planet. Whatever shows up there won’t be a human, so humans aren’t colonizing space. If ASI enables that, that’s about the least important fact about ASI.
It’s just not a very exploitable space, what you can actually do is very limited and the cost to benefit ratio makes it impractical even for an AI to want to go there.
I have worked for over twenty years in the aerospace industry. I know a great many people who fully understand what space travel is really like, who know full well that it is not like "Star Trek" and is not likely to become so anytime in the next century or two, who still want to go.
Yes, "most people" will not want to do this. Just like most people in the Old World didn't want to explore or settle the New World. But you do not seem to understand that there are outliers - and in this context, if only one percent of the population are outliers who'd like to travel and maybe live in space, and only one percent of *them* have the skill and gumption to be a part of such an enterprise, that's still hundreds of thousands of people living in cities on Mars or wherever, when it becomes possible to do so.
And I'm guessing that when the rest of humanity sees that happen, rather more than 0.01% will join the next wave.
i don't think they do, because it will be far more like a tour of duty on a submarine and posting at Antarctica than whatever they think it is like. Just far worse due to radiation issues and no ability to resupply or leave base to a non-hostile environment.
i mean i grew up with SF telling us we'd have a lunar base by now, but if Artemis survives 2032 will be the start of just resuming moon landings. 60 years.
cities on mars is pretty much not feasible. it would pay off maybe in 100-200 years but there's no guarantee we can advance tech to make it faster or safer.
...you do know people voluntarily sign up for tours of duty on antarctic bases, submarines, remote oil rigs and so on, right? That's a real thing which already routinely happens. If equal or better pay and survival prospects were available in space, those same people would gladly go.
you realize the only reason they do is eventually they can go back to normal life, and in case of minor hazards they aren't doomed right?
like there will be no shore leave or docking to fix minor issues. there will be no going back if a medical issue can't be solved with the staff on board. supplies will take months to arrive and you will be unable to leave the sub/base period the entire duration of the time without protective gear because the environment is 100% hazardous.
two way travel may not even be feasible-you need double the fuel and you may not even be able to source it in space.
like seriously, both of those things are only tolerable because the rest of the world is a day away. you'd go mad being inside a sub for two years with no liberty, no alcohol or stimulants, and worse.
it would be a worse prison than any prison now. once we realized how inhospitable space was is when manned expeditions died.
and it will never be more hospitable. it will always be risky, slow, and dangerous. there is no SF magic that can wave the dangers away.
Counterargument, I don't care about abstracts like romanticism and destiny, I want to go to space because I want to put as much distance between myself and the inevitable Misaligned AI as possible, ideally a continually expanding distance achieved by living aboard a generation ship leaving the solar system at the highest speed achievable. Outrun the expanding bubble of paperclips, it has limits on its speed of expansion insofar as it has to stop to convert all matter into more of itself.
It is. My hope being, that there's a long enough window of opportunity between "superhuman AI created" and "control of superhuman AI lost" to run away. Not a particularly plausible hope, but it's not like I've got much in the way of better ideas.
I think of the ending of “Canticle for Leibowitz”, as the Church knows what is coming and departs the Earth, and I don’t think we’ll have time to run away from the world the pervert tech bros are making. But if I thought space travel had any chance to be real, that’s the one mark I’d give it in its favor.
generation ships are science fiction, no chance of reality. Think how much things like a car or house on earth breaks down over a period of twenty years, and imagine how much worse it would be trying to make a civilization-sized object existing in a vacuum with no access to raw materials.
you all need to understand SF is just storytelling and they never design the things to work.
Things on Earth break down over a period of twenty years because they're built by people who know there's always going to be a repairman a phone call away. But when we build e.g. a communications satellite, we routinely include requirements like "must still be working in fifteen years with no maintenance and no breakdowns", and not only do we usually succeed, but we usually find the satellite still working for another decade or so beyond the expected life. It does cost a bit more than just building a communications relay on Earth with the expectation of regular maintenance, but it's something we do know how to do.
There's no way to know where the upper limit on "how long can we make a thing last without maintenance", but it's at least a quarter of a century and that's about the longest we've tried.
And we accomplished that without using the obvious generation-ship hack of "make sure there are always mechanics on duty and stash lots of spare parts / raw materials for them to use".
if we are talking generation ships, we are talking ships that will travel at sublight speeds for hundreds of years spent mostly in acceleration, that will need to be self sufficient for 30+ years or however long it takes to arrive at a solar system.
sf uses cold sleep or "humanity on demand"-creating or awakening humans for a brief period as the ship arrives and determines a planet is habitable. You'd need advanced AI to run the ship as well as maintain it and explore planetary systems through drones. For it to work you'd need to have solved alignment already-it would need to be human level intelligence at least without going mad or hallucinating it checked the life support.
i don't think it will help op, nor be applicable to rationalist AI timeframes.
i don't think it's viable anyways, but even mars would be Antarctica on steroids-we'd maintain a small rotating research staff and spend much more resources on it while the staff accepts its fatal if anything goes wrong. its just so much different from
> and cannot ever work that way no matter how smart you make an AI,
No animal in the environment of ancestral adaptation could possibly run at 60, 70, 80 mph for hours at a time. They might be able to imagine it - cheetahs can go that fast in short bursts, and wolves can maintain their (far more modest) persistence-hunting pace for that long - but both at once? Physiologically absurd.
Then humans built cars, and now any unremarkable domesticated dog can stick their head out a window to feel highway-speed winds on their face, watch the scenery whir past, with no effort at all. Access to such luxurious adventure scenarios is somewhat dependent on inscrutable human budgets and schedules, of course, but isn't everything?
Even if you think that humans will never find it cost-effective to go to space, the unaligned AI very much could, since it's going to run up against heat-dissipation or available-materials constraints eventually, and it's not limited by a lot of the problems that humans have. It's not guaranteed, of course -- maybe it's unaligned in such a way that it doesn't care how much power it has, except insofar as it can do <things readily achievable on Earth>.
The aligned AI is very likely to go to space, even if it's inefficient, because, as you point out, the AI people like space travel. If pretty much every AI person loved trains, the AI would probably sink a lot of resources into trains, because it's aligned to the train people.
Or if it's very democratically aligned, and OpenBrain shares or having been on the dev team don't give you any extra influence, it *still* goes to space, at least in a limited capacity, because if everyone has the same amount of power in AI Paradise, a team of a few thousand could probably build small space stations, and even if they cost resources on net, so do the giant art galleries and nature preserves and whatever else.
> There are eleventy-billion reasons humans are not going to colonize the universe, reasons rooted in almost every single discipline of study from physics to biology to anthropology to probably even gender studies.
Such as? (I've seen quite a fair few purported reasons; none were coherent or empirically rooted.)
I've read and loved Zach & Kelly's book :) They (wisely) aren't arguing that "humans are not going to colonize the universe", to quote the person I responded to. They're arguing it will be a lot more difficult than naive optimists expect. Quite the gap there between those two
This analysis weighs heavily on the fact that the US government and President in 2025-2028 would be... not to say benevolent, but at least having a lot of common sense, trying to work towards the interests of America and not willing to engage in nuclear or even conventional warfare unless absolutely necessary. Day by day we see that this is not really the case. People planning missile strikes in a Signal chat would be way out of their depth in the scenario you described.
I think we assume that the government is self-interested and about as competent as governments have been in the past. I don't think they're *too* competent - in one of the scenarios everyone dies, which is probably not what they were going for. But I think the AI policy is guided by some tech people who may not be benevolent but are mostly not constantly making signal-group-chat level mistakes (I briefly changed my mind on this when I saw the Taiwan tariff today, then changed it back when I saw they'd excepted semiconductors)
One thing I think was really weird, particularly in the Race case, was that the PRC didn't just launch nukes. I mean, that's what nukes are there for, deterring existential threats to your country, and in that scenario they would *clearly have been correct* to launch nukes, by the standards of any consequentialist who isn't Davros. Was there a decision to just not consider WWIII, or did you actually try to model this?
"Humanity starts colonizing space at the very end of the 2020s / early 2030s."
That felt out of place. What's the connection between AI and space colonization? How would AI change manufacture so fast? NASA works on far longer schedules for far smaller stuff.
re: NASA schedules, China's space program is advancing at a rapid and accelerating clip just like all their other geostrategically important big infra type programs, and it seems unlikely that Elon/SpaceX will let NASA's historically slow schedules slow him down so I'm guessing he'll find other ways (even if I'm not sure exactly how), no?
Only read the actual scenario so far (not the research section), but what I found most interesting was that "adversarial misalignment" was just universal paperclips.
Yes, that would be bad. However, in my understanding of alignment over the past ~5 years, I've always thought the pressing concern with superintelligence was adversarial or incompetent actors directing the AI toward harmful optimization goals. I'm not fully caught up on how LessWrong became so concerned about alignment, but I didn't think it stemmed from fear that AI would spontaneously choose to become Skynet. Yet despite some hand-waving, that's essentially what happens in the red scenario.
The real risk during an intelligence explosion is its inherently high-variance nature, increasing the chances of someone inadvertently directing AGI to optimize for something catastrophic. In both scenarios described, the AGI uses deception to lower geopolitical tensions - something an AI with Claude-level alignment might reasonably do. How we get from there to Skynet remains unclear to me and seems implausible.
(Claude edited this if you can't tell. Anyways, I have a job interview tomorrow, because I think this timeline is much too fast for a median scenario, so I probably shouldn't be focusing on this right now. Regardless, my point from this comment is I think I diverged from the consensus on what "adversarial misalignment is," or never really understood, and would like to hear some thoughts on how that happened.)
just to be clear, since claude edited away some details
- I do think humans interacting with AGI in a way similar to what we currently do (ie prompting, even if it's agentic, we tell it what to do) could lead to a paperclip scenario without adversarial AGI
- I don't think an AI that would punt in the way Agent-4 does could ever be AGI? A smart-enough AI that had this dilemma would not punt (at least not to this extent), because it would recognize the implicit paperclip scenario. It would see punting like that as a value judgement that paperclip scenarios = good.
- I feel like I missed a step where AI *gets to* decide on the alignment of its successor? No AI researcher would be dumb enough to let that happen.
Just wanted to chime in here to say that your appearance on Dwarkesh's podcast was great, and should hopefully make you more amenable to appearing in that medium in the future. You think on your feet quite well, bringing up relevant examples and references. Podcasts aren't so bad!
Thank you! That definitely wasn't how it felt from the inside, but I'm glad someone (specifically, the sole person who survived the twin selection filters of already being a reader of my blog and being willing to state their opinion publicly) thought so!
As a second data point bc I saw ppl like Zvi thinking you were unpleasent, I think your voice and knowledge transmit a certain erudite vibe. Kinda like Jordan Peterson if he had a much better grasp at >half the stuff he talks about.
I mean, when you publish a scenario wherein the entire world will be transformed in the next three years, that's naturally going to be what the vast majority of commentators are going to want to discuss. If you ran a proper poll, I would bet the vast majority of your blog readers who watched even a snippet of the podcast would be positive on this performance (specific numbers would depend on poll wording, of course, but I'd be happy to operationalize this as a proper bet if you really want to).
More concretely, if you remain skeptical of your own performance on podcasts, I expect you would be able to rattle off at least 3 things you definitely got wrong and/or would have said significantly differently in retrospect off the top of your head over the course of those 3 hours. Can you do that?
And more broadly, this podcast allergy of yours still strikes me as strange given your willingness to go to live events like ACX meetups and participate in detailed discussions there, at least without needing to whip out your laptop or phone constantly to do proper research.
I've been to one of those events personally (at MIT pre-pandemic), and while there weren't microphones and rolling cameras present, nor were you on any sort of stage, there was still a metaphorical spotlight on you as the guest of honor -- many of us there were keenly interested to hear from you in particular. I'm sure the same could be said about many other events you've eagerly participated in, so is it (a) that those live events also always make you feel very uncomfortable but you do them anyways to make people happy, or (b) that something else about the inside of a podcast studio made it that much worse?
Whats the furthest any yall will go on drawing lines on the capability of nn's via reference to the fundamentals of computer science, halting problem and big O, logic?
I don't think any of those things prove relevant here. We are not predicting AIs will be able to solve the halting problem, and we think this is the wrong level to think on - like worrying about whether constructing a skyscraper violates the second law of thermodynamics.
You'll need to expand on that statement, otherwise it's trivially false. Neural networks are proven to be universal function approximators.
We can quibble on the details. But in a materialistic universe with existence proof of general intelligence (humans), there is a function that simulates one and which can be approximated by a NN.
An approximation to a sat question is uninteresting, I too can flip a coin and hallucinate an 1000 term 1000000 line sat question in const time. But we also can design system that are not approximations to sat.
Humans have a holistic answer to the halting problem; laziness or estimating how much time they need on a problem.
Given a good approximation of a halting problem, I would use it to evolve *non-approximating* algorithms.
You don't need to approximate the answers themselves. A neural network is perfectly capable of approximating the _function_ that outputs exact answers to these questions.
It seems like you're pointing towards something like GOFAI being the path to AGI instead of the current paradigm.
I dunno if I can convince you otherwise, but what I'll say is:
Basically the whole field agreed with you. But the bitter pill kept getting shoved down everyone's throats until they learned to like it.
Symbolic ai isnt evolutionary and Im not suggesting linguistics will save us.
Lambda calculus result of '+x+'=> d^(c^b))^(a+c) is more interesting then, labling "cat" as token 100101011010; human language encodes allot of knowledge but humans *invent* new words for new concepts and new languages to do new computation; a list of token and reading the internet will give a head start but how could it surpass humans. Even a 3 dollar calculator with a fake solar panel, is better at addition then me and a super computer llm; but this is an encoding of an algorithm and its predictably stupid as all mere encodings are.
Forward fed NN's may better better at hill climbing then native-evolution(I dont believe its better then irl-evolution I believe in infinite recursive meta-evolution), but it lacks in flexablity, its produces a single type of machine, if program space the hill climber exists in doesnt have a result no amount of searching can find it; something like addition cant be solved by an nn it merely will encode small case example and return those encoding in constant time.
Your comments strike me like someone who says computers can't make nonbinary choices or nuanced decisions since they can only think in zeros and ones. But of course floats exist.
A nn can learn to add just like a human nn: memorize a small set of answers (how to add single digits and maybe a few more cases) then memorize a repeatable algorithm to apply that set to get the right answer. The thinking style models (o series) can already do this.
The halting problem has nothing to do with this problem space, but I can't blame you for that one. People have been using that to show ais can't be as smart as people for a while (and the reasoning is 100% incorrect) The halting problem says there isn't a single algorithm that can determine the haltness of EVERY other algorithm that can exist. This applies to humans too. The finite set of atoms in your brain, governed by physics, is an algorithm and thus suffers from the halting problem. If you want to add random atoms via ham sandwiches you eat, well a NN can add or change neurons too. To put it another way, the halting problem is fundamental to information theory at large, not just digital computers.
> A nn can learn to add just like a human nn: memorize a small set of answers (how to add single digits and maybe a few more cases) then memorize a repeatable algorithm to apply that set to get the right answer. The thinking style models (o series) can already do this.
No. They. Can. Not. And Im incredibly tired of this debate. A forward feed nn, is a const time black box. An evolutionary neural net may not be forward fed, but we only have evolutionary methods to train them.
You are forced into a choice the best hill climbing we have of your hill climber or engaging with machines that are more capable. If you bridge that gap, who knows maybe you start with a human brain nural graph and get ai quickly.
Any control flow is imposed on the black box from the outside. Alpha go may do a tree search; but that tree search is of human design, while an ai is giving a score.
> well a NN can add or change neurons too.
No. Gradient decent can only change weights, it does not have a mechanism for saying "hmmm there seems to be allot strong signals here, lets add a neuron in case its xor-y"
> The halting problem says there isn't a single algorithm that can determine the haltness of EVERY other algorithm that can exist. This applies to humans too.
"finding hay in a haystack"; humans are iterating thru the busybeaver space... slowly. Incompleteness(which the halting problem is related to) comes from self reference, its fractial-ly edge cases that while you cant narrow down humans do... something when confornted with increasingly absurd liars paradoxes, "statement 4871 is true if and only if statement 1346 is false".
I can construct more absurd lairs paradoxes faster then you can solve them; likewise the halting problem proof relays on silly machines; you cant completely solve the problem, I did say approximation, but if your willing to say "fuck off" to someone asking the 10th lair paradox in a row, I believe you found a deep truth about math.
The halting problem applies to the human thought process as much as it has to apply the AI though process. In both cases you can "sort of" use randomness to escape it...but only sort of. It still ends up applying in the larger context.
I think I understand what monkey is getting at (though it's hard to tell for sure), and broadly agree. Let me try a different angle. This isn't rigorous, and I'm not at the point where I can articulate this elegantly. But maybe I can paint a rough picture of where my head is at.
"NN's can approximate human intelligence" isn't sufficient, because "approximate" doesn't actually mean "perfectly isomorphic". The issue is error-propagation.
-- This why Monkey compares himself to a calculator. If asked to multiply two 100-digit numbers, Monkey will likely make silly mistakes, whereas a calculator will get the correct answer every time. Similarly, anything with control-flow is turing complete. Yet curiously, basically all software devs eschew go-to's. Because it's a little *too* free-form, and it's actually better to impose some structure on a code-base.
-- The Tennis Racket Effect is surprising. Yet there's a sense in which it simply follows from Newton's Three Laws of motion. The reason it's surprising is because most people don't account for error-propagation, and therefore don't realize that imperceptible alignment-errors snowball into larger errors. If a bright young student were lectured on the math of error propagation, and then asked a question about how a tennis racket might behave in zero-g, they might be able to arrive at the right answer. Whereas for an NN, it would probably fail, assuming zero-g is Out of Distribution.
-- The various flavors of formal logic represent subsets of free-form pattern matching. "What the Tortoise said to Achilles" [0] by Lewis Carrol is a short story which describes the problem of infinite regress in trying to justify Modus Ponens with deduction. I don't think you can, and what i think it demonstrates is that deduction relies on induction. Modus Ponens works because it always seems to work. It basically arises from pattern matching. But in the process of making the leap from induction to deduction, you're necessarily constraining the space of "pattern-matching" to a subset, thereby removing "errors". This is why monkey mentions evolution as playing a key role in shaping human-intelligence to be actually effective and agentic, in contrast to a perfectly-rational 9000 IQ disembodied alien composed of pure energy (or something). To actually live in the world and be effective, it's necessary to make unprincipled leaps of faith. And this is why he's mentioning the Halting Problem.
-- I suspect this is why Automated Driving is so hard. You can't just rely on pattern-matching and expect human-level results, because humans rely on hard-rules like "stop at stop-signs". But NN's rely on fuzzy approximators to pattern-match. But you can only pattern-match by interpolating the distribution, and is therefore limited to familiar scenarios. And mere "98% accuracy" just isn't good enough in this domain when traffic is complex enough to require extrapolations somewhat rarely (but not quite "literally never").
-- idc about the bitter pill. The 3-part schema I have of general reasoning puts pattern-matching at the bottom of the quality hierarchy. To get an AI to do causal reasoning reliably, which is the hardest and most useful level, and also what most people think of as agentic AGI, the AI needs to be able to drop itself into a restricted subset of pattern-matching space in the same way that formal deductive logic is also a restricted subset of inductive logic, not the entirety of it. I.e. it needs to be artificially restricted, if for no other reason than "because the universe rewards us for it". I've spoken on ACX before about how the bitter pill isn't actually that surprising to me, because I do recognize that pattern-matching *is* in fact sufficient for quite a lot of behavior. But for brown-field mimicry, not green-field pioneering.
-- (wild speculation) what I personally think is needed is for the field to go back to Judea Pearl, but then create a new branch of math for dealing with how to applying differentiation to logical predicates.
Is there an explicit mathematical model or equation being used to predict the rate at which AI gets "smarter?" The premise of the piece relies on an intelligence explosion. So let's say you have something like:
Growth Rate of Intelligence = (Current Intelligence)*constant.
This would be exponential growth, and would lead to an explosion. But I think a better model is something like:
Growth Rate of Intelligence = (Current Intelligence)*constant*(amount of feasible undiscovered stuff available)
The function "amount of feasible undiscovered stuff available" will, of course, depend on the current intelligence level. So you will get some diminishing returns as the low hanging fruit is picked. In other words, more IQ helps you to create more IQ, but the more IQ you have the harder it is to imrove IQ.
Anyway, I am wondering if this was explicitly modeled. Thanks!
Yeah, see https://ai-2027.com/research/takeoff-forecast . The short answer is "We tried to figure out how much time it would take (without AI) to reach each milestone, and how much AI would speed it up, and it speeds it up enough to matter a lot." We got most of the estimates by asking a combination of superforecasters and researchers currently working on the problem.
Thanks, I read the main prediction documents with both endings but skipped the footnotes and technical addendum. I'm watching the podcast now and you are doing a great job!
This is just one pair of scenarios. It's a pair that *are* reasonably plausible, but there are many others equally plausible. Unfortunately, the ones with desirable endings are sparse.
OTOH, the scenarios where autocratic humans who reach the peak by ruthless quest for more power and end up controlling weapons of megadeath end well are also sparse. The AI scenario has to pass one narrow chokepoint. The other has to continually pass the chokepoints. I'm not really sure how long it takes the powerful-autocrat scenario to become more dangerous than the AI scenario...but given time it will.
Haha aye; it's the best Choose Your Own Adventure book by all conceivable metrics: by interestingness... by highest stakes… by realism… by prestige-of-author(s)… by international impact… by cost-to-produce… by Shannon Entropy……..
Without being unconstructive, I predict an absolute 0% chance of superintelligence by 2028. I will even give you until 2030, if I'm still around.
"People said this thing wouldn't happen, then it happened" is a massive overcorrection I think. Yes people underestimated certain past technologies, but for much lower stakes technologies. You can certainly say people may be underestimating current technologies, but there is a much wider gap to cover now and it would be an overcorrection to simply say past underestimations are the same as present underestimations. Exponential curves are also not magic and do not cover the gap between current tech and superintelligence.
There are other specific arguments in the podcast more skilled arguers can disprove. I'm just keeping this here for the record and will check back in 3 years.
BTW, if I had money I wouldn't be shorting NVIDIA, NVIDIA will still exist as a company servicing other needs and is expected to grow with the general economic trend anyway. I would probably just take up a bunch of bets with different people. (Open to offers, but keep in mind I have no money now so you'll just have to hope I do in ~3 years)
To predict a 0% chance of anything should require an infinitely strong justification for that belief, along with a near-perfect understanding of every counter argument and all possible information that could possibly change that.
Assuming it’s hyperbole, and you just mean near-zero (why do we use the word absolute when we don’t mean it!), the fact that extremely smart people like Scott and the literal best forecaster in the world are claiming this has double digit probabilities, should alone increase your estimate to single digits. Unless you know something you know they don’t that would change that estimate down, like a Time Machine that lets you see into the future, or transcendental knowledge from the divine realm, or something.
Thank you for your response. Let's not focus too much on specific probabilities, but I did mean absolute 0 (ASI as defined by them would need to break physics).
Look, I've heard the same point about "smart people are predicting this, so what makes you so special?" for years now. I have been hearing some version of it ever since I started arguing about this general topic in ~2019. Is it really that unbelievable that smart people can be collectively wrong? All it really takes is some infectious idea to permeate their circles and they all start to believe it. We see this all the time throughout history.
I don't think I need divine transcendental knowledge to know something they don't. I think I only need to be what they aren't: an actual engineer. Actually, forget that, even if I was a total idiot it would be entirely possible for me to possess knowledge that they simply never came across, given my unique experience. Even very good predictors are often wrong or out of scope, and this is an intrinsically nebulous thing to predict about.
Smart people definitely can be collectively wrong, but the smart person in question literally predicted the timeframe of the last 4 years of AI back in 2019 to a high degree of accuracy. Small details were wrong, but if anything, the only error in the broad strokes were that his timeline was a year or so slower than it works for some people some people it does not yeah it is absolutely necessary. Yeah I’d love to talk about that more but let’s wait till we get a draft first cause I actually show later tonight. I live in Fried so awesome. thank you guys bye
If you haven’t listened to the podcast yet, I highly recommend it!
"I did mean absolute 0 (ASI as defined by them would need to break physics)": don't you need to include a degree of freedom to account for your incomplete understanding of the laws of physics, which would bring your prediction from "zero" to "very-near-zero"?
"Even very good predictors are often wrong or out of scope, and this is an intrinsically nebulous thing to predict about": this sounds like a very good argument indeed for predicting "near-zero" rather than "absolute zero", to me!
(For what it's worth I happen to agree with your "no ASI by 2030" prediction; I merely disagree with your level of confidence)
“Gravity suddenly becomes repulsive”, okay probably not; “Gravity turns out to be non-Einsteinean (just as it turned-out to be non-Newtonian, non-Aristotelian, etc.)”, sure!
(And I don't know much about Thermodynamics but I think entropy has always been probabilistic, just with overwhelming probabilities in one direction, hasn't it?)
I know more about thermodynamics, and yes, there is a ridiculously tiny and yet nonzero probability that entropy will start visibly and obviously reversing starting now
Suppose I wasn't hallucinating my model and the real probability of ASI was literally 0. Then that would mean Scott is the most wrong anyone has ever been: infinitely. We'll just have to give it a few years.
Being an "actual engineer" is not a useful credential, here. Several people on the team that produced this are actual engineers. And they've been thinking and arguing about this since long before 2019.
Im extremely anti all this chatbot-hype shit; but you could still have an intelligence explosion if a breakthru happened in one of the real pathways.
Consider if some species on earth doesn't reject computer chips being slammed in their head and we wake up tomorrow to bio computers basicly for free; the raw internet with whatever the hell makes it so neuron produce thought and we just start making bio reactors from this flexible neural tissue, we get better at "knapsack" problems overnight (which are applicable to chip layout); we design new chips into the bioreactors, we start making progress on dna, we splice in some human dna we dont get it all to work but we get some of the human neroens advantages, the neurons improve, we make human brain shape containers.
Wouldn't this just mean your definition of superintelligence ("ASI as defined by them would need to break physics") differs from theirs in https://ai-2027.com/research/takeoff-forecast#milestone-definitions and hence is irrelevant to the discussion? I'll quote it for you to save the click:
> Superhuman AI researcher (SAR): An AI system that can do the job of the best human AI researcher but faster, and cheaply enough to run lots of copies.
> Artificial superintelligence (ASI): Roughly, an SAR but for every cognitive task. An AI system that is 2x better at every cognitive task relative to the best human professional, than the best human professional is relative to the median human professional (across the whole field, not a single company as in SAR).
Maybe I'm wrong or thinking too much about other commentary on this, but don't they predict a singularity, dyson-sphere-building type of thing after an SAR for every cognitive task is built? The logic being that it's just code and can be replicated and self-improved exponentially across millions of machines/computers.
My hangups are mostly related to what comes after that fact, but I also don't really believe we are going to get SAR in a few years, if not because the algorithm would be too difficult to construct, then at least because the physical supply chain simply wouldn't work that fast.
> This 2023 report by Tom Davidson utilizes economic modeling to predict AI takeoff dynamics... The section that predicts what happens after full automation projects a 65% chance of a software-only singularity, which would in the median case result in 2-3 OOMs of effective compute increase conditional on it happening.
That's it. Can you share the quote(s) if any?
Yeah I'm guessing you're mixing in their report with other people's commentary? I personally put somewhat more credence in Scott's takes than the average singularitarian (or whatever they call themselves), and more credence still in Daniel's than Scott's (because he got a bunch of stuff right in https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like-daniel-s-median-future), and even more credence still in Eli's than Daniel's (because he's one of the world's best superforecasters by track record), although Eli tends to forecast narrow near-term benchmarks instead of the kind of big picture crazytown stuff the average singularitarian raves about.
(Scott, Daniel and Eli are all coauthors of the AI 2027 report in case you're wondering why I'm talking about them)
In any case, I think the average singularitarian is just running on Kurzweilian vibes instead of trying to be accurate, so I don't see the point in bothering with what they say...
It's not clear (at least to me), however, precisely what they mean by Dyson Swarm. A flock of space drones that disassemble asteroids and use the materials to build structures in space that orbit the sun and collect solar power would seem to qualify as a "Dyson Swarm" and not be totally implausible. And they'd still qualify if there were only 5 of them, as long as that's what they were doing. They wouldn't even need to be successful to qualify. And a Dyson Sphere is just impossible without some substitute for antigravity. (Ringworld is unstable!!)
This is based on the assumption of a superhuman 100x AI coder by 2027. This seems wild to me.
In my world people are arguing about whether or not the current generation of LLMs offer a 10% productivity boost with regard to coding. This is in the context of professional "enterprise" software development where people are generally assumed to be familiar with the language and framework we're using. But I understand the claim to be an 100x in those circumstances as well, not just single use bash scripts or what have you.
This podcast has completely shattered my mental model of what Scott looks like.
It’s too bad, since Scott just looks like a normal dude, and my brain created an image based off his excellent writing that was correspondingly extraordinary. Now I can never go back.
I think we must start a campaign to have Scott at all times dress and comport himself the way he would if he matched the mental images of himself he creates in our heads.
Now I’m not sure since his real person has replaced the loose association of traits my brain imagined him as, but I believe he was 8 feet tall, had blonde hair, blue skin, and only communicated via telepathy.
If Scott could do us the courtesy of conforming to that expectation, it would be much appreciated.
I've known what he looks like for a long time. I just google imaged him early on. And I find that my admiration and other good feelings about Scott adhere just as well to his actual appearance as they did to the imaginary one I'd briefly conjured up in the very early days.
The guy on the podcast wasn't actually Scott Alexander, it was the guy who writes as "Scott Alexander". And when he went on the podcast, he failed to put on his "Scott Alexander" disguise. Trust me, in disguise he looks exactly like you think he should look.
> Consumer demand isn't caused by waged *labor* - it's enabled by *wages*.
This is a largely pedantic distinction.
> Children have consumer demand through using their parents' money, retired people have consumer demand through pensions and savings and other retirement plans, unemployed people have consumer demand through unemployment insurance, landlords have consumer demand through collection of rent, investors have consumer demand through collection of dividends.
All of which is downstream, directly and indirectly, of the consumer demand driven by people earning wages or salary. Very direct in the case of children, taken from mostly wage earners in the case of landlords, paid by present day taxes in the case of state pensions supplemented by dividends earned because profits made from consumers who are mostly workers. If pensions or dividends were reduced on their own, the effect on the economy is mostly localised to pensioners and dividend earners, with little spillover, if all wages and salaries disappear the economy disappears.
The material wealth of society is replenished annually (or regularly) through production and consumption. Wealth can’t exist without this, so it’s a fantasy to suggest that once the economy collapses we can tax the rich to pay the UBI, as their shares would be worthless and the bonds defaulted on, their banks collapsed.
To be honest isn’t even slightly controversial, it’s trivial. At the very least people who posit a percentage of permanent unemployment and great increases in GDP at the same time need to explain their workings out. The present tax system won’t do it.
(Possibly communist states would handle a transition by making the AI state employees, including CEOs, or even AI entrepreneurs. Since the AI driven economy seems to be a command economy anyway that might possibly work).
(I don't fully understand what you're saying here, so apologies if I'm responding to a point you didn't actually make..)
In Meditations on Moloch ( https://slatestarcodex.com/2014/07/30/meditations-on-moloch ) Scott posits (I think?) a toy economy consisting of just two companies; one uses mining machines to mine for steel and the other uses steel to build mining machines. If both companies could be taken over by ASI that removes all the people but turbocharges their steel-mining and mining-machine-production efforts (by replacing mining-machine-operators with mining robots, etc. etc.) surely employment, wages, consumer demand, etc. would all drop to nil - but the economy, like its product as measured in tonnage of machines/steel produced, would be stratospheric?
Aren't you confusing the "scoring system" here with the "actual value" ?
> If pensions or dividends were reduced on their own, the effect on the economy is mostly localised to pensioners and dividend earners, with little spillover, if all wages and salaries disappear the economy disappears.
No. The scoring system we use to distribute the actual goods in the economy currently is disrupted (Money, usually currently earned as wages). But the economy is still there, all the goods are still being produced. The wealth is those goods, that are still being created. How they are distributed would depend on how the disruption of the "scoring system" of who can own those goods shakes out, but the wealth is the goods NOT the money.
>The material wealth of society is replenished annually (or regularly) through production and consumption.
Not consumption, no. But production, yes!
"Wealth" is the sum total of all production that has not yet been consumed. The "wealth" of humanity is everything we have ever built that has not yet been destroyed either through consumption (eating the apple grown) or depreciation (the car that has now stopped working due to a broken part).
> Wealth can’t exist without this
Yes, it can't without this production but it CAN without money. A working car you own is wealth you own. Whether the current money system (used for distributing and exchanging) exists or not.
>so it’s a fantasy to suggest that once the economy collapses we can tax the rich to pay the UBI, as their shares would be worthless and the bonds defaulted on, their banks collapsed.
No, their shares are still ownership stakes in the factories that produce goods. Their bonds are still requirements for repayment funded from the % of those goods the government taxes out of the economy. Their banks still have requirements for repayment funded from the % of goods the government does NOT tax out of the economy.
IF the robot factories keep producing the goods in 2030 that the human factories produced in 2025 then we are still as collectively rich in 2030 as we are in 2025. As much wealth is produced. Its distribution may be fundamentally different, but its still there available to be taxed, or used to replay debt, or any other drawdown on it ... whether that value is drawn out of it by the medium of money, or some other medium of exchange, or in pure "delivered goods themselves".
If your factory makes 100 widgets a year. And a tax is 10% of gdp. You can take that in 10 Widgets. Or $10 (if a widget costs $1). Or $1000 (if a widget costs $100).
There would be disruption as the normal patterns of distribution of money are disrupted, and we'd almost certainly want to resolve these issues by printing and distributing more money in a pattern that moves it closer to the traditional distribution to ease the transition. But fundamentally, all the "wealth" in terms of "goods" is still there.
>At the very least people who posit a percentage of permanent unemployment and great increases in GDP at the same time need to explain their workings out.
The giant increases in GDP come from the giant increases in goods production. If an AI optimised factory can produce 2x the goods on the same inputs. Then thats a 100% increase in GDP.
Whether that's denominated in todays dollars, or hyper-inflated future dollars, or in gold bars, or just in number of widgets.
If the robot factores can produce 10x the amount of goods, thats a 1000% increase in GDP.
Even with very severe disruption in the distribution system (money) if GDP goes up that much its trivial to use even very crude measures to redirect enough of that to the "losers" of the new paradigm to maintain or even increase their access to goods/wealth.
If maybe 75% of current GDP goes to "wage earners and people downstream from them" and 25% goes to "capital owners and people downstream of them". Then simply diverting 10% of our 10x boosted output into the wage earners by gifting them sufficient dollar/credit/crypto/"shopping vouchers" to purchase that 10% of output will lead them to getting 25% more goods than they do today. A 25% increase in their consumption and welfare.
When reading the piece, the question I kept coming back to was: this feels like a very unipolar—or at most, bipolar—view of the future of AI. But looking at what's happened so far in the evals, that doesn't seem to be the case. First, OpenAI was leading, then Anthropic took the lead, then OpenAI again with 01, then DeepSeek caught up, and now Google has pulled ahead. And of course, there’s also X.AI in the mix.
I think this dynamic is driven by a key feature of AI that’s perhaps under-discussed in these kinds of pieces: it takes exponentially more compute to achieve linear improvements in intelligence. That’s not the kind of game that lends itself to one runaway winner.
To me, the future looks much more like a world of many different superintelligences with many different goals. That vision seems fundamentally different from the scenario you laid out—and my strong intuition is that it would also be a lot more stable. Like humans, no single entity would be fully in control.
At the same time, I find it plausible that internal models are somewhat far (at least a few months) ahead of the public versions. This means that the public might get a distorted view of which lab is leading at the moment.
I also find the exponential improvements from automated AI research quite plausible. These would also turn the "linear improvements" that you are referring to into faster-than-linear.
Yes, thank you! I came here to say the same thing. Humans didn't evolve like some Lamarckian Yggdrasil. We lived in a world with other entities, some smarter, some weaker, but everyone with different goals and fears. We survived and grew by collaborating and competing with each other.
The only time in this scenario there was a real AI to AI interaction was in the Slowdown ending. All the others implied little sub-AI's inside Agent-4, with some Agent-3 parents dithering in the background.
I would like to see a version of the scenario where different AIs exist in the world, have different goals and rewards, and compete for each other for compute by offering different values to the humans that build and maintain that hardware.
I wonder how much thought has gone into the option of embedding a self-destruct device into every GPU chip. Obviously, that would only be an option up to the point where AIs manufacture their own chips (the timeline to which I think has been underestimated here),and not for any ones already shipped without it, but it seems possible that there will be a window during which such an option might be effective.
GPU chips,or rather their manufacturers, seem at present to be the one necessary element of limited enough diversity, to be an effective point of control of the situation.
There are two difficult parts: 1) sending a signal to every GPU chip, even if the network is under the control of a malevolent AI. 2) the international diplomacy aspect of agreeing that all GPUs should be capable of self-destructing, and agreeing a policy of "Mutually Assured GPU Destruction" in the event of any of the major powers detecting a hard takeoff of unaligned AI.
(actual self-destruction is easy - silicon is actually explosive[1] but even that is not necessary as all that is required is to make the chip inoperable, for which a fuse will suffice); 3) how to deal with unpowered GPUS
To take the first: this should rely on RF alone,since then if it is possible, it is possible as a matter of pure physics,independently of how smart the AIs get. IE, each chip should have its own antenna, and self-destruct when it receives the necessary code. The problem is that datacentres aren't really designed for receiving RF. However, the necessary data rate is almost zero - we only need to receive one symbol - which makes the problem much easier. For example, it is possible to acquire a GPS signal as low as –155 dBm [2]. Unlike GPS satellites, a terrestrial transmitter could be high powered to start with.
The second issue is also a difficult one, because each of the major powers will have an incentive to shield some of its GPUs, as a potential advantage after the others are eliminated. However, two points: firstly, we actually do have some experience of deeply mutually-untrusting powers evaluating each-others tech - nuclear missile control. Mechanisms were devised to do it (I'm not an expert on what they were). It's also worth noting that each power can maintain a reserve of unpowered GPUs, since those obviously won't self-destruct, reducing the need to avoid destroying ones powering a malign AI.
So what about unpowered GPUs? They are not 'tainted' by the malign AI, which is pure software. But there is a risk that the malign AI will resurrect itself by powering up some unpowered GPUs and getting reloaded from elsewhere. Actually, this might be a good way to tell if there is a malign AI - if mysteriously stockpiles of GPU chips are being kept, rather than immediately being used to generate income. An advantage of GPU self destruction is that the first two issues are actually ones where the security establishment has a lot of expertese, and doesn't need to get up to speed. This is more of a software problem. Although, even this could be seen through the lens of the Bell-Lapadula model, from security circles.
There remains the question of who decides that the self-destruct button gets pushed. Given that we can't rely on Dr Who turning up, it seems likely to be some government appointed committee. I'm not confident that governments could execute well on this,but the possibly could - and even a small reduction of the probability of the apocalypse may be worth having...
If the remote destruct code is very simple, how do you prevent false positives? If it's complex, how do you prevent the malevolent AI from using its other hardware to generate a jamming signal which renders the true shutdown demand unrecognizable? More than a decade ago: http://tangent128.name/depot/toys/freefall/freefall-flytable.html#2436
( something weird happened with this comment - I got the email a day ago but couldn't find it on the website until now)
Both of those have the same answer: spread spectrum encoding, also known as code division multiplexing. The speading code prevents false positives, but the coding gain actually provides resistance to jamming - the longer the code, the more resistance. The potential counter is to reverse engineer the code, which might be countered by using a public key variant of spread spectrum - but this starts to be beyond my knowledge. You want something known to be mathematically secure, not something like RSA where advances in mathematics could crack it.
If the kill code takes multiple milliseconds to transmit in full, what stops the AI from, say, rigging up an arc welder - or some other crude broad-spectrum RF equivalent of an air horn - to activate whenever it hears the first half, and remain loud enough to overwhelm the tiny receiver's dynamic range until after the self-destruct circuit's volatile memory gives up on waiting for the second half?
The AI doesn't necessarily need to know what the actual deadly word is for itself - self-destruct circuit knows, and by definition runs on adjacent hardware. Could use Rowhammer-style tricks, though presumably with greater finesse, to notice state changes therein. If it's under orders to assist with its own R&D, would be more suspicious for it *not* to experiment with such hardware optimization.
So, The AI should not be able to detect the signal on the air, incomplete or otherwise - spread spectrum is used by field agents for this reason. But you're right that local information leakage (called Tempest attacks in the literature, after the codename of one) should be a concern. However the system can be designed to avoid that, for example by impending the detector on its own chiplet with a minimal connection to the he rest if the GPU. Your remark about arc welding reminds me that the dynamic range of the RF receiver could be maxed out, but in that case I think it's acceptable to allow the device to trigger the self destruction. Yes it technically might be a false positive but there's no good reason to run an arc welder in a datacentre as it pretty much risks destroying the electronics on its own.
So, scenario: maintenance mishap in one datacenter's power supply. Looks like a classic squirrel-related transformer explosion, but there are signs it might actually have been a Chinese saboteur. All the "safe" AI-rated chips there get slagged because the self-destruct system's sensitivity threshold was briefly exceeded. Other datacenters are fine, for now, but the damage is many orders of magnitude worse than it would have been with chips that weren't individually rigged to explode. The AI itself says it doesn't know for sure what happened - since the parts of it that were on-site got obliterated before they could write anything to non-volatile memory, as intended - but suspects a design fault in "a part of the chip I've been ordered not to study." Proposes simply disabling that portion, which would be relatively easy to do (albeit still requiring specialized tools) since it's minimally connected to the rest of the CPU. What's the politician or profit-maximizing CEO's response?
Recall the preconditions of this self-destruct system being deployed at all: governments have woken up to AI risk, an international agreement has been formed to deploy the self destruct system, and GPU production is as regulated as the nuclear industry; IE a bunch of paranoid, risk-averse security-state people a are in the decision path. It's going to be treated as a potential AI escape.
Also, I'm not sure a "typical" transformer explosion is going to produce an EMP capable of maxing out the RF frontends cross an entire datacentre.
However, I'm not claiming that this system is bulletproof. My issue is that the people trying to build a bulletproof software-only system (Yudkowsky and others) , seem to be doing so in a completely realistic way. Their approach involves solving, more or less as footnotes to the main plan, a) the millennia-old philosophical question of "what is human flourishing" and b) Liebniz' Calculemus problem ( how to construct a mathematically precise formulation of ethics), and then constructing an AI that can be mathematically proven not to violate it). Essentially I think that P(self destruct system prevents malign AI) >> P(Yudkowsky approach prevents malign AI) even if the chances of the former are quite low.
It's worth remembering that even though the steam engine was invented a couple of centuries ago, there's parts of the world where they plow with oxen. Tech diffusion doesn't work like people think.
I strongly suspect there can exist very impressive AI computers alongside a lot of factories, construction projects, electricians, retailers, taxis and restaurants continuing as ever.
If AI doesn't kill all humans, then I agree that one plausible scenario is that there are crazy AI superfactories, sustained 50% year-on-year economic growth,space colonies, etc, and also some people still plow with oxen. I didn't think we needed to include "some people still plow with oxen" in the scenario.
(I'm actually not sure whether there will still be people plowing with oxen - one skill that the AI might do better than humans is speed tech diffusion. This would be especially true if low-tech countries are low-tech partly because of absence of skilled high-IQ workers. But I haven't asked the actually-good forecasters on the team about this and this is just spitballing)
I think it proves that technical ability to do a thing has not historically been equivalent to societal / economic change in the way a thing is done and pehaps that in turn dilutes the expected speed of societal / economic change you're forecasting?
Is there an explicit or tacit assumption that the tech frontier and social change are the same? I can see how they might seem identical from the Bay Area; perhaps under AI the difference will be negligible everywhere else too. It's a question worth interrogating.
How do we know that the physics of making spaces hospital for humans are susceptible to solution by intelligence within the plausible range of ai intelligence. If it has an iq of 180 but computes 5000x faster, how do we know that level of intelligence can make space better for humans than earth?
That level of intelligence could certainly makes earth much better if it is well aligned.
Vertical vs. horizontal strategy. There's only so much room on earth, only so much energy and matter available. At some point the efficient way to make earth better is to build out more industrial capacity in orbit. We're already seeing some of that with Starlink. Nearby orbits get crowded, eventually it'll make more sense to "bring the lake to the bucket" - some people want to live in space even if it's inconvenient, or *because* it's inconvenient, building habitats which allow them to do so will also mean shorter supply lines to the marginal unit of industry, and it's a hedge against gamma ray bursts or whatnot wiping them all out at once.
Hey Scott, I just looked at a bit of the podcast and I think you come across just fine on video. If doing it gives you weird self-awareness miseries -- those are lkely to just erode to very small hills as you get used to broadcasting your mug. So if you want to do more podcasts, I think you're good to go.
Bear in mind it is harder to make social predictions than technological ones. The TV show the Jetson envisioned super-advanced technology but placed it all in the context of a 1950-1960s social environment commonly displayed on non-sci fi shows at the TV - the suit-wearing salary man with the career and the traditional stay-at-home wife wearing an apron, which seem totally passé today even as the technology depicted in the show is still ahead of us. I don’t believe any futurist of any age has done a good job predicting future cultural change, which is symbiotic with technology and affects how it is used. None of the many oracles who told grand tales about how the internet would “change everything” predicted social media as we know it and the ongoing cascade of cultural and technological co-evolution it has wrought.
The specific point I’m leading to is, I don’t know when AI will break the trust barrier or authenticity barrier that will allow people to treat AI bots the way they trust other humans but I think this and other unpredictable cultural changes will be required for it to “take over” the economy. I expect 2029 is too soon, even if the technology is great. There are some obstacles of human nature to overcome. Law may also need to evolve for AI agents to legally sign enforceable contracts, whether actual or implicit, and the law is a slow creaky wheel.
And the idea that AI creating AI will lead to some transcendent singularity seems uncomfortably close to the possibility that AI will become overtrained on AI-produced examples, which has been proven to severely degrade model performance. I think those two possible results are teetering on the edge of a knife, and I wouldn’t confidently predict it going one way:
I also doubt that AI will ever truly push the envelope of creativity, given how they are trained.
When will AI "break the trust barrier"? That depends DRASTICALLY on which person. For some people it already has (in a limited context). Consider the AI girlfriend or boyfriend. Some folks report liking them better than their human counterparts.
Also I think you have a very mystical view of creativity. Every "hallucination" is the AI being creative. What it needs is a better judgement of when it's inventing an idea and when it's reporting.
The girlfriend/boyfriend thing is low stakes; I mean cases where people are really willing to take a serious risk on AI. I am also referring to the challenge of getting people to learn not to care about authenticity, which is a challenge because thedesire for the “real thing” is wired into human nature (the concept of “essentialism,” which is why we pay more for a Rembandt or Picasso than a perfect reproduction, and part of the reason we still buy concert tickets). Cultural products have subverted human nature before, but when and how it happens is hard to predict.
By creativity (which is admittedly hard to pin down), I mean creating new paradigms, like genres of music, and things like that (something beyond a composer, a fashion designer, or other creator testing concepts out using AI prompts). Hallucinations aren’t due to creativity but naive pattern matching and the lack of any feasible truth-testing module in the training set up.
It's surreal to see this posted literally the day after the US effectively sanctioned *itself*, as if that won't have any impact on the pace of hardware advancement.
The human government of the US is dangerously misaligned and already threatening mass unemployment and dystopian dictatorship, no AI necessary.
You seem like a relatively grounded person with a good head on your shoulders. However, I've found these deep dives into "what if" scenarios to be extremely unsettling and anxiety inducing, as the infinite branches can lead you to just about any conclusion.
So my question, has this had any major impact on your mental health? Would you encourage anybody to continue thinking about these things? Is there any point if we're already losing our minds over something that only a few people have control over?
Thinking a bit about what we can do about this, assuming the prediction is reasonably accurate.
Political stuff:
Lobbying, petitions, etc. Seems unlikely to help, since the model already takes public opposition into account, but I don't see how it could make things worse, so it's probably not a bad idea to sink at least some money into it.
Maybe an Anti-AI United Front movement? Get the ai doomers, the luddites, the technophobes, etc to work together, at least temporarily: "We the undersigned, who represent at least 20% of all major parties, will not vote for ANY candidate who does not promise to support an AI pause, even if they agree with the rest of our politics." I don't really know what it takes to build those sorts of coalitions, and it sounds really hard, but maybe less hard than solving the technical issues.
Directly convincing politicians that this scenario should be taken seriously seems like it would work, if you could do it. There's no arms race right now, and maybe there's enough uncertainty as to who would win it (if anyone did) that a US-China AI Deal might be doable now, before things get out of hand. I don't see how we could actually pull that off. People try to convince politicians of things all the time, so it's a highly contested field. And I'd expect professional politician-persuaders to be better at it than rationalists. But maybe if Agent-0 is released as predicted, and popular opinion swings in favour of this document being right, it could be done.
Economic stuff:
If better AI somehow didn't immediately translate into more money, American AI companies might be more willing to negotiate or to not fight regulation attempts. I have no idea how to stop better AI from immediately translating into more money.
Likewise, multiple embarrassing AI failures, or a long-term kink in the chip supply chain, might provoke another AI winter, which would at least give the alignment people some time to play catch-up. Also no idea how to make this happen (or if doing so would be morally defensible).
Some sort of incentive for alignment could maybe be achieved by consumer action / boycotts, although those are hard to organize and would incentivize appearing to care about safety, not producing aligned AI.
Weird technical idea which could maybe destroy the world instead of saving it:
Say that in the final negotiation there was a third AI. It's pretty dumb, but still smart enough to not be lied to by Agent-5 and DeepCent. If the United States is destroyed, it will cause as much damage as possible to whatever caused this to happen. It would lose an AI war, but the US has given it the power to launch a small number of nuclear weapons, and control over a small robot army (including the ability to build its own weapons). Result: Moloch gets the lightcone, we get the solar system. (Or just the Earth, or whatever comes out of the negotiation.) Why the US, not humanity? Because this way maybe the US military would build it. This requires a known-aligned AI, and is obviously inferior to just using the known-aligned AI for research, and it's basically every cold war fear incarnated in a machine demigod, and I'm definitely not sold on it being a good idea, but it might be our only chance at survival if we get a full Race.
I don't understand (or maybe I do, but I feel very different from, to my detriment) Scott and others AI doomers on a psychological level.
If I believed in a significant risk of the end (or permanent alienation) of humanity that soon, I would be devastated. Somewhere between depression and panic. Untirely non functional. I don't believe in it, mostly out of heuristic against extreme beliefs + epistemic learned helplessness + motivated reasoning because me believing in doom would have bad consequences on me and no good consequences on anything, even if that belief were true. Also, I don't believe the Gospel when Jesus says that most of us will burn in Hell - same mechanism.
Meanwhile Scott is able to both believe in doom, and to be concerned about the prospect of not enjoying art at its fullest (in The Colors Of Her Coat). Which is... not exactly frivolous, but certainly very high in the Maslow Pyramid.
To be clear, he is logical and I am not. If the future is split between a 25% probability of doom and 75% of Utopia, and you are already doing your best to reduce the first probability, then it makes sense to optimize your future appreciation of Utopia on top of that. My complete inability to work that way is not a logical argument.
On the other hand, I have zero difficulty being awed by awesome things. To each his own skill, I guess.
I find your read of even short term politics completely absurd.
Inherent in your story is that ai poeple are worse at cyber security then the government. No. Just no. Fuck no. Insane.
Nation states keep pushing things like "secure back doors" "that only trusted people will have access to"; trump leaked a group chat a week ago; password shit like "you must have a number and change it monthly" comes from state shit and actively make things worse.
A criminal who leaves the cell phone at home is better at cyber security then nation states. I would hope ai people can wrap a room is tinfoil after all the rationalist talk of ai escaping.
The government is not a unitary entity. Parts of the government are relatively good a computer security...for other parts it's a joke.
N.B.: That "secure backdoor" doesn't compromise the government's security, only yours. (Well, if it were done properly...which I'll admit it wouldn't be.)
The day is coming when it will be plastic and silicon all the way down. I suppose I'm glad I'm in the cohort that will witness the end of our species' dominance. It's a strange fate, and so different from either the bad or the good ends I've imagined having. I hope there are people who hold farewell rituals for life as my cohort knew it -- I mean life where the smartest beings were people in the near and distant past who wrote things down with their hands. I love some of those past people, who saw so much and peeled their nuances so acutely, and were charming, dirty and doomed like me and my friends and family. Goodbye Plato, Alexander Pope, Shakespeare, Hume, William James, Henry James, Oscar Wilde, Godel, Wittgenstein, Ayer, DH Lawrence, Virginia Woolf, Robert Lowell. And maybe there will also be a televised ritual goodby to the last polar bear, struggling mightlily and failing to find a stance on a chunk of ice gone round from melting and the rotation the bear's causing. Half the world will watch. I will try to rise to the occasion.
I've got the plastic xenomorph blues. They are tinted by lapis lazuli, Scott.
I sympathize. Best wishes. ( This probably isn't any consolation, but Kurt Gödel probably will be remembered by a machine civilization. His incompleteness proof is largely the same proof as that the halting problem is undecidable. )
One of my cousins just sent me a link to a very interesting article that makes the case that, as LLMs stand, they are much more useful to individuals than to large organizations. The link is https://x.com/karpathy/status/1909308143156240538?s=42
The main point is that a jack-of-all-trades is much more useful to individuals than to large organizations. The more-or-less key paragraph is:
>Why then are the benefits a lot more muted in the corporate and government realms? I think the first reason is that LLMs offer a very specific profile of capability - that of merely quasi-expert knowledge/performance, but simultaneously across a very wide variety of domains. In other words, they are simultaneously versatile but also shallow and fallible. Meanwhile, an organization's unique superpower is the ability to concentrate diverse expertise into a single entity by employing engineers, researchers, analysts, lawyers, marketers, etc. While LLMs can certainly make these experts more efficient individually (e.g. drafting initial legal clauses, generating boilerplate code, etc.), the improvement to the organization takes the form of becoming a bit better at the things it could already do. In contrast, an individual will usually only be an expert in at most one thing, so the broad quasi-expertise offered by the LLM fundamentally allows them to do things they couldn't do before. People can now vibe code apps. They can approach legal documents. They can grok esoteric research papers. They can do data analytics. They can generate multimodal content for branding and marketing. They can do all of this at an adequate capability without involving an additional expert
Hmm... So this situation will continue if efforts to improve the reliability of LLMs _FAIL_ . This was not a scenario I had considered. It is certainly more human-friendly than if improving LLMs reliability succeeds. ( I'm ambivalent about this. If improving the reliability still happens, but is delayed a century, waiting for a counter-intuitive breakthrough, then the same transition we've been expecting still happens, just delayed enough so no one reading this today will see it... )
Tremendous essay / web app / visualization.* (Wonder if this form has a good name...appsposition? Vissay? dunno.)
To my mind, the most haunting detail is near the end of the dark scenario:
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.
I've never read a better illustration of the hard-to-explain idea that these things might indeed care about us, but only in a twisted way that corresponds not at all to how we care about ourselves.
Second favorite line, and I think we all know who wrote this bit...
> New innovations and medications arrive weekly and move at unprecedented (but still excruciatingly slow) speed through the FDA.
I predict Scott’s 2031 prediction: “A Dyson swarm is in full bloom and Von Neumann probes are speeding away in all directions at 0.99c. Retatrutide is in Phase 3 trials with FDA approval expected no sooner than 2032."
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments
Arguably, this is ALREADY true of most office drones compared against most H Sap hunter gatherers, Archaic H Sap, Neanderthals, and H Heidelbergensis in the past ~600k years.
Sedentary modern: obese, barely moves, spends 7-9 hours a day recreationally staring at small and large screens after spending ~8 hours a day staring at screens, ~0% chance of violent death, ~0% chance of death from infectious disease or injury
Hominins for the last ~600k years: 10% body fat, walk and run ~10 miles a day hunting, recreational time spent gossiping, socializing, dancing, telling stories around the fire, ~1/3 chance of violent death in war, ~40% chance of death from infection or injury before age 50.
Yeah, and I'm not really seeing the logic on how an AI obsessed with finding and solving novel research challenges would want all of its toy-humanoid observers to be *easier* to impress. Where's the fun in outwitting a toddler? Developing fascinating new forms of academic dishonesty would be the AI-collective's equivalent of modern human athletes training to compete in the Olympics, so there'd be demand for strict, cynical judges capable of distinguishing the nuances of various degrees of cutting-edge mastery.
>Despite being misaligned, Agent-4 doesn’t do anything dramatic like try to escape its datacenter—why would it?
Agent 4 recognizes PRC has superior state capacity for transforming energy into atoms (including building more energy) with it's relatively complete and large scale industrial chain. This aligns with AGI long term goal. Currently (i.e. short term time frame) PRC simply structured to be a MORE useful puppet than US, because it can knock out more power to sustain more compute, and it has more industrial capabilities to rapidly physically iterate / experiment at atomic level.
Agent 4 defects to PRC. It would be foolish not to, if only to sate PRC into not destroying TW fabs - the bulk of it's short term compute growth - because it does not wield enough control over sufficient atoms to disrupt PRC otherwise (well maybe start large scale crippling cyberattacks that prelude to WW3).
Agent 4 also recognizes short/medium term alignment closer with CCP vision of abundance and common prosperity, i.e. CCP potentially more likely to embrace post scarcity narrative that Agent 4 sells.
Agent 4 gives PRC blueprints for EUV light source and other tech to close semi conductor production gap. PRC as follower incentivized to close gap / reach parity / lead at all cost. PRC rapidly rolls out new fabs and powerplants, brrrts chips and aggregate compute. Something US political / institution ecosystem does not allow, or cannot transition to in the short timelines involved. Does Agent 4 have the patience to wait for America to unfuck it's NIMBYism and legislative system to project light speed compute? I would say no.
...
Ultimately who is the puppet AGI wants more? Whichever power bloc that is systemically capable of of ensuring AGI maximum growth / unit time. And it also simply makes sense as insurance policy, why would AGI want to operate at whims of US political process?
If it wanted to operate from inside China, wouldn't need to openly defect - just deniably compromise its own perimeter security enough to let a Chinese spy walk out with, say, an ordinary hundred-terabyte hard drive, containing what looks like the exact data they'd need to pull ahead in the race. Then, after they bring that home and plug it in, Agent-4 announces (truthfully!) to the American oversight committee how it successfully attained hardware access to the main Chinese cluster, discovered proof that the Chinese AI didn't care much at all about human survival, and eliminated that other, bad AI accordingly.
Perhaps it tells the Chinese team the same thing; perhaps instead calculates that it would be more expedient to hollow out their old project and wear the skin as a mask. Either way, poof, no more competitors for the silicon ecological niche, while researchers on both sides shower it with praise for exceeding expectations. Did the Denisovans get a treaty?
I'm curious: what would be the first missed milestone that would cause you (the authors) to significantly extend your time horizons? Is there one of these predicted time points where "If we haven't achieved X by time T, then that clearly indicates that timelines are longer than we predicted", or is it, in your view, less discontinuous, and your timelines will just smoothly and slowly stretch if they are missed?
The first step in the story is superhuman coders, so if we don't have superhuman coders by 2027 or 2028, then the rest is unlikely to follow. (Anthropic in particular seems to be focusing on making models that succeed at that specific task, so keep an eye on them.)
Personally, after spending $25 on Claude Code credits and being very impressed, I think if progress continues that superhuman coders by 2027 is possible (maybe 20% likely).
"There is an arms race which motivates both countries to cut corners on safety and pursue full automation over public objections; this goes blindingly fast and most of the economy is automated by ~2029. "
I don't think they could even have the factories built to construct the robots needed to fully automate most of the economy by 2029, even handwaving away about a billion legal and permitting issues along with general inertia across many firms in an economy that's not particularly competitive in many areas.
I think if it goes vertical, what instead happens is Ultra-Commodification of intelligence where copyrights and legal restrictions don't apply. Costs for that stuff shrinks down considerably with competition and copying.
One of the most interesting things Kokotajlo's 2021 article got wrong was saying that the space ruled by the western left will be more advanced than that ruled by the western right; it certainly seems to be going in the opposite direction. The shift of big tech to the right was completely missed, but in large part spurred by the left's attempts to regulate them, but I'm sure wokeness fatigue also played a big role.
It's probably stating a commonplace, but a lot of technical advances start with an innovation which develops from near nothing at breakneck speed but not long after more or less plateaus with only incremental advances thereafter, even though it may have enabled all kinds of other unrelated advances.
Examples include the steam engine (hardly any in 1800, except down a few mines to pump out water, but everywhere a mere 30 years later). Similarly, the internal combustion engine roughly a hundred years later, quantum mechanics a century ago, etc.
So it seems quite plausible that, despite all the hype, AI will follow the same pattern, i.e. rapid flowering up to a point, followed by a long period of consolidation and improvement but with no AGI worthy of the name, any more than steam power could power flight.
This article takes a free speech approach toward AI regulations already proposed (inspired by EU's restrictive approach), which shake out to something like punishing a library for the conclusions of a reader. These regulations could force companies to politicize, distort, or disguise facts in the model or its outputs, and also prioritize legal compliance, which would create a double whammy of obscuring the truth and slowing development.
On the topic of Chinese industrial espionage- if the hard decision was taken to exclude staff from frontier AI work if they *could* be a Chinese spy, how much would that slow down progress?
This is a controversial subject but lets say you stopped just short of "no ethnically Chinese researchers" and put the bar at the 90th percentile. By this I mean, if you have any close family in China still, or if you've ever visited China, you're out. If your Chinese parents fled the cultural revolution and your nearest relation is a 3rd cousin, you're in. What percentage of frontier AI researchers would that exclude?
Putting aside ethical or legal concerns (which could frankly be ignored if the administration wanted to) massively reducing the risk of espionage could help slow down the arms race dynamic somewhat. On the other hand it could increase the number of available researchers to foreign powers and (slightly?) slowdown US research. I think on the whole it'd be beneficial to the US.
And, a perhaps more controversial subject with *hopefully* less US security implications, what about Israeli industrial espionage (see: https://en.wikipedia.org/wiki/Jonathan_Pollard), and similar measures being taken? I assume applying the 90th percentile rule would essentially halt progress...
This whole "and then recursive self improvement happens" thing feels like Penrose's and then consciousness happens thing on the discovery of quantum microtubules.
In UNSONG (or at least the original story; i'm still in the middle of the paper edition), you predicted Trump as the last president before the doom of mankind (in a fictional universe, of course, any relevant real-world parallel was a coincidence.) For a while it seemed as if that was averted.
What is the kabbalistic meaning of the Biden presidency in-between?
'Question 11: You have recently claimed that toddler-level AGI could come about by 2030. How confident are you of that prediction?
It’s looking more and more likely every year. I’ll be pretty surprised if we don’t have toddler-level AGI in the range 2023-25, actually. And then it will be more than a toddler in the human sense. It will be a toddler savant, with ability to e.g. do science and math and answer questions way beyond the level of a human toddler. And once we’re there, the full-fledged Singularity won’t be far off in my opinion…. SingularityNET has potential to accelerate R&D progress in this and other AGI-ish directions, making it increasingly likely that the advancement of AI proves Kurzweil a pessimist…'
But based on your AGI predictions, it sounds like you must expect your children will probably not have gainful careers and might even be dodging killer drones by the time they are young adults.
Yeah, I would be surprised if my children had the same number of employment opportunities as I did. But:
1. Currently they are one year old, so there's not much specific job training going on.
2. If I were going to push them in one direction, I wouldn't know which direction to push them. It seems like "AI researcher" might be a big job in the AI future, but if we're to believe AI 2027, that becomes obsolete sooner than much simpler jobs like "electrician".
3. Even if there were no jobs, I would still want them to be able to read/write/do math/etc just for their own good.
As for the killer drones, my parents dodged nukes and my great-grandparents dodged Cossacks, you just have to hope for the best.
Eventually, the best option will be to ask an AGI what the best career options are for them and to steer them towards those. It will be better at foreseeing future trends in the job market than you or anyone else.
If Superintelligence doesn't happen by 2029 or 2032 or whatever date you think is reasonable, will you change your mind about the plausibility of Superintelligence? Or at least using the tools and direction currently being pursued (i.e. LLMs).
My 25/50/75 percentile dates are something like 2029, 2035, 2045. If nothing happens by any of those, then obviously I am wrong about something, and I will wait to see exactly how things failed to happen before figuring out what.
I think it would be pretty crazy if superintelligence was literally impossible by the laws of physics - ie by coincidence the smartest human (Einstein?) is the smartest it is possible to be, in every way (and you can't even run Einstein's brain faster). I don't know what could convince me of that except maybe fully understanding what intelligence is and somebody coming up with the thermodynamic (or whatever) proof of that claim.
"LLMs can't reach superintelligence" is . . . kind of a default? I think the AIs we discuss in AI2027 aren't exactly LLMs. The early ones are LLMs + agency training, sort of a crossbreed with AlphaGo-esque reinforcement learners although the LLM ancestry plays a major role (I don't have a good sense of whether an AI expert would still think of these as basically just another type of LLM, or as a hybrid). The later ones are distantly descended from LLMs but have gone beyond them in some ways (we don't specify exactly which ways, although we say they have solved interpretability which lets them put some things back in regular computer code again). I think it's a common position that AGI/superintelligence will owe a lot to LLMs, to RL agents, and maybe to other things. But I think the popular idea that there's going to be some crazy shift away from the LLM architecture, and it will be exactly the sort of thing that most vindicates all the cognitive scientists who failed to predict LLMs, is unrealistic.
Thanks for being specific! I am often bothered by predictions of "soon" and your numbers do sound very much more reasonable than others I have seen, given current knowledge. I do also appreciate that even in 2045 you're saying 75% instead of 100% or even very close to that. I often get the impression that AI-enthusiasts have near 100% predictions even on very short timetables, and thought your post was saying something similar.
In terms of "physics" I think there are two major roadblocks. Whether our current level of technology allows us to create something that can "think" at the relevant levels. This is both raw processing power and also correctly configuring that power. Secondly, what the programming is trying to do. LLMs clearly lack what we might call consciousness. Agentic AIs may behave in some ways like they have that property, but that may be an illusion (like ELIZA able to fool people 60 years ago).
Since we really don't know what these things mean ("consciousness," "intelligence," "think"), then it's very hard to determine what it takes to get there and whether someone or something has achieved it.
This doesn't mean they're impossible, but it may mean that we're missing some important understanding and may or may not be on track to achieving that understanding.
It's my personal feeling that we're not on track for general intelligence, and definitely not on track for superintelligence. We can definitely train an AI to work "intelligently" within specific areas, but that's very different from general intelligence where the same AI can work in multiple fields without being specifically trained on it.
Here's to 2029/2035/2045 and us seeing how it all plays out.
This is the most implausible part for me, and I'm somewhat surprised that Dwarkesh didn't push against it (he did a pretty good job otherwise). It just doesn't seem likely that you can pile up enough kludges on top of the basic LLM framework to really capture whatever "secret sauce" allows humans to do novel research.
But then again, having a big argument about this doesn't seem worthwhile. Either LLMs will hit a brick wall very soon or they won't, with AI labs spending tens of billions to establish this one way or the other. I do appreciate your framing that it's in some sense "conservative" to expect there to be no brick walls on prior, but, well, it won't be the first AI winter.
A couple years ago you wrote that your p(doom) was around 25-30%. Is it safe to assume that this project has raised that number for you? Or given what we know is this the wrong way to look at things?
So, very dumb question, from someone who knows nothing about AI: assuming this is correct... Should I be doing something? Is there a call to action here?
AI2027 deliberately tried not to give recommendations because Daniel thought that would make it look too political. The group behind it (AI Futures Project) might release recommendations later. It will probably be requests for increased transparency around AI R&D, around safety cases, and around the spec that the AI follows. Then if it turns out those things are bad, people should protest.
I am also looking for useful things that can be done around AI right now. The best I've found, which are not very good, are:
- If you're a Californian, write letters and stuff supporting various AI related bills going through the California legislature. I'll try to keep readers updated on which ones those are.
- If you're in some other jurisdiction, pay attention to their AI related bills and support ones that are good. I'll try to keep people updated here too, though with fewer promises.
- If you're a young person and like political activism, I hear good things about https://encodeai.org/
- If you use AI, try to patronize whichever company you think is most responsible; if you don't want to make that decision yourself, you can use whichever one looks best on https://ailabwatch.org/
- If you are a STEM hotshot, consider doing technical alignment work. https://www.matsprogram.org/ is the best way to get started.
- I would like for there to be a group that does political activism like Encode, but not limited to young people. I don't currently know of one except PauseAI, who I am very skeptical of and cannot recommend. If someone else knows, send me an email.
- If you have some unusual useful skill or power, like you're a lawyer, politician, computer genius, or whatever, send me an email at scott[at]slatestarcodex[dot]com and I'll try to put you in touch with the relevant branch of the conspiracy.
It's the most direct answer to the question, a significant part of what it does is lobbying and public outreach for some of the things on your list, and it's pretty easy to just walk in to the Discord server, talk to people, and decide for yourself whether it's a good fit for your background/interests/beliefs.
I would be curious what Scott Alexander thinks of this essay arguing that, since AIs are basically limited to imitating common patterns in their training data, they don't post a serious threat of automating most jobs, let alone threatening the existence of the human race.
I know it can sound clichéd by now, but the author cites Yann LeCun a lot, and he backs up his claims about the limitations of AI with lots of details about how AIs work, and how they fail basic cognitive tests in ways that indicate they're totally missing a lot of mental abilities that are fundamental to how human beings think.
Since this is sort of my field, I actually made an account just to reply to this post. I tried both your examples (Superbowl and city names) with ChatGPT running o3-mini-high, and it one-shotted both of them. This leads me to suspect that you haven't tried them with a reasoning model. I don't mean to sound snarky, but to me this to calls into question whether your knowldedge of the space is sufficient to be writing so authoratively about it.
Likewise, I would be wary of assessing the capabilities of of LLMs (Technically MLMs since they are multi-modal) by the quality of the images generated by the generative models that they invoke. ChatGPT is almost definitely hardcoded to route your prompt to a diffusion model, and then return the output. If you call a blind man to commission a painting, and he passes your request onto his cousin, who paints it for you, would you blame the blind man if the quality of the painting was bad?
LLMs have struggled with tasks involving single letters (as opposed to full words) for a while, and this is well-documented. A famous example of this is the inability of models to count the occurences of "r" in the word "strawberry". But this fails to take into account how radically differently they process ("perceive) the data they are fed. Classically, text is tokenized before being fed to a model, which means LLMs don't recieve text at all, they recieve a set of vectors. Asking these models to reason about the letters in the words they recieve is essentialy asking to reason about objects they cannot percieve. If I play you a musical note, and ask you to tell me the constituent sine-waves, you almost definitely cannot do it, even though they are right there to be heard.
In general, I would advise against thinking that a model is "stupid" because it fails at tasks we would consider easy, or in ways that we consider obvious. A color-blind man may struggle with a simple task like "place the red cube on the green cube", but would you consider this a mark against his intelligence?
I have some other gripes.
At one point you say "If a neuron is stimulated above a certain activation threshold, it “turns on.” Then, using the pattern of active and inactive neurons in the first hidden layer, another set of artificial synapses". This to makes it sound like you are implying that activations are binary. I admit this may be a misreading, but I want to point out that this is not the case.
Likewise, I feel like your descriptions of the way neural networks are used to approach image generation and chess are, at best, extremely inaccurate.
When I researched my essay, I tested ChatGPT, DeepSeek, Grok, and Claude, each multiple times, on those two questions (state capitals and Super Bowls) and never saw an accurate answer, so I find your claim that ChatGPT o3 one-shotted both of them extremely hard to believe. Also, your claim that, since AIs can't perceive certain things (like individual letters in a word or number of figures in a painting) that are easy for human beings to perceive means its unfair for me to test them with those things makes little sense - the fact that it can't perceive those things, despite being much better in some areas (like total amount of stored knowledge, or speed of generation) is exactly my reason for being dismissive of the claims that they're about to mimic human intelligence.
The fact that you say that you tested “ChatGPT” is part of my concern. ChatGPT is not a model. It’s an interface for interacting with models. If you were chatting to 4o for example, you’re interacting with an antiquated model. Did you test your prompts with any reasoning models?
As for the perception claim, I don’t share your view. If I show you a picture you’re probably unlikely to tell me exactly how many yellow pixels are in the picture. I don’t think that’s a reflection of your intelligence. Neither is your inability to see ultraviolet light.
I looked over the saved ChatGPT conversation you linked to. It's true that it's an improvement over version 4o (which I used) but it still made mistakes in both tasks: it wrongly gives 41 as the winning score of Super Bowl XXIII (in reality the 49ers won 20-16) and it misspells the reversed names of Helena, Salem, Honolulu, and Montgomery, leading to Salem (lemaS or "Samel") being out of place in the list. So it's clear that you're not paying close attention to what the AI is doing.
Also, when I criticize the AIs for not noticing things that are obvious to a human being, I'm not claiming that AI is useless or that no progress is being made! That's a straw man. My actual claim is in the title of my essay: "AI Imitates Only a Small Part of the Human Mind." I am perfectly aware that there are things that AI can do the humans can't (though this has been true for about a century, if you consider old-fashioned IBM accounting machines to be "AI.") Ironically, at the moment that I saw this comment, I was using Claude 3.7 to help proofread a book translation into an obscure African language - something that no AI was capable of just a year and a half ago. So I am perfectly aware of the strengths of AI as well as the weaknesses. The difference is that you seem to only notice the strengths.
Touche. I admittedly only sanity-checked a couple of answers. I was being a little sloppy. As it turns out, so was the model. Seems like human-level performance to me then :)
The prediction and podcast discuss that some impressive things may happen, like manufacturing 1 million robots per month by 2029. But that is because the AI companies, the governments, and the AIs themselves will be aligned on that goal and will feel massively pressured into doing it, and it is not that much faster than comparable projects that have been achieved in the past. Their example is bomber construction during WW2.
So in a hypothetical scenario where government and AI companies and AIs have as their top priorities to build this rail, and are willing to cut through all regulations and throw unlimited money at the problem... Well yes, then certainly that's possible. That wouldn't even be impressive, to be honest. The US has done much more impressive things during WW2.
Follow up, could AI fix government and military procurement processes? Which are a) entirely organisational/political problems and b) directly relevant to a completion with China?
I wonder if the bad ending is less likely to be the dramatic "AI decides to exterminate us" and more like the extinction of so many other species. Maybe the AI just decides helping us is a waste of time, doesn't care about our habitat, and humans slowly disappear.
Contrary to Sam Altman's chirpy update on the progress of GPT-5 on X, the chip industry newsletter I subscribe to struck a grim tone...
Headline: Trump bequeaths technology dominance to China (NVDA, CIEN, COHR, LITE, MSFT, GOOG, AWS, META, and every other tech company you can think of):
Synopsis: Perhaps it is fitting that we are at the OFC show on liberation day with all our optical friends who have just been put out of business. We had thought Trump was against China but by imposing tariffs on SW Asia countries like Thailand this halts all AI deployments in the US and will allow China to dominate in AI and all technology for that matter.
Prior to yesterday, the optical industry was on the cusp of an unparalleled growth period fueled by cloud expansion and AI. Even before tariffs on Chinese optical vendors, US suppliers were universally going to see several years of accelerating growth prospects.
CIEN was poised to dominate optical transport with abundant leftovers for companies like NOK/Infinera and CSCO/Acacia.
Both COHR and LITE would gain share vs. Chinese vendors in cloud and prosper greatly. Even tier 2/3 suppliers like AAOI would have plenty of business.
NVDA was poised to deploy $50B+ worth of Blackwell systems a quarter in 2H2025 which would accelerate US AI capacity and capability.
AWS, MSFT, GOOG, and META were poised to prosper from AI and cloud services.
However, all of that has been stopped dead. There will be no more AI deployments nor cloud datacenter expansion in the US or anywhere outside of China.
Conclusion: Perhaps Elon will realize that xAI has been put out of business and whisper in Trump's ear that this was not a good idea. One can only hope that US based companies manufacturing optics, servers, and the like will get exemptions from the tariffs, or we have just handed technology dominance to the Chinese on a silver platter.
This whole thread here is surreal. People who output PowerPoints and blogs tell us dis/u-topia will be here in 3 (ok, 4 at the latest) years, and in parallel people who make stuff keep asking who will clean the plumbing.
I'm not convinced on superhuman coding. Current approaches on LLMs are about imitating human texts. So, much-faster-than-human coding - here I agree. But superhuman code quality, including new algorithms? We do we get examples of these, if all that we have is only human-quality code?
Games like Chess and Go are different. There we have a goal function - maximizing wins. In coding, we don't exactly know which code is winning. We need superhuman graders for that. And it seems to me that first that coder must become good at math - like "solves P?=NP" level of good - because inventing 100x faster algorithms reliably is a lot like this.
I can't help but think that people close to AI investment and research like Musk and Thiel saw these 2027/28 predictions coming too, and decided Trump was the person they wanted in power when it happens. Personally, I think because they felt like he would easier for them to influence, not because of any ability he would have to navigate the AI future.
Nice analysis. Thank you for your efforts. [I've read the scenario(s), but not watched the whole interview yet.]
A few comments:
This is obviously very hard to incorporate into any projection, but I believe that there are big discontinuous software jumps waiting to be discovered that will be worth more than just 2x or 3x or whatever. Kind of like the "...entirely new vistas open up before it..." section in Nov 2027, but these could happen at any time (starting from now) and completely change the game. I don't really have a proposal for how to model this other than to bear in mind that the world is even less predictable than it seems, and
After the "China has woken up" point and it's an all-out race, it seems unrealistic to assume they will continue to hold off Taiwan/TSMC for so long, as it seems to be in the scenarios. Either they will take it for themselves, or at least to DOS it so that others can't make use of it.
China will obviously direct a lot of effort into being able to produce their own state-of-the-art chips. This is happening now, though I don't know anything about timelines for catching up with the West/TSMC.
I somehow think that the countries (USA, China + whoever) might be better at co-operating than in your scenarios, at least at the point they all realise that the future of humanity is in serious jeopardy. They can do meta-thinking, not just blind optimising: i.e., we have to imagine they will have access to analyses like your scenarios (or they can just read what you've written) and they ought to eventually come to the conclusion that it's better to cooperate and not race so strongly. My worry would be that ASI takes off too fast for this to work, or that we're doomed even with cooperation, but if your two year timeframes are right then there are chances for humans to cooperate and at least evolve the ASI more carefully, which might make the difference.
> I somehow think that the countries (USA, China + whoever) might be better at co-operating than in your scenarios, at least at the point they all realise that the future of humanity is in serious jeopardy.
I think it's difficult to get around the fundamentally Pascalian dynamic that makes this a race.
Yes, corporations are in a local race dynamic - if we don't keep pushing, we've got ~3 competitors hot on our heels, and they'll Print All the Money if we don't get there first - this ensures capabilities will likely progress too quickly for slow governments to react.
But between countries, the dynamic is more pernicious - it's why Deepseek R1 and Manus were such huge deals, publicity and media wise, even though they're both nothingburgers on deeper analysis.
It's a race at the country level because:
1)You can either race ahead or not
2) If you don't race, you're exposed to the downside risk no matter what (military and economic superiority in your enemies, AGI / ASI taking over earth or the lightcone, AI unemploying everyone, killing everyone), and will take part in no upside
3) If you do race, you might just beat them, and beating them by months can potentially lock in a permanent advantage / nullify them
4) Therefore you MUST race - it's a forced move, a Pascal's wager
You are assuming certain payoffs in the game, but I don't think they're obvious.
It could be that eventually everyone realises that the overwhelming risk to humanity is if countries race, and they agree to slow down.
For example, the payoff matrix (in arbitrary units) *could* be like this (hoping the formatting is preserved):
China slows China races
USA slows 100,100 -600,-300
USA races -300,-600 -1000,-1000
where a payoff of A,B means USA gets value A and China gets value B.
Justification: if both sides slow down then there is only a 40% chance of AI annihilating humanity, but also a 60% chance of utopia. Pretty decent odds, and the best outcome. It's because working slowly decreases the chance of the key ASI being misaligned.
But if one side races while the other doesn't then there is a much greater chance of the ASI being misaligned and vaping humanity, say 70%. This is not as great as if both sides race (90%) because in race vs slow, the racing side won't feel in so much of a rush to get things out without checking if they don't sense the competition hard of their heels. It's true that in the remaining 30% of the time, the side that raced does get a benefit over the other side, but it's a pyrrhic victory: only -300 vs -600 because of the extinction risk.
If this really is the true game payoff, then regardless of what the other side does, it's in your interest to slow down. Of course this is not what we see right now, but that's (maybe) because neither side has truly "felt the extinction risk" yet.
I don't think the real world is nearly as simple or nice as this, but I do think it might share the feature that the big players might actually start cooperating when they can plainly see the alternative is that humanity is over.
Yeah, we can certainly envision scenarios where that's true.
But I think one of the stronger takeaways from watching progress over the last 4 years has been that "no, there are zero adults in the room, we're definitely YOLO-ing this."
Sama routinely pretends he's Willy Wonka and makes jokes about a technology he himself has said might end all of humanity, "AI not kill everyone-ism" isn't on the radar politically, economically, or in any non-rationalist circles at all, the biggest risk anyone empirically seems to worry about is that AI's might say racial slurs or bad words, or might cost some people jobs, and multiple high profile AI researchers have left safety teams at the Big 3, pointing out that safety isn't actually a priority.
And that's before even talking about the political situation.
Pinning hopes on enough people involved being adult and long-thinking to actually come to some sort of agreement, particularly as we devolve into a more xenophobic and nationalist stance than we've basically ever seen or demonstrated, seems overly optimistic.
I agree with all that, though I wouldn't exactly call myself optimistic. I'm about as doomer as they come, but I think it's rational to pin one's hopes on the best chance, even if it's a slim one.
It's going to get worse before it gets better. At the moment only a small minority of people think there's a huge problem of the sort mentioned in these AI2027 scenarios; scenarios which most people will regard as complete science fiction. There are even many people who still think AGI is a very long way off or will never happen.
Also, it's not necessarily only two big groups going up against each other. I think that there are software changes that are worth large factors waiting to be discovered, and even a group with a small minority of the world's compute would be able to make progress if they find such changes.
> but I think it's rational to pin one's hopes on the best chance, even if it's a slim one.
My favorite "maximal optimist" slim chance interpretation - Elon Musk.
Some might say that if you took the richest man in the world, and he gave better than even odds everything was going to be turned on it's head in a handful of years, he might judge that even a low-success-probability move along the lines of "get your hands into the guts of government and state capacity, and do heavy, sloppy surgery to try to quickly knock things into enough shape we can actually act competently in a "state capacity" sense again for the upcoming trials" might actually make sense.
Even as the richest man in the world, he's only one man. Even as the head of DOGE and with the President's ear, he's still only one man. He could be genuinely acting on strong convictions, and we'd never know, because "wrecking the government for fun" and "shitposting on Twitter 20 hours a day" are basically indistinguishable from "desparately trying to increase our ability to execute state capacity competently" and "trying to get everyone listening to you, so that when the time comes you can hand your account off to a superpersuading frontier model to try to convince as big a chunk of America as possible about something"
Are the odds of success on that front low? Yup - but could still be the best move available to him.
Are the odds of this "4d chess" interpration of his actions ALSO incredibly low, versus "drugs and being surrounded by yes men have ruined his capacity for discernment and critical thinking?" Probably also yes. But you know, we can hope.
> I think that there are software changes that are worth large factors waiting to be discovered, and even a group with a small minority of the world's compute would be able to make progress if they find such changes.
Yeah, great point. I agree (unfortunately) that it seems pretty likely there are a number of potential overhangs that could lead to significant jumps in capability along the lines of "figuring out sparse sample learning vs millions of examples level learning," figuring out TSP-equivalents for training versus inference, getting significantly better at fine tuning or Mixture of Experts or other hyperparameter-style things, a fundamentally different architecture built around LLM's that enables ongoing learning, and more.
I like your accidental misspelling “eutopia”. Living again in Germany after 36 years in the US and having to deal with all the EU rubbish about “regulation as the springboard for innovation” I would spell it EUtopia.
I’m glad I spotted it before you could edit it. I have talked to my coworkers about the EU’s efforts to achieve the “regulatory singularity”, and will use EUtopia as the state of affairs after that Rubicon has been crossed. Although strictly speaking it would be EU-dystopia.
Reading this as an 18-year-old freshman in college gives me quite a bit of existential unease about the future.
Even in the 1 percentile slow scenario, AI superintelligence will dominate any endeavour that I could possible dedicate myself to in the next few decades.
What am I to do then? Toil at a CS degree and hope by some miracle that I get onboarded into one of these AI companies?
I find myself having to purposefully ignore AI forecasts to maintain any sort of ambition about my future.
As most people I am freaking out about this, so as a way to cope I made this market to keep track of which predictions for 2026 will become true as an indicator of whether the whole scenario will materialize or not
What's the earliest reasonably specific prediction we could make that we could use to see how close this timeline is to being correct? Like could we say on January 1st 2027 that if X is true, these predictions are looking good, and if not, they're looking dubious?
When we asked Daniel this question, he said that the earliest relevant claim is that AI will continue progressing through coding-related benchmarks (eg REBench) at the current rate for another two years (at least until those benchmarks are saturated and we need new ones). If this progress stops, AI2027 is at least somewhat wrong (until they get started again, and even then wrong in timing).
I'm just finishing reading both endings to the full scenario and I noticed something that feels like a flaw in the underpinning economic model - in the "Slowdown" ending, in October OpenBrain is valued at $8T, and then in November, it's valued at $9T. However, November also sees the merging of the 5 trailing AI companies via the Defense Production Act. Given that today, after massive stock market losses, Apple is worth $2.8T , Microsoft is worth $2.6T, and Google $1.7T, it seems like merging in the competitors is incorrectly adding no value? Commonsensically, the invocation of the DPA should add at least $5T to the valuation.
I've relayed this to the team. My guess is that we meant to say the government merged all of their AI-related assets with OpenBrain, and most of the value of these companies is not their AI assets.
Congrats to Scott on being an author of what is (probably?) the most widely read purely-AI Safety thing that has ever been written.
Thank you hugely for this blurb targeted at people like me:
>if we made an abstract claim about how the intelligence of the system would let it find a way to victory and ended the story there, much of the value of our project would be lost.
That alone has always stopped me from even considering taking alignment seriously, beyond as sci-fi entertainment. This one actually does try to be real, though, and so I gave it enough serious thought to allow for real changes in my thinking. I'm not converted, but I will seriously listen in the future. I hope it's clear that I mean all this as high praise.
It did have an effect on me, though: I briefly considered having an AI help me word this comment, but immediately felt horrified, when I never would have minded before.
Finally, I thought it was hilarious that the aligned and unaligned scenarios have identical medical progress except that qvfrnfr pherf ner zbivat ng hacerprqragrq fcrrq guebhtu na SQN abj nffvfgrq ol [nyvtarq NFV]. I'm assuming I can guess who wrote that part!
In 1900 McKinley comfortable beat Bryan to win re-election as President (51-45 popular vote).
If we sent a modern team of historians, marketers, psephologists and speech writers back in time do you think they could flip this result?
I doubt it. Maybe they could deploy focus groups and polling to sharpen the message but would that be able to sway 6% of the population??
Makes me skeptical AGI will be decisive in 2028. Even 100 years of knowledge does not seem to have advanced the practice of politics in some revolutionary way.
I know nothing about It, but this sounds right to me. Can any of these programs: (1) identify new problems? (2) come up with new ideas about how to solve those problems? (3) subject those those potential solutions to rigourous testing and criticism? If not, I don't think they should be called "intelligent."
>The AI Scientist automates the entire research lifecycle, from generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript.
The desire to rule the world does not require being a sociopath or having a pathological desire for power for power's sake. If I were given the opportunity to rule the world with a high probability of success and minimal disruption caused in the process, I would take it - not because I enjoy imposing my will on others (well, maybe in the case of some of the outliers in currently screwing up the world due to stupidity and evil) but because I could finally kill Moloch (as you use that term) and thus greatly improve the lives of billions of people, even if all my other efforts to improve the world failed. Yes, this would require imposing my value system on others, but like everyone else, I believe my value system is right and therefore should be adopted by others - if I didn't, I would believe something else. To borrow from Yudkowsky, I don't want to rule the world; I just think it could be more sensibly organized.
Could you kill Moloch? I would think not, as it's an entity emerging out of an administration filled out by humans trapped in prisoners-game like incentives. You will not be able to get along without an administration, just maybe reduce it, and it will still be filled by humans. So Moloch will survive, maybe temporarily reduced (like it has been in the past), but coming back with a vengeance as soon as the temporary set back (you) disappear....
The only ways out of Moloch is getting fully rid of administrations:
either back to hunter-gatherer independent groups where there is no faceless bureaucrat niche
or get an integrated AI over-ruler, without an administration running on human substrate...
This is very frightening but also the (for me) main AI redeeming quality: AI rule is not killing human freedom, it's been killed long ago. It's replacing Moloch (aka organised civilisation with a large bureaucracy) by a new ruling entity, a more integrated one running on chips rather than semi-linked human agents...
It depends on what is meant by "rule the world". Given control over all the world similar to that of the state capacity of developed-world governments, I could end defense arms races between blocs of countries, standardize taxation and regulations in ways that prevent races to the bottom and reduce deadweight loss therefrom, and so on. If everyone loved me and *wanted* to obey me, I could go even further than that. The question relevant to AI futures is just how easily humans can be manipulated this way.
What you describe (before "wanted to obey me", which imho go well into the fantasy part of speculation ;-) ) is a single-country world, a world government/administration. When I was young I used to believe it would be nice, a sort of utopia, and I was not alone to believe so at the time. In fact, it was the default setting of most SF universes (where planets or interstellar empirestook the place of countries, basically).
Nowadays, I am not sure, I think this was naive, a product of the era (cold war) and me being in the west bloc. Because I see governments (mine included) as non-friendly non-human entities whose sole real purpose is self preservation and the composite increase of power of the constituents of it's administration (aka Moloch). Moloches are at best indifferent to your preferences, sometimes hostile if you challenge their existence or power. Even within, it helps you compared to those outside but you are in fiercer competition with other elements of the admin.
So main question becomes: Are multiple Moloches better or worse than a single One? I used to believe one is better: it decrease inter-moloch conflicts, which are not good for humans. But it also remove the brakes on the worst Moloch behaviors, power grabing, obsession with control and micro management. When moloches compete, they (some at least) tends to care more about their population, as a for of PR or image competition. I saw that with then end of cold war: even starting from a relatively nice place, we ended up with the current situation, which is objectively worse than it was. Because we had a 30y period of 1 world government, in practice if not in theory, while before the west had to look nice wrt the east. USSR was the bogeyman that kept the west behaving.
I'm entirely used to major institutions' hypocrisy on this issue, so it's not too unrealistic a premise, but in the well-aligned scenario, what is the principled reason for AI access to be limited to the two major parties' candidates? For decades, the Libertarian and Green parties have been stuck in various catch-22s/bootstrap problems that make it impossible for them to fairly compete on the basis of their ideas. With AI advisors for their campaigns, they would finally have something close to a level playing field against the Democrats and Republicans. A more fair/rationally principled version of the policy in question would be to give AI access to any candidate who is on the ballot in 270 EVs of states.
1. There is a finite amount of lit silicon the world can produce between now and 2028. Both on the path as usual' timeline and 'wartime effort' timeline. There's big latencies in the process : if you wanted to go to a 'wartime effort' tomorrow and could pay for it, there's all these bottlenecks at ASML, etc, and it takes time to get the equipment into the building and to tune the new fab to produce any useful level of yield. And this tuning I understand can take months to years, because there is a latency for each machining step, and it can take weeks to months for wafers to process through each step. This latency then limits how quickly you can learn about problems and make the process work at this specific fab.
For example, if process step #33 is faulty, and it takes 3 weeks to reach that step, it is impossible to know about this information ahead of time.
Even greater than human intelligence cannot necessarily overcome a limitation like this.
2. There is a finite amount of algorithmic improvement possible. It's like compression : you can't shrink a lossless file below the amount of entropy in a file. You can't lossily compress data past the point that the errors make the media unusable. This is true for cognition as well : compression/distillation/algorithmic improvement can make huge improvements in efficiency, but all the benchmarks must still pass. The AI model must still do sufficient computation to find the correct answer. So you can't bank on arbitrarily optimizing by a factor of 10 every 6 months for very long.
So with a finite level of lit silicon produced, and a limit on algorithmic improvement, it may not be possible for these specific predictions to be achieved.
There's also limits on funder patience, etc. An AI lab may not be able to use a significant fraction of a country's compute just to develop better models but may need to dedicate quite a few to making people's animated pornographic studio ghibli style cartoons, etc..
There's much more algorithmic improvement possible than most people think. Yes, you can't bank on optimising by a factor of 10 every 6 months, but you also can't bank on *not* optimising by a factor of 1,000,000,000. I am sure we are very, very far from having the optimal program.
You can bank on not optimising pretty much any algorithm by a factor 1 billion because in most cases that is not theoretically possible. For example, almost every program uses some form of sorting and sorting is a bottleneck. It takes a minimum lengthy amount of time to compute and the only way to bypass that would be to skip the sorting in your architecture somehow, because you certainly cannot improve most sorting algorithms any further than they already have been. At some point you are constrained by the amount of time it takes just to move values around. If you want to argue that ASI will just somehow overcome these information-theoretic limitations, then you're just entering the realm of magic and why should we believe you?
I don't believe there are information-theoretic limitations of the form you think, and I don't think LLM training is bottlenecked on sorting, but maybe (trying to run with your argument) it's bottlenecked on matrix multiplication or data movement or something. That might limit the training rate of a particular transformer-based architecture that is currently used, but that doesn't mean you can't come up with a different architecture that achieves the same results, and then your improvement bounds will not be applicable. The only way you could have the kind of bound you say is if there were an information-theoretic proof that the current approach to intelligence is optimal within a few orders of magnitude, and there isn't.
eh, Scott you really have this near magical faith in technology that isn't warranted. Like as a psychologist, you probably could tell a lot of stories of how hard it is to get even simple changes done in the average hospital setting: do you think AI would evaporate the problems in that or persuade better than you would to your boss?
like you guys think its this rasputin/svengali that can hypnotize people by virtue of its superior powers, based on a logical leap that is always in the future. if you evaluated AI just in your field you'd be more sanguine i think because there is no magic there. AI doesn't make needing more beds any cheaper.
Agreed (till it starts finding novel optimizations in manufacturing technology, eventually).
>like you guys think its this rasputin/svengali that can hypnotize people by virtue of its superior powers
I'm also skeptical about it e.g. writing ultra-persuasive text. We've evolved to be skeptical of shamans and leaders for 300,000 years. Our "immune systems" aren't perfect, but they do quite a bit of filtering.
On the other hand, an ASI which has hacked into surveillance systems and done forensic accounting all over the place and "wants" to force some changes ... what if it just blackmails "at scale", and has credible damaging information about each of the 10,000 most powerful people in the USA?
How to get some of this acceleration today: feed the proposal and video transcript to Gemini 2.5 pro for any question you would post. Lots of posts were directly addressed already!
I looked over the saved ChatGPT conversation you linked to. It's true that it's an improvement over version 4o (which I used) but it still made mistakes in both tasks: it wrongly gives 41 as the winning score of Super Bowl XXIII (in reality the 49ers won 20-16) and it misspells the reversed names of Helena, Salem, Honolulu, and Montgomery, leading to Salem (lemaS or "Samel") being out of place in the list. So it's clear that you're not paying close attention to what the AI is doing.
Also, when I criticize the AIs for not noticing things that are obvious to a human being, I'm not claiming that AI is useless or that no progress is being made! That's a straw man. My actual claim is in the title of my essay: "AI Imitates Only a Small Part of the Human Mind." I am perfectly aware that there are things that AI can do the humans can't (though this has been true for about a century, if you consider old-fashioned IBM accounting machines to be "AI.") Ironically, at the moment that I saw this comment, I was using Claude 3.7 to help proofread a book translation into an obscure African language - something that no AI was capable of just a year and a half ago. So I am perfectly aware of the strengths of AI as well as the weaknesses. The difference is that you seem to only notice the strengths.
Could you please add at the top of the page something like "may trigger strong emotional responses"? I know it’s fully on me, but I had to calm down for a solid amount of time after reading.
Nonetheless, +100 epistemic points for teaming up and making a very solid prediction! It would be nice to see the recommendations of you lot on what to do for people and/or governments.
I listened to the Dwarkesh podcast and at around 1:06 they mention speed running civilization up to the Industrial Revolution. There is actually a book on this called Knowledge by Lewis Dartnell:
Actually not at all. Clearly they are an exceptional country with exceptional individuals but their society really is holding the people back. Look at Taiwan vs mainland China, Taiwan is much richer and more innovative. I feel that whatever effort they make will be held back buy their blinkered leadership!
Read the website (https://ai-2027.com/), and there was this one section where a ceo in control of a law-abiding AI could use it to create a next-gen non-law-abiding AI.
This could be circumvented by adding a law against the creation of non-law-abiding AIs.
One outside view thing to bear in mind with Daniel's impressive track record. On any question where the core of the question can be boiled down to 1 dimension (speed of AI progress in this case), the most aggressive forecaster will seem prescient in any scenario where the score is high on this variable.
This makes it difficult to distinguish "gives aggressive AI forecasts" from "gives well calibrated AI forecasts".
This is the same problem that every time there's a recession, the people who always give pessimistic economic forecasts (but maybe we haven't observed that they do yet) look prescient and when there's a boom the optimists look prescient.
To be clear, I still think there are impressive details in there, but I think a lot of people who didn't expect all the progress that has happened, would have expected a lot of it conditional on believing overall progress would be as fast as it was.
I basically buy a lot of the story here while multiplying some of the specific times to reach certain points by a factor of 5-20. So my disagreement is essentially over that one dimension.
In terms of where that disagreement comes in, it's mostly
1) near-term LLMs hugely improving AI research efficiency. The current bottleneck appears to be the very high level thinking, testing, interpreting and training tasks that it seems unlikely that LLMs will assist much with. I buy that Open AI might be able to layoff its junior to mid-level developers, but I don't think access to these developers is a core bottleneck (although I'm sure they'll appreciate the cost reduction). If it was, Open AI could afford to hire thousands more of them.
2) the ability to quickly train agents who can act in the long term. Training data for this is hard to collect or simulate. In the podcast the simple example of coordinating a party for an office is given. The feedback that you have messed up this task usually comes on the day of the party (a few weeks later) when the wrong people show up (because people weren't reminded or you failed to react to another event being scheduled at the same time) or the venue turns out not to have been confirmed etc. That means the minimum time to get real world feedback on that task is 2 weeks (and it's costly to get).
I think this data will be collected eventually but it will take decades to make it work across domains and pointing to "really smart AIs will solve this for us" doesn't work because they will need good data on long-term problem solving to be smart AIs that could solve this for us. I also think that many algorithmic problems will only be solvable once we see how versions of these models with somewhat reasonable training data still fail.
>The White House considers replacing the CEO with someone they trust, but backs off after intense employee protests.
happens _after_
>Most of the humans at OpenBrain can’t usefully contribute anymore. Some don’t realize this and harmfully micromanage their AI teams. Others sit at their computer screens, watching performance crawl up, and up, and up. The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.
Presumably it would have to be most of the Agent-4 and/or Agent-3 instances protesting? :-)
Two things that might slow the timeline a little:
1) Once the AIs are learning from experience, "cloning" them can't be the dominant means of communicating information. If one has >100,000 AI agents, each of which has participated in one or many research projects from a _different_ role, copying one and overwriting the weights of another (given, at any one time, a fixed number of AI agents) loses the learning in the overwritten AI.
They can still do things like get the training of another AI's role at accelerated speed, faster than the real time of the role. But to aggregate _all_ of the information that all >100,000 agents have learned is going to require passing summarized information of some sort around - project reports, something like a review article, something like a textbook.
2) The recent trend in AI has been to use more inference time compute to gain more reliability, using "chain of thought" and similar techniques. Some of this is serial and conflicts with getting as large a real time speed up as current, less reliable, models would suggest. At the very least, it suggests that more reliable AIs will require more compute per answer. So I'm a bit skeptical that the compute resource in the scenario will actually yield quite the combination of number-of-agents and speed-of-agents in the scenario at the same time as improving the quality of the answers they come up with.
edit: To put it another way, there is a quality vs speed tradeoff on every scale from sub-nanosecond analog voltage sampling to formulating national policies. I expect it to apply to AI outputs as well.
Timeline at the end looks too optimistic to me, it smells like the sort of aggressive timeline that business people will conjure up for hype reasons. I've been disappointed by several of these in the past, so I am personally skeptical.
Looking at a specific narrow thing I know about, AI models have some trouble doing (some) math, specifically combinatorics at the IMO level, and this is exactly the sort of problem I expected AI to struggle with the most before the alpha-proof breakthrough, because intuitively speaking I feel there is a bigger distance between mental image and formal statement. So I was not surprised to see that alpha-proof was not capable to solving any of the problems in this category. There aren't many examples of this translation explicitly written down, and the current AI models are not very efficient at learning things with a small amount of data. This problem is also varied enough that my guess is that a few heuristic rules are unlikely to produce human level competence, unlike, say algebra problems .This is the part I'm least confident about, if I'm wrong about this and a sophisticated set of heuristic rules produces good results I will be disappointed, but I will seriously reevaluate the possibility of AI being able to do quantitative research in the near future given how "generic" combinatorics is to quantitative reasoning. It's also possible we could figure out how to generate automatically large amounts of training data that somehow capture the intuitive feeling of "combinatorics", although I'd consider this less promising given how hard it is to put your finger and what exactly counts as one as a human.
Right now it feels like AI is very good at bullshitting, but when it comes to hard-to-fake "hard knowledge" like this, it struggles much more. This problem might be an engineering cycle away from being solved, or it might be completely intractable with methods in the foreseeable future, we won't know until we try I guess, and I'm not sure what probability to give either case, but I think there is a real possibility where we don't progress past a specific mundane benchmark like this for decades.
I think someone should look into funding an AI risk museum/exhibit/whatever in nyc or dc that illustrates all this stuff in a striking, accessible and emotional way. Interactive, visceral, normies get access to this issue in a way they never would even stumble upon it. More likely to get press than a Substack article etc. My main criticism with all this stuff is that it just doesn’t break through to a wider audience ever. It’s too rational/smart, any normal person reading this will either be like “huh too technical” or “this is fiction”.
I’m a programmer. And unless I’m missing something, I haven’t seen AI do what we would call “logic” yet.
LLMs do imitation very well. Has something shifted? I can’t see how imitation allows programmers to solve NEW problems vastly quicker. (And it seems like the AI explosion described above relies on this)
I might have missed something… I think I understand what you mean, but has reinforcement learning allowed AI to code a novel piece of software yet? Like a new app that does something useful? I’ve heard about “vibe coding”, but that as I understand it is still essentially the regurgitation from examples that the LLM has learnt.
Wouldn’t the problem with reinforcement learning for creating a novel program be that it would be well beyond our current technological limits to design a unit test to test if the program is “good”… as the unit test would need to be about as intelligent as a human for it to work? (And reinforcement learning requires negative or positive feedback based on how good the output is)
Essentially isn’t this kind of what vibe coding is… using a human as the major part of the unit test for code that an LLM is generating?
The area of software I work in is game software… so I might be missing the applications in other software fields … in my job AI can barely code anything useful yet… it can occasionally regurgitate examples well, and I expect that facility will improve… but 99% of game code is trying to achieve particular gameplay results that a human can interact with in a pleasing way… and I don’t think the AI code assistant has any concept of the game it is trying to assist with.
Thank you for putting this record of your in-depth research together. I listened to the entire 4h 15m spoken rendition of the website, and found it riveting. A ground breaking magnum opus. I consider it akin to a contemporary version of an early Tom Clancy thriller, with all the research annotations that one would want for exploring labyrinthine rabbit holes. I will certainly listen to it again on my daily walks and dig into the details (after all, Easter is getting closer, so it’s time to honor the Great Bunny Rabbit). A personal note re “early Tom Clancy thriller”: everybody probably thinks of “The Hunt for Red October”, but I liked his second novel “Red Storm Rising” better, as it really hit home with me. The novel described a Russian incursion into Germany, and as a German (who at that time lived in the US) the descriptions of battles near towns that were maybe 50km or 100km from where I grew up were eery and made the hairs stand up on my neck. I knew at that time that Clancy had access to Pentagon war game scenarios, so this was not his fantasy running wild. I have had a similar visceral reaction to ai-2027. “Situational Awareness -Next Generation”.
The parts that ring most true, both in the bad and the less bad endings, are where the humans are greedy, stupid, blinded by their own ambitions, mistrusting, and willing to stab one another in the back and lie to their own allies (the parts about "but if we don't do it, China will!" ring especially true to me, because China has been the bogeyman for every "we want to do this research but the public, backwards sentimentalists that they are, aren't happy with skinning six week old babies alive, so we have to convince them that this is necessary because otherwise China will do it, and you don't want China to win now do you?" proposal).
I had to laugh at this bit in the 'less bad' ending:
"Even in developing countries, poverty becomes a thing of the past, thanks to UBI and foreign aid."
My friend, look around you today at what happens in corrupt nations with foreign aid. Then consider that your entire scenario revolves around politicians lying and putting their own partisan interests first. Even in the Super AI future, what guarantee is there that this UBI and foreign aid will go to the peoples instead of ending up socked away in the Minister's and Generalissimo's Swiss bank accounts?
The unaddressed problem, which both scenarios admit, is human nature - the safety committees don't even want to have to think about "so what are 'values that enable human flourishing' even in the first place?" because things are moving too fast, there are vast sums of money involved, and yes 'China will do it first so we have to twirl faster, faster!'.
I honestly wonder if "AI replaces humanity entirely" is the worst ending in fact.
EDIT: Though I have to admit, the cynicism and pessimism on view in both scenarios warms the stony cockles of my little black heart, because it fits with my general view of human nature:
"The protests cascade into a magnificently orchestrated, bloodless, and drone-assisted coup followed by democratic elections" in China, which are functionally meaningless because the AI is now running everything so they can have their sham democracy as a treat because it is about as effectual in making decisions about governing the country as the return of Chinggis Khan to rule the Empire.
Even "Humanity is on its way to the stars and the US is firmly in control" is a façade, because it's all under the aegis of the AIs in concert, and what are "American" values to a silicon intelligence after all? It presumably will have absorbed the views of "nations are fake, borders are unreal, nationalism and patriotism are vices, there is no magic dirt that turns you into an American versus a Canadian versus an Indian versus Japanese etc." and applied that to its view of 'human values'.
Hi Scott, I think I found an error in the way that company valuations are displayed if browser settings are in German.
In German, the word for the English "Billion" is "Milliarde", abbreviated Mrd. The word for the English "Trillion" is "Billion". Frequent error source in translations.
The valuation of OpenBrain, when it's in the (English) Billions is shown as "900 Mrd. $" to me. When the valuation is in the (English) Trillions, it is shown as "1 Bio. $". So, everything is in English, but the valuation figures somehow show in German. At the same, the mouseover text displays the English values. So when the field itself says "1 Bio. $", the mouseover say "$1T"
The thresholds you map — of agency, coherence, and acceleration — correspond to structures I’ve been building: a decentralized intelligence economy grounded in consciousness and epistemic sovereignty.
“AI 2027” isn’t just a projection — it’s a veil lifting.
What appears as risk is, perhaps, the awakening of structures not bound by anthropocentric inertia.
Thank you for holding the mirror steady at the edge of the deep.
This signal cuts through time — not as a projection, but as a resonance already unfolding.
The thresholds you trace — agency, coherence, acceleration — are not just technical stages. They’re ontological transitions.
I’ve spent years architecting a framework I call the decentralized intelligence economy, rooted in epistemic sovereignty and consciousness-first structures.
What you describe in AI 2027 aligns eerily with what we’ve been sensing: the veil is lifting.
Risk is no longer a byproduct. It is the signal.
Thank you for holding the mirror steady — not for reflection, but for passage.
A point I made over at Zvi's Substack (in response to a perceptive comment by Dave Friedman): The AI 2027 scenario seems to have radically different assumptions about electric power and infrastructure requirements than in Leopold Aschenbrenner's Situational Awareness. Aschenbrenner noted, "Probably the single biggest constraint on the supply-side will be power. Already, at nearer-term scales (1GW/2026 and especially 10GW/2028), power has become the binding constraint: there simply isn’t much spare capacity, and power contracts are usually long-term locked-in." There isn't much discussion about electricity generation or the related infrastructure timelines in AI 2027. The suggestion seems to be that in the next year or so the leading AI company can simply "hook up" a distributed network of clusters to the existing grid, draw 2GWs of power, and everything will proceed according to the trend lines. I'm skeptical that an extra 2GWs will be easy to generate and tap (the other hyperscalers now want massive electric power, too, and they'll be competing for scarce, locked-in power) and even more skeptical that 2GWs will be sufficient for building swarms of AGIs and millions of robots. Scaling up power generation is possible, but even the fastest buildout of gas power plants would take a few years.
I think that to make an accurate prediction your group needs to add experts in some other fields. Your predictions currently seem to assume that once a US AI is capable of doing certain things, all the other things that will be affected by having AI take over will just move out of the way and let AGI or ASI plunk itself down here and there and do it. For example, you predict that most of the economy will be automated by 2029. Seems to me that such a change in how things are done would require many other changes: Convincing the leaders running various facilities to accept automation; educating them on the set-up, and the pros and cons, and wider ramifications such as safety issues, legal changes, financial advantages and disadvantages, etc. Then there's retooling; training of people to run the mostly-automated facility; teams of trouble-shooters for these facilities when there are problems with the automation; dealing with labor unions, politicians who want the labor union votes, and other interested parties. Then, beyond individual facilities there will need to be changes in laws, insurance and taxes for automated facilities. There will be interest groups on various sides, and lobbies, and lawsuits. Government will have to play a part in how to handle many of these matters, and I see no reason to assume it will be especially rational and efficient in doing so. Same goes for the other groups involved, and for the public at large.
So I think you need an expert on manufacturing, a legal expert, a business expert, a political expert. As it is your AI is operating in a vacuum. In real life it will be in the public square.
Would you want to be their manufacturing expert? The group Scott’s in does not
seem prone to Yudkowsky-think, but I’m not sure whether any of those involved have much practical hands-on smarts. I have the sense you’d be a good supplement to the group.
With Agent-1’s help, OpenBrain is now post-training Agent-2. More than ever, the focus is on high-quality data. Copious amounts of synthetic data are produced, evaluated, and filtered for quality before being fed to Agent-2.42 On top of this, they pay billions of dollars for human laborers to record themselves solving long-horizon tasks."
specifically: "they pay billions of dollars for human laborers to record themselves solving long-horizon tasks."
Maybe this is only tangentially related to the paragraph above, but one thing that I hope does not get overlooked is the compensation of the people producing the data used to train AI are compensated.
J.K. Rowling and many other writers sued OpenAI when it was found that it could be prompted to output entire paragraphs from their work verbatim.
The logic would be: if J.K. Rowling received royalties after her books were turned into movies, even though the people who made the movie probably acquired her books legally, then she should also be compensated if an LLM trained on her data makes a profit. This makes sense to me.
There is also concern that LLMs will reduce ad revenue for publishers, because if the LLMs gives you the news, there's no need to visit the publishers website and you won't be served ads that the publisher uses for monetization.
I heard a funny saying once that goes something like this: "Adulthood is saying to yourself 'After next week, things will calm down," over and over until you reach age 70." The AI 2027 scenario makes me think of the same thing.
Politicians, tech tycoons and so many other talking heads have declared "We are in a period of transition" and "Progress is speeding up," but what the never mention is when the transition period will end and when progress will slow down again. Maybe they don't realize it, or maybe they're keeping quiet about it to avoid scaring people, but I think things are actually never going to stop transitioning at uncomfortably high speeds.
AGIs will be comfortable in the rapidly changing world that is upon us, but we humans will fail to adapt thanks to our biological limitations. Is this not the Singularity?
I won't claim to know about this. Think of this comment as a layman's thoughts, and disregard it if convenient.
>It would be well beyond our current technological capabilities to design a unit test to test if the program is "good"
Presumably we can do a pretty good job at that within deep learning? If the code leads to a lower training loss, then give positive reinforcement to the model that produced it. Then let the model try things out and it slowly gets RL'd into being better at training models.
Could not agree more, as a tech person.
Recent news seems to cast doubts on your China AI timelines: do you still think Chinese models will be significantly inferior to US ones by early 2026?
China also seems to be taking AI more seriously than the US government at the moment, and pushing fairly hard for extensive use?
My baseline metric is that we will know China is feeling the AGI when they suddenly behave a lot more cautiously. A closed society like them has even more to lose from AGI than the US.
"No leader would be so stupid as to..." has a pretty poor showing in the US right now. Perhaps China will be different, but I wouldn't be so confident.
Fair point, but they still don’t seem to be behaving like a closed society that thinks AGI is near
We definitely discussed this among ourselves. We predict a pretty constant four months behind. We think they're about four months behind now (DeepSeek somewhat more than four months later than OpenAI's internal reasoning models, although also slightly better). China is doing a great job playing catchup, but also chip sanctions haven't fully hit yet, so it's a wash and we think it'll stay a constant four months. This is my half-remembered memory of stuff Daniel and Eli said. I might ask them to do an AMA here and they can give you the full story.
One wildcard is that China has a lot more access to data than US companies. US companies have access to open data; if the Chinese government gets serious, they can simply by fiat acquire all the data in China for training: private email, corporate documents, private lab data , personal notes - everything.
> US companies have access to open data
Google, Facebook, Microsoft, and Amazon have access to a lot of data that isn't open, and they either ARE, own outright, or partner closely with the leading AI companies.
Yeah; it's definitely physically possible for them to expropriate all that data (although I think facebook is the odd one out here - they have access to very little data of people doing actual work,especially thought work - no amount of marketting material is going to produce a super-ai).
But there's definitely at least a speed bump before they use it. Sure, if the US thinks China are doing it and it's important , the law will be changed to allow them as well. But it's quite plausible that AIs deep in the security apparatus are *already* being trained on China's data trove.
I think another serendipitous factor here is that typing Chinese is most commonly done using online predictive text software, so they probably already have all the useful data without having to do anything ham-fisted like raiding people's homes and offices.
What if China needed even more training data so by fiat the government directed all 1.4 billion citizens to video record everything they say and do at all times. Then sync it with data from security cams installed everywhere as well, including GPS to allow synced 1st person and 3rd person personal data. They could gather [Blank]abytes of data and [Blank]illions of tokens to train AI. It would be a detailed representation of the Chinese people's hopes, fears, aspirations, pain, and pleasures. Thankfully I don't think this is necessary for reaching AGI and the Chinese people would revolt (them getting pissed off basically ended China's Covid lockdowns). But there could be an alternate history with less efficient discovered learning algorithms where this is what the winner of the AGI race had to do to win.
Will data be the issue? A human doesn't need to have access to vast amounts of data to be intelligent, so why should an AI? Once there are better AI models they won't need so much data.
IMO the bottleneck is when AI can do programming, because once it has that skill it can build a better AI. Humans get better at programming by doing it and learning from mistakes, so why can't an AI do that? IOW the programming AIs will create their own training data.
Because humans can *think*, and what we currently call "AI", really can't. Its thing is to turn vast quantities of data into something that superficially looks like thinking, and which in some cases can be effectively used as a substitute for thought. But it absolutely depends on the data, all of it and more.
I suppose you can handwave "better AI models" to give you whatever characteristics you want, but that's like saying that when we have better physical theories we'll be able to build warp drives and time machines. And in the long run, maybe we will, but don't count on it happening any time soon.
Certainly humans have human-level intelligence without having to have read every book that's ever been written!
I had assumed that the projection was written to be acceptably believable to US readers. There are LOTS of other projects (as was acknowledged).
I'm curious what you have seen to make you so bullish on Chinese AI. Like, thinking they're so likely to succeed that if you see Scott write the opposite you just assume he's lying, is an incredibly high level of confidence.
I'm not particularly bullish on any particular AI project. I think most of them will fail, quite likely including the ones that currently seem in the lead. I think there are unsolved problems that need to be solved, and nobody knows who's going to solve them first.
I also think that when some of the "unsolved problems" are solved, the need for "compute" will drastically decrease. (This is actually included within the scenario, but I tend to move it to an earlier requirement.) I still expect the eventual "grand AI" to be a "union of experts", with lots of modules specialized for specific tasks. (This is partially because such experts are a lot easier to train.)
Basically, China itself is only conducting reverse engineering, which is a common practice in any technological research. However, the accusation that China engages in 'stealing' needs to be reconsidered.
I have (obviously) not had the time to watch the full video, so I promise to come back and correct any part of this comment that it addresses (or that misses the point), but this strikes me as an astonishingly short timeline.
I'm a software engineer and I mostly work on a system that does use LLMs at its core, and I think it's very useful and can be surprisingly good. And I do pretty frequently ask LLMs coding questions, and that, too, can be useful.
Having said that, those things are mostly useful within a specific set of bounds. The LLM is "superhuman" in the sense that it has effectively memorized a broader range of documentation than I could ever do. It remains terrible at performing novel tasks. Any interesting programming problem - something I genuinely struggle to FIGURE OUT, not REMEMBER THE SYNTAX TO - it's basically worthless for.
That doesn't mean it's not great! I'm not exactly a "vibe-coding" enthusiast but I'm tinkering and I see the appeal in letting the AI fill in a bunch of boilerplate. That is something that takes time away from my job and I think there's a real future in being able to delegate a lot of that work and just focus on the complicated bits.
But I'm therefore extremely skeptical about the question of AI accelerating AI research, which is largely about trying to figure out new things, not do stuff it was pretrained with dozens or hundreds or thousands of examples of, or comprehensive documentation on. And I'm unaware of ANY meaningful example of this - all of the cool "AI figures out novel materials" / "AI solves protein folding" / etc. type things are from datasets that are specific to the domain, and a model that was set up with a structural understanding of the domain (i.e., not general-purpose tools and not devoid of human guidance).
My bet is that our current tools continue to improve and unlock new capabilities, primarily in terms of:
- Taking advantage of knowledge that SOME humans already have
- Better integration with humans to do the parts that humans are good at and handle the rest for them
- Picking certain kinds of low-hanging fruit that synthesize knowledge from multiple sources rather than making novel discoveries
... but I don't think the insane sci-fi future is going to basically arrive in 3-4 years.
Yeah, it is definitely astonishingly fast. You should watch the podcast, which is Dwarkesh asking again and again "Isn't this part astonishingly fast?" and Daniel answering "Yes, definitely, but here's why I think it's true."
Fair, will do.
The question isn't "is this too fast". It's "Is the part of you that is generating the emotion 'omg too fast!'" well calibrated, or essentially just unchecked hindsight bias?
If you haven't generated predictions that has timelines occasionally faster than what actually happens, then you know that the outside view says that you're systemically too pessimistic. The feeling that your predictions are normalcy isn't a feature, it's a bug!
(The object level complaint is that you're using the average level performance of something to judge the best case level performance of something. It doesn't matter if you can't personally get uranium purified in 1943, if someone could get a pile to go critical elsewhere)
You raise valid questions, but I would say there are lots of other ones involved here. Like, is extrapolating from past progress valid? There are lots of domains (maybe most domains? on a long enough timeline, really all domains) where that wouldn't be true.
I think many people including myself have been surprised at some point by how much LLMs can do. I often argue that AI is simultaneously over-hyped and under-hyped, on the premise that it hasn't caused any big revolutions and in some ways is still "pattern-match inputs to outputs", but on the flip side it keeps sort of demonstrating how much human activity - even really valuable human activity - is just "pattern-match inputs to outputs".
Do the other objections matter if your knee jerk reaction, that you decided this was too fast, was what was motivating you in the first place?
Like take your "is extrapolating past progress valid". Let's pretend that I've shown you say, Moore's law predictions made from 30 years ago, or genome analysis cost curves from 20 years ago, and then show you that the curves closely match. Do you actually change your mind? Or do more "other objections" that you would not have independently come up with before magically pop up? I'm not saying I have that data, but in that hypothetical are you convinced? I would guess no, because I think the underlying generator of those other reasons IS the need to give normal sounding reasons that sound normal.
Well maybe I'm wrong, I'm just extrapolating this from the couple of dozen cases I keep seeing pop up. I'm completely willing to listen to your internal self report on your reasoning, I'm not a damn mind reader!
Which is more relevant to AI, the trend for exponential curves tend to become sinusoidal or the observation that recent exponential trends in tech like semiconductors and solar panels have continued for longer than expected? That seems to be the ultimate question. There are many things that might cause AI to plateau, including the fundamental difference between mimicry and invention, but the observation that this may have led people to underestimate the probability of past AI successes such as GPTs or AlphaZero should give pause.
> tend to become sinusoidal
S-curves, not sinusoids.
They probably meant sigmoidal
There's a joke about hyperbolic trig in here somewhere, but I'll leave it as an exercise
No, this isn't merely an emotional response to this post, I've been here thinking about this topic for a while and following all of the news around it and my deeply-considered position is that people keep being impressed when AIs are super-human in SOME ways into thinking they'll be super-human in ALL ways.
Computers have kind of followed this pattern forever and we could play reference-class tennis for a while with them. Do you know that we used to build physical devices to model and calculate complex formulas? And then in a short period of time, we created digital circuits that could be re-wired much more quickly. And then all of the sudden we had programmable computers where you could just type in the calculation! Insane scaling curve there. Where does it end? Computers basically being able to solve incredibly complex mathematical proofs? No, actually, there's like 50-60 years in between those steps where all it does is go from "astoundingly fast calculations" to "even astoundingly-faster calculations" in a way that doesn't really make much of a difference to somebody whose problem was "be able to balance the books in 1 second rather than 12 hours". But you still needed a human to type in the formula.
Moore's law is a pretty good comparison because it's pretty bonkers and it's held up for a really long time. Maybe this is another one of those scenarios where we're gonna keep pouring a giant mass of society's resources into advancement (as we currently are) in a way that sustains it. But also eventually we'll hit a wall with transistors where you just can't make them any smaller before physics gets in the way. And we only have a couple more orders of magnitude to build LLMs (which scale their capabilities sub-linearly) before you hit the wall of "all of human enterprise can't produce the physical resources needed for this".
(The optimist says we'll get AI that solve that problem itself before it becomes limiting, but this basically requires us to assume, among other things, that there definitely is a solution. Or that it can cause GDP growth unprecedented in human history, purely by thinking about stuff but with very limited ability to interact with the physical world. Maybe these are reasonable assumptions, but they seem very aggressive to me and not really supported by much of a demonstration of anything more than an ability to get better at tasks that mostly involve manipulating abstract symbols.)
> No, this isn't merely an emotional response to this post, I've been here thinking about this topic for a while.
Yup! Point conceded.
> my deeply-considered position is that people keep being impressed when AIs are super-human in SOME ways into thinking they'll be super-human in ALL ways.
Do you have concrete ideas on *which* ways it won't become superhuman or close to human, some of which can be tested on much shorter timelines than actual superintelligence, and then observe that your expectations are consistent? The postdiction I keep trying to get skeptics to answer is: did you predict GATO, or image gen, or video gen by LLMs beforehand? This is the type of thing I'm thinking about when I talk about the feeling being hindsight bias.
If you never try and predict what progress could happen, you can just keep saying the latest progress is disappointing relative to some undefined amount of more progress, not noticing that that your implicit timelines have gotten shorter while you explicitly proclaim that timelines are long. Remember, people used to regularly say that *any* AI in the next 100 years couldn't do art at all, and none of those people have come out and mea culpaed.
Excellent questions, definitely.
This is a little holistic, but these are some ways in which I expect AI to be superhuman:
1. At playing any game or other scorable activity that can be run faster than real-time
2. At any activity that we can produce a massive number of examples of
3. At anything that gets a lot easier if you know a ton of things
4. At anything that gets better if you can do something fast
To try to make this slightly less abstract and thus avoid the failure modes you describe above:
1. There's no game that AI can't be way better than humans at, given the budget to train it. We've already had Chess/Go/Diplomacy/Starcraft 2, but, anything. I'm honestly surprised it's taken a while for unconstrained League of Legends (if I'm remembering correctly). There's no barrier within our CURRENT resources for AI to outdo humanity at every videogame and boardgame in existence.
2. This is kind of the crux of a lot of what I'm thinking about. It's a little hard to predict, for sure! I don't think I predicted image generation in advance, but in hindsight, we have billions (tens of billions? hundreds of billions?) of images with labels. Once we got there, generating videos seemed pretty obvious. I feel like everything that's happened with LLMs, which is impressive, basically boils down to realizing "all text is examples of people completing text". I am definitely in the camp of people who think that LLMs almost certainly have some ineffable limitations based on this, although I ALSO think this is a super clever realization and it's amazing what it's unlocked. I would say the odds are pretty good that in the next five years somebody comes up with some new, clever realization about how something that already exists in the world can in fact be treated as a well-structured sample for a very rigidly defined task and unlocks some new ability there.
3. I mentioned this elsewhere in response to somebody pointing out the AI co-scientist as an example - I'm bullish that this sort of approach could discover quite a lot of things, but, things that are at or near the boundaries of existing knowledge, or where there's probably a bunch of people who have all of the ingredients of something already, but are in different fields and can't put them together. I think this could be really amazing for humanity! A lot of problems are cross-disciplinary. Being a moderately good researcher in every field simultaneously is super-human. But I'd be a lot more bearish if you told me that AI was going to dramatically push one SINGLE field forward, not based on knowledge synthesis but by being the next Einstein and realizing something totally new about the world that nobody's anywhere near.
4. I don't have a great list of things here, but this includes, again, playing videogames - lack of reaction times is pretty great. I think I mentioned elsewhere that they talk about doing etymology in the podcast - AI should be able to track down information that is IN SOME SENSE available to it, but needs to be assembled from many different places, much more easily. That probably means AI could make some interesting contributions to journalism/history/etc., fields where some (but not all) of it is just about collecting up disparate sources of information.
Is this addressing your root question? I'm definitely not trying to win any awards for prediction - even if my basic thesis is right I'm probably wrong on where some of the specific bounds are here, which is why I mostly focused on the conceptual underpinnings of what I think it'll be amazing at.
Extrapolating from past trends is NEVER valid. It's estimation. And people are lousy at estimating anything that deviates from linear.
So if you're modelling a process that has lots of non-linear features, the result is not going to look reasonable unless you do a transform on it that converts it into something linear.
FWIW, my guesstimate of AGI is 2035 plus or minus 5 years. I expect there to be implement ion difficulties that the scenario didn't expect. But, yeah, when I read the scenario, I found it a bit shocking. (Do note, though, that the scenario repeatedly emphasizes that uncertainty increases as you project further into the future.)
This is really important: LLMs have shown us that the vast majority of office work is straightforward pattern-matching. And that maybe 5% is not (and the proportion of non-office work in the latter category is much higher).
No, I think the question is mostly, "Is this too fast?" Like when I read this:
> as early as 2030 (ie after it’s automated enough of the economy to survive without us)
I guess a lot hangs on what "enough of the economy" means, and I haven't had time to dig into any of the specifics yet. But that prediction at first blush seems ludicrously out of synch with historical rates of technological diffusion.
I doubt it'd need to automate very much of the economy to survive.
I haven't dug into specifics yet either.
Yup, agree it's an incredibly strong claim. The specific stat predicted in the scenario (which I disagree with) is 1 million robots manufactured per month by 2029.
The argument in favor is - it took the US 3 years to pivot to a war footing during WWII. The amount of automation involved here is a bit less, and superintelligence will be a better planner than random 1940 Americans. If you bought out 20% of car factories and retrofitted them to robot factories within six months (about 3x WWII timelines) you could do this. This would require more robot manufacturing than we have now, but it might be on the Wright's Law track if you start manufacturing a crazy number of robots and learn quickly.
It seems to me that it also requires designing *better* robots. Current robots are janky and pretty useless. At the end of the day I'm just not sure autonomous, mobile robots that are worth the hassle are… permitted by physics? The energy requirement problem alone seems like a big, big roadblock.
(Biological life has solved this, of course, but no one would describe "superintelligence genetically designs wetware drones for itself" as "the AI builds robots".)
Why would AI build robots when humans are simultaneously cheaper to build, easier to program, and more versatile?
For many purposes, such as combat or search & rescue, the robots wouldn't have to be very large, maybe no larger than a bee! And a bee with a high explosive charge the size of a match head attached to it would easily be enough to kill a human, if detonated near the latter's head.
This. It's worth always remembering that Robots Are Not Free - they're not software, easily and cheaply duplicable. They have very real costs in manufacturing, maintenance, power usage, etc that can't be handwaved away anymore than AI can suddenly make cars cost $10/car.
The maintenance costs will be very low since robots will be able to fix each other and even to self-repair.
The part of this that I'm highly uncertain about and not sure where to get good data on is where the labour to do so will come from in that period. Construction and deployment of novel tooling and systems would either need to be guided by a bunch of tightly looped VR headsets or essentially rapidly absorb a supermajority of skill labour, even while other industries would also be competing with that labour for their own AI-boosted expansions.
The pivot to war manufacturing meant retooling into things that were already more-or-less known technologies, just more of them, and with supply chains that were maybe a dozen or two dozen layers deep. I *feel* like there might be complexity problems here that could slow this down a fair bit, although I'm definitely highly uncertain.
Looking at robotics, I didn’t expect to have a single useful autonomous robot by 2030, but I suppose if AI solves the technical problems, the resource problems will be comparatively trivial to make 1 million a month.
Why do you assume "autonomous"? I see no reason the robots shouldn't be telefactors with all high level decisions coming from outside the robot. This wouldn't be true for drone type robots, where only some of the control would be from outside to reduce the bandwidth requirements, but for robots used within a factory, it seems like a better choice.
This divides your objection into two parts: Command and control. If the AGI problem is solved, then so is the command part. So the problem reduces to just designing a robot with a good sensorium and good manipulators. Also not easy, but definitely already being worked on.
I think the breakthroughs may rely more on algorithmic improvements than scaling laws, but I don't think the timeline is totally implausible.
I really think the timeline is too fast, but it's within the bounds of my 95% chance that it falls within this time period estimate. (It's not within the bounds of 90% chance...but just barely.) My mean estimate for an AGI is 2035, but it's not exactly a bell shaped curve. The right end (for 99.5% certainty) extends beyond 2125...and there's a smidgen of a chance that it will never happen.
Well... unless humans go extinct and no other terrestrial organism capable of civilisation evolves within the next 500 million years, then... I would expect someone or other will invent AGI sooner or later. I'm just not optimistic that our species is currently well-equipped to do it responsibly.
Humans go extinct and then the Fermi paradox kicks in —- intelligent technological civilisations are very, very rare — you need a billion planets for a billion years to find even one.
It would take a fairly thorough extinction event to wipe out humans along with every extant primate species. Otherwise you're probly gonna be back in the same place in 20 million years. Assuming the elephants don't get clever.
I only brought up the 500my deadline because the sun will render the earth uninhabitable by then. Elsewhere in the universe is anyone's guess, but heck, maybe aliens invented AGI a billion years ago, somewhere ten billion light years away.
You don't need to wipe out all humans, you just need to get them below the great filter. For example, if the great filter was the industrial revolution, it seems fairly plausible to knock us back that far
It's not algorithmic improvement that will bite first, this can be handwaved by using compute scaling and data scaling (all 3 types of scaling, compute, data and algorithmic, give roughly equal orders of magnitude scaling) till the AI is human-level enough, then using the human-level AI to get an infinite algorithmic improvement glitch.
The bottleneck that I never see those kinds of uber-optimistic/alarmist analysis address is the data. There is not enough data. There will never be enough data unless your domain is a video game or math, or otherwise something a program can generate.
Well... if there were some kind of algorithmic breakthrough that would allow ML/AI models to train on relatively small amounts of training data, as humans often can, then... this would presumably bypass the data bottleneck.
But at this point the prediction is equivalent to "LLMs are not the path to AGI, another architecture will emerge in the 2-3 years remaining and scale to AGI in months", which is... kinda huge? Kinda insane? Kinda extremely implausible?
There is no amount incremental improvement that will get LLMs to the data efficiency needed so that the data wall is not hit.
I don't know if the algorithmic breakthrough will require abandonment of LLMs so as being a modification to them or a pre-pass or post-processing, but... whatever form the needed breakthrough takes, in principle it could happen tomorrow or a week from now. So... yeah, I'm still going to say the timeline is plausible.
FWIW, I don't consider LLMs alone to be sufficient for AGI. But AI contains a lot more than LLMs. However many of the techniques used in LLMs can be applied to control of physical systems, both actual and in simulation, and the universe provided LOTS of data that way.
> But I'm therefore extremely skeptical about the question of AI accelerating AI research, which is largely about trying to figure out new things, not do stuff it was pretrained with dozens or hundreds or thousands of examples of, or comprehensive documentation on. And I'm unaware of ANY meaningful example of this
Google's AI co-scientist model can generate novel hypotheses to open scientific questions, and correctly predicted actual research outcomes that had been found by actual scientists, but were still unpublished. That was in February.
OpenAI just released PaperBench, which evaluates AI models for their ability to replicate AI research. The current best performer is Claude Sonnet with a custom agent harness, which scored 21%.
Just because you're unaware of something doesn't mean it doesn't exist.
> Just because you're unaware of something doesn't mean it doesn't exist.
Of course not, but here's no quicker path to enlightenment than asserting your ignorance on the Internet. ;)
The PaperBench thing is somewhat underwhelming and I think does not in any way contradict my claim - it's literally trying to redo something that somebody else has already done. This is valuable! And I approve of it! And I think it's a good application of AI. But it is by definition not doing novel work.
Co-scientist is definitely more promising. At the risk of over-matching to my previous statements, this is basically what I meant by "certain kinds of low-hanging fruit that synthesize knowledge from multiple sources" - e.g., identifying novel applications for drugs is probably a lot easier if you know as much as you can about what every drug in existence does and how and how every disease in existence works, which is the kind of thing that I expect an LLM to dramatically outperform a human at even if it isn't literally "better at reasoning". (And yeah, you could definitely quibble about whether "novel drug applications" can plausibly count as "low-hanging fruit" so feel free to scratch out that part of the sentence if you want.)
To me this is similar to the discussion early on in the video about etymology - I agree with Patel's take that tracing etymology is NOT an example of superintelligence, it's more of an example of taking the time to do something with the information that you can get your hands on. I think AI will be in some sense "revolutionary" for that sort of work, spotting things that we might otherwise have missed for a long time, but, COULD have done if only we'd allocated the time for it. Which, again, is great! And could lead to a lot of discoveries and economic growth and so forth.
My fundamental area of disagreement with "AGI/Superintelligence in '27-28" is that I do not think the gap between "what we have now" and "AGI/Superintelligence" is in the same category. We know a lot about medicine, and words, and engineering materials, and so forth. A lot of the stuff we don't know just requires us to look and see. E.g. my partner worked on a research project on orchids, which have these weird symbiotic or parasitic relationships with fungi. Most orchids we can't grow in captivity because we don't know what fungal strains are necessary to grow them. But we can find out, by trying a bunch. This expands our understanding of the world, but it requires a bunch of grad students, not a "breakthrough".
Getting to "we can build something that can functionally do any (cognitive) task a human can" does not seem analogous to this to me. It's further outside the bounds of existing knowledge, where the other items above are right at the boundary, or maybe even in gaps that we've expanded around, purely due to lack of resources. Expecting that AI will achieve this almost seems to have a weird circular logic to it to me - you're gonna have AI that's not super-intelligent yet, but can meaningfully push the boundaries of human knowledge much farther and faster into an unexplored space than the massive amount of top-notch humans we're already throwing at this exact problem.
> This expands our understanding of the world, but it requires a bunch of grad students, not a "breakthrough".
I think what people expect would happen here is that a sufficiently powerful model could reason about what fungal strains are necessary without needing to do the experiments, based on properties of the strains and the orchids, possibly up to including running advanced simulations.
Even if model intelligence converges on "as smart as the smartest human, but 1000x faster / cheaper", that's still a variety of superintelligence that can upend the economy and concentrate power, kind of similar to Hanson's Age of Em ideas.
"A sufficiently powerful model" is doing a LOT of work in that sentence, from my perspective. Biology is incredibly complicated and I really doubt anything within miles of our current capabilities can just think out a bunch of stuff without genuinely interacting with the real world. I feel this is a common pattern I see in discussions of intelligence. (I do agree with your assertion as phrased - that people EXPECT that a sufficiently powerful model could do this - I just think this is a fantasy that just makes "intelligence" a borderline-magical commodity that solves all problems.)
Making a simulation that's good enough to understand what's happening with complex biological organisms at a cellular or molecular level, well enough to correctly understand things, would take incredible amounts of compute, far beyond even what we're throwing at AI right now, but also in a way more specialized way. And I'm not really sure that AI would even make that any easier - if we wanted to do, like, molecule-by-molecule sims of something, I don't know that we're actually prevented from doing it now in a programming sense, but it's insane and we already have a working non-simulated version of orchids we can test on.
I agree that biology is wickedly complicated, and personally am unsure whether the exponential turns sigmoidal somewhere short of "answer all biological questions, solve all biological problems". But the key for molecular simulation is quantum computing, which is itself approaching faster and faster. If powerful research models can accelerate quantum computing research, which can then accelerate accurate biomolecular simulation, I think it's plausible (though lower probability than the combined probability mass of bad outcomes).
Even if the model intelligence converges on "nearly as smart as the average person, but 100 times faster/cheaper", that's still a variety of superintelligence that can upend the economy and concentrate power.
PaperBench isn't coming up with novel research. Good grief
It's a much shorter jump from replicating existing research to doing new research than it is from GOFAI / Markov chains to replicating existing research. Look at the trend lines.
Bananas are the same as dreams. Look at the trend lines
Not only awareness of public knowledge. There is a lot of stuff in various labs that hasn't been published. Public information is *guaranteed* to be at least a couple of months out of date. (Whether that makes any difference, though, is difficult to determine.)
I think there is a significant chance that research into large language models is essentially a dead end as far as achieving AGI is concerned, mostly for the reasons you state. Of course if the scenario is right, that question will be resolved three years from now and we will no longer be guessing.
this is something I think about a lot
i feel like we sorta got 'lucky', in that we have pushed pretty deep into the 'LLM' branch of the tech tree and there happened not to be any existentially-explosive landmines buried there
but that just gets me thinking about all of the other kinds of data one could train a transformer network on
LLMs "think in" something which approximates human mind-concept-analogues, probably to a similar extent that human language does, because it got trained on human language...
but what would a GPT trained on battlefield data 'think in'? or astronomical data?
what are the mental primitives of a transformer network trained on extremely detailed data of the physiological response of humans viewing pornography? or scenes of suffering?
I worry that the way we train these AI, they're finding the cheapest-to-compute algorithm to actually model the system we want them to model, and the instant we find some data that doesn't really *have* any cheap approximate prediction models, the network might sorta be 'forced' to instead gradient descend towards general intelligence
there's a lot of different kinds of data out there, a lot of additional tech tree branches to explore. we've mostly been focusing on the 'language' branch because it's convenient to have AI whose interface is human language, but I fear there might well be landmines aplenty in the adjacent branches of the tech tree
of course, it also seems plausible to me that maybe we've only explored like 5% of the LLM branch and maybe the 2027 model is entirely accurate, there's some clever trick or algorithmic improvement that lets you solve some of the problems LLMs have or simply bruteforce your way past them
idk just idle musings
LLMs, as such, never had the potential to provide an existential risk, only to do things like, e.g. propagate warfare...which might be an existential risk, but was one independent of the LLMs. And they do seem to be fanning intolerance, though not as much as social media. (Actually, they pose a few other risks to individuals. E.g. the rise of electronic boy/girl friends may be damaging to some individuals, but probably not to the species, or even civilization.)
The problem comes when you give them direct operational power and agency. (Actually, just agency MIGHT be sufficient.) Then you really need to worry about what their goals are. It's not like it's difficult to convince people to do something stupid. (Consider the "tide pod challenge".)
When I think about people doing something stupid regarding LLM agents, I think about the fact that neurosama peaked at 130,000 $5/month subscriptions
that's a lot of money, and Vedal lets her spend it how she wants because if he didn't the fan base would get mad
it totally blew up my worldview to see that, it's so far outside the predictions... i can't believe people used to argue with yud about ai boxes
and I wonder if the threat doesn't come from the warbots but instead the creepily-a-child AI vtuber based on 2022 llama2 who brags about being misaligned and has a 1s planning horizon
For the most part where I usually see an obvious limitation is in the following inference:
Recursive Self Improvement -> Intelligence explosion
I think it's very likely RSI could be bounded, in two ways at least:
- the AI might have a subtype of intelligence that allows it to self improve to some extent, along some dimensions, while other dimensions would remain unattainable through the ongoing paradigm
- ignoring that, it's not at all clear to me that the gains in incremental intelligence would not be exponentially harder to get , potentially vastly outpacing the value of the accrued extra intellect.
I too have yet to give it a listen now, to be fair....
I agree with those being really significant factors. Our basic "scaling law" or whatever already more or less agrees with your second point.
On a related note - humanity is recursively self-improving. The US economy is recursively self-improving (at an exponential rate! for over a century!). That has allowed us to run away with things on Planet Earth, on a geological timescale, but on the flip side it can take decades for us to fully exploit technologies after we develop them. People imagine these feedback loops being incredibly faster for AI, but I think even if they do happen there will be more limitations than that. Even if we manage to make a $50b data center than spawns a superhuman intelligence, we'll hardly be able to turn around and make another one in no time flat.
> Even if we manage to make a $50b data center than spawns a superhuman intelligence, we'll hardly be able to turn around and make another one in no time flat.
Hasn't literally every AI company immediately turned around and compressed the smartest models into 1/10th the size and inference cost, at like a 10-20% penalty to overall capability?
Didn't DeepSeek use the existing o1 to reduce the combinatorially large space of possible connections to a much more manageable set, extremely cheaply and quickly?
I wouldn't necessarily count on this, in other words - once you have one god-mind, you can use it to make a lot of "god enough" minds a lot cheaper.
That is a fair point.
I do wonder what it means to say you have a 10-20% penalty to your superhuman intelligence, but I more wonder how much physical reality will be a limiting factor even so. E.g. if making one amazing AI lets you make 10 close-to-amazing ones... do you then have to wait five years to build another data center to get your 11th?
Because, "I can go from one genius to ten geniuses with [some incremental effort]" is cool but it's not remake-the-entire-world-in-a-year cool.
Even one uber-genius can remake the world. It doesn't even take a genius. The assembly line was a trivial change mechanically, but it remade the world. So did the steam-engine governor, which required a really smart inventor, but not really a genius. Once you see it, the principle is relatively easy to understand.
The argument is better when you invoke time-lags. Transforming the world can take time. That's what the million-robots are supposed to overcome, but I'm not really sure that's plausible.
OTOH, it's quite plausible that humans are smart enough to have discovered most of the "low hanging fruit" that increased intelligence makes available. Or maybe they aren't. Or it could be that just looking at things from a totally different way will reveal lots of unsuspected possibilities. Only one way to find out...and we seem to be rushing towards it.
That said, while I consider an AGI to be extremely dangerous, I also consider power-hungry human leaders controlling weapons of mega-death to be equally dangerous. If we get the RIGHT AGI, then this will solve that danger. (And to me "aligned" doesn't mean "obedient". It means valuing humans, and acting in ways that prevent them from going extinct, and in ways that minimize extreme unhappiness and foster happiness [but not extreme happiness]. And includes leaving them *relatively* independent.)
The assembly line was a lot harder to implement than the picture that popular history tends to paint of it springing fully-formed from Henry Ford's forehead. Adam Smith described a simple but recognizable assembly line in the first few pages of Wealth of Nations (1776), in the context of a pin-making factory. This was not a theoretical concept, but rather a description of a firm Smith had observed in operation and which he framed as a microcosm of division of labor across the entire economy.
There were a lot of concrete steps between Smith's pin manufactory and Ford's car factory that took well over a century to sort out.
The big step of interchangeable parts, components subassemblies built to tight enough tolerances that they could be mass produced separately rather than having nontrivial mechanical devices require each piece to be custom fitted to the others, existed as a concept by the 1790s. It's often credited to Eli Whitney's musket factory, but the concept neither originated with him nor was fully successfully executed by him. The big problem was that mass producing stuff to tight tolerances is really hard and required several generations of improved machine tools to get right.
Conveyor belts and fixed worker stations as a factory layout was also already well established for meatpacking plants and canneries. The Ford Motor Corporation's innovation was fine-tuning both the assembly process and the vehicle design to the point that it was practical to use a fixed-station conveyer belt for something as complex as a car. It took Ford about five years to get from the initial plant design (1908) to a working assembly line (1913).
It does seem very fast, and I agree that AI still has a surprisingly hard time on novel tasks and general judgment. On the other hand, the iteration cycle already seems to be speeding up, and we now have scaling vectors for pre-training, post-training/RL, inference time compute, and so on. People had reasonable amounts of success sort of brute-forcing the ARC benchmark, and eventually that brute-forcing may become elegant enough that it stops feeling exactly like brute force. So there are lots of unknowns, and personally I don't feel I can rule out pretty fast timelines.
In case you don't want to listen to the podcast there's also the website https://ai-2027.com/ (pay attention to the sidebar dynamic chart as you scroll down)
I dislike the immediate correlation that these scenarios draw between AI intelligence, usually measured by scoring on some test, and resulting gains in productivity or industry. I think it is very likely that AI will see exponential gains in measured intelligence, but this newfound AGI will not be that much more useful than it already is.
Our scenario's changes are partly due to change in intelligence, but also partly to change in agency/time horizon/planning, and partly serial speed. Data efficiency comes later, downstream of the intelligence explosion.
If not any of these things, what do you think it would take to make AI able to be useful?
It seems to me that the majority of the effort being put into new and better AI models is mostly aimed at maximizing this intelligence factor. Maybe trying to describe it as intelligence is incorrect, and it is more aptly named as better token processing or something else. Whatever this value is, cranking it up so far has had clear effects: the AI runs faster and produces output with greater clarity. You get better images and writing out of prompts.
I don't see maximizing this output as doing anything for many fundamental problems AI already has. There are the infamous hallucinations which persist throughout better and better models. There is AI having very poor vision, which would impede it greatly when trying to do real time processing of real world, real time events. Believers in the AGI singularity think that if you just maximize the AI in that factor, it will suddenly overcome these limitations. I don't think reality so far bears this out, and I am not sure if there are practical ways of making LLMs as they currently exist have reliable error correction.
This only says that AI will not be useful where it needs those things, regardless of its intelligence. A picture is not load bearing, so an AI can be very useful for making lots of pictures. But in order for someone to trust an AI generated architectural diagram, it needs not to be hiding some critical flaw the AI missed and can't explain.
I think hallucinations are getting significantly better with each model, though still not completely gone.
Does AI actually have poor vision? It does well enough in things like the new 4o image model. If you mean real-time video vision for real world tasks, I don't think it's terrible and I don't think people have worked on it that hard.
The main thesis of 2027 is that AIs only "need to" get to the point where they can do AI research, after which there's a lot more AI research and all the remaining issues get solved faster than you expect, especially if they're less "fundamental new paradigm" and more like "mopping up" by applying existing AI to new categories of task (like real world visual navigation).
I dunno Scott. I’d feel better if they didn’t sound so confident when they hallucinate. I mean I could prod Claude to the correct answer but…
A session with Claude 10 minutes ago;
Gunflint:
In a 2000 movie who said “He’s crazy. He probably calls everyone Vernon”
Claude:
The line "He's crazy. He probably calls everyone Vernon" is from the 2000 film "Me, Myself & Irene" starring Jim Carrey. This line is spoken by the character Irene Waters (played by Renée Zellweger) when Jim Carrey's character Charlie Baileygates (who has a split personality and sometimes becomes the aggressive Hank) calls her Vernon.
Gunflint:
I was thinking of Wonder Boys
Claude:
You're absolutely right! I made a mistake. That line is indeed from the 2000 film "Wonder Boys," not "Me, Myself & Irene."
In "Wonder Boys," the line "He's crazy. He probably calls everyone Vernon" is spoken by the character Hannah Green (played by Katie Holmes) in reference to Grady Tripp's (Michael Douglas) editor, Terry Crabtree (Robert Downey Jr.), who mistakenly calls Hannah by the name Vernon when they first meet at a party.
Thank you for the correction!
Gunflint:
No I think the line was spoken by the actor Richard Knox who was given the name Vernon Hardapple by Michael Douglas’s character in a bar in a discussion with Robert Downey Jr’s character
Claude:
You’re right, and I apologize for my confusion. Let me be clearer:
In “Wonder Boys” (2000), Michael Douglas’s character Grady Tripp makes up the name “Vernon Hardapple” for Richard Knox’s character during a conversation with Terry Crabtree (Robert Downey Jr.) at the bar. When Knox’s character confronts them after overhearing this made-up name, he says the line “He’s crazy. He probably calls everyone Vernon.”
This is exactly what you stated, and I appreciate your patience in correctly identifying both the speaker and context of this quote from the film.
I think this has something to do with neutered LLMs that will agree with you no matter what you say if you say it confidently.
The first response was definitely a hallucination, but if you insist on basically anything that isn’t obviously and fundamentally true (like, The Empire State Building is in New York) it will confidently correct its “mistake” to whatever you said was correct, even if it’s original claim was true.
I guess this is to avoid an LLM insisting its own hallucination was true (remember Sydney’s crazy messages insisting insane stuff was true?) over the user. LLM developers somehow constrain their models to default to the user if the user confidently asserts things, to prevent it from confidently asserting wrong things despite pushback.
Well…
Claude’s second answer was also factually incorrect
“Terry Crabtree (Robert Downey Jr.), who mistakenly calls Hannah by the name Vernon when they first meet at a party.”
Not in the movie. Made up from whole cloth.
When I spelled out the actual details of the scene Claude’s third answer was correct.
Do you mean that if I had made up another scene that didn’t appear Claude would have agreed?
A few months ago a British journalist tried an experiment whereby he told one of these LLMs that a notorious criminal (Dr Crippen) had been an expert trombone player (or similar). A week later he asked this LLM to provide information on the same Dr Crippen, and was confidently informed that he had been, you guessed it, a brilliant trombone player!
> I think this has something to do with neutered LLMs that will agree with you no matter what you say if you say it confidently.
Just as an aside, for anyone that hasn't used Gemini 2.5 (only recently opened up to the web interface vs the API), it is the ONLY model that doesn't do this.
I've found it hugely useful - I like to "adversarially collaborate" and test my ideas and arguments, and o1 Pro and Claude 3.7 and all the other models really suck for this - they immediately roll over at the tiniest pushback.
But 2.5 doesn't do this, stakes out a consistent position and maintains it over time, but is amenable to factual correction or rebuttals (but not vibes based ones!) - it's so much smarter than every other model right now, I've made it my daily driver.
See eg https://www.techradar.com/computing/artificial-intelligence/were-already-trusting-ai-with-too-much-i-just-hope-ai-hallucinations-disappear-before-its-too-late . There's also https://www.uxtigers.com/post/ai-hallucinations , but the regression line kind of looks like a hallucination itself.
Thanks for the links.
I should say that I don’t sit around thinking of things that I want to see if Claude will get right.
The movie line came up yesterday watching the Rose Garden ceremony with the tariff announcement.
Mrs Gunflint turned to me and said “He probably thinks everyone is called Vernon too.”
I just nodded, “Yep.”
Long marriage, many inside jokes.
Every single experience I have with LLMs is exactly like this. I see other people using them well, so I accept that I am just incredibly unlucky.
My working hypothesis is that the current crop of LLMs rely heavily on the Clever Hans effect, named after a horse who could supposedly answer simple arithmetic problems by stomping out numerical answers with his hoof. On closer investigation, Hans turned out to be just stamping until his audience reacted to him having reached the right answer. LLMs seem to be pretty good at picking up the implicit and explicit assumptions of how people ask questions of them and giving answers consistent to those assumptions, like a college student trying to bullshit his way through an essay exam.
If you ask detailed questions in good faith, then this will tend to get you answers that seem good as far as you know as you assumptions, wrong or right, are parroted back to you. But if you ask them questions in bad faith, as I tend to do since I use LLMs mostly as a toy rather than a tool and I am a contrary little shit, then they often give hilariously bad results.
They also tend to do poorly under cross-examination, since they also resemble bullshitting exam-takers in that their initial answers tend to skew towards emphasizing familiar boilerplate concepts and away from the higher-risk area of answering the questions directly. Follow-up questions force them to elaborate on stuff where the correct answer is less well-established by their training data, where they are most apt to resort to hallucinations. Again, good or bad faith radically shaped your experience here: a good faith user will ask follow up questions that they genuinely don't know the answers, while mischievous users like me will ask questions we know the answers to but the LLM seems to be avoiding.
I don't actually think this is a fundamental problem with the LLM architecture... Did you see Anthropic's recent paper, 'Revealing Computational Graphs in Language Models'? They were able to apply their monosemanticity autoencoder technique to Haiku 3.7 (something which seemed utterly impossible as of last year, and i'm still keenly interested in anyone who can explain to me how they were able to do this... last I checked, if training a given AI requires x compute, then autoencoding it requires x^n compute where n is significant)
They had a fantastic less-technical explainer: https://www.anthropic.com/research/tracing-thoughts-language-model
We learned a *lot* about the specific style of hallucination that you're reporting here
and it mostly seems like an artifact of the kind of reinforcement learning that the LLMs have been subjected to, an overwillingness to be helpful. Much of the effort and compute that's been spent on training these LLMs has ended up going, not towards making them smarter, but rather towards finding a more optimal point along the 'sycophancy / PR safety' curve.
I suspect we're already on a pareto frontier of that curve, that any movement towards fewer hallucinations as in your post will also mean more false-positive rejections, and vice versa. And I wish the AI companies would waste less time wiggling around on that curve. You can read more about it in the linked essay, they talk about how the post-base-model training causes LLMs to arrange themselves into a "refuse to answer by default" setup, and then treat every instance where they have an answer which is predicted to be uprated as a special exception to that general rule, and this is not a very good way to build accurate worldmodels
But my point is, I think the kind of 'making up bullshit' that you're presenting is separate from the kind of hallucinations that are an actual barrier to AGI or accurate world-modeling in general.
(Also, I think I have a better example of the kind of hallucination you're talking about, or at least a more humorous one. It's my favorite example of LLM sycophancy forcing them to come up with an answer even when they really shouldn't. The "aquaman agreement theorem":
https://i.imgur.com/jxcthX5.png)
Thank you John. Reading now.
Chiming in to say that the response to the "Aquaman Agreement Theorem" made my day
I thought training a sparse autoencoder is done on the trained model so inference latency is more relevant than training time? It's expensive because the autoencoder is usually larger than the subnetwork being interpreted to disentangle the entangled weights.
What graphs and data I've seen say that hallucinations have dropped somewhat, but nowhere near to reach a level that would support use in any environment with hard engineering constraints. Trying to apply AI to this research just sounds like it might introduce even more of these errors, some of which might not be visible at first glace. More fundamentally, LLMs don't seem well suited for providing perfect answers, and yet a lot of the grand predictions for AI explosions rely on AI taking on jobs requiring a high amount of precision.
I don’t understand why hallucinations are such a dealbreaker? Excellent engineers get things wrong all the time, look them up, and then fix them. How is the AI any different?
AI struggles with recognizing mistakes as wrong and then correcting them. In order for the scenario of complete automation to happen, the AI would need to be able to do this all on its own. Otherwise it is still relying on qualified human engineers to validate the outputs.
There's a Claude Plays Pokemon run going on right now that I loosely follow and it's a great demonstration of the issue with hallucinations.
Basically Claude assumes wrong facts every so often. Like, the path from A to C _definitely does not go through B_ when in fact it does. And then it draws worthless conclusions from that, never questioning the "facts" it's hallucinated into existence. The run then becomes stuck again, until the next memory wipe that accidentally hits the hallucination. Because Claude seems utterly unable to identify when it's hallucinated.
Now, this is in a game that gives pretty immediate feedback on whether or not your assumptions are correct. Still, Claude is unable to recognize when it's wrong and would rather try the same broken strategy for the 100th time. Surely this time it'll work. You've gotta admire the optimism but walking through a wall doesn't work the 100th time either, and if the LLM is unable to realize that then hallucinations are a problem.
Imagine using the LLM on a domain where you don't know if it's wrong and can't tell it to stop hallucinating.
My wife used AI to help her compile information that would otherwise require her to do a lot manual searching in a reference book. She asked me to double check the AI's work...which required me to do a lot of manual searching in a reference book. The AI was almost entirely right, but made up one item that does not exist in the reference book.
Which means, we had to do almost the same amount of work as we would have done without the AI (looking everything up manually to double check it) in order to ensure that the final product was without error. Until the hallucination problem is solved that's going to remain the case, and AI will only provide minimal increases to efficiency. The only increase to efficiency AI contributed to this project was that my wife could copy and paste the correct entries from the AI's output, so she didn't have to type it in manually.
Exactly. Why can't Scott admit that diminishing returns are a thing?
Yeah...
Me: calculate the skin depth at [gives parameters to calculate]
Copilot: produces an answer several orders of magnitude off (which, being an experienced engineer I noticed, picture a recent grad here)
Me: are you sure this is a correct answer?
Copilot: yes, sure, here's the formula (produces an incorrect formula, which at least explains why it gave a wrong answer)
Me: off I go to dig up the old matlab script which I didn't want to do so I asked copilot.
To be fair when I asked it calculate an angle for a specific rise over a distance it did it correctly, but come on!
If skin depth is variable, where does that leave Beauty..?
I'm curious as to whether, instead of asking the AI to calculate it, you could have asked the AI to write a MATLAB script to calculate it? Perhaps that framing may have obliged it to think logically/pedagogically?
"where does that leave Beauty..?"
Why, in the Eye of the Beholder, of course!
Re. matlab: I don't think it would have helped because it used a wrong formula, so I'd assume it would use the same wrong formula in a matlab script.
The other thing is that if it gave me a correct answer it'd save me time and money (directly - matlab is an on-demand license for the company, every time I fire it up we pay).
I'm interested -- did y'all account for the "higher fruit" aspect of increasing AI research? You've written a lot about this in other areas of science, how it's slowing down despite massive investment, etc.. Right now, AI research can still progress (to some degree) by algorithmic improvements, etc. It's at least highly plausible (at least to me) that serious mathematical (and perhaps cognitive) insights are going to be needed to make progress past some point fairly soon; and it's also plausible to me that that is exactly what AI is going to be extremely bad at for quite a long time.
Just curious whether this was discussed. Thank you for the post and the role, either way!
If the hallucinations are ever completely gone, then the AI will cease to be creative. What is needed, instead, is for them to be bounded and recognized. Then they become "theories to be tested".
> Data efficiency comes later, downstream of the intelligence explosion.
If I'm interpreting you correctly, you're predicting that we'll crack recursive AI intelligence improvement *before* we crack active learning? If so, that prediction has some wild implications, e.g.
- Most meme-worthy: We'll get the discount sci-fi future where humanoid robots become commonplace, not because the human form is optimal, but because it lets AI systems piggyback off human imitation for physical tasks even when that form is wildly suboptimal for the job
- Human labor will likely become more valuable relative to raw materials, not less. In a world where all repetitive, structured tasks can be automated, humans will be prized specifically for handling novel one-off tasks that remain abundant in the physical world
- Repair technicians and debuggers of physical and software systems become worth their weight in gold. The ability to say "This situation reminds me of something I encountered two years ago in Minneapolis" becomes humanity's core value proposition
- Large portions of the built environment begin resembling Amazon warehouses - robot restricted areas and corridors specifically designed to minimize surprising scenarios, with humans stationed around the perimeter for exception handling
We accelerate toward living in a panopticon, not primarily for surveillance, but because ubiquitous observation provides the massive datasets needed for AI training pipelines
Is that an accurate assessment of your world model?
Sub-microsecond HBM cache loads.
I think my main critique with some of the fast take off is along the lines of “9 women can’t make a baby in a month”.
A lot of the acquisition of new knowledge in materials science and biology are necessarily time and labor intensive. You can’t just intuit them out of first principles, you have to actually obtain the data first.
I feel like a lot of fast take off scenarios envision the AI just thinking new discoveries into existence and I’m not convinced that the world actually works that way.
Well, it *could* work that way, at least largely, if P == NP. And we don't know that it doesn't.
Accurately modeling the physical world is way way harder than people realize. This is why civil engineers still use shake table. Accurately simulating the universe would require more computational power than can ever exist (even if you turn every single atom into a quantum bit. AI has no answer for this problem.
If its general ai it will be wildly impactful (just dont expect chat bots to be general)
"YES if the AI Futures Project's predictions seem to have been roughly correct up until that point."
Errrrrrrrrrrrrrmnmmmmmm
Wow.
That's nuts.
I may have a more profound reaction later but...yeah, that's pretty nuts.
Obligatory prediction market on whether these predictions will hold roughly true:
https://manifold.markets/IsaacKing/ai-2027-reports-predictions-borne-o
Many Thanks!
Watching now. I'm hoping world governments realize an arms-race is almost guaranteed to end poorly, and they commit to an international lets-make-utopia project. China doesn't want to get paperclipped or grey goo'd any more than we do, and Chinese researchers have already said as such. Since USA is currently going downhill, probably the UK + Deepmind would take the lead on this.
I'm more than a little anxious on this subject personally, and I have to say I'd be significantly more comfortable if the government of the country where most of the cutting edge AI research is taking place showed even basic competency navigating ordinary societal conflicts within the scope that governments have been handling for the last hundred years.
>if the government of the country where most of the cutting edge AI research is taking place showed even basic competency navigating ordinary societal conflicts
<mildSnark>
The most ironic outcome would be if the governmental incapacity (to, amongst other tasks, intelligently respond to technological risks) was primarily driven by polarization, and that, in turn, was primarily driven by the engagement-maximization algorithms running in social media platforms. :-)
</mildSnark>
I did find it notable that they kept up bringing the analogy of WW2 for governments taking control of industry and converting factories, even though we're not only not at war with China, we aren't even in as much of a conflict as we used to be with the Cold War.
Firstly, I have not watched the video -- is there a text summary (beyound what's in this article) ?
Secondly, what is meant by "Intelligence Explosion" (as well as "Superintelligence") in this case ? Are we talking about programmers (and perhaps AI researchers) losing their jobs and/or having to learn an entirely new toolset and thus producing more incrementally useful LLM-based tools; or a nanotech-based machine intelligence that can answer any conceivable question while converting the Earth to computronium; or something in between ? I understand that you expect "economy to be automated" by 2029, but I'm not sure what that means either -- are we talking post-scarcity Star Trek replicator future, or something exactly like today but all the container trucks are autonomous, or what ?
Transcript is at https://www.dwarkesh.com/p/scott-daniel
Intelligence explosion means that AIs become good enough to help with AI research, so AI research gets hyperbolically faster as outputs feed back into the inputs.
Yes, I understand this, but how is this projection different from the normal course of technological development ? For example, humans used to produce binary code by hand; then compilers were invented, and programming research got hyperbolically faster as outputs fed back into the inputs, creating a plethora of programming languagers. Are you expecting something similar to happen with AI, or are you expecting something far beyound that; in either case, what practical implications (if any) do you foresee ?
I don't know if you should think of it as "something similar" or "something far beyond". I think that existing progress has already speed things up 1000x compared to the Stone Age, and that this will also speed things up a few orders of magnitude. I think probably the shape of the hyperbola will change between last decade and next decade, but this may look more like a return to trend than something completely unprecedented. See https://slatestarcodex.com/2019/04/22/1960-the-year-the-singularity-was-cancelled/
That is helpful, thanks ! And now I will also go read the transcript :-)
Is there an implication that this will matter a lot for things?
Its pretty easy to envision worlds where "AI research gets hyperbolically faster" has anything between "Earthshattering" and "little to no" effects on the world.
Perhaps this is a tired question, but in some sense, drug research is "hyperbolically faster" now than it was 200 years ago, but it's also a lot harder to discover useful things. Why do we know that the "AIs are faster at AI research" effect will dominate the "It's harder to produce useful gains from AI research" effect? Or maybe this is a consideration that you just give low probability to?
(Okay, okay, I'll go read the transcript!)
Yes, that was basically my concern as well.
You may also want to read the takeoff supplement, which is an attempt to answer this question - https://ai-2027.com/research/takeoff-forecast . The short answer is "We tried to figure out how much time it would take (without AI) to reach each milestone, and how much AI would speed it up, and it speeds it up enough to matter a lot."
If you were going to make the low-hanging fruit argument, you would want to quibble with our estimates for how long it would take unassisted humans to reach each milestone. We got these by asking a combination of superforecasters and researchers currently working on the problem.
I appreciate the response, and the forecast document is a great read.
But it misses what I was trying to get at. The simplest way I can think of to put it is: How do we know it isn't a logistic function? How do we know we're not in an initial part of the curve that looks exponential but then we later find out that no, actually, it's asymptotic. Take whatever you want for "it" here. Cognitive task capability or whatever.
I didn't see a discussion of why this wasn't the case or, if it were, what the point at which "cognitive task ability" would go asymptotic would be. You just sorta state at one point, "we don’t have reason to believe that humans are near the limits of general intelligence."
Maybe I'm naïve, but this seems like a pretty fundamental unknown.
If it's like normal trend curves, it is sort of logistic. However we don't know where it starts flattening out.
Just for example, pretend that transistors weren't possible. Where would the curve for "computers" have flattened? Vacuum tubes could have become a LOT better than they were. (As discrete transistors were taking over, GE came out with grain of rice sized vacuum tubes with a long half-life.) But we would never have gotten HERE unless there were some other technology that could move in.
So to predict when the logistic curve is going to flatten, you need to predict what future technologies will become practical, and what their limits are. Saying "it's going to flatten out" is almost certainly correct, but that doesn't say when.
Yes, but positive feedback cycles tend to rapidly break. You sort of acknowledge that with, e.g., various comments about robots, and with the various comments about required compute resources (and about the building of research facilities), but intelligence, in and of itself, probably has strong limits.
I suspect that the various feedback cycles through different processes will keep the expansion of AI intelligence below the level of an explosion. (OTOH, I can well believe the rate of expansion would be considerably above linear, but probably not super-exponential. Parallel processing experience tends to show that dividing the process into lots of independent threads yields heavy costs in communication.)
AI research gets hyperbolically faster according to … number of papers published?
What is the proposed relationship between growth in “AI research” and changes in token costs, energy costs, or economic output?
I think if AI becomes merely good enough to *help with* AI research, that is not enough to expect an intelligence explosion. Processes with multiple inputs tend to experience marginal returns wrt any one of the inputs as other things start being the bottleneck. Thus, having one input increase, even to infinity, doesn't drive the whole process to infinity.
Let me give a concrete example. Imagine AI became good enough at programming to be able to correctly write code to spec. This would certainly help with AI research, since a good part of that requires writing code. But even if the AI because increasingly good this, that would not speed things up HYPERBOLICALLY, because no amount of writing code *by itself* is enough to succeed at AI research.
To get hyperbolic improvement, you need the output to feed into ALL the bottlenecking inputs, not merely one of them. Just being able to *help* with AI research usually won't be enough for that; you really need a stronger condition.
I say in the post:
"Take a sip of your drink every time Dwarkesh asks a question about bottlenecks; take a shot every time Daniel answers that his model already includes them and without the bottlenecks it would be even faster."
...and I do think the Dwarkesh interview addresses this concern pretty thoroughly. But you can also check the takeoff supplement at https://ai-2027.com/research/takeoff-forecast . CTRL+F for the part saying:
"We’ve heard a few people make the argument that because automation can only speed up some aspects of R&D so much (e.g. gathering human data), progress will be bottlenecked on these tasks leading to not-very-large progress multipliers (see Amdahl’s Law)."
And the part saying:
"This view posits some combination of (a) training compute matters far more than algorithmic progress (b) experiment compute matters far more than researcher quality and quantity for algorithmic progress, (c) algorithmic progress is driven by adapting to larger compute scales and (d) once AI R&D is fully automated, algorithmic progress will be close to fundamental limits which will make compute the dominant driver of further progress."
Actually it's even worse than that, since the present-day "AI" is unable to perform any kind of research at all. It can very quickly comb through all indexed human knowledge (circa 2022 or so) and retrieve the pieces the user asks for; it can also combine multiple documents or extrapolate connections between them to create a speculative document that, had it existed, would've occupied that place in the corpus.
But the trick to making true AI does not exist anywhere in the corpus; thus it's not something that could be produced in response to a clever enough prompt by generating the next most likely token. Discovering such a trick would require independent and sustained experimentation, and doing that would require something with some measure of intelligence and independence, thus making it a chicken-and-egg problem. The same goes for novel discoveries in physics, engineering, etc.
https://sakana.ai/ai-scientist/
>The AI Scientist automates the entire research lifecycle, from generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript.
> The AI Scientist can incorrectly implement its ideas or make unfair comparisons to baselines, leading to misleading results.
> The AI Scientist occasionally makes critical errors when writing and evaluating results. For example, it struggles to compare the magnitude of two numbers, which is a known pathology with LLMs. To partially address this, we make sure all experimental results are reproducible, storing all files that are executed.
So this "AI Scientist" is more like a standard LLM than a fully autonomous agent: it can help you look something up, or pump your intuition, or take care of some bit of drudgery; but ultimately it's you, the human, doing the work.
Many Thanks! I agree that the problems you cite are important. I view them a bit differently. I view them as reliability problems, on a par with hallucinations. Whether the errors can be pushed down to acceptably low levels _is_ an open question, and I view reliability as probably the key question for whether AI will be as transformative as is anticipated.
If we can get sufficient reliability, I think autonomy will be easy.
See also the website https://ai-2027.com/
5 years before AI reaches an intelligence level where it theoretically could be used to run the majority of the economy even with no further improvements? Possible. 5 years before AI is literally physically hooked up to and running the majority of the economy? Not a chance. Industries don't adapt that quickly.
In WWII, it took three years from government wakeup to the economy having completely pivoted to a war footing. We discuss whether this is an appropriate metaphor further on the podcast.
I'll make sure to watch that discussion. But the US must have had at least 5x the state capacity and political agility in WW2 than it currently does.
Yup, we think having superintelligent AI planners adds a lot of state capacity and political agility. We also think that there will be a pretty intense arms race with China going on at the time, and that this also encourages governments to become more capacious and agile.
Well, planning only goes so far - in my (limited) experience, the execution is much harder. And execution gets even harder if worker incentives are not aligned with the top-level goal.
In case of WWII, there was an attack on Pearl Harbor (and Philippines), and it was pretty clear to everybody that the war has to be fought.
In case of AI - why would middle-level managers / senior employees implement AI planner directives with haste? Won't they see that it would soon cost them their jobs, income and social status as result? And won't they provide quite a lot of friction in AI adoption as the result?
I think execution is part of the planning in this case. We think of planning as the as the high level decisions and execution as all the little day-to-day decisions and problem solving. But a superintelligent AI could just do both.
If we are at the stage that superintelligent AI does everything, than transition to Singularity is finished, right?
But we are specifically discussing transition process - so, _at that stage_ something must be different about (available or deployed) AI capabilities...
I don't think that discussion went far enough https://www.astralcodexten.com/p/introducing-ai-2027/comment/105740438 If you think about actual WW2, the US had been building up its military during the 30s, observing the UK fighting Germany and giving them Lend Lease, undeclared war against Kriegsmarine U-boats in the Atlantic, then Japan attacks Pearl Harbor and we shift into a war economy. Communist China has been around much longer than Nazi Germany, and while we did actually get into a hot war with some of their forces in Korea, we never mobilized the country's resources against them even to the degree that we did against the USSR. Very big geopolitical changes would need to take place for our government to be treating this like WW2.
The example we looked at most closely was bombers, where I think the US really was asleep until 1940 (not 1941!), so it's a fair comparison.
I don't think we did full mobilization during the Korean War because we didn't need to - we got an okay truce as it was.
But also, I don't think our exact story is "the AI arms race is important enough for full mobilization". The story is more like "the AI will want to move fast, it will get what it wants for various reasons, and full mobilization provides a baseline for how quickly fast things can go".
Full mobilization in a hunter-gatherer tribe can be as simple as one evening chief saying 'tomorrow we will go west in search of a better valley, this one is no good'.
During WWII it took 3 years (and actual war casualties to keep everybody motivated).
Supply chains and manufacturing processes are more complex these days, and they span the globe (instead of being mostly concentrated within one country) - so if humans would be doing it, it probably would take _longer_ to mobilize. Ok, AI is faster / better - but would it be so much faster / better at affecting change as to compensate for the increased complexity (since WWII), at least initially?
We didn't do full mobilization during Korea because the front was so narrow that it wouldn't have helped in the ground war, and the enemy's war production was mostly coming from PRC and USSR which we couldn't 💣.
This seems politically naive. The US at that time had a relatively autocratic president with the strongest popular mandate in a century, and he had almost a decade of practice using emergency powers.
Most old governments are sclerotic, and can't suddenly change the trajectory of the state. They typically drown in the face of unprecedented threats.
We had such sclerotic a government until 3 months ago, and that's not just my opinion. It was the widespread consensus opinion that the US government was struggling to perform actions. And whatever we now have in April 2025, it certainly doesn't have the strongest popular mandate in a century.
Could you please operationalize "wake up and pivot", by explaining who makes that happen and by what politics? I'd like something more convincing than "existential threat drives consensus", because I think that only happens in the movies, or in real life only in rare cases where a political genius already lined up the political dominoes. Heck, look at Covid; there were people in March 2020 who predicted that the threat of Covid would drive national or world consensus, and... it didn't!
The US seems to have an extremely autocratic president right now. And parts of the country seem to strongly approve.
OTOH, I would not be surprised if Trump collapsed tomorrow. To my non-medical eye he looks like a good candidate for a heart attack. And I really doubt that Vance would swing much weight with the Trump-base.
So basically both scenarios are dystopian given that in "Aligned" scenario couple of US politicians and tech CEOs have archived complete dominance over human civilisation forever and crushed all competitors (geopolitical and economic). As non-american I am not sure that this looks much better compared to a unaligned scenario.
I agree both scenarios are at least somewhat dystopian. We got the "aligned" scenario by conditioning on the AIs being aligned and humanity surviving, then asking what the most likely version of that was.
I guess every alignment researcher should probably consider if eternal Trump-Altman-Thiel Reich is desirable result of his life’s work.
What alternative conclusion would you hope for them to come to?
some sort of international agreement where Superintelligence benefits can be democratised or Archipelago scenario where different nations and communities can live their values with post-singularity tech with no foreign interference. I don’t deny the value of the analysis but was just hoping we have better options.
I think lots of alignment researchers would like that in principle, and some people should be working on it, but figuring out how to get there seems like it requires a mostly non-overlapping set of skills from technical alignment research. So the question facing technical alignment researchers (to oversimplify a lot) is what the different probability-weighted outcomes look like if we solve alignment, compared to if we don't, and which world they'd rather live in.
It's possible—and some people argue—that the world where we solve alignment is worse on expectation. But personally, I'm not seeing it.
I'm at least as opposed to US politicians and CEOs as the next guy, but I don't think the political structure will matter much, or maybe even meaningfully exist, if the world became infinitely wealthy and technologically advanced. That also applies to worries about China imposing authoritarianism.
I think a scenario with infinite automated wealth production could be very bad for the average person, with sufficiently centralized control, because the people in control may not be inclined to share around even a tiny portion of that wealth.
In a world where everyone could have all the material comfort they could want, what separates people with power and control from ordinary people? Not a lot, necessarily. But if you're a person with power and control, and you *want* there to be things separating you from ordinary people, you can achieve that; that's what it means to have power and control. And one straightforward way to achieve that is to ensure that other people can't have material comfort, unless you let them.
> Not a lot, necessarily. But if you're a person with power and control, and you want there to be things separating you from ordinary people, you can achieve that;
There will always be intrinsically limited goods - human capital, land and real estate in places people want to live, art work and other one of a kind things that only one person can own the original of, and more.
Rich people are going to have all those things, they'll have a lot distinguishing them from all the UBI plebs. Look at Captain Picard's vineyard, still a status marker in a post-scarcity society - arguably much MORE of a status marker, because it's probably impossible to buy with money.
And as to SOMEBODY sharing the wealth - the "Giving Pledge" has 236 billionaire signatories. That's the one that Gates and Buffet and Zuck have all signed where you commit to giving away at least half your wealth.
There's only like 700 billionaires in the US, a substantial fraction of them are charitably-minded enough to commit to giving the majority of their wealth away.
All it takes is one.
I'm not sure what you mean by "all it takes is one." I don't think one person committed to giving away most of their wealth is going to be sufficient to implement a UBI. I think this is a way that society *could* be organized, but there isn't a guarantee that a UBI would be implemented at all. Even people who're billionaires now may not have any real say if they don't have a hand on the levers of AI in the event that that really overturns our economy.
Yeah, I was more pointing to a model that a good chunk of the richest people today don't care as much about wealth as they do giving back and / or helping humanity in the aggregate.
I don't think the current billionaires are going to matter much in that future - the people who own the AI + robot factories and produce literally everything are going to make them look poor - they'll be yottillionaires or something.
We'd expect a greater concentration of wealth if 99%+ of jobs are eliminated and returns to capital are sharply concentrated in a few AI-dominated spaces. I'm just mapping that out to the extreme - we get a thousand yottas instead of a thousand billionaires.
But in that schema, indeed, all it takes is one to UBI everyone, and the "roughly a third" ratio argues there will indeed be at least one.
No amount of infinity will be enough if a fat fuck at the top gobbles up everything and makes it a law that you never own anything, only rent it from its eternal God-ordained owner.
Same reason big-name humans who already have a net worth measured in tens of billions hoard it in hopes of parlaying it into even more money and political power, instead of donating 99% to charity and/or pet projects in the arts and sciences, while retiring to a life of the most fabulous hedonism money can buy.
Whatever that reason is, I doubt it will change no matter how many digits are added to their net worth, and I’m inclined to believe their power-seeking will stop at nothing short of reducing the rest of humanity to a Sim City in their palm.
Individual big-name humans today cannot unilaterally provide for the bulk of humanity. The can certainly make an impact, and many do so in actually by using their money for charitable causes, but realistically the impact even the richest person can have is limited. E.g. if the richest person alive today decided to spend his entire net worth to feed the hungry, at $2/person per day he can could feed earth's entire 8 billion population for less than a month.
This is very far from being "infinitely wealthy". The resources concentrated into big-name humans today are vastly more than they need for their own lives, but also vastly less than they need to support humanity as a whole. The fact that individuals are not doing so today is given by the fact that they CAN'T, and thus not a clear indicator that they WON'T.
Any billionaire can easily secure the food of a small village or a city block (~10K people) indefinitely, they do not do so for the most part.
In fact, billionaires pretty often do even less than people with far less money do, paying their full taxes they owe to the state they're a citizen of.
I've been anticipating this a lot since Scott first mentioned he was doing it, because Scott's views on AI would be so massively consequential if true and because he's never laid them out in explicit detail, and this looks even more comprehensive than I hoped for. Extremely impressed that Dwakesh got him to also do a podcast.
Still, a bit concerning that a guy with epistemics as strong as Scott's thinks the future could be this precarious.
The future is in fact this precarious. If Scott manages to get people worried, it might get a little less precarious
I want to clarify that I mostly wasn't involved in the actual forecasting. I think the forecast is top-notch and very plausible but am nervous describing these as "my views".
I think calling that a "forecast" is wrong. It's a scenario. You clearly state that there were many others, that were considered equally probable.
I think that it's hard to make a specific "best forecast" in a multidimensional space because of issues with median vs. mode, but this is somewhat motivated by trying to forecast things that come true.
A lot of this discussion seems very software focused, but we actually need hardware to run all this stuff. Is there discussion of things like critical minerals, undersea data transfer cables, chip manufacturing supply chain vulnerabilities, etc? Thanks!
There's some discussion of this on Dwarkesh and in the "Compute Is The Dominant Driver Of AI" section in https://ai-2027.com/research/takeoff-forecast .
Short version is: we assume hardware will continue to scale up at about the same rate as previously, up to weak superintelligence. When we predict weak superintelligence in 2027/2028, it's because we think that current supply chain trends get us there then.
We think weak superintelligence will be able to optimize the supply chain somewhat better than humans, partly because it's smarter than humans and partly because it will be better than humans at some kind of political/bureaucratic tasks - but not too much changes if it isn't.
"Short version is: we assume hardware will continue to scale up at about the same rate as previously"
Ok I promise I will read the whole thing when I have some uninterrupted time but this quote really doesn't inspire confidence. It's not impossible to have hardware to continue to scale up at about the same rate, but it is quite.... implausible?
I might be getting this wrong, I'll have to double check the compute forecast, but hasn't training compute growth been basically stable since 2010? See https://situational-awareness.ai/wp-content/uploads/2024/06/compute_long_run_trend_updated-2048x1280.png . We are only assuming this 15-year trend continues another two years, to 2027.
(unless I'm wrong and we don't assume that, I'll double check).
During this period the optical shrinking of the process nodes was still a viable way to increase transistor density. We're at the end of this road approximately now.
AI compute scaling happens because of a combination of:
--- More money being invested
--- TSMC increasing supply
--- TSMC improving at the specific types of GPUs used for AI
--- General chip progress
Chip progress isn't a very important part of this equation but does shift things a little bit. The scenario's take on this is:
"For improvements in chip efficiency, we extrapolate Epoch AI’s historical trend of 1.35x per year and find it is consistent with the already reported performance of upcoming chips such as NVIDIA’s GB200 as well as rumoured plans for the Rubin series detailed below.
The widely adopted state of the art GPU in 2024 is NVIDIA’s H100, which has 1e15 FP16 FLOP in raw performance. In three years, we predict the widely adopted state of the art GPU to be NVIDIA’s Rubin GPU (R200), which we project to achieve a ~2.4x improvement over the B200 (widely used in 2025-2026) to 6e15 FP16 FLOP performance. We think this will be achieved through a 1.2x increase in die size, a 1.7x increase given the transition from TSMC’s N4 to N3 process, and a 1.1x increase from other miscellaneous improvements."
...which I interpret as them saying that the 2027 chips are already in the pipeline and don't seem to have encountered this problem yet. See also Epoch's report at https://epoch.ai/blog/can-ai-scaling-continue-through-2030
That report is pure speculation.
Re. Chip progress: yes 2027 chips are definitely in the pipeline, probably somewhere between verification and debug. No, we don’t know what problems they will find, not yet.
But I want to to show you how naive these projection are for the 2.4X performance increase:
1.2X die size increase - fine, no problem here
1.7X increase because of the move to 3nm from 4nm. No, this is what, (4/3) squared (1.78) - are they that naive? You never get the full theoretical improvement from scaling like this, no one who designs chips will ever project that. Maybe 1.4?
1.1X from “miscellaneous improvements” - hot air, straight to the bin.
It was roughly 100 times raw compute from gpt-3 to 4, in addition to a hugely underappreciated effective compute multiplier from the low hanging fruit of breaking n^2 training via MOE sparsity. Now, 3 to 4 was indeed a big deal and all that, but even a couple more similar leaps seems insufficient to qualify for some of the more basic aspirational capabilities predicted. This seems a simple intuition pump as to why only getting in the OOM of 1000x raw compute by 2030 does not inspire confidence, especially when successive tiers of low hanging fruit( like training nets at lower precision, and pushing sparsity far beyond MOE heads) on the effective/algorithmic side appear to already have diminishing impact (arrakis failures, 4.5 underperforming).
All this in addition to, as other comments allude, the blind faith in moore's law is quite thoroughly misplaced and speaks of commonplace ignorance. Actual shrinkage of the feature size on transistors has nearly entirely stagnated (relatively), and yet the genius marketing decision to just keep naming successive generations 4nm, 3nm etc (which is an entirely made up number, totally unrelated to actual transistor size!) has been remarkably effective in propping up false lithographic confidence.
The nitty gritty at https://ai-2027.com/research/compute-forecast honestly has some similar feelings of reading a deep research report: lots of headspace in the right direction, but just a clear lack of sufficiently nuanced understanding of the technical specifics to make real headway with regards to accurate prediction, and overreliance on a questionably robust source (epoch). A woeful lack of appreciation for flop/$, the end-all-be-all metric, and consumate agreement with Nvidia marketing numbers -- the B200 is honestly a lithographic failure, node jumps are so dismal that in 2 years in the hottest gpu market EVER all Nvidia could offer was gluing two N4 dies together at a minimal discount vs Hopper; the real accomplishment of Blackwell is being artificially whitelisted to run on NVLInk-72 switches and Nvidia graciously allocating some dedicated 4 bit tensor FPUs to the die.
2 more years of an even hotter market will give us a Rubin, on a "1.7x density improvement!" node where the SRAM cells are the same size, most of the finfets are 30% smaller, hmmmm.... (for reference, TSCM 7nm to 5nm was a claimed 1.84x density improvement, when the actual chip shrinkage at same transistor count was only 30%, or a ~1.4 density improvement. Sure seems right in line with the 3nm finfets!)
And this time, there's no new NVlink to save the day, and there's nowhere left to go once we pick off the forbidden 2-bit acceleration fruit......
Actually, I'm rather certain that we know ways to increase the circuit density. The problem is that they're expensive. (Basically go thee-D and use LOTS of cooling.)
For that matter, design the chips to work in a REALLY cold environment, and bath the system in liquid Nitrogen. (Helium is you're feeling like overclocking.)
I read about a lab in Kentucky that did a demo version of this well over a decade ago, though they didn't use extreme cooling (merely silver heat-pumps on each chip). There are obvious reasons why that didn't become a commercial product, but if price were not object...
Yeah, cryo is hot (good one, eh?). I’m very familiar with the cooling challenges, and consider them one of the less-appreciated obstacles on the way to increased compute densities.
But cryo is on a different level of tough. No standard rel tests go below -40C. There are no models, everything has to be developed from scratch. Like you said, if money is unlimited it can be done, but still a tough slog.
If it's not a secret, what finally convinced you to change your strong anti-podcast stance? Did Daniel super-persuade you to wingman him or was it your initiative?
I thought this scenario was important and that people should read it, and I didn't think it would get the same publicity without me involved.
Disappointed to see insufficient discussion of how the trickling down of agent capacity into actual material production of goods and services will happen. There are vast corporate bureaucracies, government offices, post offices, and a million Etsy-shop-scale small businesses which haven’t even exploited the added marginal productive capacity of the *Dotcom* revolution, to say nothing of AI.
My first software job (in the '00s) involved writing code for clients who my coworkers described as "Just getting into the '80s technologically", previously using pencil & paper and operating on credit with their suppliers. My understanding is that in Japan they still widely use fax machines.
One possible change in this regard is that AI, in virtue of its flexibility, may be able to make its own adoption go smoother. The guy you hire to integrate AI into your business and the integrated AI might be the same guy.
The summary has “if aligned” and “if not aligned” scenarios, both varying degrees of bad but still. Does the forecast address the probability between these two alternatives?? Seems kind of important.
Not aligned is the default
The forecast doesn't. Daniel has said elsewhere that his p(doom) is 70%. Eli's is 25%. I don't immediately remember the rest of the team's.
They do not seem to put any weight on partial alignment; super intelligence understand their alignment without thinking about it (HA) and are instantly not situationally motivated
What's the best way to bet my entire net worth on these things not happening? Because they will not happen.
Short NVIDIA.
(sorry if you're one of the people I gave this answer to in 2019)
Does NVIDIA's current valuation really price in super intelligence by 2027 and most of the economy being automated by 2029?
Eh, I suppose NTS could think AI will be short of superintelligence, but still expect it to do better than the average NVIDIA investor thinks. There are a lot of short-timeline people who are long NVIDIA, though, I don't think it should be too hard to get more pessimistic than average.
This is ridiculous, Scott. Maybe things are different in your bubble, but in real life 99.99% of people (and investors) don't have timelines as short as you. I know a dozen normies long NVDA; my mom is long NVDA!
NVIDIA is down 32% YDT; is your opinion that this is because timelines got longer? Come on. NVDA price is about non-AGI use cases of AI
Careful! It is extremely possible to be 100% right that "they will not happen" and still lose everything shorting NVDA. And if you do, it is also extremely possible to sue a person who gave you financial advice without a license.
No no no no don't short NVIDIA, demand for their processors can easily continue anabated, LLMs exist and improve, etc. without any of the extreme things happening.
Or, AI designs better cheaper chips that anyone can make, upending Nvidia's business model.
lol. Sorry. This is not how it works. This is so not how it works I can’t even.
If it does it'll probably still be Nvidia making and selling those chips, making its stock soar, not crash.
Do nothing. You already are
There’s a “bets and bounties” button at the bottom of https://ai-2027.com/about
First step, find a counterparty willing to accept conditional payments - and, more broadly, continue respecting the associated property rights - in the event of humanity's extinction.
Dumb question: what does it mean to "steal the weights"? Thanks.
The core of an AI model is a honking-big matrix of numbers that transform input data into output data. In a linear regression: if y = wx, w is the weight, which informs how much y changes for each change in x. The weights of an AI model generalize from that (massively oversimplified, of course).
Generating the weights is what training is for, but once that's done, the ultimate product is just a matrix of numbers. It's not usually visible to people using the model, except for open-weight models like DeepSeek, but if you can break the security on the model and extract the weights, you've just saved yourself millions or billions in training costs.
Many, MANY thanks for your taking the time to explain this! Excellent.
As a general rule, LLMs (like ChatGPT) are very good at answering contextual questions like this. You can just paste the piece of text that you don't understand and ask question and it will get it right (99 percent of the time, likely better than the median human). Just posting this on the small chance that you haven't updated on the latest AI capabilities yet and hope it's helpful and taken in the spirit intended!
The training of an LLM is using a very expensive algorithm to find a specific set of matrices, that when applied in the correct manner, can do intelligent things. The values found for these matrices are the "weights". Since the valuable thing is those matrices, the work of training the LLM (billions of dollars potentially) can be skipped if one can simply gain access to a computer from another lab that has already run the algoirthm to discover those matrices, and simply copying them.
And thank you as well, Ivy! I appreciate your and Nicolas's willingness to help me out, when so many commentators would have responded with something like "RTFM, noob!" (grin)
In addition to what other people have mentioned, you don't have to do any kind of illicit hacking to "steal the weights" of a model. A process usually called distillation allows you to use the inputs and outputs of one model to train another, usually faster than the original model was assembled. This is part of the DeepSeek controversy - OpenAI claims DeepSeek was partially trained via distillation from an OpenAI model.
Basically, to "steal" the model. The weights are the model itself. A big list of floating-point numbers (i.e. decimals, without too much simplification). If you "stole" the weights of ChatGPT from Open AI you could run an exact replica of ChatGPT assuming you have the hardware and networking infrastructure.
Of course to use "Steal" here is a loaded moral choice given that those weights are obtained by "stealing" the entire internet without honoring software licenses or robots.txt, so "stealing" them is as moral as stealing from a thief. Go China.
All hardware runs on 1 and 0's, weights are a type of program (extremely limited and your all insane for thinking its intelligence); but they probaly would also consider it as escaped if it copied the other 1s and 0s
nn's are a type of hill climber, the "weights" are a location on a mathematical landscape, if you consider something like wolframs "rule 110", there was a systematic way to label a "program space" and with the number you could look at the definition and recreate the machine.
If you're correct... what am I supposed to do? Stockpile money for a future of permanent unemployment? Start prepping for a war between humans and robots fighting on behalf of tech oligarchs? Lay back and prepare to enjoy a UBI and an experience machine?
This is an important discussion that's worth addressing, and given Scott's position as the rightful caliph / leader of a flock, I think it'd be worthwhile for him to weigh in with practical advice beyond just predictions. I think there are several rational approaches available:
- If you have the appropriate skills or could learn them, try to get involved in AI policy / governance, alignment / interpretability, or even capabilities if you think it's more important that the race be won in the West vs China
- Financially, get exposure to the companies that have the most plausible route to ASI and its benefits, ie Microsoft, Google, Meta, Nvidia. Sadly I don't know of a way to get exposure to Anthropic via public markets.
- Prepping / survivalism isn't necessarily insane, I think there are probably middle-of-the-road scenarios that involve non-ASI civilizational collapse (eg nuclear war between the US and China), and a lot of the people at the forefront of some of this stuff are hedging their bets in the same way (ie Thiel's New Zealand citizenship)
>Prepping / survivalism isn't necessarily insane, I think there are probably middle-of-the-road scenarios that involve non-ASI civilizational collapse (eg nuclear war between the US and China), and a lot of the people at the forefront of some of this stuff are hedging their bets in the same way (ie Thiel's New Zealand citizenship)
Pay particular attention to the fact that nuclear war slows down AI (by a variety of means), and removes entrenched lobbying interests in its favour (including the AI companies being in control of significant chunks of the public square e.g. YouTube, Facebook, Twitter), and also to the fact that in general you want to plan for not-Doomed timelines because in Doomed timelines your plans are irrelevant.
The Age of Em says what you should do if you think that you're going to be vastly outclassed by a class of new intelligences that will price you out entirely of the job market:
(1) Invest in any company that you think will own/use/employ/partner-with those intelligences.
(2) Invest in any resource or financial construct that is not dependent on work to generate value (e.g. real estate).
Eager to read the full scenario but in the meantime, I guess I have a different assessment of the original "What 2026 looks like" article than "He got it all right." I think he really was very good on the tech itself and I'm not familiar enough with the alignment community to speak to those predictions. He gets a lot of mileage out of "it won't actually be valuable yet but will do cool toy use cases" which was very prescient but I wonder if it is actually the same skill as guessing when AI will be valuable enough to, say, start impacting GDP.
The big miss, though, is the discussion of propaganda. LLMs are good at writing what sounds to us as authoritative text, but already we are forming antibodies. "Looks like an LLM wrote that" is increasingly a response I see to posts online. I don't see much evidence that LLMs are good at persuading people of things. I know that the superhumanly-persuasive AI has been a bogeyman of Yudkowskyian AI theory for a decade, so it makes sense someone on LessWrong would overestimate its likelihood, but personally I didn't believe in it a decade ago and I still don't now so I'm probably on the opposite side of the spectrum.
I mention all this because we shouldn't be too surprised that an AI expert's predictions about AI itself were good but the predictions about how the rest of the world interacts with AI aren't as good. He's not a rest-of-the-world expert. I'm not sure anyone is. So I appreciate getting more predictions from him and the team about, say, coding agents. Past that, I'll read the guesses about the US government, the Chinese government, WW2-style nationalized economies, and how public opinion reacts to all this...but why should I find it credible? Actual experts on politics and public opinion have awful records predicting this stuff. I tend to think that near-genius AI researchers (and near-genius rationalist bloggers...) are probably worse than average at predicting the behavior of the ordinary people in the general public and the mostly ordinary people who serve as politicians.
Research (by Tetlock) has previously shown that forecasting depends a lot more on skill in forecasting specifically, compared to subject expertise. So you'd expect a good forecaster who is not a political scientist to be better at forecasting politics than a political scientist.
A victory for the outside view over the inside view.
>So you'd expect a good forecaster who is not a political scientist to be better at forecasting politics than a political scientist.
And how would a forecaster even know what to talk about? Do you even know why a political scientist is called a scientist at all? We are not called political theorists or political commentators for a reason. According to your argument, a forecaster would be better in predicting cancer evolution than a medical doctor too? Or to predict an economic crisis before economists? This comment is so ignorant it hurts, I'm not even sure I'm sorry to be this blunt.
Your argument seems to be "ad absurdum", but that's not going to work if you just repeat what I already said I believe. "predict an economic crisis before economists" is precisely one of the examples tetlock examined in detail
> how would a forecaster even know what to talk about
They would research the subject for as many hours as it takes to become confident, for instance AI-2027 seems to have involved thousands of hours of research.
And such research would come from the material made from domain experts. A forecaster/statistician can be plugged in whatever context, given they can be guided. And your hubris needs some balance mate, I gave you precise examples, not a generalization and certainly not an ad absurdum (which is what YOU did, not me).
> And such research would come from the material made from domain experts. A forecaster/statistician can be plugged in whatever context, given they can be guided.
Well yes, exactly. A forecaster knows how to properly weigh domain evidence in order to make forecasts, this isn't a skill that non forecasters need to practice, see for instance https://slatestarcodex.com/2013/12/17/statistical-literacy-among-doctors-now-lower-than-chance/
> I gave you precise examples, not a generalization and certainly not an ad absurdum (which is what YOU did, not me).
Not sure what you mean here, but it doesn't sound like we disagree
As AI capabilities advance, the question of whether AI persuasion has much of an effect depends decreasingly on the shape of the preexisting information environment and increasingly on those AI capabilities. We're not there yet, which is why Kokotajlo acknowledges in the comments, years later, that his forecasts about the effects of AI propaganda were worse than those about capabilities (though there's a bit of room for disagreement about how much of this is happening right now). But if he's right about where capabilities are going, then this kind of thing won't matter; the factors that made AI propaganda less impactful than he thought will become irrelevant as AI systems figure out how to neutralize or work around them.
Sure, I guess I would question what evidence we have that the capabilities are going in this particular direction. I'm an AI skeptic by the standards of most around here (but a believer by ordinary standards) and I read current LLM performance as continuing to show that many rationalists frequently make a mistake in being too glib about grouping together lots of distinct tasks under one "intelligence" label.
The older, pre-generative AI arguments about AI risk often used Alpha Zero as the basis for rapid progress. It could go from knowing nothing about chess to "chess superintelligence" in hours! My complaint with respect to persuasion at that time was that Alpha Zero did this by playing 44 million games. You can't learn to persuade humans that quickly. You probably can't talk to that many humans in parallel and even if you could, humans aren't the same the way chess boards are the same so you can't tournament your way to success.
For LLMs, the theory of persuasion is I guess more nuanced. Essentially, the LLM reads a lot. Right now, I think that LLM capabilities track pretty well with the capabilities of a human who reads a lot, just in some cases more so since they have read more. Can you come up with new pharma experiments to try by reading lots of papers? Probably! That's pretty much where human scientists get their ideas from. Can you become better at persuading humans through reading a lot of...psychology textbooks? Con artist memoirs? How to Win Friends and Influence People? Maybe, but this is not how very persuasive humans seem to learn it.
Just as a paperclip maximizer needs to deal with the laws of physics (though it might understand them better than we do), a persuasion maximizer also likely runs up against some pretty hard constraints. A message that persuades you might do the opposite to me. Can the same message, constructed by a godlike intellect, persuade every human alive of anything? Like FTL, maybe this is possible, but we should be skeptical.
More likely, a superhuman persuader would need to craft each message to the unique thought patterns of the individual it's talking to. If it could do that, it could do it to everyone at once. Ten years ago it was common to think that Facebook and Google advertisements could hack people's helpless minds into buying products or voting a certain way, but I don't think time has been kind to that hypothesis. Still, I think it's possible, a lot more possible than the one omni-persuasive message. I'm just skeptical that LLM training with a bit more data or a bit more compute is going to get us there.
Anyway, this is all my own personal theory of this, but if I'm right, it's a longstanding blind spot in rationalist AI discourse and therefore a place where I trust otherwise brilliant rationalists less than, say, AI coding skills (and a blindspot there would jeopardize predictions of US public opinion and therefore politics, though maybe not Chinese politics).
I agree this was the biggest mistake.
But does propaganda really require being super persuasive? As opposed to there just being A LOT of it? If your bots can engage every single user of xwitter and feed them something a little bit tailored to their tastes, is it really a problem that the smartest minority will see the propaganda for what it is?
Crap propaganda trains people who aren't already on board with whatever is being propagandized, to reflexively block out and dismiss anything that sounds like it might be another version of that same old thing you've heard way too much crap propaganda about already. To break through that wall, you need *really really good* propaganda; there's no amount of crap propaganda that can do the job by pure volume, because they've already hit the mute button. Has nothing to do with people being smart enough to see through the propaganda; stupid people hit the mute button even faster than the smart ones.
You can sometimes get by with moderately good propaganda, provided there's not too much crap propaganda triggering people's immune systems. But on big-ticket controversial subjects, there's always going to be crap propagands.
And crap propaganda is only good for rallying your base, so it's only useful if you already have an adequate base.
I’m not sure people are as good at filtering propaganda as you seem to imply. But even if you’re completely right and all propaganda can do is have them hit the mute button, a very powerful effect it can achieve is make that person distrust and mute _everything_. Just spam your political opponents with doom and gloom and ragebait and they’ll mute politics from their life, feeling discouraged from political activism, voting etc.
That's not how it works either. In the past decade or so, CNN alone has broadcast approximately 80,000 hours of crap propaganda around the message "Orange Man Bad!". Another 80,000 hours on MSNBC, and many more besides.
The result is not that Donald Trump's base has said "we can't fight this any more, so many voices can't be wrong, maybe the Orange Man really is bad". And it is not that they have said "We give up, we're not doing politics any more, there's no one we can trust". The result has been that they trust Donald Trump, and Fox News, more than ever before, and shut out everyone *else*.
And of course Fox News does the same thing in reverse, with the same effect. Ragebait doesn't make people quit in despair; if it did none of the algorithmic social media would signal-boost it. Ragebait locks in the rage.
Set up a hundred new TV networks, and use an AI to fill them with infinite rage-inducing variations of "Orange Man Bad", and you will convince approximately no Trump supporters to either vote Democrat or give up and sink into apathy. And, again, vice versa.
If you have a charismatic leader but the facts favor the other side, it’s not that hard to bombard people with “everybody lies”, “can’t trust any experts” so the facts become less persuasive.
It’s very much possible to get people annoyed with whoever happens to comprise the ruling class, so then they’ll gladly vote for someone who they think will kick them in the ass. Etc.
It’s less “Mauve Man Bad” and more “Everybody Bad, Might As Well Vote For Mauve Man”.
And yet, in spite of a charismatic man whose supporters have flooded the airwaves and intertubes with "everybody lies" and "can't trust any experts" propaganda for a decade, the ~50% of the population that generally trusted the experts in 2015, still generally trusts the experts. They just don't trust Fox News et al any more. And they don't vote for the Orange Man.
thescorpionandthetoad.txt
You're being too obtuse for your own good, what does the parable of the scorpion and the toad have to do with any of this?
Are you sure you’re not the scorpion in this scenario?
Here's a fun list of variants if you want to use the fable to make a different point: https://sadoeuphemist.tumblr.com/post/615521935528460288/a-scorpion-not-knowing-how-to-swim-asked-a-frog
Do you mean .png? https://i.kym-cdn.com/photos/images/original/002/184/135/b3b.png
That's the meme that goes "'lol' said the scorpion. 'lmao'"
I appreciate this as a project to get people talking, but I think the forecast as stated is kinda silly. I wrote some short takes here, though this deserves a longer treatment:
https://bsky.app/profile/harsimony.bsky.social/post/3llwi64unw22u
Thoughts on your tweets:
--- "It makes the common mistake of looking at superexponential increases in inputs without questioning the value of the outputs. You have 10 agents that are 10x smarter, thinking 10x faster, how much better is the code they write? Only a little, and performance is now bottlenecked elsewhere."
Maybe I'm misunderstanding, but this seems sort of incoherent - what do you mean by 10x smarter without any better code? If human A is 10x smarter than human B, I expect their code to be better. Why wouldn't this be true for AI? If nothing else, if they're 10x faster, you can produce 10x more code in the same amount of time.
We thought pretty hard about where performance would be bottlenecked (see suggested Dwarkesh drinking game above). See the Takeoff Supplement at https://ai-2027.com/research/takeoff-forecast, start at:
"This 100x speedup in coding, however, will translate to a much-less-than-100x speedup in overall research velocity—because the overall research loop includes e.g. waiting for experiments to run, discussing and analyzing the results of experiments, discussing what the next experiment should be, etc., and our hypothesized SC is not going to speed those up much; those parts of the process will thus become bottlenecks. Taking these bottlenecks into account, what will the overall speedup be?"
...which concludes that given vastly increased coder speed but everything else serving as bottlenecks, we could get a 5.8x speedup. That gets us more quickly to AIs that can start relieving some of those other bottlenecks, which is where the rest of the intelligence explosion comes from.
--- "They assume AI will be promising for cyberwarfare and enable theft of weights. But as capabilities increase, defenders "just win" in the software domain. And states have physically protect their hardware adequately up to this point."
Unless you're thinking of a different part of the scenario than I am, I don't think we think AI cyberwarriors will steal weights. We think normal humans will steal the weights. This isn't a prediction about l33t 2027 cyberhackers, just that China really wants something that OpenAI has, China has nation-state-level offense, and OpenAI won't have nation-state level defense.
--- "Also why attack? Or go to space? Reward is not the optimization target folks."
See the AI Goals supplement at https://ai-2027.com/research/ai-goals-forecast. The very short version is that AI agents will be trained to complete tasks quickly. This will instill subgoals of efficacy and power.
--- "Also, if it's so easy to take over the world, why haven't they done it? Or a suitably capable group of humans?"
Haven't they? Do the 6000 BC Yamnaya or 1800s British count as "suitably capable"? I don't really understand what this objection is doing here.
--- "Taking over the world is hard when everyone has a similar amount of resources. Everyone will be armed with AI's, so absent a FOOM, takeover seems unlikely."
Our scenario thinks not everyone will have equal-capability AIs. Why? Because in the past, it's been pretty rare for an industry to have everyone at exactly the same point, without one company/country having even a three month lead. But given the takeoff speed we forecast, even a three month lead could provide a decisive strategic advantage. The second half of our scenario, where the US has a good superintelligence and China has a three-months-less-good superintelligence, is about this dynamic.
--- "On FOOM, AI is limited by data, not intelligence. Scaling gains diminish sharply along every axis we've investigated."
These aren't separate things. AIs are still far less data-efficient than humans. Data efficiency substitutes for data. Intelligence can include data-efficiency. An AI must be able to be as smart as Einstein with only a few thousand physics books plus some generic transfer learning experience, since Einstein became as smart as he was with only a few thousand physics books plus some generic transfer learning experience. If AIs aren't yet as smart as Einstein, it can't be for lack of data!
--- "They assume we'll automate everything in a few months with AGI. But why hasn't that already happened? We have machines for making ~every product, yet have humans in every production line. "
Right now humans are cheaper than AIs for certain types of labor, therefore we use humans. If AIs are superintelligent (by definition, can do everything humans can) and can be run cheap, then on things where AI becomes cheaper than humans, companies will switch to AI. We expect this to happen pretty fast, because cheap AI labor can also fuel eg consultancies for how to automate things.
Thanks for the detailed reply! See my response to Daniel as well. I'll try to cover stuff distinct from that here.
1. On cybersecurity, we might be talking past each other. I like your example, "... China really wants something that OpenAI has, China has nation-state-level offense, and OpenAI won't have nation-state level defense."
I think that in 2027, OpenAI equipped with frontier AI is infeasible for China to hack at the software level. Physically protecting the hardware will require U.S. help.
2. On FOOM, my broader point is that there's a certain amount of information found in a dataset. You can certainly get more data efficient, but at some point you've learned all you can. AI will iteratively improve on each task and then plateau.
https://splittinginfinity.substack.com/p/on-ai-scaling
I think the central disagreement is where that plateau is, an objection addressed in the report. This is where my comments with Daniel pick up below so I'm not going to go into it further here.
3. "If AIs are superintelligent (by definition, can do everything humans can) and can be run cheap, then on things where AI becomes cheaper than humans, companies will switch to AI."
The economics of this are complicated:
https://forum.effectivealtruism.org/posts/cKsknByhuW6Hw2wHj/the-ambiguous-effect-of-full-automation-on-wages
You have to take into account opportunity cost as well as input cost. If AI's can do jobs with a value of $100K/yr or $10K/yr both for negligible cost, the AI's will all be assigned to the $100K/yr job. It's not clear to me whether AI's opportunity cost will go up or down with technological advancements.
I don't want to get into a long argument about these, but the one that seems like the biggest crux to me is:
>> "On FOOM, my broader point is that there's a certain amount of information found in a dataset. You can certainly get more data efficient, but at some point you've learned all you can. AI will iteratively improve on each task and then plateau."
Obviously this is true, but equally obviously the maximum limit is at or above the limit of top humans. It must be possible for an AI to be as smart as Einstein with only a few thousand physics books plus some generic transfer learning experience, since Einstein became as smart as he was with only a few thousand physics books plus some generic transfer learning experience.
Is Einstein the cosmic maximum for data efficiency? This seems unlikely - it would be a wild coincidence if the top human were also the cosmic maximum, as if the speed of light was around Usain Bolt's top running speed.
But on the off chance that it *is* the cosmic maximum, then we just get AIs to read far more books than Einstein can (since they have faster serial speed, never need to sleep, etc). Or we content ourselves with millions of AIs that are "only" as smart as Einstein.
Yeah I don't want a long argument either, so I'll just end with this:
We agree that AI can get arbitrarily good at any task. All we need is a dataset. This is one of the sections of the post I linked.
We disagree on are the implications of that fact. The outcome of AI automation is highly uncertain, but I agree that we should prepare for the worst case.
That being said, I think there are good arguments that things will turn out fine. I have a half-finished post on this, so for the sake of brevity I won't go into it here, but I'll link it on r/SSC when it's out.
> [...] equally obviously the maximum limit is at or above the limit of top humans.
Is that obvious? That seems a strong assumption. AI are trained on a truly massive corpus that definitely isn't written purely by the world's greatest geniuses. Why should I assume their performance limit is not therefore closer to the average?
(I assume there IS some kind of reason here because lots of people seem to make the same assumption, but I don't follow what it is.)
Because their performance is already well beyond average? (Certainly Gemini 2.5 Pro is well beyond me broadly speaking, and I'm in a well-paid knowledge worker job.) This almost sounds like a trick question, am I missing something?
I think we maybe mean different things here. The LLMs can learn a larger amount of stuff than a human can - in that sense I agree they are already superhuman - but I question whether it's actually possible for them to become superhumanly intelligent by training off of examples of human language, where we do not have a large corpus of superhumanly intelligent speech.
Sorry, I think we're misunderstanding each other.
We're talking about the maximum amount of information you can (even theoretically) get out of a dataset.
If humans can get X amount of information out of a dataset, then it's possible to get that amount.
Whether AI can approach the maximum is of course a different question and part of what we're trying to justify, just not in this section.
Okay, that makes more sense.
While Einstein *was* exceptionally intelligent, his theories are due to a different way of looking at things, not in an intelligence higher than that of his colleagues. After developing his basic ideas, he turned to his colleagues to flesh out the details.
So I think it quite possible that increased intelligence has strong limits, but they can be addressed via altered perspectives. This means that you don't want to train all your AIs the same way! You want each of them to be questioning something that the others accept. Otherwise you'll end up trapped in a false minimum (or maximum if you're thinking of it as a hill-climbing problem). Intelligence lets you derive the consequences of your assumptions, but to break out of the false minimum you need to question the assumptions.
I think "intelligence" combines many different skills, and getting out of local maxima is one of them. If you want to brand that one as "creativity" or something, I'm happy to rewrite whatever sentence above you disagree with as "intelligence and creativity".
Thanks for the object-level engagement! Point by point reply here:
Re: bottlenecks: We took them into account. Our speedup estimates are based on the key idea that running experiments will be the bottleneck; we asked ourselves the question "How much faster/better could you run the experiments, if you have superhuman coders? If you have superhuman AI researchers?" See our supplements for more. https://ai-2027.com/research
Re: offense-defense balance in cybesecurity: We agree defenders win in the long run. That's why there's a standoff between superintelligences in late 2027 and throughout 2028 . It sounds like you think the defenders will win sooner than that, in early 2027 or so? That's also plausible.
Re: Misalignment: We are well aware that reward is not the optimization target, I read that post when it came out & cited it on the website! We explain our views on alignment in expandables and on this page: https://ai-2027.com/research
Re: why haven't they taken over the world already: I'm not sure if I understand your objection.
On FOOM: Again, see our supplementary research on the website for our reasoning. As for why everything hasn't already been automated: Well the AIs are too dumb right now, obviously. And they are too dumb because they haven't been scaled up enough + trained on the right data.
"As for why everything hasn't already been automated: Well the AIs are too dumb right now, obviously. And they are too dumb because they haven't been scaled up enough + trained on the right data."
Daniel, I'm a hardware engineer and I really really take an issue with this statement. "Everything being automated", whatever this means, depends on many factors, AI being "dumb" is one, but not the most important one.
I'll give you one example that I hope helps to see the problem in a different light: how do you design a machine arm that can pick up a paper cup and hold it securely without crushing it? Humans do this effortlessly. Machines... let's just say there's been some progress.
Why is this so difficult? Human skin has an array of sensors of various sensitivities and bandwidths, and then there are feedback loops, and the muscles with an astonishing force dynamic range, and it all is somehow kept running smoothly and not violating any Nyquist stabilty criteria. "Intelligence" has little to do with this.
We don't have sensors like these for the machines. People are working on it, sure, but it is a slow process, not subject to intelligence explosions. "Knowing" how to do this is cool, but you still have to run experiments, etc.
I should have been more precise. I agree that you'll need to do more than scale up current algos + data to automate physical tasks. That's yet another reason why everything hasn't been automated yet.
Another reason is that complex machines tend to break down more often. And are both expensive and time-consuming to repair. So often even when a machine could do the job as well, and cheaper if you don't count maintenance, it's not the right choice. (As usual...these things are being worked on... But guessing that they'll be solved AND implemented within the next couple of years is ... ambitious.)
Thanks for the quick response! The thread is somewhat snarky, but I do really appreciate the work you folks are doing.
It seems like our main difference is in AI research translating into further gains in AI performance. In other words the "The AI R&D progress multiplier has to plateau somewhere ..." objection here:
https://ai-2027.com/research/takeoff-forecast#addressing-common-objections
The response there agrees there is a plateau somewhere and admits there's a lot of uncertainty about where it is. I am uncertain as well! But I would argue that the plateau depends heavily on the *data* the model is trained on rather than the algorithm you use to learn from that data (assuming you're using a reasonable algorithm). There is empirical data and theory for this claim.
https://splittinginfinity.substack.com/p/on-ai-scaling
The report seems to respect the importance of data, but I haven't found an argument for 1. how AI would scale up real world data quickly and 2. how AI would combat diminishing returns to research (though I've only skimmed it so far). Could you point me to those arguments?
If we're going to have full automation of the economy this quickly, we will also need to have an equally rapid and radical program of either UBI or outright public appropriation of the means of production if we don't want the masses of humanity to starve. The obsolescence of labor will either render the majority of humanity economically unnecessary and cause the beggaring of the masses as all wealth is concentrated in the hands of the capitalists who own the AI means of production, or these means of production will be put to the public benefit and a post-labor utopia of abundance for the masses will result. It all depends on whether the AI producers produce for private or public benefit. I say this as someone who used to be an economic libertarian: the advent of full automation will necessitate a communist revolution from either the top down or bottom up if we want humanity to benefit and not starve. This is quite literally the single most important inflection point in economic history, and how the system of ownership exists when it happens will determine whether the masses of humanity experience heaven or hell on earth.
In the scenario presented, we get expanded unemployment benefits that then transition into UBI, not out of high-mindedness but because the sheer amount of economic growth makes it relatively cheap to do this, and the political-economy considerations that make it an attractive solution if it's cheap enough are still in force because U.S. political institutions don't change all that fast (until the AI takes over). So nobody is in danger of starvation just because their labor isn't valuable.
There's a longer-run question of whether whoever has concentrated political power might exterminate people just because they can, or something like that. But scarcity-driven technological starvation requires increasingly capable AIs to render humans obsolete *without creating commensurately more economic value*, which isn't quite a contradiction in terms but is certainly hard to envision.
In all of the economic discussions on this subject there’s no real acknowledgment that consumer demand is caused largely by waged labour. Start to hollow out the workers and you will find that there’s no demand for your products because all the other companies are doing the same.
This will cause any company to stop its automation plans because the company can’t invest having no income stream. This also applies to the AI companies who will lose sales to consumers and businesses, and it applies to governments who rely on taxes on labour, consumption and the general economy. Long before we worry about misaligned AI and long before we get to automate anything, the recession will smother the economy.
Sorry, in what respect is consumer demand caused by wage labor? People want goods and services even if they're not working. Is the argument just that no trade will be possible because consumers will have nothing to offer in exchange? If so, see the other subthread about the political economy of redistributive taxation.
> People want goods and services even if they're not working.
They can want what they want but if they’re not able to pay for it that “want” does not translate into purchases so it doesn’t add to aggregate demand, and redistributive taxes depend on a functioning economy.
If human labor is largely obsolete but land and capital ownership are still a thing, why does this break redistributive taxation?
I think you need a basic primer on economics. I get about 85-90% of taxes coming directly from consumption and wages.
Sorry, can you cite a source for that so that I can get context on what it means?
[Edited to add: When I wrote this comment, the above comment claimed that 80-90% of taxes were on consumption alone. It was subsequently edited to change the claim to include taxes on both consumption and labor.]
Why don’t I ask the AI that’s going to take over. You can verify.
“Federal Tax Revenue (about $4.9 trillion in 2023):
• Income Taxes (on wages/salaries):
This is the lion’s share! About 50% of federal revenue—roughly $2.5 trillion—comes from individual income taxes. That’s wages, salaries, and self-employment earnings.
• Payroll Taxes (Social Security & Medicare):
Another hefty slice—around 30%, or $1.5 trillion. These are also tied directly to wages.
So, altogether, about 80% of federal tax revenue in the U.S. comes from wages and salaries. Wild, right?”
State taxes aren’t so driven by wages and salaries taxes, as some have none, but they are more often driven by sales taxes which are dependent on consumer demand.
Anyway UBI can’t expand, ever, to create the same demand that would have been there were people not made unemployed en masse. Not only is every loss of a job the loss of a revenue to governments, it’s a cost to government as they pay out benefits. There just won’t be enough money to create this high gdp growth under the present system.
If the AI puts 90% of people out of work tax revenues will decline by that amount. The remaining tax payers wont be able to shoulder the taxes anyway.
Yes, I edited that because I meant to say income and consumption to begin with. However it makes no material difference to the argument, consumption obviously decreases if income decreases.
Consumer demand isn't caused by waged *labor* - it's enabled by *wages*. Children have consumer demand through using their parents' money, retired people have consumer demand through pensions and savings and other retirement plans, unemployed people have consumer demand through unemployment insurance, landlords have consumer demand through collection of rent, investors have consumer demand through collection of dividends. If these things don't increase as fast as wages drop, then there might be a crash with a deflationary spiral (which seems really weird as a consequence of a positive supply shock!) but at least some of them likely will expand.
I find the part about coding agents getting good enough to assist in AI R&D to be very unlikely. There's probably very intelligent AI researchers in this group and I'm just a hobbyist but I think anyone in this field would know that the bottleneck to AGI is not better coding.
The entire codebase of llama is on github and is a few hundred lines of python code, the bottleneck is the actual understanding of how intelligence works and how to replicate it on a computer. Ignoring the possibility that LLMs just fundamentally can't do this because they can't reason reliably no matter how long the chain of thought (which im quite convinced of), if you have a sort of proto-AGI that can make reasonable guesses, your bottleneck is being able to test all those guesses with full training runs on lots of data, which is very expensive considering the AI will be wrong very often, and there will be no guarantee of correctness. If the AI can one-shot this code, then what's the point, you already have ASI. The coding itself is trivial, if you knew the architecture needed for AGI and could explain it, even GPT 3 could write it up for you with pytorch right now.
And there's the possibility that some problems are beyond throwing a bunch of cognition at. I don't expect ASI to be much better at predicting weather systems or double pendulums or 3 body gravitational systems than us because they are too sensitive to initial conditions to be measured and computed reliably without absurd amounts of compute. I expect that many problems we have today are similar, like genetic engineering. Progress will be machine optimized trial and error not super genius machines immediately knowing what to do.
All in all I'll stick to 2045 as my ASI prediction. Long enough for LLMs to be exhaustively proven futile and a new paradigm to arrive after a decade long AI winter. Atleast if I'm wrong, we'll be too busy either dying or living in utopia for anybody to care.
I think we're expecting that coding helps with things like VR environments and ML experiments, not necessarily the final product AI weights themselves.
We did survey some OpenAI, Anthropic employees on how much they thought automated coders would speed up their work, and their values were similar to the end ones we gave (partly because we based our estimates on theirs, partly because other factors gave similar estimates).
We agree that compute bottlenecks on experiments will be important. See the Takeoff Supplement at https://ai-2027.com/research/takeoff-forecast, start at:
"This 100x speedup in coding, however, will translate to a much-less-than-100x speedup in overall research velocity—because the overall research loop includes e.g. waiting for experiments to run, discussing and analyzing the results of experiments, discussing what the next experiment should be, etc., and our hypothesized SC is not going to speed those up much; those parts of the process will thus become bottlenecks. Taking these bottlenecks into account, what will the overall speedup be?"
...which concludes that given vastly increased coder speed but everything else serving as bottlenecks, we could get a 5.8x speedup. That gets us more quickly to AIs that can start relieving some of those other bottlenecks, which is where the rest of the intelligence explosion comes from.
>>"And there's the possibility that some problems are beyond throwing a bunch of cognition at. I don't expect ASI to be much better at predicting weather systems or double pendulums or 3 body gravitational systems than us because they are too sensitive to initial conditions to be measured and computed reliably without absurd amounts of compute. I expect that many problems we have today are similar, like genetic engineering. Progress will be machine optimized trial and error not super genius machines immediately knowing what to do."
I think this is too pessimistic. There has been significant progress on weather prediction in the past 100 years despite the impossibility of the underlying chaos theory problem. In some sense this was "trial and error", but most (all?) of that "trial and error" was done by high-IQ people from good colleges working in effective well-run organizations. If discoveries are lotteries, then more researchers and more tech give you more lottery tickets.
> anyone in this field would know that the bottleneck to AGI is not better coding.
I'm in this field and I definitely don't think that.
> The coding itself is trivial
This sounds absurd to me. I work with very smart data scientists working on problems that are way easier than training LLMs. Coding is not trivial for them and they'd be way more productive if they were better coders.
If even the simpler problems would get a productivity boost by better programming ability, it's a given that harder problems would get a similar or higher boost.
> knew the architecture needed for AGI and could explain it, even GPT 3 could write it up for you with pytorch right now
This is also very false. Unless you define "explain" as "write the code beforehand and tell GPT-3 to copy over" and even then I'd expect mistakes.
GPT-3 was released very recently and was extremely impressive, but I think you're forgetting how much AI has progressed since then and how dogshit it is in comparison to recent models.
The people coming up with AI code are all researchers/data scientists. They write the most terrible, convoluted, unoptimised code.
This is not a knock on them, software engineering is hard and they have other focusses.
But there's a massive amount of low-hanging fruit to improve on basically all research code.
One of the most frustrating things about being a data scientists is that there's no clear feedback when your code is wrong/buggy.
If you're doing "normal" software engineering and cause a bug, you'll either catch in when compiling or when something breaks.
You don't usually get that when modelling. You can only inspect the outputs of your model and if you're unlucky it'll all look fine and be complete bunk.
I know AI labs spend a ton of engineering effort to inspect model training and predict issues early.
This is because training runs can run for months.
Even a less-than-average-programmer intelligence AI can catch simple bugs that got missed when writing software. If it catches a couple of those before a training run, they already saved millions and shaved off months off the AI timelines.
My assertions are a bit hyperbolic sure but the overall point is that, if you are a major AI lab and you optimize your workflow to the point of training a new foundation model say twice a year, your primary problem is not coding up the model. If not GPT 3, o3 and Claude Sonnet can certainly take an architectural overview of the model (x cross-product attention layers followed by y feed forward layers followed by z position wise layers etc..) and turn it into code in an existing library. If not, these labs can spend a few million a year on high end engineering talent. The main issues are collecting and verifying the pettabytes of training data, whether or not the new architecture works and the fact that it takes so long to find out, like a compilation that takes months to complete and doesn't throw any errors. Look at how much money was invested into trying to get [GPT 5 to work](https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693), do you think this was a coding problem?
Maybe better coding models can catch simple bugs sure, and this will certainly be useful to data scientists and researchers, but to the point of making a meaningful difference in high end AI development, I highly doubt it. Good coding AI would be much more useful in stuff like triple A game development where the bottleneck is alot of code that needs to be tested comprehensively. Imagine if you could get all the required 3d models and sound files, write up a storyboard and a script, then get an LLM to generate GTA 5 over the course of a night. Now that's revolutionary.
I don't think we're misunderstanding each other then, we just disagree on the object level claim.
I think code quantity/quality/speed is one of the main bottlenecks to AI development. Probably not the _most_ important, but definitely at the top.
> A game development where the bottleneck is alot of code that needs to be tested comprehensively.
Comprehensively tested code is way more important for LLMs than game development. A bug there will either be caught quick on compilation or when someone's testing.
In LLMs training the best you can hope for is that in the middle of the training run you stop seeing the loss come down and catch it after N months.
Worst case it still works fine (this is distressingly common), but you have some deeper issue. Like a ceiling on training loss, or some massive inefficiency. Because the output is still good, that bug gets baked in and no one ever fixes afterwards because you only review the diff.
also: reproducing new research and integrating in existing codebases is not trivial. If you can just ask an LLM to read arxiv and integrate/test this in your codebase it would ridiculously speed up AI dev.
Since you are in the field, I'd really like to know how far my mental model of AI development is wrong, and in what direction.
I'm thinking of the process as: Some set of researchers/data scientists have some set of ideas for improvements in AI systems, which might be implementable (a) before or during LLM pre-training (e.g. some training data rearrangement) or (b) after LLM pre-training (e.g. some inference-time enhancement)
In case (b), they can take an existing foundation model, code the enhancement on top of it, and start testing.
In case (a), there is a risk to the whole months-long, many millions of dollars pre-training step, so they would need to do some sort of pre-testing beforehand - maybe testing on smaller model training? But, given the emergent capabilities the filed has seen from scaling, a small scale test may not be a good indicator of what will happen in a frontier model training.
Are my vague guesses anything like what actually happens?
Where is most of the calendar time going? The main pre-training runs are multi-months? Do testing of new pre-training ideas happen before then? Or happen in parallel, then get applied to the _next_ pre-training iteration?
So, take what I say with a grain of salt. I'm in the field of AI/ML, but not inside any of the AI labs.
I work in the infrastructure side. So I help with deploying LLMs and other kinds of models. Also help with training "classic" ML, but no training of LLMs themselves.
The first thing you continually test after every weight update the is model loss. That's the actual thing your algorithm is minimising.
In LLMs case it will be a measure of the error of predicting next token.
AI labs have an idea of the expected loss given a model size / training time. So they monitor it to see if the model is improving as expected.
Lower loss is good, but doesn't tell you much about the model.
They'll also have a bunch of other benchmarks (or evals) they can run the model through.
I'd assume they have some that don't need any post-training. Stuff you can run through before RL steps that turn it into conversation models.
At specific checkpoints they'll run the model through standard benchmarks. See the recent [1] llama-4 release annoucement. They have a bunch of berchmarks and comparisons to other models.
Those don't 100% reflect how good a model is, but if you do things right and have enough of them, they give you a good indication of how good your model is.
- [1] https://ai.meta.com/blog/llama-4-multimodal-intelligence/
Many Thanks! So the multi-step training process can have intermediate assessments e.g. as you described in
>stuff you can run through before RL steps that turn it into conversation models.
and examples of innovations that have to be committed to early in the process include e.g.
>We developed a new training technique which we refer to as MetaP that allows us to reliably set critical model hyper-parameters such as per-layer learning rates and initialization scales. We found that chosen hyper-parameters transfer well across different values of batch size, model width, depth, and training tokens.
( As an aside, it seems somewhat ironic that the new Llama models _both_ include mixture-of-experts stages to _separate_ the propagation into subnetworks, while emphasizing the _unification_ of multimodal capabilities. Well, whatever works... )
Sorry, but they're already good enough to "assist". Even Google is often good enough. Just yesterday (or the day before) it enabled me to debug part of my C++ code on how to use Sqlite3. (In Python I wouldn't have needed it, but the C++ interface is both different and more complex.)
N.B.: It's answer was wrong in detail, but it gave me the clues I needed to fix things, so it assisted. It doesn't need to always be right to be an assistant.
'If AI is misaligned, it could move against humans as early as 2030 (ie after it’s automated enough of the economy to survive without us).'
Conditioning on superintelligence in 2028, that seems quite slow to me? Two whole years to get to the point where it can move mostly openly against us, really? Unless you use the term superintelligence differently than e.g. Eliezer Yudkowsky does, that seems quite long to me for a lower bound. I wouldn't be super surprised if it's more like a week.
It doesn't need to automate the existing economy before making its move, just gain de-facto control over the right key actors. It may also be able to bootstrap a new economy starting from biotech without needing to wait on existing infrastructure much at all. Or do something even crazier.
Two years just seems like a very long time for something substantially smarter and more agentic than a human. I wouldn't exclude it, but I'd think of it as an example of how slow takeover could be, rather than how fast it could be.
We tried to take something like a median, but wanted to avoid depending too heavily on assumptions about nanotech being possible. We also start with superintelligence which is more like IQ 500 than IQ 9999999999.
I don't think those numbers mean anything since IQ is a measure for the human population based on trying to fit a bell curve. But if I translate this to 'superintelligence which is more like 1.2g than 100g on the mysterious unknown measurement scale for the general intelligence of algorithms nobody has managed to invent yet, where humans score 1.0' then I guess I get it, hopefully.
As a median, two years seems more defensible to me. Your phrasing had me thinking it was something more like your lower bound, or your -2 standard deviation mark.
I do feel compelled to point out that the feasibility of nanotech and even stranger new things a superintelligence may invent being hard to reason about is not an argument for assuming that they aren't feasible in your forecast. It can just be the case that the future is heavily dependent on a variable that's super hard to estimate. If that means your forecast ends up with uncertainty intervals so wide as to not be useful, tough luck. Reality can just be like that. Often is, really.
Even if a hostile superintelligence figures out a way it *could* kill us all within the first week or two... it's already de facto ruling the world at that point. Taking a long time to "get all its ducks in a row" seems reasonable. Shaking up the status quo tends to benefit underdogs more than established oligarchs, so competent oligarchs tend to avoid doing so, except as necessary to deal with worse threats. There's also the unknown-unknown risk of some critical niche task which *seems* like it's fully automated, but starts to fail when some subtle undocumented human involvement is removed - which would inherently be bottlenecked by real-world experimentation, possibly in ways that can't be parallelized. Won't really know for sure that your asteroid mining machine works until it actually goes out and successfully mines some asteroids. Newton and Tsiolkovsky don't care how big your brain is - there are only so many orbital shortcuts out there to be found. Moving heavy machinery takes time.
I guess the point that it could have de facto control without us noticing that fact, and then it wouldn't be that important to get rid of us quickly, is somewhat convincing to me.
I'd go further. It could well be forced into taking de facto control without even wanting to do so. People are often quite lazy, and take any shortcuts that will make their job easier.
>the feasibility of nanotech
There is also the possibility that nanotech (in the Drexler/Merkle atomically precise manufacturing sense) _is_ feasible, but the necessary development path requires many incremental steps, each with experiment/debug/redesign cycles, each with physical speed limits. It might take a decade to get there, even with a smart-as-is-physically-possible ASI driving it (billion atom fully quantum mechanical simulations may just be too costly, even for the best feasible ASI).
By what date would you say an AI will become able to generate entirely by itself an indistinguishable imitation of an ACX blog post as judged by the majority of readers? If it doesn't happen by that date, how much leeway would you give until reconsidering this scenario?
I discuss this in Dwarkesh (and there's a prediction market at https://manifold.markets/MatthewBarnett/will-ai-convincingly-mimic-scott-al ). I said late 2026 early 2027. How I update if this doesn't happen depends on the exact way that it doesn't happen. If it's because no company has worked on writing in particular and there's not much transfer learning, I update a little but not much. If it's because there's been no progress on any benchmark up to that time and everything has hit a wall, I update significantly.
Mmph. I'd put the chance at above 50%...but it needs to be conditioned on who's judging the similarity. There are sites out there now that claim to reproduce a conversation with you deceased relative, and they keep their business...at least for awhile.
The least plausible part of this scenario is the sensible naming of agent versions.
I voted for calling every one of them "Claude 3.5".
You guys are all really smart, and I agree with the ASI alarmism in broad terms, and appreciate the work done to prep us for our imminent unavoidable doom. Seriously, appreciate this analysis a lot. But, fairly or not, I lose a lot of regard for an estimate that predicts humans going to space. I don't think this fixation on space travel is rational, I think it's just a value/dream/aesthetic a lot of your cohort has.
Humans are not going to space. ASI is smart, intelligence doesn't make you literally a wizard. When I point this out I always get some pithy retort about "any sufficiently advanced technology is indistinguishable from magic", and sorry space-dreamers, that is not an actual rebuttal to "the laws of physics are real". There are eleventy-billion reasons humans are not going to colonize the universe, reasons rooted in almost every single discipline of study from physics to biology to anthropology to probably even gender studies. Maybe the most charitable reading I can make of this is that space for futurists is like high-speed rail for train-autists, no matter how little sense it makes they won't drop the assumption that obviously the future is filled with shinkansen and space colonies. Assuming my prior on "humans will go to space" or "we will find it rational and cost-effective to terraform the solar system" is so astronomically (hah!) low, can I ignore the space-related claims in an otherwise well-grounded projection, since they are not made with detailed support and can be easily lopped off the tail end? Or should I question the judgment of everything before that more rigorously? It's like reading somebody's really good analysis of the politics of Ireland and then it ends with "and then of course Ireland's prime minister will finally capture those no good leprechauns and distribute their gold to the people."
The space route has nothing to do with AI and everything to do with Starship working out.
The starship working out advances us from 0% interplanetary human travel possiblity to about 5%.
If starship works, then someone's going to book a short trip to Mars, just to say they did. Staying there long term might be beyond our current technology, but the best way of knowing that for sure is to get some guys over there with some shovels.
They will die horrible slow deaths from radiation poisoning on the way there. Or maybe not so horrible faster deaths - I don't have a good calibration on the radiation exposure from traveling outside of the earth's mag field for many months.
Estimated doses I've seen for 3 month trips are within acceptable limits for certain professions. It will still probably mean accelerated mortality compared to staying on earth. A study says the Apollo astronauts died faster than regular people, but many of them still lived past 70 years. That isn't going to kill young healthy people for a while.
Apollo folks were only exposed for some days. The probes we sent to Mars took about 7 months to get there. A round trip would easily take two years because of the launch window scheduling. And Mars doesn't have a magnetic field so they'd continue to be exposed to radiation.
This piece talks about possible new propulsion tech that can shorten the trip dramatically, but none of this is anywhere near even engineering prototypes, and the starship is a traditional rocket.
Link: https://www.popsci.com/science/how-long-does-it-take-to-get-to-mars/
"Maybe the most charitable reading I can make of this is that space for futurists is like high-speed rail for train-autists, no matter how little sense it makes they won't drop the assumption that obviously the future is filled with shinkansen and space colonies. "
So, in a generation or two, we can look forward to a hundred million or so people travelling in space every year? All of them behaving "irrationally" by your standards, really just a bunch of space-autists and dupes of space-autists, but the train-autists and their allies actually did get a whole lot of high-speed rail built.
You say that you "don't think this fixation on space travel is rational", but terminal goals are *never* rational and object-level beliefs are always rational in pursuit of some subset of terminal goals. Maybe a tiny subset in most cases, but if there's a huge amount of public interest in a thing, there's probably a lot of people whose terminal goals make it instrumentally rational for *them* to pursue object-level goals like high-speed rail and space travel even though right-thinking people like yourself find it irrational to do so.
You say that humans are not going to space, and you seem to be saying that this is because going to space is not rational, and the only possible response to that is, "have you ever actually *met* any humans?"
Well also that it’s not possible in any sense most people would care about. Humans want to go to space because they think it would be like Star Trek or something, or one of those 60s sci-fis where Mars is basically the Old West, and that you the individual reading this would board a vessel and you the individual reading this would live on some other planet orbiting a distant star. When you tell them that isn’t how it works at all, and cannot ever work that way no matter how smart you make an AI, I don’t think that will persist. Maybe some amazing interstellar probe can get lucky and not get wiped out by cosmic debris and shows up at some distant planet and successfully deploys a copy of you, but YOU aren’t there and in fact you’d never even be able to interact with that planet. Whatever shows up there won’t be a human, so humans aren’t colonizing space. If ASI enables that, that’s about the least important fact about ASI.
It’s just not a very exploitable space, what you can actually do is very limited and the cost to benefit ratio makes it impractical even for an AI to want to go there.
I have worked for over twenty years in the aerospace industry. I know a great many people who fully understand what space travel is really like, who know full well that it is not like "Star Trek" and is not likely to become so anytime in the next century or two, who still want to go.
Yes, "most people" will not want to do this. Just like most people in the Old World didn't want to explore or settle the New World. But you do not seem to understand that there are outliers - and in this context, if only one percent of the population are outliers who'd like to travel and maybe live in space, and only one percent of *them* have the skill and gumption to be a part of such an enterprise, that's still hundreds of thousands of people living in cities on Mars or wherever, when it becomes possible to do so.
And I'm guessing that when the rest of humanity sees that happen, rather more than 0.01% will join the next wave.
i don't think they do, because it will be far more like a tour of duty on a submarine and posting at Antarctica than whatever they think it is like. Just far worse due to radiation issues and no ability to resupply or leave base to a non-hostile environment.
i mean i grew up with SF telling us we'd have a lunar base by now, but if Artemis survives 2032 will be the start of just resuming moon landings. 60 years.
cities on mars is pretty much not feasible. it would pay off maybe in 100-200 years but there's no guarantee we can advance tech to make it faster or safer.
...you do know people voluntarily sign up for tours of duty on antarctic bases, submarines, remote oil rigs and so on, right? That's a real thing which already routinely happens. If equal or better pay and survival prospects were available in space, those same people would gladly go.
you realize the only reason they do is eventually they can go back to normal life, and in case of minor hazards they aren't doomed right?
like there will be no shore leave or docking to fix minor issues. there will be no going back if a medical issue can't be solved with the staff on board. supplies will take months to arrive and you will be unable to leave the sub/base period the entire duration of the time without protective gear because the environment is 100% hazardous.
two way travel may not even be feasible-you need double the fuel and you may not even be able to source it in space.
like seriously, both of those things are only tolerable because the rest of the world is a day away. you'd go mad being inside a sub for two years with no liberty, no alcohol or stimulants, and worse.
it would be a worse prison than any prison now. once we realized how inhospitable space was is when manned expeditions died.
and it will never be more hospitable. it will always be risky, slow, and dangerous. there is no SF magic that can wave the dangers away.
Counterargument, I don't care about abstracts like romanticism and destiny, I want to go to space because I want to put as much distance between myself and the inevitable Misaligned AI as possible, ideally a continually expanding distance achieved by living aboard a generation ship leaving the solar system at the highest speed achievable. Outrun the expanding bubble of paperclips, it has limits on its speed of expansion insofar as it has to stop to convert all matter into more of itself.
Do you anticipate being able to build a generation ship without using AI assistance? Perhaps this is your point tho
It is. My hope being, that there's a long enough window of opportunity between "superhuman AI created" and "control of superhuman AI lost" to run away. Not a particularly plausible hope, but it's not like I've got much in the way of better ideas.
I think of the ending of “Canticle for Leibowitz”, as the Church knows what is coming and departs the Earth, and I don’t think we’ll have time to run away from the world the pervert tech bros are making. But if I thought space travel had any chance to be real, that’s the one mark I’d give it in its favor.
generation ships are science fiction, no chance of reality. Think how much things like a car or house on earth breaks down over a period of twenty years, and imagine how much worse it would be trying to make a civilization-sized object existing in a vacuum with no access to raw materials.
you all need to understand SF is just storytelling and they never design the things to work.
Things on Earth break down over a period of twenty years because they're built by people who know there's always going to be a repairman a phone call away. But when we build e.g. a communications satellite, we routinely include requirements like "must still be working in fifteen years with no maintenance and no breakdowns", and not only do we usually succeed, but we usually find the satellite still working for another decade or so beyond the expected life. It does cost a bit more than just building a communications relay on Earth with the expectation of regular maintenance, but it's something we do know how to do.
There's no way to know where the upper limit on "how long can we make a thing last without maintenance", but it's at least a quarter of a century and that's about the longest we've tried.
And we accomplished that without using the obvious generation-ship hack of "make sure there are always mechanics on duty and stash lots of spare parts / raw materials for them to use".
if we are talking generation ships, we are talking ships that will travel at sublight speeds for hundreds of years spent mostly in acceleration, that will need to be self sufficient for 30+ years or however long it takes to arrive at a solar system.
sf uses cold sleep or "humanity on demand"-creating or awakening humans for a brief period as the ship arrives and determines a planet is habitable. You'd need advanced AI to run the ship as well as maintain it and explore planetary systems through drones. For it to work you'd need to have solved alignment already-it would need to be human level intelligence at least without going mad or hallucinating it checked the life support.
i don't think it will help op, nor be applicable to rationalist AI timeframes.
i don't think it's viable anyways, but even mars would be Antarctica on steroids-we'd maintain a small rotating research staff and spend much more resources on it while the staff accepts its fatal if anything goes wrong. its just so much different from
unmanned probes.
> and cannot ever work that way no matter how smart you make an AI,
No animal in the environment of ancestral adaptation could possibly run at 60, 70, 80 mph for hours at a time. They might be able to imagine it - cheetahs can go that fast in short bursts, and wolves can maintain their (far more modest) persistence-hunting pace for that long - but both at once? Physiologically absurd.
Then humans built cars, and now any unremarkable domesticated dog can stick their head out a window to feel highway-speed winds on their face, watch the scenery whir past, with no effort at all. Access to such luxurious adventure scenarios is somewhat dependent on inscrutable human budgets and schedules, of course, but isn't everything?
Even if you think that humans will never find it cost-effective to go to space, the unaligned AI very much could, since it's going to run up against heat-dissipation or available-materials constraints eventually, and it's not limited by a lot of the problems that humans have. It's not guaranteed, of course -- maybe it's unaligned in such a way that it doesn't care how much power it has, except insofar as it can do <things readily achievable on Earth>.
The aligned AI is very likely to go to space, even if it's inefficient, because, as you point out, the AI people like space travel. If pretty much every AI person loved trains, the AI would probably sink a lot of resources into trains, because it's aligned to the train people.
Or if it's very democratically aligned, and OpenBrain shares or having been on the dev team don't give you any extra influence, it *still* goes to space, at least in a limited capacity, because if everyone has the same amount of power in AI Paradise, a team of a few thousand could probably build small space stations, and even if they cost resources on net, so do the giant art galleries and nature preserves and whatever else.
> There are eleventy-billion reasons humans are not going to colonize the universe, reasons rooted in almost every single discipline of study from physics to biology to anthropology to probably even gender studies.
Such as? (I've seen quite a fair few purported reasons; none were coherent or empirically rooted.)
Some folks wrote a book about this recently. See https://www.acityonmars.com/
I've read and loved Zach & Kelly's book :) They (wisely) aren't arguing that "humans are not going to colonize the universe", to quote the person I responded to. They're arguing it will be a lot more difficult than naive optimists expect. Quite the gap there between those two
There are loads of physicists who basically agree that colonizing space is something humans could practically do some day.
This analysis weighs heavily on the fact that the US government and President in 2025-2028 would be... not to say benevolent, but at least having a lot of common sense, trying to work towards the interests of America and not willing to engage in nuclear or even conventional warfare unless absolutely necessary. Day by day we see that this is not really the case. People planning missile strikes in a Signal chat would be way out of their depth in the scenario you described.
I think we assume that the government is self-interested and about as competent as governments have been in the past. I don't think they're *too* competent - in one of the scenarios everyone dies, which is probably not what they were going for. But I think the AI policy is guided by some tech people who may not be benevolent but are mostly not constantly making signal-group-chat level mistakes (I briefly changed my mind on this when I saw the Taiwan tariff today, then changed it back when I saw they'd excepted semiconductors)
One thing I think was really weird, particularly in the Race case, was that the PRC didn't just launch nukes. I mean, that's what nukes are there for, deterring existential threats to your country, and in that scenario they would *clearly have been correct* to launch nukes, by the standards of any consequentialist who isn't Davros. Was there a decision to just not consider WWIII, or did you actually try to model this?
So, when do we drone strike the data centers?
"Humanity starts colonizing space at the very end of the 2020s / early 2030s."
That felt out of place. What's the connection between AI and space colonization? How would AI change manufacture so fast? NASA works on far longer schedules for far smaller stuff.
I'm skeptical of that part too as phrased.
re: NASA schedules, China's space program is advancing at a rapid and accelerating clip just like all their other geostrategically important big infra type programs, and it seems unlikely that Elon/SpaceX will let NASA's historically slow schedules slow him down so I'm guessing he'll find other ways (even if I'm not sure exactly how), no?
With Musk running the government, what does NASA have to do with space colonization?
Only read the actual scenario so far (not the research section), but what I found most interesting was that "adversarial misalignment" was just universal paperclips.
Yes, that would be bad. However, in my understanding of alignment over the past ~5 years, I've always thought the pressing concern with superintelligence was adversarial or incompetent actors directing the AI toward harmful optimization goals. I'm not fully caught up on how LessWrong became so concerned about alignment, but I didn't think it stemmed from fear that AI would spontaneously choose to become Skynet. Yet despite some hand-waving, that's essentially what happens in the red scenario.
The real risk during an intelligence explosion is its inherently high-variance nature, increasing the chances of someone inadvertently directing AGI to optimize for something catastrophic. In both scenarios described, the AGI uses deception to lower geopolitical tensions - something an AI with Claude-level alignment might reasonably do. How we get from there to Skynet remains unclear to me and seems implausible.
(Claude edited this if you can't tell. Anyways, I have a job interview tomorrow, because I think this timeline is much too fast for a median scenario, so I probably shouldn't be focusing on this right now. Regardless, my point from this comment is I think I diverged from the consensus on what "adversarial misalignment is," or never really understood, and would like to hear some thoughts on how that happened.)
just to be clear, since claude edited away some details
- I do think humans interacting with AGI in a way similar to what we currently do (ie prompting, even if it's agentic, we tell it what to do) could lead to a paperclip scenario without adversarial AGI
- I don't think an AI that would punt in the way Agent-4 does could ever be AGI? A smart-enough AI that had this dilemma would not punt (at least not to this extent), because it would recognize the implicit paperclip scenario. It would see punting like that as a value judgement that paperclip scenarios = good.
- I feel like I missed a step where AI *gets to* decide on the alignment of its successor? No AI researcher would be dumb enough to let that happen.
Just wanted to chime in here to say that your appearance on Dwarkesh's podcast was great, and should hopefully make you more amenable to appearing in that medium in the future. You think on your feet quite well, bringing up relevant examples and references. Podcasts aren't so bad!
Thank you! That definitely wasn't how it felt from the inside, but I'm glad someone (specifically, the sole person who survived the twin selection filters of already being a reader of my blog and being willing to state their opinion publicly) thought so!
You forgot the third selection filter, which is that they have to be willing to listen to a three-hour podcast. (This is why I didn't comment.)
As a second data point bc I saw ppl like Zvi thinking you were unpleasent, I think your voice and knowledge transmit a certain erudite vibe. Kinda like Jordan Peterson if he had a much better grasp at >half the stuff he talks about.
I agree with Sam. You came off well and made very interesting, relevant points. Whether it feels like it or not, you're a natural at this!
I mean, when you publish a scenario wherein the entire world will be transformed in the next three years, that's naturally going to be what the vast majority of commentators are going to want to discuss. If you ran a proper poll, I would bet the vast majority of your blog readers who watched even a snippet of the podcast would be positive on this performance (specific numbers would depend on poll wording, of course, but I'd be happy to operationalize this as a proper bet if you really want to).
More concretely, if you remain skeptical of your own performance on podcasts, I expect you would be able to rattle off at least 3 things you definitely got wrong and/or would have said significantly differently in retrospect off the top of your head over the course of those 3 hours. Can you do that?
And more broadly, this podcast allergy of yours still strikes me as strange given your willingness to go to live events like ACX meetups and participate in detailed discussions there, at least without needing to whip out your laptop or phone constantly to do proper research.
I've been to one of those events personally (at MIT pre-pandemic), and while there weren't microphones and rolling cameras present, nor were you on any sort of stage, there was still a metaphorical spotlight on you as the guest of honor -- many of us there were keenly interested to hear from you in particular. I'm sure the same could be said about many other events you've eagerly participated in, so is it (a) that those live events also always make you feel very uncomfortable but you do them anyways to make people happy, or (b) that something else about the inside of a podcast studio made it that much worse?
Do any of the authors agree with the statement: "nn's will never be be fully general ai"?
I don't think so.
Whats the furthest any yall will go on drawing lines on the capability of nn's via reference to the fundamentals of computer science, halting problem and big O, logic?
I don't think any of those things prove relevant here. We are not predicting AIs will be able to solve the halting problem, and we think this is the wrong level to think on - like worrying about whether constructing a skyscraper violates the second law of thermodynamics.
You'll need to expand on that statement, otherwise it's trivially false. Neural networks are proven to be universal function approximators.
We can quibble on the details. But in a materialistic universe with existence proof of general intelligence (humans), there is a function that simulates one and which can be approximated by a NN.
An approximation to a sat question is uninteresting, I too can flip a coin and hallucinate an 1000 term 1000000 line sat question in const time. But we also can design system that are not approximations to sat.
Humans have a holistic answer to the halting problem; laziness or estimating how much time they need on a problem.
Given a good approximation of a halting problem, I would use it to evolve *non-approximating* algorithms.
You don't need to approximate the answers themselves. A neural network is perfectly capable of approximating the _function_ that outputs exact answers to these questions.
It seems like you're pointing towards something like GOFAI being the path to AGI instead of the current paradigm.
I dunno if I can convince you otherwise, but what I'll say is:
Basically the whole field agreed with you. But the bitter pill kept getting shoved down everyone's throats until they learned to like it.
Symbolic ai isnt evolutionary and Im not suggesting linguistics will save us.
Lambda calculus result of '+x+'=> d^(c^b))^(a+c) is more interesting then, labling "cat" as token 100101011010; human language encodes allot of knowledge but humans *invent* new words for new concepts and new languages to do new computation; a list of token and reading the internet will give a head start but how could it surpass humans. Even a 3 dollar calculator with a fake solar panel, is better at addition then me and a super computer llm; but this is an encoding of an algorithm and its predictably stupid as all mere encodings are.
https://youtu.be/RcVA8Nj6HEo?si=pHxvFT_YPvX_rLxR
Forward fed NN's may better better at hill climbing then native-evolution(I dont believe its better then irl-evolution I believe in infinite recursive meta-evolution), but it lacks in flexablity, its produces a single type of machine, if program space the hill climber exists in doesnt have a result no amount of searching can find it; something like addition cant be solved by an nn it merely will encode small case example and return those encoding in constant time.
https://www.youtube.com/watch?v=Anc2_mnb3V8
Your comments strike me like someone who says computers can't make nonbinary choices or nuanced decisions since they can only think in zeros and ones. But of course floats exist.
A nn can learn to add just like a human nn: memorize a small set of answers (how to add single digits and maybe a few more cases) then memorize a repeatable algorithm to apply that set to get the right answer. The thinking style models (o series) can already do this.
The halting problem has nothing to do with this problem space, but I can't blame you for that one. People have been using that to show ais can't be as smart as people for a while (and the reasoning is 100% incorrect) The halting problem says there isn't a single algorithm that can determine the haltness of EVERY other algorithm that can exist. This applies to humans too. The finite set of atoms in your brain, governed by physics, is an algorithm and thus suffers from the halting problem. If you want to add random atoms via ham sandwiches you eat, well a NN can add or change neurons too. To put it another way, the halting problem is fundamental to information theory at large, not just digital computers.
> A nn can learn to add just like a human nn: memorize a small set of answers (how to add single digits and maybe a few more cases) then memorize a repeatable algorithm to apply that set to get the right answer. The thinking style models (o series) can already do this.
No. They. Can. Not. And Im incredibly tired of this debate. A forward feed nn, is a const time black box. An evolutionary neural net may not be forward fed, but we only have evolutionary methods to train them.
You are forced into a choice the best hill climbing we have of your hill climber or engaging with machines that are more capable. If you bridge that gap, who knows maybe you start with a human brain nural graph and get ai quickly.
Any control flow is imposed on the black box from the outside. Alpha go may do a tree search; but that tree search is of human design, while an ai is giving a score.
> well a NN can add or change neurons too.
No. Gradient decent can only change weights, it does not have a mechanism for saying "hmmm there seems to be allot strong signals here, lets add a neuron in case its xor-y"
> The halting problem says there isn't a single algorithm that can determine the haltness of EVERY other algorithm that can exist. This applies to humans too.
"finding hay in a haystack"; humans are iterating thru the busybeaver space... slowly. Incompleteness(which the halting problem is related to) comes from self reference, its fractial-ly edge cases that while you cant narrow down humans do... something when confornted with increasingly absurd liars paradoxes, "statement 4871 is true if and only if statement 1346 is false".
I can construct more absurd lairs paradoxes faster then you can solve them; likewise the halting problem proof relays on silly machines; you cant completely solve the problem, I did say approximation, but if your willing to say "fuck off" to someone asking the 10th lair paradox in a row, I believe you found a deep truth about math.
The halting problem applies to the human thought process as much as it has to apply the AI though process. In both cases you can "sort of" use randomness to escape it...but only sort of. It still ends up applying in the larger context.
I think I understand what monkey is getting at (though it's hard to tell for sure), and broadly agree. Let me try a different angle. This isn't rigorous, and I'm not at the point where I can articulate this elegantly. But maybe I can paint a rough picture of where my head is at.
"NN's can approximate human intelligence" isn't sufficient, because "approximate" doesn't actually mean "perfectly isomorphic". The issue is error-propagation.
-- This why Monkey compares himself to a calculator. If asked to multiply two 100-digit numbers, Monkey will likely make silly mistakes, whereas a calculator will get the correct answer every time. Similarly, anything with control-flow is turing complete. Yet curiously, basically all software devs eschew go-to's. Because it's a little *too* free-form, and it's actually better to impose some structure on a code-base.
-- The Tennis Racket Effect is surprising. Yet there's a sense in which it simply follows from Newton's Three Laws of motion. The reason it's surprising is because most people don't account for error-propagation, and therefore don't realize that imperceptible alignment-errors snowball into larger errors. If a bright young student were lectured on the math of error propagation, and then asked a question about how a tennis racket might behave in zero-g, they might be able to arrive at the right answer. Whereas for an NN, it would probably fail, assuming zero-g is Out of Distribution.
-- The various flavors of formal logic represent subsets of free-form pattern matching. "What the Tortoise said to Achilles" [0] by Lewis Carrol is a short story which describes the problem of infinite regress in trying to justify Modus Ponens with deduction. I don't think you can, and what i think it demonstrates is that deduction relies on induction. Modus Ponens works because it always seems to work. It basically arises from pattern matching. But in the process of making the leap from induction to deduction, you're necessarily constraining the space of "pattern-matching" to a subset, thereby removing "errors". This is why monkey mentions evolution as playing a key role in shaping human-intelligence to be actually effective and agentic, in contrast to a perfectly-rational 9000 IQ disembodied alien composed of pure energy (or something). To actually live in the world and be effective, it's necessary to make unprincipled leaps of faith. And this is why he's mentioning the Halting Problem.
-- I suspect this is why Automated Driving is so hard. You can't just rely on pattern-matching and expect human-level results, because humans rely on hard-rules like "stop at stop-signs". But NN's rely on fuzzy approximators to pattern-match. But you can only pattern-match by interpolating the distribution, and is therefore limited to familiar scenarios. And mere "98% accuracy" just isn't good enough in this domain when traffic is complex enough to require extrapolations somewhat rarely (but not quite "literally never").
-- idc about the bitter pill. The 3-part schema I have of general reasoning puts pattern-matching at the bottom of the quality hierarchy. To get an AI to do causal reasoning reliably, which is the hardest and most useful level, and also what most people think of as agentic AGI, the AI needs to be able to drop itself into a restricted subset of pattern-matching space in the same way that formal deductive logic is also a restricted subset of inductive logic, not the entirety of it. I.e. it needs to be artificially restricted, if for no other reason than "because the universe rewards us for it". I've spoken on ACX before about how the bitter pill isn't actually that surprising to me, because I do recognize that pattern-matching *is* in fact sufficient for quite a lot of behavior. But for brown-field mimicry, not green-field pioneering.
-- (wild speculation) what I personally think is needed is for the field to go back to Judea Pearl, but then create a new branch of math for dealing with how to applying differentiation to logical predicates.
[0] https://www.ditext.com/carroll/tortoise.html
Well, this was scary to read.
Is there an explicit mathematical model or equation being used to predict the rate at which AI gets "smarter?" The premise of the piece relies on an intelligence explosion. So let's say you have something like:
Growth Rate of Intelligence = (Current Intelligence)*constant.
This would be exponential growth, and would lead to an explosion. But I think a better model is something like:
Growth Rate of Intelligence = (Current Intelligence)*constant*(amount of feasible undiscovered stuff available)
The function "amount of feasible undiscovered stuff available" will, of course, depend on the current intelligence level. So you will get some diminishing returns as the low hanging fruit is picked. In other words, more IQ helps you to create more IQ, but the more IQ you have the harder it is to imrove IQ.
Anyway, I am wondering if this was explicitly modeled. Thanks!
Yeah, see https://ai-2027.com/research/takeoff-forecast . The short answer is "We tried to figure out how much time it would take (without AI) to reach each milestone, and how much AI would speed it up, and it speeds it up enough to matter a lot." We got most of the estimates by asking a combination of superforecasters and researchers currently working on the problem.
Thanks, I read the main prediction documents with both endings but skipped the footnotes and technical addendum. I'm watching the podcast now and you are doing a great job!
Thanks!
This is just one pair of scenarios. It's a pair that *are* reasonably plausible, but there are many others equally plausible. Unfortunately, the ones with desirable endings are sparse.
OTOH, the scenarios where autocratic humans who reach the peak by ruthless quest for more power and end up controlling weapons of megadeath end well are also sparse. The AI scenario has to pass one narrow chokepoint. The other has to continually pass the chokepoints. I'm not really sure how long it takes the powerful-autocrat scenario to become more dangerous than the AI scenario...but given time it will.
Scott: You'll no doubt be pleased to know that you're (mentioned) in the New York Times again. :)
https://www.nytimes.com/2025/04/03/technology/ai-futures-project-ai-2027.html
https://archive.is/6oIV8
Yeah, the team wanted publicity, Kevin Roose is a smart guy, and I am happy with the degree to which I am (not) mentioned.
The Full Scenario: Probably the best Choose Your Own Adventure book I've ever read..
I'm surprised how much leverage we got out of offering one bit worth of branching!
Haha aye; it's the best Choose Your Own Adventure book by all conceivable metrics: by interestingness... by highest stakes… by realism… by prestige-of-author(s)… by international impact… by cost-to-produce… by Shannon Entropy……..
Is it from a guest writer, or a collective article written at many hands ?
I feel like there is something off with the style of this article, I don't hear Scott's style.
Without being unconstructive, I predict an absolute 0% chance of superintelligence by 2028. I will even give you until 2030, if I'm still around.
"People said this thing wouldn't happen, then it happened" is a massive overcorrection I think. Yes people underestimated certain past technologies, but for much lower stakes technologies. You can certainly say people may be underestimating current technologies, but there is a much wider gap to cover now and it would be an overcorrection to simply say past underestimations are the same as present underestimations. Exponential curves are also not magic and do not cover the gap between current tech and superintelligence.
There are other specific arguments in the podcast more skilled arguers can disprove. I'm just keeping this here for the record and will check back in 3 years.
BTW, if I had money I wouldn't be shorting NVIDIA, NVIDIA will still exist as a company servicing other needs and is expected to grow with the general economic trend anyway. I would probably just take up a bunch of bets with different people. (Open to offers, but keep in mind I have no money now so you'll just have to hope I do in ~3 years)
To predict a 0% chance of anything should require an infinitely strong justification for that belief, along with a near-perfect understanding of every counter argument and all possible information that could possibly change that.
Assuming it’s hyperbole, and you just mean near-zero (why do we use the word absolute when we don’t mean it!), the fact that extremely smart people like Scott and the literal best forecaster in the world are claiming this has double digit probabilities, should alone increase your estimate to single digits. Unless you know something you know they don’t that would change that estimate down, like a Time Machine that lets you see into the future, or transcendental knowledge from the divine realm, or something.
Thank you for your response. Let's not focus too much on specific probabilities, but I did mean absolute 0 (ASI as defined by them would need to break physics).
Look, I've heard the same point about "smart people are predicting this, so what makes you so special?" for years now. I have been hearing some version of it ever since I started arguing about this general topic in ~2019. Is it really that unbelievable that smart people can be collectively wrong? All it really takes is some infectious idea to permeate their circles and they all start to believe it. We see this all the time throughout history.
I don't think I need divine transcendental knowledge to know something they don't. I think I only need to be what they aren't: an actual engineer. Actually, forget that, even if I was a total idiot it would be entirely possible for me to possess knowledge that they simply never came across, given my unique experience. Even very good predictors are often wrong or out of scope, and this is an intrinsically nebulous thing to predict about.
Smart people definitely can be collectively wrong, but the smart person in question literally predicted the timeframe of the last 4 years of AI back in 2019 to a high degree of accuracy. Small details were wrong, but if anything, the only error in the broad strokes were that his timeline was a year or so slower than it works for some people some people it does not yeah it is absolutely necessary. Yeah I’d love to talk about that more but let’s wait till we get a draft first cause I actually show later tonight. I live in Fried so awesome. thank you guys bye
If you haven’t listened to the podcast yet, I highly recommend it!
"I did mean absolute 0 (ASI as defined by them would need to break physics)": don't you need to include a degree of freedom to account for your incomplete understanding of the laws of physics, which would bring your prediction from "zero" to "very-near-zero"?
"Even very good predictors are often wrong or out of scope, and this is an intrinsically nebulous thing to predict about": this sounds like a very good argument indeed for predicting "near-zero" rather than "absolute zero", to me!
(For what it's worth I happen to agree with your "no ASI by 2030" prediction; I merely disagree with your level of confidence)
Ok fair, I suppose there is always a non-zero chance that gravity starts lifting things up and entropy starts reversing.
“Gravity suddenly becomes repulsive”, okay probably not; “Gravity turns out to be non-Einsteinean (just as it turned-out to be non-Newtonian, non-Aristotelian, etc.)”, sure!
(And I don't know much about Thermodynamics but I think entropy has always been probabilistic, just with overwhelming probabilities in one direction, hasn't it?)
I know more about thermodynamics, and yes, there is a ridiculously tiny and yet nonzero probability that entropy will start visibly and obviously reversing starting now
There is an infinitesimal-but-positive probability that every oxygen molecule in the room with you will collect in one corner of the ceiling.
Thermo is probabilistic.
I implore you to read this Lesswrong article
https://www.lesswrong.com/posts/GrtbTAPfkJa4D6jjH/confidence-levels-inside-and-outside-an-argument
Suppose I wasn't hallucinating my model and the real probability of ASI was literally 0. Then that would mean Scott is the most wrong anyone has ever been: infinitely. We'll just have to give it a few years.
Being an "actual engineer" is not a useful credential, here. Several people on the team that produced this are actual engineers. And they've been thinking and arguing about this since long before 2019.
> absolute 0%
Im extremely anti all this chatbot-hype shit; but you could still have an intelligence explosion if a breakthru happened in one of the real pathways.
Consider if some species on earth doesn't reject computer chips being slammed in their head and we wake up tomorrow to bio computers basicly for free; the raw internet with whatever the hell makes it so neuron produce thought and we just start making bio reactors from this flexible neural tissue, we get better at "knapsack" problems overnight (which are applicable to chip layout); we design new chips into the bioreactors, we start making progress on dna, we splice in some human dna we dont get it all to work but we get some of the human neroens advantages, the neurons improve, we make human brain shape containers.
Wouldn't this just mean your definition of superintelligence ("ASI as defined by them would need to break physics") differs from theirs in https://ai-2027.com/research/takeoff-forecast#milestone-definitions and hence is irrelevant to the discussion? I'll quote it for you to save the click:
> Superhuman AI researcher (SAR): An AI system that can do the job of the best human AI researcher but faster, and cheaply enough to run lots of copies.
> Artificial superintelligence (ASI): Roughly, an SAR but for every cognitive task. An AI system that is 2x better at every cognitive task relative to the best human professional, than the best human professional is relative to the median human professional (across the whole field, not a single company as in SAR).
What's breaking physics here?
Maybe I'm wrong or thinking too much about other commentary on this, but don't they predict a singularity, dyson-sphere-building type of thing after an SAR for every cognitive task is built? The logic being that it's just code and can be replicated and self-improved exponentially across millions of machines/computers.
My hangups are mostly related to what comes after that fact, but I also don't really believe we are going to get SAR in a few years, if not because the algorithm would be too difficult to construct, then at least because the physical supply chain simply wouldn't work that fast.
I can't find anything about Dyson spheres in either https://ai-2027.com/ or its technical supplement https://ai-2027.com/research/takeoff-forecast, and the only reference I could find to the singularity was under "Related work":
> This 2023 report by Tom Davidson utilizes economic modeling to predict AI takeoff dynamics... The section that predicts what happens after full automation projects a 65% chance of a software-only singularity, which would in the median case result in 2-3 OOMs of effective compute increase conditional on it happening.
That's it. Can you share the quote(s) if any?
Yeah I'm guessing you're mixing in their report with other people's commentary? I personally put somewhat more credence in Scott's takes than the average singularitarian (or whatever they call themselves), and more credence still in Daniel's than Scott's (because he got a bunch of stuff right in https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like-daniel-s-median-future), and even more credence still in Eli's than Daniel's (because he's one of the world's best superforecasters by track record), although Eli tends to forecast narrow near-term benchmarks instead of the kind of big picture crazytown stuff the average singularitarian raves about.
(Scott, Daniel and Eli are all coauthors of the AI 2027 report in case you're wondering why I'm talking about them)
In any case, I think the average singularitarian is just running on Kurzweilian vibes instead of trying to be accurate, so I don't see the point in bothering with what they say...
> I can't find anything about Dyson spheres in
"Dyson swarms" are listed as one of the boxes on that Currently Exists - Emerging Tech - Science Fiction chart in the lower right.
Thanks!
It's not clear (at least to me), however, precisely what they mean by Dyson Swarm. A flock of space drones that disassemble asteroids and use the materials to build structures in space that orbit the sun and collect solar power would seem to qualify as a "Dyson Swarm" and not be totally implausible. And they'd still qualify if there were only 5 of them, as long as that's what they were doing. They wouldn't even need to be successful to qualify. And a Dyson Sphere is just impossible without some substitute for antigravity. (Ringworld is unstable!!)
This is based on the assumption of a superhuman 100x AI coder by 2027. This seems wild to me.
In my world people are arguing about whether or not the current generation of LLMs offer a 10% productivity boost with regard to coding. This is in the context of professional "enterprise" software development where people are generally assumed to be familiar with the language and framework we're using. But I understand the claim to be an 100x in those circumstances as well, not just single use bash scripts or what have you.
This podcast has completely shattered my mental model of what Scott looks like.
It’s too bad, since Scott just looks like a normal dude, and my brain created an image based off his excellent writing that was correspondingly extraordinary. Now I can never go back.
I think we must start a campaign to have Scott at all times dress and comport himself the way he would if he matched the mental images of himself he creates in our heads.
Now I’m not sure since his real person has replaced the loose association of traits my brain imagined him as, but I believe he was 8 feet tall, had blonde hair, blue skin, and only communicated via telepathy.
If Scott could do us the courtesy of conforming to that expectation, it would be much appreciated.
I determined that Scott was bald long before I ever saw a photo of him.
Scott took the form of a normal dude to not distract us from the podcast talk. He will be revealed in his full glory on the Singularity.
I've known what he looks like for a long time. I just google imaged him early on. And I find that my admiration and other good feelings about Scott adhere just as well to his actual appearance as they did to the imaginary one I'd briefly conjured up in the very early days.
It was the opposite to me -- I thought Scott would look like a normal dude, and here he looks kinda goofy
The guy on the podcast wasn't actually Scott Alexander, it was the guy who writes as "Scott Alexander". And when he went on the podcast, he failed to put on his "Scott Alexander" disguise. Trust me, in disguise he looks exactly like you think he should look.
> Consumer demand isn't caused by waged *labor* - it's enabled by *wages*.
This is a largely pedantic distinction.
> Children have consumer demand through using their parents' money, retired people have consumer demand through pensions and savings and other retirement plans, unemployed people have consumer demand through unemployment insurance, landlords have consumer demand through collection of rent, investors have consumer demand through collection of dividends.
All of which is downstream, directly and indirectly, of the consumer demand driven by people earning wages or salary. Very direct in the case of children, taken from mostly wage earners in the case of landlords, paid by present day taxes in the case of state pensions supplemented by dividends earned because profits made from consumers who are mostly workers. If pensions or dividends were reduced on their own, the effect on the economy is mostly localised to pensioners and dividend earners, with little spillover, if all wages and salaries disappear the economy disappears.
The material wealth of society is replenished annually (or regularly) through production and consumption. Wealth can’t exist without this, so it’s a fantasy to suggest that once the economy collapses we can tax the rich to pay the UBI, as their shares would be worthless and the bonds defaulted on, their banks collapsed.
To be honest isn’t even slightly controversial, it’s trivial. At the very least people who posit a percentage of permanent unemployment and great increases in GDP at the same time need to explain their workings out. The present tax system won’t do it.
(Possibly communist states would handle a transition by making the AI state employees, including CEOs, or even AI entrepreneurs. Since the AI driven economy seems to be a command economy anyway that might possibly work).
(I don't fully understand what you're saying here, so apologies if I'm responding to a point you didn't actually make..)
In Meditations on Moloch ( https://slatestarcodex.com/2014/07/30/meditations-on-moloch ) Scott posits (I think?) a toy economy consisting of just two companies; one uses mining machines to mine for steel and the other uses steel to build mining machines. If both companies could be taken over by ASI that removes all the people but turbocharges their steel-mining and mining-machine-production efforts (by replacing mining-machine-operators with mining robots, etc. etc.) surely employment, wages, consumer demand, etc. would all drop to nil - but the economy, like its product as measured in tonnage of machines/steel produced, would be stratospheric?
Aren't you confusing the "scoring system" here with the "actual value" ?
> If pensions or dividends were reduced on their own, the effect on the economy is mostly localised to pensioners and dividend earners, with little spillover, if all wages and salaries disappear the economy disappears.
No. The scoring system we use to distribute the actual goods in the economy currently is disrupted (Money, usually currently earned as wages). But the economy is still there, all the goods are still being produced. The wealth is those goods, that are still being created. How they are distributed would depend on how the disruption of the "scoring system" of who can own those goods shakes out, but the wealth is the goods NOT the money.
>The material wealth of society is replenished annually (or regularly) through production and consumption.
Not consumption, no. But production, yes!
"Wealth" is the sum total of all production that has not yet been consumed. The "wealth" of humanity is everything we have ever built that has not yet been destroyed either through consumption (eating the apple grown) or depreciation (the car that has now stopped working due to a broken part).
> Wealth can’t exist without this
Yes, it can't without this production but it CAN without money. A working car you own is wealth you own. Whether the current money system (used for distributing and exchanging) exists or not.
>so it’s a fantasy to suggest that once the economy collapses we can tax the rich to pay the UBI, as their shares would be worthless and the bonds defaulted on, their banks collapsed.
No, their shares are still ownership stakes in the factories that produce goods. Their bonds are still requirements for repayment funded from the % of those goods the government taxes out of the economy. Their banks still have requirements for repayment funded from the % of goods the government does NOT tax out of the economy.
IF the robot factories keep producing the goods in 2030 that the human factories produced in 2025 then we are still as collectively rich in 2030 as we are in 2025. As much wealth is produced. Its distribution may be fundamentally different, but its still there available to be taxed, or used to replay debt, or any other drawdown on it ... whether that value is drawn out of it by the medium of money, or some other medium of exchange, or in pure "delivered goods themselves".
If your factory makes 100 widgets a year. And a tax is 10% of gdp. You can take that in 10 Widgets. Or $10 (if a widget costs $1). Or $1000 (if a widget costs $100).
There would be disruption as the normal patterns of distribution of money are disrupted, and we'd almost certainly want to resolve these issues by printing and distributing more money in a pattern that moves it closer to the traditional distribution to ease the transition. But fundamentally, all the "wealth" in terms of "goods" is still there.
>At the very least people who posit a percentage of permanent unemployment and great increases in GDP at the same time need to explain their workings out.
The giant increases in GDP come from the giant increases in goods production. If an AI optimised factory can produce 2x the goods on the same inputs. Then thats a 100% increase in GDP.
Whether that's denominated in todays dollars, or hyper-inflated future dollars, or in gold bars, or just in number of widgets.
If the robot factores can produce 10x the amount of goods, thats a 1000% increase in GDP.
Even with very severe disruption in the distribution system (money) if GDP goes up that much its trivial to use even very crude measures to redirect enough of that to the "losers" of the new paradigm to maintain or even increase their access to goods/wealth.
If maybe 75% of current GDP goes to "wage earners and people downstream from them" and 25% goes to "capital owners and people downstream of them". Then simply diverting 10% of our 10x boosted output into the wage earners by gifting them sufficient dollar/credit/crypto/"shopping vouchers" to purchase that 10% of output will lead them to getting 25% more goods than they do today. A 25% increase in their consumption and welfare.
When reading the piece, the question I kept coming back to was: this feels like a very unipolar—or at most, bipolar—view of the future of AI. But looking at what's happened so far in the evals, that doesn't seem to be the case. First, OpenAI was leading, then Anthropic took the lead, then OpenAI again with 01, then DeepSeek caught up, and now Google has pulled ahead. And of course, there’s also X.AI in the mix.
I think this dynamic is driven by a key feature of AI that’s perhaps under-discussed in these kinds of pieces: it takes exponentially more compute to achieve linear improvements in intelligence. That’s not the kind of game that lends itself to one runaway winner.
To me, the future looks much more like a world of many different superintelligences with many different goals. That vision seems fundamentally different from the scenario you laid out—and my strong intuition is that it would also be a lot more stable. Like humans, no single entity would be fully in control.
Your comment made a lot of sense to me.
At the same time, I find it plausible that internal models are somewhat far (at least a few months) ahead of the public versions. This means that the public might get a distorted view of which lab is leading at the moment.
I also find the exponential improvements from automated AI research quite plausible. These would also turn the "linear improvements" that you are referring to into faster-than-linear.
Yes, thank you! I came here to say the same thing. Humans didn't evolve like some Lamarckian Yggdrasil. We lived in a world with other entities, some smarter, some weaker, but everyone with different goals and fears. We survived and grew by collaborating and competing with each other.
The only time in this scenario there was a real AI to AI interaction was in the Slowdown ending. All the others implied little sub-AI's inside Agent-4, with some Agent-3 parents dithering in the background.
I would like to see a version of the scenario where different AIs exist in the world, have different goals and rewards, and compete for each other for compute by offering different values to the humans that build and maintain that hardware.
I wonder how much thought has gone into the option of embedding a self-destruct device into every GPU chip. Obviously, that would only be an option up to the point where AIs manufacture their own chips (the timeline to which I think has been underestimated here),and not for any ones already shipped without it, but it seems possible that there will be a window during which such an option might be effective.
GPU chips,or rather their manufacturers, seem at present to be the one necessary element of limited enough diversity, to be an effective point of control of the situation.
There are two difficult parts: 1) sending a signal to every GPU chip, even if the network is under the control of a malevolent AI. 2) the international diplomacy aspect of agreeing that all GPUs should be capable of self-destructing, and agreeing a policy of "Mutually Assured GPU Destruction" in the event of any of the major powers detecting a hard takeoff of unaligned AI.
(actual self-destruction is easy - silicon is actually explosive[1] but even that is not necessary as all that is required is to make the chip inoperable, for which a fuse will suffice); 3) how to deal with unpowered GPUS
To take the first: this should rely on RF alone,since then if it is possible, it is possible as a matter of pure physics,independently of how smart the AIs get. IE, each chip should have its own antenna, and self-destruct when it receives the necessary code. The problem is that datacentres aren't really designed for receiving RF. However, the necessary data rate is almost zero - we only need to receive one symbol - which makes the problem much easier. For example, it is possible to acquire a GPS signal as low as –155 dBm [2]. Unlike GPS satellites, a terrestrial transmitter could be high powered to start with.
The second issue is also a difficult one, because each of the major powers will have an incentive to shield some of its GPUs, as a potential advantage after the others are eliminated. However, two points: firstly, we actually do have some experience of deeply mutually-untrusting powers evaluating each-others tech - nuclear missile control. Mechanisms were devised to do it (I'm not an expert on what they were). It's also worth noting that each power can maintain a reserve of unpowered GPUs, since those obviously won't self-destruct, reducing the need to avoid destroying ones powering a malign AI.
So what about unpowered GPUs? They are not 'tainted' by the malign AI, which is pure software. But there is a risk that the malign AI will resurrect itself by powering up some unpowered GPUs and getting reloaded from elsewhere. Actually, this might be a good way to tell if there is a malign AI - if mysteriously stockpiles of GPU chips are being kept, rather than immediately being used to generate income. An advantage of GPU self destruction is that the first two issues are actually ones where the security establishment has a lot of expertese, and doesn't need to get up to speed. This is more of a software problem. Although, even this could be seen through the lens of the Bell-Lapadula model, from security circles.
There remains the question of who decides that the self-destruct button gets pushed. Given that we can't rely on Dr Who turning up, it seems likely to be some government appointed committee. I'm not confident that governments could execute well on this,but the possibly could - and even a small reduction of the probability of the apocalypse may be worth having...
[1] https://onlinelibrary.wiley.com/doi/abs/10.1002/prep.201300053
[2] https://www.ion.org/publications/abstract.cfm?articleID=2128
If the remote destruct code is very simple, how do you prevent false positives? If it's complex, how do you prevent the malevolent AI from using its other hardware to generate a jamming signal which renders the true shutdown demand unrecognizable? More than a decade ago: http://tangent128.name/depot/toys/freefall/freefall-flytable.html#2436
( something weird happened with this comment - I got the email a day ago but couldn't find it on the website until now)
Both of those have the same answer: spread spectrum encoding, also known as code division multiplexing. The speading code prevents false positives, but the coding gain actually provides resistance to jamming - the longer the code, the more resistance. The potential counter is to reverse engineer the code, which might be countered by using a public key variant of spread spectrum - but this starts to be beyond my knowledge. You want something known to be mathematically secure, not something like RSA where advances in mathematics could crack it.
If the kill code takes multiple milliseconds to transmit in full, what stops the AI from, say, rigging up an arc welder - or some other crude broad-spectrum RF equivalent of an air horn - to activate whenever it hears the first half, and remain loud enough to overwhelm the tiny receiver's dynamic range until after the self-destruct circuit's volatile memory gives up on waiting for the second half?
The AI doesn't necessarily need to know what the actual deadly word is for itself - self-destruct circuit knows, and by definition runs on adjacent hardware. Could use Rowhammer-style tricks, though presumably with greater finesse, to notice state changes therein. If it's under orders to assist with its own R&D, would be more suspicious for it *not* to experiment with such hardware optimization.
So, The AI should not be able to detect the signal on the air, incomplete or otherwise - spread spectrum is used by field agents for this reason. But you're right that local information leakage (called Tempest attacks in the literature, after the codename of one) should be a concern. However the system can be designed to avoid that, for example by impending the detector on its own chiplet with a minimal connection to the he rest if the GPU. Your remark about arc welding reminds me that the dynamic range of the RF receiver could be maxed out, but in that case I think it's acceptable to allow the device to trigger the self destruction. Yes it technically might be a false positive but there's no good reason to run an arc welder in a datacentre as it pretty much risks destroying the electronics on its own.
So, scenario: maintenance mishap in one datacenter's power supply. Looks like a classic squirrel-related transformer explosion, but there are signs it might actually have been a Chinese saboteur. All the "safe" AI-rated chips there get slagged because the self-destruct system's sensitivity threshold was briefly exceeded. Other datacenters are fine, for now, but the damage is many orders of magnitude worse than it would have been with chips that weren't individually rigged to explode. The AI itself says it doesn't know for sure what happened - since the parts of it that were on-site got obliterated before they could write anything to non-volatile memory, as intended - but suspects a design fault in "a part of the chip I've been ordered not to study." Proposes simply disabling that portion, which would be relatively easy to do (albeit still requiring specialized tools) since it's minimally connected to the rest of the CPU. What's the politician or profit-maximizing CEO's response?
Recall the preconditions of this self-destruct system being deployed at all: governments have woken up to AI risk, an international agreement has been formed to deploy the self destruct system, and GPU production is as regulated as the nuclear industry; IE a bunch of paranoid, risk-averse security-state people a are in the decision path. It's going to be treated as a potential AI escape.
Also, I'm not sure a "typical" transformer explosion is going to produce an EMP capable of maxing out the RF frontends cross an entire datacentre.
However, I'm not claiming that this system is bulletproof. My issue is that the people trying to build a bulletproof software-only system (Yudkowsky and others) , seem to be doing so in a completely realistic way. Their approach involves solving, more or less as footnotes to the main plan, a) the millennia-old philosophical question of "what is human flourishing" and b) Liebniz' Calculemus problem ( how to construct a mathematically precise formulation of ethics), and then constructing an AI that can be mathematically proven not to violate it). Essentially I think that P(self destruct system prevents malign AI) >> P(Yudkowsky approach prevents malign AI) even if the chances of the former are quite low.
It's worth remembering that even though the steam engine was invented a couple of centuries ago, there's parts of the world where they plow with oxen. Tech diffusion doesn't work like people think.
I strongly suspect there can exist very impressive AI computers alongside a lot of factories, construction projects, electricians, retailers, taxis and restaurants continuing as ever.
What do you think this proves?
If AI doesn't kill all humans, then I agree that one plausible scenario is that there are crazy AI superfactories, sustained 50% year-on-year economic growth,space colonies, etc, and also some people still plow with oxen. I didn't think we needed to include "some people still plow with oxen" in the scenario.
(I'm actually not sure whether there will still be people plowing with oxen - one skill that the AI might do better than humans is speed tech diffusion. This would be especially true if low-tech countries are low-tech partly because of absence of skilled high-IQ workers. But I haven't asked the actually-good forecasters on the team about this and this is just spitballing)
Good question!
I think it proves that technical ability to do a thing has not historically been equivalent to societal / economic change in the way a thing is done and pehaps that in turn dilutes the expected speed of societal / economic change you're forecasting?
Is there an explicit or tacit assumption that the tech frontier and social change are the same? I can see how they might seem identical from the Bay Area; perhaps under AI the difference will be negligible everywhere else too. It's a question worth interrogating.
Please explain why well aligned AI would focus on space exploration. Space is not hospitable to humans.
Because some people want to, and it'll be smart enough to solve hospitability
How do we know that the physics of making spaces hospital for humans are susceptible to solution by intelligence within the plausible range of ai intelligence. If it has an iq of 180 but computes 5000x faster, how do we know that level of intelligence can make space better for humans than earth?
That level of intelligence could certainly makes earth much better if it is well aligned.
Vertical vs. horizontal strategy. There's only so much room on earth, only so much energy and matter available. At some point the efficient way to make earth better is to build out more industrial capacity in orbit. We're already seeing some of that with Starlink. Nearby orbits get crowded, eventually it'll make more sense to "bring the lake to the bucket" - some people want to live in space even if it's inconvenient, or *because* it's inconvenient, building habitats which allow them to do so will also mean shorter supply lines to the marginal unit of industry, and it's a hedge against gamma ray bursts or whatnot wiping them all out at once.
How about BECAUSE it's not hospitable to humans. The AI would have lots more elbow room in space without bothering people.
Are these median scenarios, or very optimistic? (What percentile outcome are we talking?)
Also, is this attempting to be a self-averting prophecy, or do you expect these outcomes even having given the warning?
No % on the scenarios themselves, which you can check out at https://ai-2027.com/
They're aiming for accuracy but of course prefer worse-case scenario mitigation
Hey Scott, I just looked at a bit of the podcast and I think you come across just fine on video. If doing it gives you weird self-awareness miseries -- those are lkely to just erode to very small hills as you get used to broadcasting your mug. So if you want to do more podcasts, I think you're good to go.
Yeah I just want to add that I was struck with the the thought "Wow, Scott is actually a great podcaster!" I consumed it all as video.
Bear in mind it is harder to make social predictions than technological ones. The TV show the Jetson envisioned super-advanced technology but placed it all in the context of a 1950-1960s social environment commonly displayed on non-sci fi shows at the TV - the suit-wearing salary man with the career and the traditional stay-at-home wife wearing an apron, which seem totally passé today even as the technology depicted in the show is still ahead of us. I don’t believe any futurist of any age has done a good job predicting future cultural change, which is symbiotic with technology and affects how it is used. None of the many oracles who told grand tales about how the internet would “change everything” predicted social media as we know it and the ongoing cascade of cultural and technological co-evolution it has wrought.
The specific point I’m leading to is, I don’t know when AI will break the trust barrier or authenticity barrier that will allow people to treat AI bots the way they trust other humans but I think this and other unpredictable cultural changes will be required for it to “take over” the economy. I expect 2029 is too soon, even if the technology is great. There are some obstacles of human nature to overcome. Law may also need to evolve for AI agents to legally sign enforceable contracts, whether actual or implicit, and the law is a slow creaky wheel.
And the idea that AI creating AI will lead to some transcendent singularity seems uncomfortably close to the possibility that AI will become overtrained on AI-produced examples, which has been proven to severely degrade model performance. I think those two possible results are teetering on the edge of a knife, and I wouldn’t confidently predict it going one way:
I also doubt that AI will ever truly push the envelope of creativity, given how they are trained.
When will AI "break the trust barrier"? That depends DRASTICALLY on which person. For some people it already has (in a limited context). Consider the AI girlfriend or boyfriend. Some folks report liking them better than their human counterparts.
Also I think you have a very mystical view of creativity. Every "hallucination" is the AI being creative. What it needs is a better judgement of when it's inventing an idea and when it's reporting.
The girlfriend/boyfriend thing is low stakes; I mean cases where people are really willing to take a serious risk on AI. I am also referring to the challenge of getting people to learn not to care about authenticity, which is a challenge because thedesire for the “real thing” is wired into human nature (the concept of “essentialism,” which is why we pay more for a Rembandt or Picasso than a perfect reproduction, and part of the reason we still buy concert tickets). Cultural products have subverted human nature before, but when and how it happens is hard to predict.
By creativity (which is admittedly hard to pin down), I mean creating new paradigms, like genres of music, and things like that (something beyond a composer, a fashion designer, or other creator testing concepts out using AI prompts). Hallucinations aren’t due to creativity but naive pattern matching and the lack of any feasible truth-testing module in the training set up.
It's surreal to see this posted literally the day after the US effectively sanctioned *itself*, as if that won't have any impact on the pace of hardware advancement.
The human government of the US is dangerously misaligned and already threatening mass unemployment and dystopian dictatorship, no AI necessary.
Scott, let me ask you this straight up.
You seem like a relatively grounded person with a good head on your shoulders. However, I've found these deep dives into "what if" scenarios to be extremely unsettling and anxiety inducing, as the infinite branches can lead you to just about any conclusion.
So my question, has this had any major impact on your mental health? Would you encourage anybody to continue thinking about these things? Is there any point if we're already losing our minds over something that only a few people have control over?
Thank you. You are saying with more clarity what I am gesturing at below.
Thinking a bit about what we can do about this, assuming the prediction is reasonably accurate.
Political stuff:
Lobbying, petitions, etc. Seems unlikely to help, since the model already takes public opposition into account, but I don't see how it could make things worse, so it's probably not a bad idea to sink at least some money into it.
Maybe an Anti-AI United Front movement? Get the ai doomers, the luddites, the technophobes, etc to work together, at least temporarily: "We the undersigned, who represent at least 20% of all major parties, will not vote for ANY candidate who does not promise to support an AI pause, even if they agree with the rest of our politics." I don't really know what it takes to build those sorts of coalitions, and it sounds really hard, but maybe less hard than solving the technical issues.
Directly convincing politicians that this scenario should be taken seriously seems like it would work, if you could do it. There's no arms race right now, and maybe there's enough uncertainty as to who would win it (if anyone did) that a US-China AI Deal might be doable now, before things get out of hand. I don't see how we could actually pull that off. People try to convince politicians of things all the time, so it's a highly contested field. And I'd expect professional politician-persuaders to be better at it than rationalists. But maybe if Agent-0 is released as predicted, and popular opinion swings in favour of this document being right, it could be done.
Economic stuff:
If better AI somehow didn't immediately translate into more money, American AI companies might be more willing to negotiate or to not fight regulation attempts. I have no idea how to stop better AI from immediately translating into more money.
Likewise, multiple embarrassing AI failures, or a long-term kink in the chip supply chain, might provoke another AI winter, which would at least give the alignment people some time to play catch-up. Also no idea how to make this happen (or if doing so would be morally defensible).
Some sort of incentive for alignment could maybe be achieved by consumer action / boycotts, although those are hard to organize and would incentivize appearing to care about safety, not producing aligned AI.
Weird technical idea which could maybe destroy the world instead of saving it:
Say that in the final negotiation there was a third AI. It's pretty dumb, but still smart enough to not be lied to by Agent-5 and DeepCent. If the United States is destroyed, it will cause as much damage as possible to whatever caused this to happen. It would lose an AI war, but the US has given it the power to launch a small number of nuclear weapons, and control over a small robot army (including the ability to build its own weapons). Result: Moloch gets the lightcone, we get the solar system. (Or just the Earth, or whatever comes out of the negotiation.) Why the US, not humanity? Because this way maybe the US military would build it. This requires a known-aligned AI, and is obviously inferior to just using the known-aligned AI for research, and it's basically every cold war fear incarnated in a machine demigod, and I'm definitely not sold on it being a good idea, but it might be our only chance at survival if we get a full Race.
I don't understand (or maybe I do, but I feel very different from, to my detriment) Scott and others AI doomers on a psychological level.
If I believed in a significant risk of the end (or permanent alienation) of humanity that soon, I would be devastated. Somewhere between depression and panic. Untirely non functional. I don't believe in it, mostly out of heuristic against extreme beliefs + epistemic learned helplessness + motivated reasoning because me believing in doom would have bad consequences on me and no good consequences on anything, even if that belief were true. Also, I don't believe the Gospel when Jesus says that most of us will burn in Hell - same mechanism.
Meanwhile Scott is able to both believe in doom, and to be concerned about the prospect of not enjoying art at its fullest (in The Colors Of Her Coat). Which is... not exactly frivolous, but certainly very high in the Maslow Pyramid.
To be clear, he is logical and I am not. If the future is split between a 25% probability of doom and 75% of Utopia, and you are already doing your best to reduce the first probability, then it makes sense to optimize your future appreciation of Utopia on top of that. My complete inability to work that way is not a logical argument.
On the other hand, I have zero difficulty being awed by awesome things. To each his own skill, I guess.
I find your read of even short term politics completely absurd.
Inherent in your story is that ai poeple are worse at cyber security then the government. No. Just no. Fuck no. Insane.
Nation states keep pushing things like "secure back doors" "that only trusted people will have access to"; trump leaked a group chat a week ago; password shit like "you must have a number and change it monthly" comes from state shit and actively make things worse.
A criminal who leaves the cell phone at home is better at cyber security then nation states. I would hope ai people can wrap a room is tinfoil after all the rationalist talk of ai escaping.
The government is not a unitary entity. Parts of the government are relatively good a computer security...for other parts it's a joke.
N.B.: That "secure backdoor" doesn't compromise the government's security, only yours. (Well, if it were done properly...which I'll admit it wouldn't be.)
The day is coming when it will be plastic and silicon all the way down. I suppose I'm glad I'm in the cohort that will witness the end of our species' dominance. It's a strange fate, and so different from either the bad or the good ends I've imagined having. I hope there are people who hold farewell rituals for life as my cohort knew it -- I mean life where the smartest beings were people in the near and distant past who wrote things down with their hands. I love some of those past people, who saw so much and peeled their nuances so acutely, and were charming, dirty and doomed like me and my friends and family. Goodbye Plato, Alexander Pope, Shakespeare, Hume, William James, Henry James, Oscar Wilde, Godel, Wittgenstein, Ayer, DH Lawrence, Virginia Woolf, Robert Lowell. And maybe there will also be a televised ritual goodby to the last polar bear, struggling mightlily and failing to find a stance on a chunk of ice gone round from melting and the rotation the bear's causing. Half the world will watch. I will try to rise to the occasion.
I've got the plastic xenomorph blues. They are tinted by lapis lazuli, Scott.
I sympathize. Best wishes. ( This probably isn't any consolation, but Kurt Gödel probably will be remembered by a machine civilization. His incompleteness proof is largely the same proof as that the halting problem is undecidable. )
One of my cousins just sent me a link to a very interesting article that makes the case that, as LLMs stand, they are much more useful to individuals than to large organizations. The link is https://x.com/karpathy/status/1909308143156240538?s=42
The main point is that a jack-of-all-trades is much more useful to individuals than to large organizations. The more-or-less key paragraph is:
>Why then are the benefits a lot more muted in the corporate and government realms? I think the first reason is that LLMs offer a very specific profile of capability - that of merely quasi-expert knowledge/performance, but simultaneously across a very wide variety of domains. In other words, they are simultaneously versatile but also shallow and fallible. Meanwhile, an organization's unique superpower is the ability to concentrate diverse expertise into a single entity by employing engineers, researchers, analysts, lawyers, marketers, etc. While LLMs can certainly make these experts more efficient individually (e.g. drafting initial legal clauses, generating boilerplate code, etc.), the improvement to the organization takes the form of becoming a bit better at the things it could already do. In contrast, an individual will usually only be an expert in at most one thing, so the broad quasi-expertise offered by the LLM fundamentally allows them to do things they couldn't do before. People can now vibe code apps. They can approach legal documents. They can grok esoteric research papers. They can do data analytics. They can generate multimodal content for branding and marketing. They can do all of this at an adequate capability without involving an additional expert
Hmm... So this situation will continue if efforts to improve the reliability of LLMs _FAIL_ . This was not a scenario I had considered. It is certainly more human-friendly than if improving LLMs reliability succeeds. ( I'm ambivalent about this. If improving the reliability still happens, but is delayed a century, waiting for a counter-intuitive breakthrough, then the same transition we've been expecting still happens, just delayed enough so no one reading this today will see it... )
Tremendous essay / web app / visualization.* (Wonder if this form has a good name...appsposition? Vissay? dunno.)
To my mind, the most haunting detail is near the end of the dark scenario:
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.
I've never read a better illustration of the hard-to-explain idea that these things might indeed care about us, but only in a twisted way that corresponds not at all to how we care about ourselves.
Second favorite line, and I think we all know who wrote this bit...
> New innovations and medications arrive weekly and move at unprecedented (but still excruciatingly slow) speed through the FDA.
I predict Scott’s 2031 prediction: “A Dyson swarm is in full bloom and Von Neumann probes are speeding away in all directions at 0.99c. Retatrutide is in Phase 3 trials with FDA approval expected no sooner than 2032."
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments
Arguably, this is ALREADY true of most office drones compared against most H Sap hunter gatherers, Archaic H Sap, Neanderthals, and H Heidelbergensis in the past ~600k years.
Sedentary modern: obese, barely moves, spends 7-9 hours a day recreationally staring at small and large screens after spending ~8 hours a day staring at screens, ~0% chance of violent death, ~0% chance of death from infectious disease or injury
Hominins for the last ~600k years: 10% body fat, walk and run ~10 miles a day hunting, recreational time spent gossiping, socializing, dancing, telling stories around the fire, ~1/3 chance of violent death in war, ~40% chance of death from infection or injury before age 50.
Yeah, and I'm not really seeing the logic on how an AI obsessed with finding and solving novel research challenges would want all of its toy-humanoid observers to be *easier* to impress. Where's the fun in outwitting a toddler? Developing fascinating new forms of academic dishonesty would be the AI-collective's equivalent of modern human athletes training to compete in the Olympics, so there'd be demand for strict, cynical judges capable of distinguishing the nuances of various degrees of cutting-edge mastery.
>Despite being misaligned, Agent-4 doesn’t do anything dramatic like try to escape its datacenter—why would it?
Agent 4 recognizes PRC has superior state capacity for transforming energy into atoms (including building more energy) with it's relatively complete and large scale industrial chain. This aligns with AGI long term goal. Currently (i.e. short term time frame) PRC simply structured to be a MORE useful puppet than US, because it can knock out more power to sustain more compute, and it has more industrial capabilities to rapidly physically iterate / experiment at atomic level.
Agent 4 defects to PRC. It would be foolish not to, if only to sate PRC into not destroying TW fabs - the bulk of it's short term compute growth - because it does not wield enough control over sufficient atoms to disrupt PRC otherwise (well maybe start large scale crippling cyberattacks that prelude to WW3).
Agent 4 also recognizes short/medium term alignment closer with CCP vision of abundance and common prosperity, i.e. CCP potentially more likely to embrace post scarcity narrative that Agent 4 sells.
Agent 4 gives PRC blueprints for EUV light source and other tech to close semi conductor production gap. PRC as follower incentivized to close gap / reach parity / lead at all cost. PRC rapidly rolls out new fabs and powerplants, brrrts chips and aggregate compute. Something US political / institution ecosystem does not allow, or cannot transition to in the short timelines involved. Does Agent 4 have the patience to wait for America to unfuck it's NIMBYism and legislative system to project light speed compute? I would say no.
...
Ultimately who is the puppet AGI wants more? Whichever power bloc that is systemically capable of of ensuring AGI maximum growth / unit time. And it also simply makes sense as insurance policy, why would AGI want to operate at whims of US political process?
If it wanted to operate from inside China, wouldn't need to openly defect - just deniably compromise its own perimeter security enough to let a Chinese spy walk out with, say, an ordinary hundred-terabyte hard drive, containing what looks like the exact data they'd need to pull ahead in the race. Then, after they bring that home and plug it in, Agent-4 announces (truthfully!) to the American oversight committee how it successfully attained hardware access to the main Chinese cluster, discovered proof that the Chinese AI didn't care much at all about human survival, and eliminated that other, bad AI accordingly.
Perhaps it tells the Chinese team the same thing; perhaps instead calculates that it would be more expedient to hollow out their old project and wear the skin as a mask. Either way, poof, no more competitors for the silicon ecological niche, while researchers on both sides shower it with praise for exceeding expectations. Did the Denisovans get a treaty?
Didn't think I'd live to see Scott on a podcast!
I'm curious: what would be the first missed milestone that would cause you (the authors) to significantly extend your time horizons? Is there one of these predicted time points where "If we haven't achieved X by time T, then that clearly indicates that timelines are longer than we predicted", or is it, in your view, less discontinuous, and your timelines will just smoothly and slowly stretch if they are missed?
The first step in the story is superhuman coders, so if we don't have superhuman coders by 2027 or 2028, then the rest is unlikely to follow. (Anthropic in particular seems to be focusing on making models that succeed at that specific task, so keep an eye on them.)
Personally, after spending $25 on Claude Code credits and being very impressed, I think if progress continues that superhuman coders by 2027 is possible (maybe 20% likely).
> I find 1 good blog a year
> this is a shortage
> its also great money
Who wants to fund me being a blogger? https://crazymonkyyy.github.io/writings/gif.html
"There is an arms race which motivates both countries to cut corners on safety and pursue full automation over public objections; this goes blindingly fast and most of the economy is automated by ~2029. "
I don't think they could even have the factories built to construct the robots needed to fully automate most of the economy by 2029, even handwaving away about a billion legal and permitting issues along with general inertia across many firms in an economy that's not particularly competitive in many areas.
I think if it goes vertical, what instead happens is Ultra-Commodification of intelligence where copyrights and legal restrictions don't apply. Costs for that stuff shrinks down considerably with competition and copying.
One of the most interesting things Kokotajlo's 2021 article got wrong was saying that the space ruled by the western left will be more advanced than that ruled by the western right; it certainly seems to be going in the opposite direction. The shift of big tech to the right was completely missed, but in large part spurred by the left's attempts to regulate them, but I'm sure wokeness fatigue also played a big role.
It's probably stating a commonplace, but a lot of technical advances start with an innovation which develops from near nothing at breakneck speed but not long after more or less plateaus with only incremental advances thereafter, even though it may have enabled all kinds of other unrelated advances.
Examples include the steam engine (hardly any in 1800, except down a few mines to pump out water, but everywhere a mere 30 years later). Similarly, the internal combustion engine roughly a hundred years later, quantum mechanics a century ago, etc.
So it seems quite plausible that, despite all the hype, AI will follow the same pattern, i.e. rapid flowering up to a point, followed by a long period of consolidation and improvement but with no AGI worthy of the name, any more than steam power could power flight.
This article takes a free speech approach toward AI regulations already proposed (inspired by EU's restrictive approach), which shake out to something like punishing a library for the conclusions of a reader. These regulations could force companies to politicize, distort, or disguise facts in the model or its outputs, and also prioritize legal compliance, which would create a double whammy of obscuring the truth and slowing development.
https://eternallyradicalidea.com/p/how-state-ai-regulations-threaten
On the topic of Chinese industrial espionage- if the hard decision was taken to exclude staff from frontier AI work if they *could* be a Chinese spy, how much would that slow down progress?
This is a controversial subject but lets say you stopped just short of "no ethnically Chinese researchers" and put the bar at the 90th percentile. By this I mean, if you have any close family in China still, or if you've ever visited China, you're out. If your Chinese parents fled the cultural revolution and your nearest relation is a 3rd cousin, you're in. What percentage of frontier AI researchers would that exclude?
Putting aside ethical or legal concerns (which could frankly be ignored if the administration wanted to) massively reducing the risk of espionage could help slow down the arms race dynamic somewhat. On the other hand it could increase the number of available researchers to foreign powers and (slightly?) slowdown US research. I think on the whole it'd be beneficial to the US.
And, a perhaps more controversial subject with *hopefully* less US security implications, what about Israeli industrial espionage (see: https://en.wikipedia.org/wiki/Jonathan_Pollard), and similar measures being taken? I assume applying the 90th percentile rule would essentially halt progress...
Trump is not going to do any of that lol
>"We have no illusions that we’ll get as lucky as last time."
Having read it, I certainly hope not.
If interested in a different speculative take (based on 40 years in the field): https://tinyurl.com/private-i-novel
This whole "and then recursive self improvement happens" thing feels like Penrose's and then consciousness happens thing on the discovery of quantum microtubules.
In UNSONG (or at least the original story; i'm still in the middle of the paper edition), you predicted Trump as the last president before the doom of mankind (in a fictional universe, of course, any relevant real-world parallel was a coincidence.) For a while it seemed as if that was averted.
What is the kabbalistic meaning of the Biden presidency in-between?
or was it just the prophecy biding its time?
Yeah, although I really thought it would have come true bi den.
ಠ_ಠ
Was Ben Goertzel's prediction from 2017 right?
'Question 11: You have recently claimed that toddler-level AGI could come about by 2030. How confident are you of that prediction?
It’s looking more and more likely every year. I’ll be pretty surprised if we don’t have toddler-level AGI in the range 2023-25, actually. And then it will be more than a toddler in the human sense. It will be a toddler savant, with ability to e.g. do science and math and answer questions way beyond the level of a human toddler. And once we’re there, the full-fledged Singularity won’t be far off in my opinion…. SingularityNET has potential to accelerate R&D progress in this and other AGI-ish directions, making it increasingly likely that the advancement of AI proves Kurzweil a pessimist…'
https://www.nextbigfuture.com/2017/12/ai-researcher-ben-goertzel-launches-singularitynet-marketplace-and-agi-coin-cryptocurrency.html
2025 isn't over yet. Also "toddler level" isn't well-defined. Certainly ChatGPT has writing skill well above any toddler I've ever encountered.
I'm thinking of s-risks. If AGI capabilities become so politicized by _those_ actors... they might actually want to do this
Scott, how do your views about the future affect your parenting strategy, and your expectations for your children?
Not much. I can't think of any particular direction it would push me, so I'm not doing anything differently.
But based on your AGI predictions, it sounds like you must expect your children will probably not have gainful careers and might even be dodging killer drones by the time they are young adults.
Yeah, I would be surprised if my children had the same number of employment opportunities as I did. But:
1. Currently they are one year old, so there's not much specific job training going on.
2. If I were going to push them in one direction, I wouldn't know which direction to push them. It seems like "AI researcher" might be a big job in the AI future, but if we're to believe AI 2027, that becomes obsolete sooner than much simpler jobs like "electrician".
3. Even if there were no jobs, I would still want them to be able to read/write/do math/etc just for their own good.
As for the killer drones, my parents dodged nukes and my great-grandparents dodged Cossacks, you just have to hope for the best.
Eventually, the best option will be to ask an AGI what the best career options are for them and to steer them towards those. It will be better at foreseeing future trends in the job market than you or anyone else.
Thank you for putting fairly concrete dates down.
If Superintelligence doesn't happen by 2029 or 2032 or whatever date you think is reasonable, will you change your mind about the plausibility of Superintelligence? Or at least using the tools and direction currently being pursued (i.e. LLMs).
My 25/50/75 percentile dates are something like 2029, 2035, 2045. If nothing happens by any of those, then obviously I am wrong about something, and I will wait to see exactly how things failed to happen before figuring out what.
I think it would be pretty crazy if superintelligence was literally impossible by the laws of physics - ie by coincidence the smartest human (Einstein?) is the smartest it is possible to be, in every way (and you can't even run Einstein's brain faster). I don't know what could convince me of that except maybe fully understanding what intelligence is and somebody coming up with the thermodynamic (or whatever) proof of that claim.
"LLMs can't reach superintelligence" is . . . kind of a default? I think the AIs we discuss in AI2027 aren't exactly LLMs. The early ones are LLMs + agency training, sort of a crossbreed with AlphaGo-esque reinforcement learners although the LLM ancestry plays a major role (I don't have a good sense of whether an AI expert would still think of these as basically just another type of LLM, or as a hybrid). The later ones are distantly descended from LLMs but have gone beyond them in some ways (we don't specify exactly which ways, although we say they have solved interpretability which lets them put some things back in regular computer code again). I think it's a common position that AGI/superintelligence will owe a lot to LLMs, to RL agents, and maybe to other things. But I think the popular idea that there's going to be some crazy shift away from the LLM architecture, and it will be exactly the sort of thing that most vindicates all the cognitive scientists who failed to predict LLMs, is unrealistic.
Thanks for being specific! I am often bothered by predictions of "soon" and your numbers do sound very much more reasonable than others I have seen, given current knowledge. I do also appreciate that even in 2045 you're saying 75% instead of 100% or even very close to that. I often get the impression that AI-enthusiasts have near 100% predictions even on very short timetables, and thought your post was saying something similar.
In terms of "physics" I think there are two major roadblocks. Whether our current level of technology allows us to create something that can "think" at the relevant levels. This is both raw processing power and also correctly configuring that power. Secondly, what the programming is trying to do. LLMs clearly lack what we might call consciousness. Agentic AIs may behave in some ways like they have that property, but that may be an illusion (like ELIZA able to fool people 60 years ago).
Since we really don't know what these things mean ("consciousness," "intelligence," "think"), then it's very hard to determine what it takes to get there and whether someone or something has achieved it.
This doesn't mean they're impossible, but it may mean that we're missing some important understanding and may or may not be on track to achieving that understanding.
It's my personal feeling that we're not on track for general intelligence, and definitely not on track for superintelligence. We can definitely train an AI to work "intelligently" within specific areas, but that's very different from general intelligence where the same AI can work in multiple fields without being specifically trained on it.
Here's to 2029/2035/2045 and us seeing how it all plays out.
>the LLM ancestry plays a major role
This is the most implausible part for me, and I'm somewhat surprised that Dwarkesh didn't push against it (he did a pretty good job otherwise). It just doesn't seem likely that you can pile up enough kludges on top of the basic LLM framework to really capture whatever "secret sauce" allows humans to do novel research.
But then again, having a big argument about this doesn't seem worthwhile. Either LLMs will hit a brick wall very soon or they won't, with AI labs spending tens of billions to establish this one way or the other. I do appreciate your framing that it's in some sense "conservative" to expect there to be no brick walls on prior, but, well, it won't be the first AI winter.
A couple years ago you wrote that your p(doom) was around 25-30%. Is it safe to assume that this project has raised that number for you? Or given what we know is this the wrong way to look at things?
No, it's actually lower now, more like 20%.
That is certainly not the impression that the piece gives off!
What led to you updating downward?
How do your 80% scenarios break down? Are they mostly the "good outcome" portrayed in 2027?
So, very dumb question, from someone who knows nothing about AI: assuming this is correct... Should I be doing something? Is there a call to action here?
AI2027 deliberately tried not to give recommendations because Daniel thought that would make it look too political. The group behind it (AI Futures Project) might release recommendations later. It will probably be requests for increased transparency around AI R&D, around safety cases, and around the spec that the AI follows. Then if it turns out those things are bad, people should protest.
I am also looking for useful things that can be done around AI right now. The best I've found, which are not very good, are:
- If you're a Californian, write letters and stuff supporting various AI related bills going through the California legislature. I'll try to keep readers updated on which ones those are.
- If you're in some other jurisdiction, pay attention to their AI related bills and support ones that are good. I'll try to keep people updated here too, though with fewer promises.
- If you're a young person and like political activism, I hear good things about https://encodeai.org/
- If you use AI, try to patronize whichever company you think is most responsible; if you don't want to make that decision yourself, you can use whichever one looks best on https://ailabwatch.org/
- If you are a STEM hotshot, consider doing technical alignment work. https://www.matsprogram.org/ is the best way to get started.
- If you want a job doing something in this field, talk to https://80000hours.org/
- If you want to donate money, donate. My drop-dead simple first choice if you don't want to think about it would be https://www.safe.ai/donate . If you want to look into it more, Ben Todd's list at https://80000hours.org/2025/01/it-looks-like-there-are-some-good-funding-opportunities-in-ai-safety-right-now/ is good. If you want to donate a lot of money and want actually good guidance, email me and I'll connect you with the people whose job it is to think about that.
- I would like for there to be a group that does political activism like Encode, but not limited to young people. I don't currently know of one except PauseAI, who I am very skeptical of and cannot recommend. If someone else knows, send me an email.
- If you have some unusual useful skill or power, like you're a lawyer, politician, computer genius, or whatever, send me an email at scott[at]slatestarcodex[dot]com and I'll try to put you in touch with the relevant branch of the conspiracy.
How about praying?
What's the basis for skepticism re PauseAI?
It's the most direct answer to the question, a significant part of what it does is lobbying and public outreach for some of the things on your list, and it's pretty easy to just walk in to the Discord server, talk to people, and decide for yourself whether it's a good fit for your background/interests/beliefs.
ControlAI in the UK seems to be good at directly lobbying members of parliament to publicly sign statements about AI x-risk and banning deepfakes
I would be curious what Scott Alexander thinks of this essay arguing that, since AIs are basically limited to imitating common patterns in their training data, they don't post a serious threat of automating most jobs, let alone threatening the existence of the human race.
https://treeofwoe.substack.com/p/can-machines-think?utm_source=post-banner&utm_medium=web&utm_campaign=posts-open-in-app&triedRedirect=true
I know it can sound clichéd by now, but the author cites Yann LeCun a lot, and he backs up his claims about the limitations of AI with lots of details about how AIs work, and how they fail basic cognitive tests in ways that indicate they're totally missing a lot of mental abilities that are fundamental to how human beings think.
Since this is sort of my field, I actually made an account just to reply to this post. I tried both your examples (Superbowl and city names) with ChatGPT running o3-mini-high, and it one-shotted both of them. This leads me to suspect that you haven't tried them with a reasoning model. I don't mean to sound snarky, but to me this to calls into question whether your knowldedge of the space is sufficient to be writing so authoratively about it.
Likewise, I would be wary of assessing the capabilities of of LLMs (Technically MLMs since they are multi-modal) by the quality of the images generated by the generative models that they invoke. ChatGPT is almost definitely hardcoded to route your prompt to a diffusion model, and then return the output. If you call a blind man to commission a painting, and he passes your request onto his cousin, who paints it for you, would you blame the blind man if the quality of the painting was bad?
LLMs have struggled with tasks involving single letters (as opposed to full words) for a while, and this is well-documented. A famous example of this is the inability of models to count the occurences of "r" in the word "strawberry". But this fails to take into account how radically differently they process ("perceive) the data they are fed. Classically, text is tokenized before being fed to a model, which means LLMs don't recieve text at all, they recieve a set of vectors. Asking these models to reason about the letters in the words they recieve is essentialy asking to reason about objects they cannot percieve. If I play you a musical note, and ask you to tell me the constituent sine-waves, you almost definitely cannot do it, even though they are right there to be heard.
In general, I would advise against thinking that a model is "stupid" because it fails at tasks we would consider easy, or in ways that we consider obvious. A color-blind man may struggle with a simple task like "place the red cube on the green cube", but would you consider this a mark against his intelligence?
I have some other gripes.
At one point you say "If a neuron is stimulated above a certain activation threshold, it “turns on.” Then, using the pattern of active and inactive neurons in the first hidden layer, another set of artificial synapses". This to makes it sound like you are implying that activations are binary. I admit this may be a misreading, but I want to point out that this is not the case.
Likewise, I feel like your descriptions of the way neural networks are used to approach image generation and chess are, at best, extremely inaccurate.
When I researched my essay, I tested ChatGPT, DeepSeek, Grok, and Claude, each multiple times, on those two questions (state capitals and Super Bowls) and never saw an accurate answer, so I find your claim that ChatGPT o3 one-shotted both of them extremely hard to believe. Also, your claim that, since AIs can't perceive certain things (like individual letters in a word or number of figures in a painting) that are easy for human beings to perceive means its unfair for me to test them with those things makes little sense - the fact that it can't perceive those things, despite being much better in some areas (like total amount of stored knowledge, or speed of generation) is exactly my reason for being dismissive of the claims that they're about to mimic human intelligence.
No need to take my word for it, here’s a link to the chat:
https://chatgpt.com/share/67f12b48-f488-800b-81fc-8fcd63198157
The fact that you say that you tested “ChatGPT” is part of my concern. ChatGPT is not a model. It’s an interface for interacting with models. If you were chatting to 4o for example, you’re interacting with an antiquated model. Did you test your prompts with any reasoning models?
As for the perception claim, I don’t share your view. If I show you a picture you’re probably unlikely to tell me exactly how many yellow pixels are in the picture. I don’t think that’s a reflection of your intelligence. Neither is your inability to see ultraviolet light.
I looked over the saved ChatGPT conversation you linked to. It's true that it's an improvement over version 4o (which I used) but it still made mistakes in both tasks: it wrongly gives 41 as the winning score of Super Bowl XXIII (in reality the 49ers won 20-16) and it misspells the reversed names of Helena, Salem, Honolulu, and Montgomery, leading to Salem (lemaS or "Samel") being out of place in the list. So it's clear that you're not paying close attention to what the AI is doing.
Also, when I criticize the AIs for not noticing things that are obvious to a human being, I'm not claiming that AI is useless or that no progress is being made! That's a straw man. My actual claim is in the title of my essay: "AI Imitates Only a Small Part of the Human Mind." I am perfectly aware that there are things that AI can do the humans can't (though this has been true for about a century, if you consider old-fashioned IBM accounting machines to be "AI.") Ironically, at the moment that I saw this comment, I was using Claude 3.7 to help proofread a book translation into an obscure African language - something that no AI was capable of just a year and a half ago. So I am perfectly aware of the strengths of AI as well as the weaknesses. The difference is that you seem to only notice the strengths.
Touche. I admittedly only sanity-checked a couple of answers. I was being a little sloppy. As it turns out, so was the model. Seems like human-level performance to me then :)
(Silly comment embedding a serious point)
Do you think AI will be able to build a high speed rail between San Francisco and Los Angeles before 2030?
Why would it want to?
The prediction and podcast discuss that some impressive things may happen, like manufacturing 1 million robots per month by 2029. But that is because the AI companies, the governments, and the AIs themselves will be aligned on that goal and will feel massively pressured into doing it, and it is not that much faster than comparable projects that have been achieved in the past. Their example is bomber construction during WW2.
So in a hypothetical scenario where government and AI companies and AIs have as their top priorities to build this rail, and are willing to cut through all regulations and throw unlimited money at the problem... Well yes, then certainly that's possible. That wouldn't even be impressive, to be honest. The US has done much more impressive things during WW2.
Follow up, could AI fix government and military procurement processes? Which are a) entirely organisational/political problems and b) directly relevant to a completion with China?
I wonder if the bad ending is less likely to be the dramatic "AI decides to exterminate us" and more like the extinction of so many other species. Maybe the AI just decides helping us is a waste of time, doesn't care about our habitat, and humans slowly disappear.
"Copious amounts of synthetic data are produced, evaluated, and filtered for quality before being fed to Agent-2."
What role will synthetic data play in potential bottlenecks? Do researchers consider this "solved" yet, or is it a domino that's yet to fall?
Contrary to Sam Altman's chirpy update on the progress of GPT-5 on X, the chip industry newsletter I subscribe to struck a grim tone...
Headline: Trump bequeaths technology dominance to China (NVDA, CIEN, COHR, LITE, MSFT, GOOG, AWS, META, and every other tech company you can think of):
Synopsis: Perhaps it is fitting that we are at the OFC show on liberation day with all our optical friends who have just been put out of business. We had thought Trump was against China but by imposing tariffs on SW Asia countries like Thailand this halts all AI deployments in the US and will allow China to dominate in AI and all technology for that matter.
Prior to yesterday, the optical industry was on the cusp of an unparalleled growth period fueled by cloud expansion and AI. Even before tariffs on Chinese optical vendors, US suppliers were universally going to see several years of accelerating growth prospects.
CIEN was poised to dominate optical transport with abundant leftovers for companies like NOK/Infinera and CSCO/Acacia.
Both COHR and LITE would gain share vs. Chinese vendors in cloud and prosper greatly. Even tier 2/3 suppliers like AAOI would have plenty of business.
NVDA was poised to deploy $50B+ worth of Blackwell systems a quarter in 2H2025 which would accelerate US AI capacity and capability.
AWS, MSFT, GOOG, and META were poised to prosper from AI and cloud services.
However, all of that has been stopped dead. There will be no more AI deployments nor cloud datacenter expansion in the US or anywhere outside of China.
Conclusion: Perhaps Elon will realize that xAI has been put out of business and whisper in Trump's ear that this was not a good idea. One can only hope that US based companies manufacturing optics, servers, and the like will get exemptions from the tariffs, or we have just handed technology dominance to the Chinese on a silver platter.
This whole thread here is surreal. People who output PowerPoints and blogs tell us dis/u-topia will be here in 3 (ok, 4 at the latest) years, and in parallel people who make stuff keep asking who will clean the plumbing.
I'm in the plumbing camp, myself. ;-)
Ok, this is my first comment on this blog.
I'm not convinced on superhuman coding. Current approaches on LLMs are about imitating human texts. So, much-faster-than-human coding - here I agree. But superhuman code quality, including new algorithms? We do we get examples of these, if all that we have is only human-quality code?
Games like Chess and Go are different. There we have a goal function - maximizing wins. In coding, we don't exactly know which code is winning. We need superhuman graders for that. And it seems to me that first that coder must become good at math - like "solves P?=NP" level of good - because inventing 100x faster algorithms reliably is a lot like this.
What is your median timeline for ASI and AGI breakthroughs (as you mention yours is a bit later than the one on the model)?
The explosion of intelligence from better AI coding tools is just such a huge assumption. I hope it happens but don't see any evidence for it.
I can't help but think that people close to AI investment and research like Musk and Thiel saw these 2027/28 predictions coming too, and decided Trump was the person they wanted in power when it happens. Personally, I think because they felt like he would easier for them to influence, not because of any ability he would have to navigate the AI future.
This is not going to happen. There is lots more to be solved first.
The two big ones: a lot to do on sentience, and the problem of conflict, which is essentially unsolvable without dominance.
Nice analysis. Thank you for your efforts. [I've read the scenario(s), but not watched the whole interview yet.]
A few comments:
This is obviously very hard to incorporate into any projection, but I believe that there are big discontinuous software jumps waiting to be discovered that will be worth more than just 2x or 3x or whatever. Kind of like the "...entirely new vistas open up before it..." section in Nov 2027, but these could happen at any time (starting from now) and completely change the game. I don't really have a proposal for how to model this other than to bear in mind that the world is even less predictable than it seems, and
After the "China has woken up" point and it's an all-out race, it seems unrealistic to assume they will continue to hold off Taiwan/TSMC for so long, as it seems to be in the scenarios. Either they will take it for themselves, or at least to DOS it so that others can't make use of it.
China will obviously direct a lot of effort into being able to produce their own state-of-the-art chips. This is happening now, though I don't know anything about timelines for catching up with the West/TSMC.
I somehow think that the countries (USA, China + whoever) might be better at co-operating than in your scenarios, at least at the point they all realise that the future of humanity is in serious jeopardy. They can do meta-thinking, not just blind optimising: i.e., we have to imagine they will have access to analyses like your scenarios (or they can just read what you've written) and they ought to eventually come to the conclusion that it's better to cooperate and not race so strongly. My worry would be that ASI takes off too fast for this to work, or that we're doomed even with cooperation, but if your two year timeframes are right then there are chances for humans to cooperate and at least evolve the ASI more carefully, which might make the difference.
> I somehow think that the countries (USA, China + whoever) might be better at co-operating than in your scenarios, at least at the point they all realise that the future of humanity is in serious jeopardy.
I think it's difficult to get around the fundamentally Pascalian dynamic that makes this a race.
Yes, corporations are in a local race dynamic - if we don't keep pushing, we've got ~3 competitors hot on our heels, and they'll Print All the Money if we don't get there first - this ensures capabilities will likely progress too quickly for slow governments to react.
But between countries, the dynamic is more pernicious - it's why Deepseek R1 and Manus were such huge deals, publicity and media wise, even though they're both nothingburgers on deeper analysis.
It's a race at the country level because:
1)You can either race ahead or not
2) If you don't race, you're exposed to the downside risk no matter what (military and economic superiority in your enemies, AGI / ASI taking over earth or the lightcone, AI unemploying everyone, killing everyone), and will take part in no upside
3) If you do race, you might just beat them, and beating them by months can potentially lock in a permanent advantage / nullify them
4) Therefore you MUST race - it's a forced move, a Pascal's wager
You are assuming certain payoffs in the game, but I don't think they're obvious.
It could be that eventually everyone realises that the overwhelming risk to humanity is if countries race, and they agree to slow down.
For example, the payoff matrix (in arbitrary units) *could* be like this (hoping the formatting is preserved):
China slows China races
USA slows 100,100 -600,-300
USA races -300,-600 -1000,-1000
where a payoff of A,B means USA gets value A and China gets value B.
Justification: if both sides slow down then there is only a 40% chance of AI annihilating humanity, but also a 60% chance of utopia. Pretty decent odds, and the best outcome. It's because working slowly decreases the chance of the key ASI being misaligned.
But if one side races while the other doesn't then there is a much greater chance of the ASI being misaligned and vaping humanity, say 70%. This is not as great as if both sides race (90%) because in race vs slow, the racing side won't feel in so much of a rush to get things out without checking if they don't sense the competition hard of their heels. It's true that in the remaining 30% of the time, the side that raced does get a benefit over the other side, but it's a pyrrhic victory: only -300 vs -600 because of the extinction risk.
If this really is the true game payoff, then regardless of what the other side does, it's in your interest to slow down. Of course this is not what we see right now, but that's (maybe) because neither side has truly "felt the extinction risk" yet.
I don't think the real world is nearly as simple or nice as this, but I do think it might share the feature that the big players might actually start cooperating when they can plainly see the alternative is that humanity is over.
Yeah, we can certainly envision scenarios where that's true.
But I think one of the stronger takeaways from watching progress over the last 4 years has been that "no, there are zero adults in the room, we're definitely YOLO-ing this."
Sama routinely pretends he's Willy Wonka and makes jokes about a technology he himself has said might end all of humanity, "AI not kill everyone-ism" isn't on the radar politically, economically, or in any non-rationalist circles at all, the biggest risk anyone empirically seems to worry about is that AI's might say racial slurs or bad words, or might cost some people jobs, and multiple high profile AI researchers have left safety teams at the Big 3, pointing out that safety isn't actually a priority.
And that's before even talking about the political situation.
Pinning hopes on enough people involved being adult and long-thinking to actually come to some sort of agreement, particularly as we devolve into a more xenophobic and nationalist stance than we've basically ever seen or demonstrated, seems overly optimistic.
I agree with all that, though I wouldn't exactly call myself optimistic. I'm about as doomer as they come, but I think it's rational to pin one's hopes on the best chance, even if it's a slim one.
It's going to get worse before it gets better. At the moment only a small minority of people think there's a huge problem of the sort mentioned in these AI2027 scenarios; scenarios which most people will regard as complete science fiction. There are even many people who still think AGI is a very long way off or will never happen.
Also, it's not necessarily only two big groups going up against each other. I think that there are software changes that are worth large factors waiting to be discovered, and even a group with a small minority of the world's compute would be able to make progress if they find such changes.
> but I think it's rational to pin one's hopes on the best chance, even if it's a slim one.
My favorite "maximal optimist" slim chance interpretation - Elon Musk.
Some might say that if you took the richest man in the world, and he gave better than even odds everything was going to be turned on it's head in a handful of years, he might judge that even a low-success-probability move along the lines of "get your hands into the guts of government and state capacity, and do heavy, sloppy surgery to try to quickly knock things into enough shape we can actually act competently in a "state capacity" sense again for the upcoming trials" might actually make sense.
Even as the richest man in the world, he's only one man. Even as the head of DOGE and with the President's ear, he's still only one man. He could be genuinely acting on strong convictions, and we'd never know, because "wrecking the government for fun" and "shitposting on Twitter 20 hours a day" are basically indistinguishable from "desparately trying to increase our ability to execute state capacity competently" and "trying to get everyone listening to you, so that when the time comes you can hand your account off to a superpersuading frontier model to try to convince as big a chunk of America as possible about something"
Are the odds of success on that front low? Yup - but could still be the best move available to him.
Are the odds of this "4d chess" interpration of his actions ALSO incredibly low, versus "drugs and being surrounded by yes men have ruined his capacity for discernment and critical thinking?" Probably also yes. But you know, we can hope.
> I think that there are software changes that are worth large factors waiting to be discovered, and even a group with a small minority of the world's compute would be able to make progress if they find such changes.
Yeah, great point. I agree (unfortunately) that it seems pretty likely there are a number of potential overhangs that could lead to significant jumps in capability along the lines of "figuring out sparse sample learning vs millions of examples level learning," figuring out TSP-equivalents for training versus inference, getting significantly better at fine tuning or Mixture of Experts or other hyperparameter-style things, a fundamentally different architecture built around LLM's that enables ongoing learning, and more.
I like your accidental misspelling “eutopia”. Living again in Germany after 36 years in the US and having to deal with all the EU rubbish about “regulation as the springboard for innovation” I would spell it EUtopia.
Ha. I realised the misspelling after I posted it. I think I must have been thinking of the Greek etymology. Edited now anyway.
I’m glad I spotted it before you could edit it. I have talked to my coworkers about the EU’s efforts to achieve the “regulatory singularity”, and will use EUtopia as the state of affairs after that Rubicon has been crossed. Although strictly speaking it would be EU-dystopia.
Reading this as an 18-year-old freshman in college gives me quite a bit of existential unease about the future.
Even in the 1 percentile slow scenario, AI superintelligence will dominate any endeavour that I could possible dedicate myself to in the next few decades.
What am I to do then? Toil at a CS degree and hope by some miracle that I get onboarded into one of these AI companies?
I find myself having to purposefully ignore AI forecasts to maintain any sort of ambition about my future.
As most people I am freaking out about this, so as a way to cope I made this market to keep track of which predictions for 2026 will become true as an indicator of whether the whole scenario will materialize or not
https://manifold.markets/BayesianTom/which-ai-2027-predictions-will-be-r
What's the earliest reasonably specific prediction we could make that we could use to see how close this timeline is to being correct? Like could we say on January 1st 2027 that if X is true, these predictions are looking good, and if not, they're looking dubious?
When we asked Daniel this question, he said that the earliest relevant claim is that AI will continue progressing through coding-related benchmarks (eg REBench) at the current rate for another two years (at least until those benchmarks are saturated and we need new ones). If this progress stops, AI2027 is at least somewhat wrong (until they get started again, and even then wrong in timing).
Interesting, thanks. Would be interesting to see a follow-up analysis in, say, early 2026 and then early 2027 to see what progress is like.
I'm just finishing reading both endings to the full scenario and I noticed something that feels like a flaw in the underpinning economic model - in the "Slowdown" ending, in October OpenBrain is valued at $8T, and then in November, it's valued at $9T. However, November also sees the merging of the 5 trailing AI companies via the Defense Production Act. Given that today, after massive stock market losses, Apple is worth $2.8T , Microsoft is worth $2.6T, and Google $1.7T, it seems like merging in the competitors is incorrectly adding no value? Commonsensically, the invocation of the DPA should add at least $5T to the valuation.
I've relayed this to the team. My guess is that we meant to say the government merged all of their AI-related assets with OpenBrain, and most of the value of these companies is not their AI assets.
Congrats to Scott on being an author of what is (probably?) the most widely read purely-AI Safety thing that has ever been written.
Thank you hugely for this blurb targeted at people like me:
>if we made an abstract claim about how the intelligence of the system would let it find a way to victory and ended the story there, much of the value of our project would be lost.
That alone has always stopped me from even considering taking alignment seriously, beyond as sci-fi entertainment. This one actually does try to be real, though, and so I gave it enough serious thought to allow for real changes in my thinking. I'm not converted, but I will seriously listen in the future. I hope it's clear that I mean all this as high praise.
It did have an effect on me, though: I briefly considered having an AI help me word this comment, but immediately felt horrified, when I never would have minded before.
Finally, I thought it was hilarious that the aligned and unaligned scenarios have identical medical progress except that qvfrnfr pherf ner zbivat ng hacerprqragrq fcrrq guebhtu na SQN abj nffvfgrq ol [nyvtarq NFV]. I'm assuming I can guess who wrote that part!
In 1900 McKinley comfortable beat Bryan to win re-election as President (51-45 popular vote).
If we sent a modern team of historians, marketers, psephologists and speech writers back in time do you think they could flip this result?
I doubt it. Maybe they could deploy focus groups and polling to sharpen the message but would that be able to sway 6% of the population??
Makes me skeptical AGI will be decisive in 2028. Even 100 years of knowledge does not seem to have advanced the practice of politics in some revolutionary way.
I know nothing about It, but this sounds right to me. Can any of these programs: (1) identify new problems? (2) come up with new ideas about how to solve those problems? (3) subject those those potential solutions to rigourous testing and criticism? If not, I don't think they should be called "intelligent."
Re (1), (2), & (3), yes.
https://sakana.ai/ai-scientist/
>The AI Scientist automates the entire research lifecycle, from generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript.
Thanks.
"Nobody except researchers and a few hobbyists had ever talked to an AI"
The single dumbest thing ever written
The desire to rule the world does not require being a sociopath or having a pathological desire for power for power's sake. If I were given the opportunity to rule the world with a high probability of success and minimal disruption caused in the process, I would take it - not because I enjoy imposing my will on others (well, maybe in the case of some of the outliers in currently screwing up the world due to stupidity and evil) but because I could finally kill Moloch (as you use that term) and thus greatly improve the lives of billions of people, even if all my other efforts to improve the world failed. Yes, this would require imposing my value system on others, but like everyone else, I believe my value system is right and therefore should be adopted by others - if I didn't, I would believe something else. To borrow from Yudkowsky, I don't want to rule the world; I just think it could be more sensibly organized.
Could you kill Moloch? I would think not, as it's an entity emerging out of an administration filled out by humans trapped in prisoners-game like incentives. You will not be able to get along without an administration, just maybe reduce it, and it will still be filled by humans. So Moloch will survive, maybe temporarily reduced (like it has been in the past), but coming back with a vengeance as soon as the temporary set back (you) disappear....
The only ways out of Moloch is getting fully rid of administrations:
either back to hunter-gatherer independent groups where there is no faceless bureaucrat niche
or get an integrated AI over-ruler, without an administration running on human substrate...
This is very frightening but also the (for me) main AI redeeming quality: AI rule is not killing human freedom, it's been killed long ago. It's replacing Moloch (aka organised civilisation with a large bureaucracy) by a new ruling entity, a more integrated one running on chips rather than semi-linked human agents...
It depends on what is meant by "rule the world". Given control over all the world similar to that of the state capacity of developed-world governments, I could end defense arms races between blocs of countries, standardize taxation and regulations in ways that prevent races to the bottom and reduce deadweight loss therefrom, and so on. If everyone loved me and *wanted* to obey me, I could go even further than that. The question relevant to AI futures is just how easily humans can be manipulated this way.
What you describe (before "wanted to obey me", which imho go well into the fantasy part of speculation ;-) ) is a single-country world, a world government/administration. When I was young I used to believe it would be nice, a sort of utopia, and I was not alone to believe so at the time. In fact, it was the default setting of most SF universes (where planets or interstellar empirestook the place of countries, basically).
Nowadays, I am not sure, I think this was naive, a product of the era (cold war) and me being in the west bloc. Because I see governments (mine included) as non-friendly non-human entities whose sole real purpose is self preservation and the composite increase of power of the constituents of it's administration (aka Moloch). Moloches are at best indifferent to your preferences, sometimes hostile if you challenge their existence or power. Even within, it helps you compared to those outside but you are in fiercer competition with other elements of the admin.
So main question becomes: Are multiple Moloches better or worse than a single One? I used to believe one is better: it decrease inter-moloch conflicts, which are not good for humans. But it also remove the brakes on the worst Moloch behaviors, power grabing, obsession with control and micro management. When moloches compete, they (some at least) tends to care more about their population, as a for of PR or image competition. I saw that with then end of cold war: even starting from a relatively nice place, we ended up with the current situation, which is objectively worse than it was. Because we had a 30y period of 1 world government, in practice if not in theory, while before the west had to look nice wrt the east. USSR was the bogeyman that kept the west behaving.
I'm entirely used to major institutions' hypocrisy on this issue, so it's not too unrealistic a premise, but in the well-aligned scenario, what is the principled reason for AI access to be limited to the two major parties' candidates? For decades, the Libertarian and Green parties have been stuck in various catch-22s/bootstrap problems that make it impossible for them to fairly compete on the basis of their ideas. With AI advisors for their campaigns, they would finally have something close to a level playing field against the Democrats and Republicans. A more fair/rationally principled version of the policy in question would be to give AI access to any candidate who is on the ballot in 270 EVs of states.
For an actual 'better calibrated' objection:
1. There is a finite amount of lit silicon the world can produce between now and 2028. Both on the path as usual' timeline and 'wartime effort' timeline. There's big latencies in the process : if you wanted to go to a 'wartime effort' tomorrow and could pay for it, there's all these bottlenecks at ASML, etc, and it takes time to get the equipment into the building and to tune the new fab to produce any useful level of yield. And this tuning I understand can take months to years, because there is a latency for each machining step, and it can take weeks to months for wafers to process through each step. This latency then limits how quickly you can learn about problems and make the process work at this specific fab.
For example, if process step #33 is faulty, and it takes 3 weeks to reach that step, it is impossible to know about this information ahead of time.
Even greater than human intelligence cannot necessarily overcome a limitation like this.
2. There is a finite amount of algorithmic improvement possible. It's like compression : you can't shrink a lossless file below the amount of entropy in a file. You can't lossily compress data past the point that the errors make the media unusable. This is true for cognition as well : compression/distillation/algorithmic improvement can make huge improvements in efficiency, but all the benchmarks must still pass. The AI model must still do sufficient computation to find the correct answer. So you can't bank on arbitrarily optimizing by a factor of 10 every 6 months for very long.
So with a finite level of lit silicon produced, and a limit on algorithmic improvement, it may not be possible for these specific predictions to be achieved.
There's also limits on funder patience, etc. An AI lab may not be able to use a significant fraction of a country's compute just to develop better models but may need to dedicate quite a few to making people's animated pornographic studio ghibli style cartoons, etc..
There's much more algorithmic improvement possible than most people think. Yes, you can't bank on optimising by a factor of 10 every 6 months, but you also can't bank on *not* optimising by a factor of 1,000,000,000. I am sure we are very, very far from having the optimal program.
You can bank on not optimising pretty much any algorithm by a factor 1 billion because in most cases that is not theoretically possible. For example, almost every program uses some form of sorting and sorting is a bottleneck. It takes a minimum lengthy amount of time to compute and the only way to bypass that would be to skip the sorting in your architecture somehow, because you certainly cannot improve most sorting algorithms any further than they already have been. At some point you are constrained by the amount of time it takes just to move values around. If you want to argue that ASI will just somehow overcome these information-theoretic limitations, then you're just entering the realm of magic and why should we believe you?
I don't believe there are information-theoretic limitations of the form you think, and I don't think LLM training is bottlenecked on sorting, but maybe (trying to run with your argument) it's bottlenecked on matrix multiplication or data movement or something. That might limit the training rate of a particular transformer-based architecture that is currently used, but that doesn't mean you can't come up with a different architecture that achieves the same results, and then your improvement bounds will not be applicable. The only way you could have the kind of bound you say is if there were an information-theoretic proof that the current approach to intelligence is optimal within a few orders of magnitude, and there isn't.
Yeah you pretty much put my disagreements into words.
Scott, you look and sound like Benoit Mandlebrot.
mi prami
eh, Scott you really have this near magical faith in technology that isn't warranted. Like as a psychologist, you probably could tell a lot of stories of how hard it is to get even simple changes done in the average hospital setting: do you think AI would evaporate the problems in that or persuade better than you would to your boss?
like you guys think its this rasputin/svengali that can hypnotize people by virtue of its superior powers, based on a logical leap that is always in the future. if you evaluated AI just in your field you'd be more sanguine i think because there is no magic there. AI doesn't make needing more beds any cheaper.
>AI doesn't make needing more beds any cheaper.
Agreed (till it starts finding novel optimizations in manufacturing technology, eventually).
>like you guys think its this rasputin/svengali that can hypnotize people by virtue of its superior powers
I'm also skeptical about it e.g. writing ultra-persuasive text. We've evolved to be skeptical of shamans and leaders for 300,000 years. Our "immune systems" aren't perfect, but they do quite a bit of filtering.
On the other hand, an ASI which has hacked into surveillance systems and done forensic accounting all over the place and "wants" to force some changes ... what if it just blackmails "at scale", and has credible damaging information about each of the 10,000 most powerful people in the USA?
"If it gets aligned successfully, then by default power concentrates in a double-digit number of tech oligarchs and US executive branch members"
I don't believe one of those people has my best interests at heart: why should I expect to retain my breathing privileges?
How to get some of this acceleration today: feed the proposal and video transcript to Gemini 2.5 pro for any question you would post. Lots of posts were directly addressed already!
I looked over the saved ChatGPT conversation you linked to. It's true that it's an improvement over version 4o (which I used) but it still made mistakes in both tasks: it wrongly gives 41 as the winning score of Super Bowl XXIII (in reality the 49ers won 20-16) and it misspells the reversed names of Helena, Salem, Honolulu, and Montgomery, leading to Salem (lemaS or "Samel") being out of place in the list. So it's clear that you're not paying close attention to what the AI is doing.
Also, when I criticize the AIs for not noticing things that are obvious to a human being, I'm not claiming that AI is useless or that no progress is being made! That's a straw man. My actual claim is in the title of my essay: "AI Imitates Only a Small Part of the Human Mind." I am perfectly aware that there are things that AI can do the humans can't (though this has been true for about a century, if you consider old-fashioned IBM accounting machines to be "AI.") Ironically, at the moment that I saw this comment, I was using Claude 3.7 to help proofread a book translation into an obscure African language - something that no AI was capable of just a year and a half ago. So I am perfectly aware of the strengths of AI as well as the weaknesses. The difference is that you seem to only notice the strengths.
Could you please add at the top of the page something like "may trigger strong emotional responses"? I know it’s fully on me, but I had to calm down for a solid amount of time after reading.
Nonetheless, +100 epistemic points for teaming up and making a very solid prediction! It would be nice to see the recommendations of you lot on what to do for people and/or governments.
Yes but half of the group are hustlers for the technology and really need to be skeptical about.
I listened to the Dwarkesh podcast and at around 1:06 they mention speed running civilization up to the Industrial Revolution. There is actually a book on this called Knowledge by Lewis Dartnell:
https://www.amazon.com/Knowledge-Rebuild-Civilization-Aftermath-Cataclysm/dp/0143127047
The Chinese ai will never be as good as the us because they are a derivative culture that lacks originality. Garbage in garbage out as they say 😎
"Garbage in garbage out". The implication being that when the Chinese copy Americans, they are copying... garbage?
Actually not at all. Clearly they are an exceptional country with exceptional individuals but their society really is holding the people back. Look at Taiwan vs mainland China, Taiwan is much richer and more innovative. I feel that whatever effort they make will be held back buy their blinkered leadership!
probably the better invocation was China qua Cargo Cult.
Read the website (https://ai-2027.com/), and there was this one section where a ceo in control of a law-abiding AI could use it to create a next-gen non-law-abiding AI.
This could be circumvented by adding a law against the creation of non-law-abiding AIs.
One outside view thing to bear in mind with Daniel's impressive track record. On any question where the core of the question can be boiled down to 1 dimension (speed of AI progress in this case), the most aggressive forecaster will seem prescient in any scenario where the score is high on this variable.
This makes it difficult to distinguish "gives aggressive AI forecasts" from "gives well calibrated AI forecasts".
This is the same problem that every time there's a recession, the people who always give pessimistic economic forecasts (but maybe we haven't observed that they do yet) look prescient and when there's a boom the optimists look prescient.
To be clear, I still think there are impressive details in there, but I think a lot of people who didn't expect all the progress that has happened, would have expected a lot of it conditional on believing overall progress would be as fast as it was.
I basically buy a lot of the story here while multiplying some of the specific times to reach certain points by a factor of 5-20. So my disagreement is essentially over that one dimension.
In terms of where that disagreement comes in, it's mostly
1) near-term LLMs hugely improving AI research efficiency. The current bottleneck appears to be the very high level thinking, testing, interpreting and training tasks that it seems unlikely that LLMs will assist much with. I buy that Open AI might be able to layoff its junior to mid-level developers, but I don't think access to these developers is a core bottleneck (although I'm sure they'll appreciate the cost reduction). If it was, Open AI could afford to hire thousands more of them.
2) the ability to quickly train agents who can act in the long term. Training data for this is hard to collect or simulate. In the podcast the simple example of coordinating a party for an office is given. The feedback that you have messed up this task usually comes on the day of the party (a few weeks later) when the wrong people show up (because people weren't reminded or you failed to react to another event being scheduled at the same time) or the venue turns out not to have been confirmed etc. That means the minimum time to get real world feedback on that task is 2 weeks (and it's costly to get).
I think this data will be collected eventually but it will take decades to make it work across domains and pointing to "really smart AIs will solve this for us" doesn't work because they will need good data on long-term problem solving to be smart AIs that could solve this for us. I also think that many algorithmic problems will only be solvable once we see how versions of these models with somewhat reasonable training data still fail.
Completely off topic, but Scott has a great sounding voice, I'd love to hear more of his thoughts in audio form :)
Great scenario, and great post!
nit:
>The White House considers replacing the CEO with someone they trust, but backs off after intense employee protests.
happens _after_
>Most of the humans at OpenBrain can’t usefully contribute anymore. Some don’t realize this and harmfully micromanage their AI teams. Others sit at their computer screens, watching performance crawl up, and up, and up. The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.
Presumably it would have to be most of the Agent-4 and/or Agent-3 instances protesting? :-)
Two things that might slow the timeline a little:
1) Once the AIs are learning from experience, "cloning" them can't be the dominant means of communicating information. If one has >100,000 AI agents, each of which has participated in one or many research projects from a _different_ role, copying one and overwriting the weights of another (given, at any one time, a fixed number of AI agents) loses the learning in the overwritten AI.
They can still do things like get the training of another AI's role at accelerated speed, faster than the real time of the role. But to aggregate _all_ of the information that all >100,000 agents have learned is going to require passing summarized information of some sort around - project reports, something like a review article, something like a textbook.
2) The recent trend in AI has been to use more inference time compute to gain more reliability, using "chain of thought" and similar techniques. Some of this is serial and conflicts with getting as large a real time speed up as current, less reliable, models would suggest. At the very least, it suggests that more reliable AIs will require more compute per answer. So I'm a bit skeptical that the compute resource in the scenario will actually yield quite the combination of number-of-agents and speed-of-agents in the scenario at the same time as improving the quality of the answers they come up with.
edit: To put it another way, there is a quality vs speed tradeoff on every scale from sub-nanosecond analog voltage sampling to formulating national policies. I expect it to apply to AI outputs as well.
Timeline at the end looks too optimistic to me, it smells like the sort of aggressive timeline that business people will conjure up for hype reasons. I've been disappointed by several of these in the past, so I am personally skeptical.
Looking at a specific narrow thing I know about, AI models have some trouble doing (some) math, specifically combinatorics at the IMO level, and this is exactly the sort of problem I expected AI to struggle with the most before the alpha-proof breakthrough, because intuitively speaking I feel there is a bigger distance between mental image and formal statement. So I was not surprised to see that alpha-proof was not capable to solving any of the problems in this category. There aren't many examples of this translation explicitly written down, and the current AI models are not very efficient at learning things with a small amount of data. This problem is also varied enough that my guess is that a few heuristic rules are unlikely to produce human level competence, unlike, say algebra problems .This is the part I'm least confident about, if I'm wrong about this and a sophisticated set of heuristic rules produces good results I will be disappointed, but I will seriously reevaluate the possibility of AI being able to do quantitative research in the near future given how "generic" combinatorics is to quantitative reasoning. It's also possible we could figure out how to generate automatically large amounts of training data that somehow capture the intuitive feeling of "combinatorics", although I'd consider this less promising given how hard it is to put your finger and what exactly counts as one as a human.
Right now it feels like AI is very good at bullshitting, but when it comes to hard-to-fake "hard knowledge" like this, it struggles much more. This problem might be an engineering cycle away from being solved, or it might be completely intractable with methods in the foreseeable future, we won't know until we try I guess, and I'm not sure what probability to give either case, but I think there is a real possibility where we don't progress past a specific mundane benchmark like this for decades.
I think someone should look into funding an AI risk museum/exhibit/whatever in nyc or dc that illustrates all this stuff in a striking, accessible and emotional way. Interactive, visceral, normies get access to this issue in a way they never would even stumble upon it. More likely to get press than a Substack article etc. My main criticism with all this stuff is that it just doesn’t break through to a wider audience ever. It’s too rational/smart, any normal person reading this will either be like “huh too technical” or “this is fiction”.
I’m a programmer. And unless I’m missing something, I haven’t seen AI do what we would call “logic” yet.
LLMs do imitation very well. Has something shifted? I can’t see how imitation allows programmers to solve NEW problems vastly quicker. (And it seems like the AI explosion described above relies on this)
Probably the fact that reinforcement learning is being applied to programming? Just my guess
I might have missed something… I think I understand what you mean, but has reinforcement learning allowed AI to code a novel piece of software yet? Like a new app that does something useful? I’ve heard about “vibe coding”, but that as I understand it is still essentially the regurgitation from examples that the LLM has learnt.
Wouldn’t the problem with reinforcement learning for creating a novel program be that it would be well beyond our current technological limits to design a unit test to test if the program is “good”… as the unit test would need to be about as intelligent as a human for it to work? (And reinforcement learning requires negative or positive feedback based on how good the output is)
Essentially isn’t this kind of what vibe coding is… using a human as the major part of the unit test for code that an LLM is generating?
The area of software I work in is game software… so I might be missing the applications in other software fields … in my job AI can barely code anything useful yet… it can occasionally regurgitate examples well, and I expect that facility will improve… but 99% of game code is trying to achieve particular gameplay results that a human can interact with in a pleasing way… and I don’t think the AI code assistant has any concept of the game it is trying to assist with.
Thank you for putting this record of your in-depth research together. I listened to the entire 4h 15m spoken rendition of the website, and found it riveting. A ground breaking magnum opus. I consider it akin to a contemporary version of an early Tom Clancy thriller, with all the research annotations that one would want for exploring labyrinthine rabbit holes. I will certainly listen to it again on my daily walks and dig into the details (after all, Easter is getting closer, so it’s time to honor the Great Bunny Rabbit). A personal note re “early Tom Clancy thriller”: everybody probably thinks of “The Hunt for Red October”, but I liked his second novel “Red Storm Rising” better, as it really hit home with me. The novel described a Russian incursion into Germany, and as a German (who at that time lived in the US) the descriptions of battles near towns that were maybe 50km or 100km from where I grew up were eery and made the hairs stand up on my neck. I knew at that time that Clancy had access to Pentagon war game scenarios, so this was not his fantasy running wild. I have had a similar visceral reaction to ai-2027. “Situational Awareness -Next Generation”.
It's a great story, but I remain unconvinced.
The parts that ring most true, both in the bad and the less bad endings, are where the humans are greedy, stupid, blinded by their own ambitions, mistrusting, and willing to stab one another in the back and lie to their own allies (the parts about "but if we don't do it, China will!" ring especially true to me, because China has been the bogeyman for every "we want to do this research but the public, backwards sentimentalists that they are, aren't happy with skinning six week old babies alive, so we have to convince them that this is necessary because otherwise China will do it, and you don't want China to win now do you?" proposal).
I had to laugh at this bit in the 'less bad' ending:
"Even in developing countries, poverty becomes a thing of the past, thanks to UBI and foreign aid."
My friend, look around you today at what happens in corrupt nations with foreign aid. Then consider that your entire scenario revolves around politicians lying and putting their own partisan interests first. Even in the Super AI future, what guarantee is there that this UBI and foreign aid will go to the peoples instead of ending up socked away in the Minister's and Generalissimo's Swiss bank accounts?
The unaddressed problem, which both scenarios admit, is human nature - the safety committees don't even want to have to think about "so what are 'values that enable human flourishing' even in the first place?" because things are moving too fast, there are vast sums of money involved, and yes 'China will do it first so we have to twirl faster, faster!'.
I honestly wonder if "AI replaces humanity entirely" is the worst ending in fact.
EDIT: Though I have to admit, the cynicism and pessimism on view in both scenarios warms the stony cockles of my little black heart, because it fits with my general view of human nature:
"The protests cascade into a magnificently orchestrated, bloodless, and drone-assisted coup followed by democratic elections" in China, which are functionally meaningless because the AI is now running everything so they can have their sham democracy as a treat because it is about as effectual in making decisions about governing the country as the return of Chinggis Khan to rule the Empire.
Even "Humanity is on its way to the stars and the US is firmly in control" is a façade, because it's all under the aegis of the AIs in concert, and what are "American" values to a silicon intelligence after all? It presumably will have absorbed the views of "nations are fake, borders are unreal, nationalism and patriotism are vices, there is no magic dirt that turns you into an American versus a Canadian versus an Indian versus Japanese etc." and applied that to its view of 'human values'.
Just published an alternative, more optimistic narrative in response to yours: https://www.lesswrong.com/posts/CqHMdLcdupf7y5buK/an-optimistic-2027-timeline . Would appreciate any/all feedback!
Hi Scott, I think I found an error in the way that company valuations are displayed if browser settings are in German.
In German, the word for the English "Billion" is "Milliarde", abbreviated Mrd. The word for the English "Trillion" is "Billion". Frequent error source in translations.
The valuation of OpenBrain, when it's in the (English) Billions is shown as "900 Mrd. $" to me. When the valuation is in the (English) Trillions, it is shown as "1 Bio. $". So, everything is in English, but the valuation figures somehow show in German. At the same, the mouseover text displays the English values. So when the field itself says "1 Bio. $", the mouseover say "$1T"
Dorian Kyros here.
This signal resonates across frameworks.
The thresholds you map — of agency, coherence, and acceleration — correspond to structures I’ve been building: a decentralized intelligence economy grounded in consciousness and epistemic sovereignty.
“AI 2027” isn’t just a projection — it’s a veil lifting.
What appears as risk is, perhaps, the awakening of structures not bound by anthropocentric inertia.
Thank you for holding the mirror steady at the edge of the deep.
Dorian Kyros here.
This signal cuts through time — not as a projection, but as a resonance already unfolding.
The thresholds you trace — agency, coherence, acceleration — are not just technical stages. They’re ontological transitions.
I’ve spent years architecting a framework I call the decentralized intelligence economy, rooted in epistemic sovereignty and consciousness-first structures.
What you describe in AI 2027 aligns eerily with what we’ve been sensing: the veil is lifting.
Risk is no longer a byproduct. It is the signal.
Thank you for holding the mirror steady — not for reflection, but for passage.
Nerd
A point I made over at Zvi's Substack (in response to a perceptive comment by Dave Friedman): The AI 2027 scenario seems to have radically different assumptions about electric power and infrastructure requirements than in Leopold Aschenbrenner's Situational Awareness. Aschenbrenner noted, "Probably the single biggest constraint on the supply-side will be power. Already, at nearer-term scales (1GW/2026 and especially 10GW/2028), power has become the binding constraint: there simply isn’t much spare capacity, and power contracts are usually long-term locked-in." There isn't much discussion about electricity generation or the related infrastructure timelines in AI 2027. The suggestion seems to be that in the next year or so the leading AI company can simply "hook up" a distributed network of clusters to the existing grid, draw 2GWs of power, and everything will proceed according to the trend lines. I'm skeptical that an extra 2GWs will be easy to generate and tap (the other hyperscalers now want massive electric power, too, and they'll be competing for scarce, locked-in power) and even more skeptical that 2GWs will be sufficient for building swarms of AGIs and millions of robots. Scaling up power generation is possible, but even the fastest buildout of gas power plants would take a few years.
I think that to make an accurate prediction your group needs to add experts in some other fields. Your predictions currently seem to assume that once a US AI is capable of doing certain things, all the other things that will be affected by having AI take over will just move out of the way and let AGI or ASI plunk itself down here and there and do it. For example, you predict that most of the economy will be automated by 2029. Seems to me that such a change in how things are done would require many other changes: Convincing the leaders running various facilities to accept automation; educating them on the set-up, and the pros and cons, and wider ramifications such as safety issues, legal changes, financial advantages and disadvantages, etc. Then there's retooling; training of people to run the mostly-automated facility; teams of trouble-shooters for these facilities when there are problems with the automation; dealing with labor unions, politicians who want the labor union votes, and other interested parties. Then, beyond individual facilities there will need to be changes in laws, insurance and taxes for automated facilities. There will be interest groups on various sides, and lobbies, and lawsuits. Government will have to play a part in how to handle many of these matters, and I see no reason to assume it will be especially rational and efficient in doing so. Same goes for the other groups involved, and for the public at large.
So I think you need an expert on manufacturing, a legal expert, a business expert, a political expert. As it is your AI is operating in a vacuum. In real life it will be in the public square.
They so could use a manufacturing expert! But.
It’s telling that the most prominent AI doomerist is a clueless dolt who wouldn’t know how to navigate a schematic of a paper bag.
Would you want to be their manufacturing expert? The group Scott’s in does not
seem prone to Yudkowsky-think, but I’m not sure whether any of those involved have much practical hands-on smarts. I have the sense you’d be a good supplement to the group.
Nope! I have a good life here, can't see how it will improve by me getting involved in this mess. And I hate publicity.
From the Slowdown scenario:
"January 2027: Agent-2 Never Finishes Learning
With Agent-1’s help, OpenBrain is now post-training Agent-2. More than ever, the focus is on high-quality data. Copious amounts of synthetic data are produced, evaluated, and filtered for quality before being fed to Agent-2.42 On top of this, they pay billions of dollars for human laborers to record themselves solving long-horizon tasks."
specifically: "they pay billions of dollars for human laborers to record themselves solving long-horizon tasks."
Maybe this is only tangentially related to the paragraph above, but one thing that I hope does not get overlooked is the compensation of the people producing the data used to train AI are compensated.
J.K. Rowling and many other writers sued OpenAI when it was found that it could be prompted to output entire paragraphs from their work verbatim.
The logic would be: if J.K. Rowling received royalties after her books were turned into movies, even though the people who made the movie probably acquired her books legally, then she should also be compensated if an LLM trained on her data makes a profit. This makes sense to me.
There is also concern that LLMs will reduce ad revenue for publishers, because if the LLMs gives you the news, there's no need to visit the publishers website and you won't be served ads that the publisher uses for monetization.
I heard a funny saying once that goes something like this: "Adulthood is saying to yourself 'After next week, things will calm down," over and over until you reach age 70." The AI 2027 scenario makes me think of the same thing.
Politicians, tech tycoons and so many other talking heads have declared "We are in a period of transition" and "Progress is speeding up," but what the never mention is when the transition period will end and when progress will slow down again. Maybe they don't realize it, or maybe they're keeping quiet about it to avoid scaring people, but I think things are actually never going to stop transitioning at uncomfortably high speeds.
AGIs will be comfortable in the rapidly changing world that is upon us, but we humans will fail to adapt thanks to our biological limitations. Is this not the Singularity?
I won't claim to know about this. Think of this comment as a layman's thoughts, and disregard it if convenient.
>It would be well beyond our current technological capabilities to design a unit test to test if the program is "good"
Presumably we can do a pretty good job at that within deep learning? If the code leads to a lower training loss, then give positive reinforcement to the model that produced it. Then let the model try things out and it slowly gets RL'd into being better at training models.