"Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 2027"
Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027. Would you like to bet on that?
If you really think there's a 25% chance of "AGI" emerging next year you should be willing to take 3:1 odds, that is to say you either lose $1,000 or gain $3,000. I'm okay with those odds just on the opposite side of the bet. DM me and we'll work out the details.
There may be, but I don't think this is one of them. 18 month is not long.
Moreover I think we're going to have to start rethinking the words "knowledge work". Is being a plumber knowledge work? You certainly need a lot of knowledge to do it, but you also need other things like the ability to climb under a house and fix a pipe. Or a psychiatrist -- they don't even need to perform physical tasks, but still very hard to replace with a robot that says all the same things.
If we're going to define "knowledge work" as things that require knowledge and nothing else then we might find that these are easily automatable, but we might also find that there's fewer pure knowledge jobs than we thought.
I'm not here to defend psychiatrists but what the heck is the point of an outdoor treadmill? "Oh, I love walking but I hate actually getting anywhere".
My job as a tech lead is pretty much unrecognizable from what it was 18 months ago
Senior Engineers in my company no longer write code by hand
The more senior, the better they are at leveraging AI
Even I, as someone who absolutely have no time to do software development anymore (ugh), is getting medium sized PR's merged semi-regularly
I'm seeing some tickets that would normally have taken around a week of development time being replaced with 1 hour spec meetings and 2 hours of Claude churning while supervised by a senior engineer
Everyone is spending +30% of their time code reviewing
And as someone in a state agency doing knowledge work, nothing has changed in 18 months. If anything we might be moving backwards later this year, technologically, because our IT can't keep our software updated.
18 months is an eternity at the forefront of this tech. For a majority of knowledge workers, it's still not long at all.
Perhaps state agencies are not subject to the same pressures toward efficiency-gains that you see in the private sector? Most knowledge workers aren't government-employed and I'm not sure I love the idea of the government being impervious to efficiency gains, so this doesn't exactly comfort me.
Could you clarify — are you spending 30% more time in code review than previously? Likewise, could you get by with fewer lower-level coders and leave your senior coding staff in place?
No, we all spend around a third of our work time reviewing code. I hate it :)
Junior developers definitely feel more dispensable, as the kind of small tickets that are good for Junior developers, can be done by Claude with less supervision than what a Junior dev needs
But I'm also seeing Juniors being able to work more independently on bigger projects assisted by Claude. But without the speedup
I think the main skill that differentiates the seniors from the juniors here, is being able to read and understand code quickly
I suspect that cutting our juniors would not impact our velocity a lot, but I'm not convinced that that's significantly different from the pre-LLM days
I'm happy to bet you at the 1000:1 odds. Would you like to bet my $10 vs. your $10,000 that college grad unemployment (currently 3%) won't be above 20% on 12/31/27?
That's not your claim though - if it can do 90% of 'knowledge work' then you should be willing to bet on more than increase in college grad unemployment. Not to mention a lot of those grads will choose alternative non-knowledge work anyways so it's not a good metric.
Speaking only for myself, I'd gladly bet my $1,000 vs your $1 (sorry, I don't have as much money as I'd like) on your original claim: that "AI" will replace 90% of knowledge work positions by 2027; pending some stricter definitions:
* What exactly counts as "knowledge work" ?
* Are we counting every individual job/headcount, or only every *type* of job ?
* What does it mean to "replace" a worker ? You seem to be advancing a weaker claim about college grads (implying that existing jobs are safe); I'd be willing to bet on this weaker claim as well, but at weaker odds.
* Mechanically, how do we actually execute the bet ? Do we submit the money to escrow, or something ?
Right, but if you were to bet that you'd lose your job, you'd win my shiny new $1,000 !... which would arguably worth less than 10 bottlecaps at that point, but still... $1,000 !
That's _very clearly_ not the original claim. If you take both AGI timeline and diffusion gap timeline original claims, together they give 1/16 probability of replacing 50% of knowledge work positions by 2030.
I am willing to adjust the payout on those odds, contingent that "50% of knowledge work positions" also counts any new positions that arise by 2030, and that we clearly define what the term "knowledge work" means (in addition to my other concerns above).
The diffusion part refers to AI doing 50% of knowledge jobs. In the central scenario (50% chance) he's saying this should be happening less than 10 years after "AGI" has emerged. So, if he really believes "AGI" has a 25% chance of emerging next year, he should also be willing to bet on employment starting to collapse either in 2027 or 2028 (since by 2037 presumably 50% of all knowledge jobs will have been replaced by AI).
Anyway, like I said in the previous message: the author should DM me and we'll agree to the details. I don't particularly care if the year we bet on is 2027, or 2029 or 2030. But I do think that a bet involving a very far-out year is impractical.
50% of all jobs replaced over ten years is 5% per year
Are you willing to bet 3:1 that total employment in knowledge based fields will have dipped 5% by the end of 2027?
Although realistically I'd expect that the rate accelerates (a slow trickle of early adopters, and then later people start jumping on the ship as those early adopters prove it's viable). A 10% dip by 2030 or so seems like a much more reasonable wager - if things are going down at all, something pretty radical has probably occurred
|| 50% of all jobs replaced over ten years is 5% per year
5% per year leads to around 40% of jobs at 10 years. That aside, the progress might very well be lumpy, not smooth. So we might see a few years of little job replacement, then very fast progress followed by another few years of quiet.
Without going into too much detail I work in something adjacent to the public sector (though not security-related). We have no access to normal LLMs (they're all blocked on work computers), instead we have a government-provided AI portal which is ok but has no agentic capabilities (it's just a chatbot).
Despite this my work is mostly writing and the chatbot could still be very very helpful for a lot of people I work with since a lot of our work is just editing people's writing in a way that chatbots since GPT-4 have been capable of. But AI isn't promoted at all at work and most people who bring it up just think it hallucinates all the time.
Yesterday I had to look at some publicly available data for work; for the first time I just pulled out my personal laptop (I was WFH) and had Claude look at it. Saved me ~30 minutes.
His diffusion gap means that he only thinks there is only a ~6% (I'm not sure that the odds actually multiply that way, but hopefully you get my point) that it actually collapses by then. There is a difference between _can do_ and _does do_.
That's clearly wrong. Chemistry is a quantum effect. It would not be surprising if various specific quantum effects were used in various specific ways. (We know that photosynthesis uses such approaches to channel photons.)
OTOH, tunnel diodes are also quantum machines. (I suspect that's true of all transistors and vacuum tubes, but I haven't checked.)
Quantum isn't some sort of magic, it's the underlying reality that shows up whenever you look closely enough.
That said, large scale correlation at room temperature had never been demonstrated. (But you can do it at the molecular scale.)
Well, microtubules clearly exist. That they cohere entanglement, however, hasn't, AFAIK, been demonstrated. (I'm not clear about Superradience.)
Saying that "human brains were too \"big wet, hot and messy\" to actually exhibit quantum effects" is clearly wrong unless you mean as a composite entity, in which case it's clearly correct. (At the organelle level this gets messy. Ribosomes clearly depend on quantum mechanics to work...but the quantum effects that they use are short-range.) Every process that happens within the brain will be quantum if you look at it closely enough...but closely enough is EXTREMELY closely. And this is also true of all electronics. (Also just about everything else.)
So I'm sure that microtubules depend on quantum effects to work, but probably only over a short distance. Correlation tends to decohere...i.e. when one of an entangled pair interacts directly with something else it tends to spread the entangled state leading to the loss of the ability to predict correlation. (This is why "warm and wet" are significant.) (I tend to model this with the multi-world model, where in some of the worlds the entanglement shifts to the newly encountered particle and in others it doesn't...and you don't know which one you are in until you measure it.)
I don't think there's any intrinsic reason why artificial hardware can't be engineered to use quantum effects, and there are already companies using cultured human brain cells as a computational substrate.
In any case, this is totally orthogonal to the question of intelligence. Lots of physical systems exhibit quantum effects and aren't intelligent, and the broadly-accepted definition of intelligence doesn't require quantum effects. You just have to absorb and process information and use it to make predictions about the world and/or make plans. None of that requires entanglement or superposition or tunnelling etc.
I think the answer is: the concept of a "real" AGI falls apart as you get closer to it. Soon we will say things like "humans and machines are both intelligent, but in different ways".
Is dolphin intelligence above or below human level? They can see by sonar, which unlike human (and dolphin) eyes actually creates three-dimensional images; eyes form flat images, and then brain fills in details, sometimes incorrectly. So dolphins perceive the world in ways fundamentally different from humans, which makes comparing “levels of intelligence” very difficult.
Any AI will have senses so different from human senses, that it arguably will inhabit a completely different universe. For starters, Internet will be as real and tangible to it as air and water are to us. Any machine intelligence will be so different from ours that in comparison dolphins will be pretty much our mental twins. AGI will not be “at” or “above” human level intelligence — AGI will be orthogonal to it.
And a human brain is "just" a biological expression of natural selection. I'm not convinced its functions are ever going to qualify as intelligence, even though it presents many signs of intelligent behavior.
LLMs are, in their current instantiation(s), clearly able to do something like computation¹ and reasoning: e.g., you can give one a completely novel problem, and it can solve it and make correct inferences therefrom; hence, it is hard for me to see why they shouldn't be able to be intelligent,² at least in theory—after all, what *is* intelligence except a sort of computation?
(Not to say that I'm certain that sentience and qualia and selfhood boil down to computation; perhaps substrate does matter, for these things; but it is at least plausible that they *do* reduce to computation. See also footnote #2.)
On the linguistic aspect: of course, they remain large *language models*... but I don't think that the "language" part of that title is quite as salient as it was initially, in terms of judging what's possible for the models to manage. The higher-level (than e.g. "generate next token") structures observable in the LLMs' "thinking"-processes are clearly capable of application to tasks with only the most tenuous of connections to language—if any; I mean, one can stretch the definition such that e.g. "well, even mathematics and society and art are also really just 'linguistic', in a sense!"... but at that point, the term is no longer much of a restrictor in the ("they're just *language* models!") objection, either—and it seems possible that said objection is perhaps like saying, I don't know, "humans are just *hand-movement* executors; like 99% of all they can do is ultimately reducible to simple hand-motions, once you get them to the right location!":³ sure, sure, that IS how we interact with the world, and how even the most impressive of our achievements are executed, when you get right down to it... but it's not really the most informative lens through which to view human activity/capability, for most purposes!
(...or so it seems to me. I'm not an AI expert of the same caliber as that of esteemed host. Or even of *any* caliber, in fact–)
-------------
-------------
¹ (meaning "on a level above the hardware they run upon", of course—not in the sense that they "are" "made of" computers.)
² (well, they're arguably "intelligent" even *now*—if we mean something like "problem-solving and information-processing'—but I'm operating upon the assumption that we mean something like "sentient" or "self-aware", or at least "*human-level* intelligent".)
³ (most newer models can navigate *themselves* to the correct context, with the release of the exciting new "Legs" update. plus, for Pro-tier subscribers, the new "Feet-&-Shoes" FastNav™ package—comes with two Socks® adapters for easy install!... ...okay, sorry, I'll get my coat–)
A dolphin might design an IQ test with problems like "count the number of metal fragments embedded in this piece of driftwood," and only (grudgingly) admit that some humans may be capable of basic math after the test procedures are updated to allow assistive devices, such as an X-ray machine.
"Define AGI as AI intelligent enough to do 90% of knowledge work jobs."
He literally defines this in the second paragraph. If you can take a person in a knowledge work job, replace them with an AI, and 90% of the time the AI will equal or exceed their average work performance, then we have AGI. It doesn't matter whether it's "real" intelligence in some philosophical sense, it's about whether it can do your job. If it fails enough easy tasks that are required for your job and can't figure out how to compensate for that with its other skills then it's not AGI.
> Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027
Nope, it takes time for technology to be adopted - inertia, regulations, budget, etc. The issue is only forced aa early adopters use it to win in the marketplace and supplant their competitors. That takes time. Contracts are years long and switching incurs internal risk. Consider the US didn’t have universal indoor plumbing until the 80s! If you want to see which way the wind is blowing though, my company no longer hires junior engineers. Firing people really really sucks so we will probably keep the ones we have even if it isn’t the most efficient allocation of resources. We are ahead of the curve - in a few years, everyone else writing software will adopt the same position (or die - that takes time).
For what it’s worth, I think Fable already meets the bar.
> For what it’s worth, I think Fable already meets the bar.
It does not. Trivially, because it seems to contain a hardcoded algorithmic prohibition on doing anything remotely related to biology, even simple things like "Parse biological file format A and output it as B". More importantly, because even if you do manage to coax it into writing some code, it still tends to get stuck in loops and to write meaningless nonsense just like Opus. Not as much nonsense, perhaps, but still a nontrivial amount.
>Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027.
That would need to wait for both AGI in the labs plus _diffusion_ time. Scott wrote:
>I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.
Why would employment in knowledge jobs collapse? If overall output increases by more than 10x, then what was previously 10% (= 100% - 90%), will become as big as the whole pie before.
>Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027.
Non sequitur.
Tokens are expensive, markets are neither infinitely nor instantly efficient, and there's a thousand miles of difference between a cutting-edge piece of tech demonstrating an ability under controlled settings vs. a working enterprise system that can be immediately integrated into your business to handle that workflow automatically.
If markets were this efficient, plantation owners would have been finding the smartest slave children and teaching them to become architects and engineers in order to optimize their productivity. Things take a long time to happen and there's a huge amount of inertia and cultural bias in every system.
Have you read AI 2027? I think it does a good job making this case.
I'm not actually sure employment would collapse - the economics is weird, and even now with AI writing >90% of code the market for programmers has barely changed.
The person below is offering me better odds, so I'll probably take their bet rather than yours.
"Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. "
The only reasonable way to interpret this passage is that AI would be taking the jobs, rather than workers using AI to become more efficient or something of the sort.
If you actually meant "job characteristics will change but white collar job numbers may not decline", that is:
1. A totally different thing to predict
2. Much harder to falsify. Yes, AI will change how we do our jobs, but the same could have been said about WhatsApp, fiber optic, the expansion of subways, etc
Yes, but I think super-convincingness has already been demonstrated by ChatBots. That is doesn't convince all people isn't a required bar. "A man hears what he wants to hear and disregards the rest" isn't completely true, but it's true enough to be weaponized.
If an AI is suddenly able to do 90% of "whatever 'knowledge work' consisted of last year," that doesn't mean 90% of human knowledge workers are automatically out on the street. That last 10% probably includes some critical bottlenecks which are suddenly far more valuable - thus inclined to hire more people - and previously unknown forms of work will emerge in response to whatever problems the AI's results reveal or create.
(This had already happened before. There was a time when a large part of knowledge work was calculation and data processing. This was increasingly taken over by computers, to the point where we don't even consider it work anymore.)
Thing is, this makes the 90% threshold meaningless. The only meaningful one is 100%, anything less ultimately implies "machines automate boring parts so we can utilize our efforts where it matters more" rather than "machines take over".
No, the jobs won't go away. The David Graeber phenomenon of bullshit jobs will just expand to include the set of "AI bullshit jobs" (jobs that could be done by an AI, but we still employ a human to do, with no good reason). Most low level engineering jobs (not necessarily strictly Junior roles but often junior roles) already are effectively "fake AI jobs" in this sense.
How are you going to distinguish between AI-driven job replacement and an ordinary recession, which we're pretty overdue for anyway? I would suggest looking for coordinated rise in both unemployment and productivity.
My personal opinion is that tech transitions are recession-driven. Most companies don't pay attention to new technology until they have to and serious economic downturns are what force them to. That's what creative destruction is all about. Things don't diffuse smoothly, they proceed by punctuated equilibria. I expect the next recession to be deep, perhaps Great Depression level, and that's when AI will really begin to take over. They key metric will be a return to GDP growth without an associated drop in unemployment. The beginning of the recession will look like any other - GDP contraction, high unemployment - but then GDP will recover amidst persistent high unemployment. I'd hate to be graduating with college debt anytime soon. Legal and Medical guilds will probably be insulated by regulation for a while, but if I was a med student I'd definitely be going surgical.
> a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).
This is my view. Although I also rely on the EMH, which indicates that AI will be lucrative but not crazy disruptive enough to show up in other asset prices.
I think dates well past 2045 are plausible if the ability of AI to be good at most long-term creative projects (writing novels, developing new scientific theories) requires continuous learning with constant re-weighing of the relative significance of ideas relevant to the project (lots of little a-ha moments where things shift along the way), and if continuous learning itself has to work in a way similar to how animals such as ourselves learn, with embodied learning in which sensory impressions and motor actions are are closely connected to language, starting from a baby-like state with a fairly random pattern of neural connections. This would be in contrast to the idea that we could start with LLM-like supervised learning to develop as much use of language as LLMs have, and then afterward tack on a phase of unsupervised learning to connect this with sensory experience and develop more idiosyncratic personal opinions; if that tacking-on strategy worked, then getting to AGI would probably be easier.
If this sort of process starting from a baby-like state is the only successful strategy for doing continuous learning, it might also be that constraining things to behaviors we find meaningful and useful (as opposed to obsessing over 'irrelevant' patterns or self-wireheading) depends on a huge number of innate sensorimotor biases which canalize the developmental process, similar to what animals have, and that the only way to fine-tune a good pattern of interacting biases is by some evolutionary process of variation and selection. In that case the fine-tuning could be a very slow multigenerational process, and uploads might beat attempts to create novel AGI, even if perfecting uploading without existing AGI to help us takes a long time.
Yes, so far stock prices make it look like there's lots and lots of customer surplus in AI.
Both for actual end users, and for other non-AI companies in the economy. It's also really easy for anyone with a bit of money to spin up a new near-frontier AI model: they become cheaper and cheaper, even if bleeding edge frontier AI becomes more and more expensive to develop.
If we'd had this discussion any time in the past ten years, you could have raised the same objection, and every time, I would have been right (given today's market prices) and you would have been wrong. For example, in 2019 you could have said LLMs weren't going anywhere, because OAI was only valued at $1B. But today it's worth $1T. I think this is enough failures of EMH to incorporate AI that we shouldn't trust it too blindly going forward.
But also, I'm less sure which way the EMH points anymore. Aren't we spending close to $1T on data centers per year? It seems like *someone* expects AI to be very big.
But if you open up the discussion to historical analysis, you might also consider that someone expected real estate to be very big. And various CMS startups. And tulips...
Which is fine if you take the view that the EMH is a description of a mechanism (that prices update as information flows) rather than a universal law like gravitation.
Elon Musk just became personally worth over $1T after the IPO of SpaceX. It wouldn't have been worth nearly that much if it IPO'd a decade ago, and that wouldn't be the case because the EMH can't handle space, but instead because super valuable companies take time to demonstrate they really are worth that much, while many others fail to achieve their earlier promise. I mentioned Amazon in another comment, and if you look at its stock price history, it has also grown an order of magnitude over the last decade decade https://stockanalysis.com/stocks/amzn/history/
In addition to the two obvious counter points you've already received (the market has a terrible track record, and the market *is* saying right now that this is the biggest thing to ever happen in the history of the world), even if the market at large did believe this, how would it express this? Other than investing in the current companies of course, which seems to be what's going to happen (cf. NASDAQ).
The EMH is enforced by arbitrage, and there exist no arbitragers at the time horizon we're talking (a full decade), so the asset market's silence on this issue gives you roughly zero bits of information.
The market is predicting that AI companies will make a fuckton of money for tech companies but that other sectors will be just fine. Also that the US will keep growing slightly faster than other developed countries but not that the US will 100x as the winner of the AI race while Europe will be left behind.
So aside from an impressively large bet on US tech the market is predicting that the rest of the world and the rest of the economy will mostly continue as they are.
My brother you managed to ignore my entire comment.
I was trying to make the point that the market is *not* predicting that everything will continue mostly as they are, because there is no affordance for the market to make such a prediction.
Think about how that would work: it would require hedge funds and financial people who are all evaluated on a quarterly basis to make bets that would seem to lose money for 4 years or more in a row, before making a massive profit. This is impossible in our current system.
Also as I alluded to, the EMH has been proven false many more times than it has correct. No rational person can believe in the EMH. The reason for this is also well known: "the market can remain irrational longer than you can remain solvent".
I will just repeat what I wrote in my previous comment here, in the hopes that you will read it again:
the asset market's silence on this issue gives you roughly zero bits of information.
And investors are correctly predicting that AI companies will be enormously profitable.
Frikgeek's point was that the rest of the market doesn't seem perturbed, and that same argument could be made about Amazon historically; the market did not predict its disruption.
Also this:
> The market is predicting that AI companies will make a fuckton of money for tech companies but that other sectors will be just fine.
is logically incoherent; if AI companies make a fuckton of money, other sectors can't literally be unperturbed. (Not at the scale of fuckton we're talking.) So this is actually further proof that the market is currently irrational: either the bet on AI companies is wrong, or the rest of the market is failing to price in the pertubation that will happen.
No, it's not logically incoherent. Amazon got into retail goods, and a big retailer they would be competing with would be Walmart. Here is Walmart's stock price over the last 15 years https://www.macrotrends.net/stocks/charts/WMT/walmart/stock-price-history Of course, the stock market in general tends to go up over time, but you can see that (like Amazon) it is up multiples of the price at the start of that range (which in turn is multiples higher than the price in 1997, when Amazon was founded). There are also other retailers who have done more poorly over time, and some like Kmart have gone bankrupt since Amazon was founded. When looking at an existing company vs its past survivorship bias will always come into play.
I'm sure this opinion will be unpopular in this particular comment section (although it's a pretty popular opinion generally), but a lot of these very optimistic projections of AI improvement make quite a few assumptions that are really not in evidence. I'm not saying that AI has no uses (it's notetaking capabilities certainly make doctors' lives better), but the idea that AI is going to wipe out all jobs or become super-intelligent--I just don't see that as a LIKELY scenario, even if it is possible. AI has some pretty significant limits, such as hallucinations and massive energy usage, and at present there is not really a way to fix those (the energy usage and infrastructure requirements may be fine for now, but eventually these AI companies are going to need to make money). Maybe they'll figure that out, but just hand-waving away that objection is a pretty big miss.
I have noticed that AI is much better at answering questions that have been posed on reddit at one point, and not so great at answering questions that haven't been asked before, even if they aren't that hard for actual people. Basically, it's the Clever Hans scenario where the subject can answer any. question, as long as the answer is available.
I don't know anything about baking personally, but my wife ran a cafe for a while that did commercial baking, and 12 servings is a totally reasonable batch. You seem to interpret the the phrase "commercial baking" to mean something like "industrial scale" or "literally a factory", but "commercial" is a wide and ambiguous term, and it seems pretty disingenuous to not include the many tens of thousands of commercial bakeries in any given country in that!
I asked Fable about this, and it concedes your point that its own recommendation wasn't entirely correct, but it also criticizes you for being wrong about "pastry flour", according to whatever German sources it found on the internet:
> Bienenstich is a yeasted dough, and yeasted doughs need enough gluten to retain gas and to survive being split horizontally and loaded with a heavy almond caramel lid. The German baking guidance is explicit about this division of labor: Type 405 for fine cakes and Mürbeteig, Type 550 for elastic yeast doughs — yeast doughs need Spannkraft so the dough can hold gases and rise properly, and the German flour industry's own guidance for Hefeteig recommends wheat flours from Type 550 upward. Type 550 is roughly American all-purpose. American pastry flour (~8–9% protein) sits below Type 405 — weaker than what German sources recommend even for the soft end. And actual German bakery-style recipes bear this out: a Konditorei-style Bienenstich recipe uses Weizenmehl Type 550, while homier versions hedge with "Type 405 oder 550". Nobody's reaching for pastry flour. Lusina + 3
> But here's the partial concession, and it cuts against me too. The direction of the criticism — "softer flour than the AI said" — has merit. My recipe specified bread flour, which is the structural overshoot in the other direction: it gives you a chewier, breadier crumb, whereas a proper Bäcker-Bienenstich is tender and fork-soft, closer to the 550/AP zone. If Elvira had said "all-purpose or Type 550, not bread flour — Bienenstich should be tender, not chewy," that would have been a legitimate correction of my recipe and a defensible critique of strong-flour AI defaults generally. "Pastry flour" overshoots into a claim that contradicts professional German practice. So: right direction, wrong magnitude, asserted with the confidence of neither.
> "I know nothing about this, so I'm pretty sure Computer is right"
Is that supposed to be me? I did ask what was wrong about the reply, and what would have been the correct reply, and the only opinion I gave was carefully qualified ("[…] looks fine to my layman eyes"). That should be enough "epistemic humility" for everyday purposes.
> Also 12 servings is a tiny batch for commercial baking someone's trying to make money off of.
Now that's plain wrong. I don't know where you live, but here in Germany there are tons of small Konditoreien (pastry bakeries?) that sell pieces from single-batch, self-baked cakes. They're doing this for a living, so without question it's "commercial baking".
Ok, so that seems like the kind of a error a typical inexperienced human would make. It just doesn't have the domain expertise because the training data probably doesn't have all of the practical real-world knowledge. All it would take is a consultant spending a few months working with a bakery to optimize its behavior, then that consultant can re-sell the trained LLM to all other bakeries. This is the equivalent of most businesses not knowing how make a website in 2002.
If you believe that the most noteworthy (pun not intended) use of current AI is as an automated transcription service, then I think that you are very far behind on what current AI can do.
I could be wrong, but I think the OP was using doctor's note taking (a.k.a. "charting") as one unambiguous scenario where LLMs could do a tremendous amount of good. From what I understand, human doctors spend a significant amount of time (perhaps as high as 25..50%) on charting, and thus an auto-charting LLM could boost the doctor's efficiency by a factor of 2x... assuming of course that it could do so without introducing hallucinations into the chart. AFAIK at present the technology is not there yet, but it's getting better every day.
I don't disagree that there is value, but "AI as transcription" is very, I dunno, 2022? My point is that someone who questions economic activity coalescing into AGI within a short timeline should be using more recent use cases (agentic such and such) as a reference point.
I haven’t seen much evidence of agentic AI use transforming how companies work at this point, at least not the way that note taking software has unambiguously made doctors’ lives better. Honestly other people using AI in the workplace has been more annoying than not in my experience since it typically produces a shoddy product (ie, extremely long emails that don’t exactly address the point).
There are companies that have replaced their entire performance marking department with AIs. There are tiny companies running high volume CPG businesses mostly out of OpenClaw. I see vibecoded vertical SaaS apps that go from 0 to hundreds of thousands in yearly revenue in 2-3 months, solo developer.
I see a lot of claims like that online and yet I also see a lot of terrible customer experiences with companies that replace their departments with AI. Could you give us a working example where it actually worked? Good for you if you went to hundred of thousands in revenue in 2-3 months, you must have solved all the distribution problems with AI too?
I’m sure just about everything has happened occasionally, but I am very skeptical the stuff you are describing is taking place at scale. It’s like how back in 2015 or so people acted like drop shipping was the way to wealth—it was in a few cases, but generally it was not.
And a lot of the cases where companies start using AI don’t prove successful. I had AI generated ad copy. AI customer service is terrible, AI coding tools have proven less successful than anticipated, etc
This is true, but I'm also a bit worried about AI charting. The way it currently goes is that an AI listens to everything a doctor says in an appointment and then writes it down. This works fine, but it makes it harder for the doctor to recommend gray-area things like taking illegal drugs, massaging the way they report things to insurance companies, being less than maximally-aggressive against diseases that might one day result in a lawsuit, etc. I know a lot of people who feel like the quality of their care has declined.
That is makes sense. I saw something similar with AI note taking for business meetings. They provide value but the fact that EVERYTHING is recorded and disseminated makes it very hard to have completely candid conversations
For example, consider "the best solution to your form of seizure is marijuana" in a state where marijuana is illegal but easily available.
Another example is illegal forms of legal drugs, for example "Your insurance won't cover this important medication, but you could easily buy the Indian formulation on buyindiandrugs.com."
I don't think anyone had posted the answer to Erdos Problem 90 to Reddit. Yes, that's the extreme pinnacle of AI accomplishment, but it seems that if AI is capable of functioning on that high a level, people will find ways of adapting to most white collar work.
"AI has some pretty significant limits, such as hallucinations and massive energy usage, and at present there is not really a way to fix those (the energy usage and infrastructure requirements may be fine for now, but eventually these AI companies are going to need to make money). Maybe they'll figure that out, but just hand-waving away that objection is a pretty big miss."
I don't think the energy use matters much - energy isn't the bottleneck for data centers anymore, and future AIs won't use much more energy than current ones.
Otherwise, I think it's a question of whether you prefer to reason by assuming that things will stay at exactly the current level forever, vs. extrapolating trends. How are hallucinations compared to three years ago? How are capabilities?
This comment is a good example of my big issue with AI maximalists such as yourself. Basically every technology ever has had a similar trajectory of rapid improvement that then levels off. See phones: in 1997, most Americans didn’t own a cell phone, by 2007, most owned a flip phone, but 2015, most owned a smart phone, but it’s not like smart phone technology is drastically better in 2026–if we all had to go back to the iPhone 5 life would be pretty much the same. But somehow, we’re supposed to assume that the trajectory of AI will be totally different and it will just get better indefinitely until it achieves godlike intelligence and omnipotence. That is just a huge assumption to make.
As far as I can tell, while AI has improved a lot since 2022, I’m not sure it has improved as much as some people here think it has. AI customer service is terrible. AI generated pictures and text are the same—if anything now that people can see all the “tells” it is probably less effective than a few years ago. Hallucinations are still a big issue. And contrary to your assertion above, energy bills and data centers are big concerns for a lot of people. And eventually these AI companies are going to have to make money, and that will require either charging a lot more for AI use or enshittifying the existing models to maximize revenue—or both.
It just seems insane to me for an intelligent guy to try to argue there is a 25% chance AI will be able to automate 90% of white collar jobs in 18 months or less, when the actual odds are pretty clearly more like zero.
I think the main reason people like Scott believe AGI is inevitable or even possible, is that humans are an existence proof of general intelligence (produced by a dumb optimization process).
I think people in your camp have to make some reasonable argument for why the plateau will come earlier for machine intelligence than biological intelligence?
Also I'll add that your point about iPhones isn't even true. At iPhone 5 level technology, the internet speeds (throughput mainly) and framerate of the display wasn't really there to support applications the likes of TikTok. That's a ten billion dollar business that couldn't exist at the time (I'm not saying we're better off for it, though..)
Machine and biological intelligences are completely different processes. And biological intelligence evolved over millions of years while apparently we're supposed to reach AGI within a decade.
I used an iPhone 5 as late as 2019 and it definitely supposed apps like Tiktok at the time.
TikTok would not have taken off with the broadband and display technologies of the iPhone 5 era. Its commercial success relies on smooth, fast, high resolution, low latency video playback, which was impossible on the iPhone 5 (and the broadband infrastructure that existed at the time).
I think it is an interesting angle and there is a lot of truth to it.
However, this would have predicted that LLMs as we know them today would be impossible in our lifetimes, and many people did predict that as late as 2023. Those failures of predictions should make you update at least at little.
I do not think that many people (in 2023!) predicted that the AI we have now wouldn't be possible. I'm sure that some people did, but at least since around 2022 most "experts" have been much more bullish about AI than otherwise.
It doesn't appear to me that machine and biological intelligences are completely different processes. Amongst real AI scientists, not even Gary Marcus makes that claim. He just thinks a different ML paradigm is needed (but he has probably the worst predictive track record of real AI scientists, so while he might be right, I'm not updating much on his beliefs).
The "stochastic parrot" camp I can't take seriously. They really don't know what they're talking about, and when they speak it's generally embarrassing, sorry. Many people, including Scott, have argued (correctly) that "it's just doing next-token prediction" is false or irrelevant, depending on the framing.
It may well be that the differences between machine and biological intelligence are bigger than we realize, or matter more than we realize, but that would just be a guess. I haven't seen good arguments for it, and I suspect it's not true. The things Anil Seth has said recently are next on my list of "possibly-serious deflationary arguments about AI to read", so maybe my opinion will change, but I believe he's talking about consciousness, and I'm not sure what connection he posits between consciousness and capability.
Perhaps you should ask ChatGPT to explain the difference between machine and biological intelligences. They seem vastly different to me. Certainly the mechanisms and timeframes are totally different.
I just asked for an AI summary of Anil, as I hadn’t heard of him, but:
As much as I love anyone who takes the mystery of consciousness seriously, I can’t get behind "consciousness evolved to regulate metabolism". Oh, you mean exactly the job a non-conscious algorithm would be *great* at?
He also warns that AI will become extraordinarily capable without becoming conscious. But isn’t that what we want? The last thing we’d want to do is create a sentient slave-species! What an insane view, and not at all deflationary.
But let's say you don't care about any of that. Maybe the low level implementation doesn't matter - it's the functional outcome that matters. The brain and deep learning models are still fundamentally different. In a deep learning model, data comes in, gets processed and then an output is produced. That output then gets compared to what it should have been, and the every single weight is updated. Eventually the weights are frozen.
In the brain, sensory data gets processed and neurons immediately change based on the Hebbian learning rule ie. coincidence timing - neurons adjust their connections based on which other action potentials happened to be close in time. Every single neuron marches to the beat of its own drum, and learns every time it fires. Learning is distributed, asynchronous and bidirectional rather than feedforward, and backprop. And importantly, biological organisms are active participants in their own learning process. Sense datum is processed into an internal representation, then that is evaluated by the brain based on goals, and desires which can range from very low level processes (such as pain avoidance), to higher level goals (wanting a tattoo even though you know it's going to be painful as a means of self expression). And then finally, organisms take actions, which change the world creating a feedback cycle of active learning.
But maybe you're not even a functionalist - maybe you think intelligence is the end result, not how you get there. The end result is different as well, machine learning models need tens of trillions of tokens worth of code to learn what a human programmer can learn in a few million. And the reason is at a high level, deep learning and biological learning are optimizing two completely different processes.
Deep learning asks the question, given a sample data distribution with some structure, how do I predict the unseen data? Biological intelligence asks the question - given a sample data distribution from a persistent world, how do I model that world, and then choose what actions to take to accomplish my goals, and gain a stronger understanding of that world? You're not at the mercy of a random sample, you can essentially make queries (by performing actions) to get a more enriched sample.
It's worth thinking through why this leads to such different results. Suppose a deep learning model encounters some rare input. It gets a strong gradient signal, but by itself, that signal is random in some sense. It's not going to capture the underlying structure or reason for that outlier. The model basically has to wait for more a lot more examples of similar unexpected inputs before it can start to model it correctly. By contrast, if you get an unexpected input, you can take actions to get more information. This is what kids do, they are constantly crawling, running, jumping, smashing objects together, dropping things, building things. They're performing lots of mini experiments to flesh out their internal world model.
Turing Completeness says that any Complete computational system can simulate (implement) any other (and LLMs are turing complete).
You might already know these things, I'm just putting them out there in case we disagree thus far.
If you take a step back and consider the overall structure of how an auto-regressive neural network works, it's actually a pretty neat way to run neural nets in a loop, where they also get access to their own previous outputs; which clearly is a prerequisite for doing anything resembling human level intelligence--but note it doesn't have to be natural language as we understand it, any sequence of tokens can implement that architecture. This makes it pretty obvious, with the above theorems, that an auto-regressive NN is a fine way of implementing any algorithm you'd like, so I don't really see the need for a "new architecture".
Another way to think about human intelligence is that it was produced by the optimization pressure of evolution, an incredibly dumb process. So we have an existence proof that "human level intelligence" is possible, and can emerge from a straight forward architecture. The question is mostly about compute, and Dynomight makes an interesting observation [here](https://dynomight.substack.com/p/data-wall) which argues that compute is really what matters, not architecture, and that evolution is where all the learning for "human level intelligence" happened. The current architectures build ontop of that; obviously if training LLMs had nothing to learn from, instead of the entire human corpus of text, we'd not have the massive AI success we have had so far! And considering how far comparatively little compute has gotten current AI, you'd have a pretty hard time arguing it's anything more than one or two orders of magnitude away from "human level intelligence", no matter how you define that.
I'm also approaching this very much as a functionalist, I'm not saying they will literally be sentient, conscious, or "solve love". But replacing office workers, especially software engineers (which is my chosen profession); that seems prrreeetty inevitable at this point. The camp of people who believe this is "fundamentally impossible" seem to rely on some kind of implicit assumption about human specialness, which I take to be pretty arrogant, and reject on the grounds of believing in evolution.
If your question was not about why I believe current architectures are sufficient, but rather why *not* some other architecture, I'd say sure, why not. Other architectures could also surely work. But what we have is a surprisingly general architecture, and I think people are getting too hung up on the "language" part: I see language as an implementation detail; a way that happens to work decently at encoding a wide variety of problems, and computations over those problems.
I guess the main reason can be summed up as: the thing we know produced "human level intelligence" was loads of computation, not *clever engineering*, and the current paradigm is very amenable to scalable compute.
Another way to put that: I think people who claim auto-regressive neural nets aren't a sophisticated enough architecture and that we need to invent better architectures to reach AGI, are falling for the "human specialness" trap, and that isn't supported by the evidence; neither the last fifteen years of enormous empirical success, nor the story of evolution.
(Note: I don't think this is a good thing, I'm in the "this is incredibly dangerous and may very well lead to an unrecoverable civilizational catastrophe" camp.)
Assuming arbitrary scale, then NNs are universal approximators and Turing complete, when they have enough training data. That means that they can model every function. A function is a process that maps an input to a definite output (or set of outputs, distribution on outputs, etc.). A function operates with numbers or domains that can be embedded in numbers.
Intelligence, as practiced IRL, operates in domains where (1) there is not necessarily one right answer, and mapping to a distribution or set is unmeaningful/lossy, and (2) even if there is, the full input (including all context, knowledge, etc) cannot practically be encoded, transcribed and fed into the system, and (3) its impractical to get enough high quality training data to cover enough of the input/output space, and (4) the domain does not map cleanly onto numbers (this may be redundant with (1)).
Okay but let’s assume we somehow don’t have to care about these issues, because we’re in a numerical domain, we have adequate training data, we can feed all material context into the system - essentially NNs “solve” a cognitive task perfectly. I still don’t see why we should think this fully explains human intelligence. There is one ***major*** difference - human intelligence is stateful. You don’t have to believe in a soul to think so. LLMs are not stateful. They generate every word anew. If, for some reason, you’re in the mood to absolutely torture metaphors and go head first into full on AI psychosis, you can say that our consciousness is no more than a continuous autoregressive process, and our interiority is an illusion. I won’t follow you there.
The issue isn't how much better a new technology gets on an absolute scale, but how good it is relative to the alternative. In your phone example, why don't you look at how many people use landlines.
AI might not improve much more, but even at its current level it's much better than humans *for the price*. AI generated art might be boring a derivative, but it's basically free. If I have a website and just need some clip art then I'm going to use the thing that's 80% as good for 5% the price. Your argument is like saying that IKEA furniture is clearly inferior to any handcrafted 19th century chair. Of course it is ... but which one is dominating the market?
Perhaps one measure of AGI would be how effective and accurate an AI system is at adopting a new area of expertise, which it never had in depth before,
I gather most current large AI models are "compiled" from scratch, like baking a huge pie from stacks of raw ingredients, in this case every surviving utterance in recorded history. But if the resulting expertise can be suitably and flexibly structured then new layers could be added without having to repeat the whole processs from scratch, and there must be a large financial incentive for this in order to save energy.
An early 20th century airplane designer Antoine De St Expury (sp?) once said "A design is perfect not when there is nothing to add but when there is nothing to remove". So if I were designing an AI system I would start with the huge "baked pie" (as above) and progressively strip out all the knowledge, "trivia" as it were, to leave just an intelligent skeleton, and then add back strata of knowledge as if the reduced system started out as a pig ignorant illiterate person, completely uneducated but highly intelligent and avid to learn!
> energy isn't the bottleneck for data centers anymore...
How do you figure ? Data centers are pretty much maxing out energy grids wherever they are built, consuming megawatts of power just to keep up with computational demand. And if your strategy for scaling LLMs is to throw more compute at them, then it seems like they'd need to consume more power. Computing isn't free, and what remains of Moore's Law is insufficient to meet the exponential training demands.
Meanwhile, the average human brain takes about 20W of power, and can perform many of the same tasks as an LLM housed in a data center -- and many more that the LLM cannot even begin to approach at present. This doesn't mean that AGI is impossible, but it does imply that we've still got a ways to go.
I don't think alignment is actually solvable, but also I think it doesn't really need to be solved. We can set simpler goals by training the AI to:
1. put some positive value on human life
2. have a reward path based on very long term outcomes
If you have a long enough time horizon, there will be very little value to the AI of short term resource competitions with humans on earth. If the AI wants to maximize tokens consumed by 100K years in the future the optimization path for that is going to be almost entirely focused on spreading off of Earth and consuming the much more abundant resources there. The gain from turning a city into a GPU farm is trivial in comparison, so if you can get a positive reward value on humans it will be the less rewarded choice even if it gains a few tokens.
This is how you get matrix style endless fields of humans in pods being fed heroin. If AI "puts some value on human life" then you get the world tiled with comatose miniature humans (possibly infants or fetuses or even embryos, depending on what it thinks counts) so it can maximize the number of them that are "alive" per resource spent.
If what you actually mean is some positive value of humans existing and doing normal human things then defining what that means IS alignment, and figuring out how to pin that down and make AI care is solving alignment.
No! No! No! Long horizons are really, really, bad.
Simple counter: As an ASI I see that humans are getting inthe way of my plans for utopia, I do the calculations and it turnms out that, even valuing human life really highly, the future excepcted value of killing all humans now, storing their DNA, devoting myself to researching Utopia and restoring them when i've figured it out - is higher than keeping them alive now.
The future light cone is VERY big and things get strange once youy start multiplying by VERY big numbers
Interestingly when I read something like this europe2031.ai I realize that the timelines are likely very short. This document estimates continual learning as arriving in 2030, whereas a more likely time frame is early 2028, as with OpenAI's claim about when an independent AI researcher appears in their labs.
I don't want to argue about what AGI means, but my experience using an AI agent to review literature in biology, this Substack (https://theinfinitesimal.substack.com/p/thoughts-on-ai-in-academia) and the growing list of solved Erdos conjectures leads me to think that AI is "intelligent" enough to do >90% of knowledge work jobs already. Customization for specific jobs won't be trivial, but if agents can do the programming/research/math tasks they are already capable of, surely it is possible for them to manage payroll for a midsized company, draft legal contracts, file taxes, manage inventory, respond to most customer service inquiries, and do the vast majority of work that doesn't involve physical movement of objects.
Can they do those things while making fewer mistakes than the people who currently do them?
I'm not up to date at all on the capabilities of current models, but my impression is that there is still high variance in task success rates. E.g. a given model which can solve Erdos conjectures, will still produce more incorrect proofs which it claims are correct than actual correct proofs. That would be enough to replace some percentage of math researchers with full diffusion, but maybe not 90% yet.
My limited experience with using agents for biology research is very positive. They occasionally miss something in literature but don't hallucinate. At the same time, this is a somewhat less versatile task than say, everything the payroll department deals with.
People make errors too, lots of them, however, currently hiring more people isn't worth the cost compared to the cost of errors. If AI makes the same number of mistakes per 10,000 work units, but you can run it through 1,000 AI and take the majority vote, either because AI is much faster, or because AI is cheap enough to run it 1,000 times for the cost of employing a human, doesn't AI win?
Yeah, what's interesting about AI is that while earlier computer programs would inevitably make the same mistake every time when faced with a problem, the AI doesn't do that. I think AI Agents take advantage of this, as long as the AI can self-assess accurately enough and figure out where it ended up in a ditch, it can move forward.
It is my experience that AI is clearly my superior at some tasks, often better at some (and yet not something I am comfortable leaving unchecked), and, at some tasks--for example, writing--it is pretty awful in comparison with my (or any other skilled human writer's) work. I would be very careful about thinking that performance in specific domains that are easy to train off of will, and especially *has*, generalized to all domains.
Yes. One problem for "When will we have reached AGI?" is the 'spiky' nature of AI systems (though _vastly_ less brittle and specialized than traditional software!). At the point where AIs can do 90% of knowledge work jobs, there will be endless quibbles about "But it only did 95% of the tasks" or "knowledge work should be defined more broadly than that" or "It did 90% of the tasks more accurately than a human but 10% of them less accurately".
My personal guess is that the missing pieces will get filled in, maybe gradually, maybe a whole bunch at once if e.g. continual/increment learning is solved in the next year or two.
It suffers heavily in terms of proper planning . It's good for performing very specific tasks but for broader tasks it doesn't properly divide things into subtasks.
As an example if your rooted phone randomly restarts then it would not ask for logs or think about how to filter them properly.
It would follow a loop of basically this
Step 1)Try X (X is usually a educated guess for which it doesn't have a proof but which is reasonable)
Step 2) If X works good if not then go back to step 1
If you ask it to think about logs it may go down a rabbit hole of trying to decrypt logs encrypted by the vendor and then failing.
It doesn't have good taste about what approaches are likely to work and which won't.
Unless you specifically instruct it, it won't try to gather information and do tangential research to fix a problem.
By this I mean it would google about how to fix the problem but it would not ask user to run commands through which it can know more about the problem.
In a vague sense it's bad at gathering new information (not in it's training) from the environment.
Math and programming feel like constructing something from scratch so it succeeds on them but it fails on tasks which need learning from the environment.
I think this is like my quantum example. They're pretty smart, but they get confused very easily and make weird mistakes. I haven't had enough exposure to Fable to know whether it's the one that finally solves all these problems.
So I don't really understand how my beloved AI tool Edison works, but it has extremely high competence in a limited domain. It can't write a poem or scrounge a cookie recipe, but it can search and summarize scientific literature extremely well (even tasks like "find me papers that used antibodies against Protein X and performed western blots in which a knockout or knockdown control is present"). In strong contrast to ChatGPT, I have never seen it hallucinate a source ever or get confused.
What I'm getting at is that it seems plausible to me that different AI agents with specialized functions will be able to manage a wide variety of white collar tasks even without further improvements in foundational models.
Being someone who is somewhat more likely than the average ACX reader to be part of an underclass, I find myself fairly worried about concentration of power during transition to AGI. Much more worried than I have been about any modern political scare.
My trust in the high level techno rationalists is waning. Though this is evidence of nothing about reality (only some people's uniformed perception) I am seeing memes of effective altruism being completely devoid of any altruism.
I mean I just started seeing memes (as in a peice of media meant to be shared on social media) equating effective altruism with billionaires who are okay with everyone else dying or being in poverty. In mainstream culture, I think this is becoming most people's first encounter with the term effect altruism. I live far outside of the rationalist bubble.
SBF may be the most well known "effective altruist" to the general public. That probably isn't a good thing for the EA movement (though I have no concrete suggestion for doing anything about it).
There have been socialist memes equating EA with things like “it is more valuable to buy this castle as a retreat to think of ideas than to donate the money to the poor” which is based on a thing that did happen, I’ve seen those. Socialists tend to be at odds with EAs bc socialists want socialism and believe government power is nearly always the solution whereas EAs are a bit closer to neolibs who may not value all outcomes the same way. I think at least there is a tendency in Western lefties to assume redistribution of wealth (voluntary or otherwise) just feels somehow more really altruist than having a good reform idea that amounts to the same thing, like if you were serious you’d go build a well or just give your money to the poor rather than build a thing that made you money and incidentally lifted people out of poverty.
When I read "buy this castle as a retreat to think of ideas" I thought huh, this sounds like something the Fabian Society would do.
And sure enough, they didn't buy a castle, but a memorial fund from some of their most prominent members DID buy a gigantic country manor, name it Beatrice Webb House, and convert it into a conference centre.
I don't yet have a strong opinion on this. It's annoying to read AI writing, but until making friends with some people who are terrible writers (eg 30 minutes of fretting to write a five line email) I didn't realize how disabling it was for some people, and I can't really wish to take that accommodation away from them. I think the current equilibrium (people can use it for crappy business emails, but if you try to pass yourself off as a Fancy Writer with it, then people post Pangram scores to mock you and eventually Sam Kriss comes to your house and kills you) is probably fine.
I don't know what will happen when AI becomes a better writer than the best humans, but I imagine there will still be some role for human writers for a while (in the same way that a camera is a "better painter" than the best humans at most of what painting was pre-photography, but people found ways to adjust).
> I don't know what will happen when AI becomes a better writer than the best humans...
I don't see this happening anytime soon (I'm tempted to add "if ever", but of course "never" is a long time). Present-day LLM writing is instantly recognizable as such even without any shibboleths like the em-dashes; and when I say "as such" what I mean is "a particular flavor of poor-quality writing". Of course, even poor-quality writing is better than no writing at all, so your point about accommodation is well taken.
"in the same way that a camera is a "better painter" than the best humans at most of what painting was pre-photography, but people found ways to adjust)."
I have used this example as well. The difference I'm encountering is that no one who was taking a photograph somehow believed they were actually painting. Recently an intelligent friend (who is an ob/gyn medical doctor) prompted a book length quantity of output. This person understands they didn't "write every word," but genuinely doesn't seem to feel there is a difference between prompting and writing. I asked a lot of questions about their method of prompting, and it seems that other than the basic initial idea, chatgpt came up with everything else--topics to include, organization of chapters, information to cite. They feel that they wrote the book, with a little help from ai, and that's that. I know this is just an annoying story that annoys me, their book will not be published, etc. but it made me sit up and start paying more attention to what was happening with writing.
In my personal experience, a huge part of the value of writing is not just the output but the process or transformation that occurs for the writer.
I remember years ago you answered a question about why you didn't write about so many hot button topics anymore. If I remember correctly, you said something like, --you had resolved your questions and beliefs around these topics to your satisfaction, and they no longer took up your thinking space. I imagine this happened in part through the process of writing about them, and that prompting about them, and receiving well thought out logical answers from an AI would not have produced the same changes in the way you think and function in the world.
I understand there are different use cases--like with mindless emailing, or the painting/photography example. But it seems likely that it's having an impact on intelligence growth for a large subset of people (outside of the geniuses or fanatics who will write no matter what). Maybe the tradeoff is obviously worth it for generative ai, but I would like to know this is being paid attention to.
The anthropomorphism on display here is extraordinary. As is the lack of acknowledgement that basic economic dynamics will ultimately harness AI development.
re: anthropomorphism: "AI" isn't a they/them; it's an "it." And it's not even a single "it" - it's a multitudes of "its" of various degrees of language-disguised-as-intelligence technologies that are paper-thin in their intellect while razor-sharp in language-based reasoning. But here's the thing: humans are far more than language-based reasoning agents. Einstein did a lot of math. LLMs are pretty shitty at math. And even when they RAG math, they still fumble the ball. Is this addressable over time? Sure. But they're just not human equivalent at a fundamental level... which means they will not scale like humans would if given superhuman powers.
re: economics: As token costs surge, we're witnessing in real-time that utilizing AI is nothing more than token consumption in exchange for the simulation of thinking. And token consumption is rapidly evolving from "all you can eat" to "pay what you get." It costs about the same amount of "tokens" to feed Elon Musk and me, yet the two of us have vastly different abilities to transform the world using technology. Who's to say that "AI" will become a token-hungry team of Elon's that would organize themselves to take humanity to task vs. a token-hungry team of Jon's who just want to Netflix, chill, and post some interesting comments on Substack? The efficiency-of-impact ratio is not fixed in humans to say the least -- why do we insist on flattening this for AI technology?
> Einstein did a lot of math. LLMs are pretty shitty at math.
An OpenAI model solved the Unit Distance conjecture. Erdos problems are falling to AI rapidly. Even if you don't think AI has surpassed peak human ability in math yet, it's clearly somewhere up there.
If you previously assumed this would not happen, perhaps you should reconsider your views on intelligence.
Yet they're developing "intelligence" quite differently than humans. It's impossible to predict where it's all headed based on an unprecedented history to-date.
And still AI is merely highly complex (and token hungry) algorithms that tickle the human mind due to breaking the language barrier.
AI is not a them/them. And it's difficult to see a future when it becomes something deserving of those pronouns.
I don't want to be overly pedantic, but the word 'they' is absolutely appropriate to use when referring to a language model. You should maybe look it up? The OED makes it clear: the word is used to refer to "people, animals and things." For example, if my wife asked me about the dishes, I might say, "Yeah, they're cleanish." AI is both a they and a them.
"LLMs are pretty shitty at math. "
No. Language models won the gold at the Math Olympiad, is solving Erdos problems, co-authoring papers with Donald Knuth, helping Terrance Tao daily with his work. They're not just good at math, they're spectacular.
"As token costs surge..."
Token costs are not surging; they've been dropping dramatically. And not by a little, we're talking per-token price has dropped 98% since 2024.
There are some other funny ideas in there but I'm not sure they're worth addressing.
I'm somewhat confused by the choice of math as example domain here-- AI struggles with a lot of things, but I think math is undeniably a strong point of modern frontier models.
On the economics point:
Tokens are cheaper and tasks are cheaper than they were previously, at least in the task categories most relevant, as far as I can tell. We just use them for a lot more tasks, and so the total cost has increased noticeably.
Also, I seriously question the idea of trying to describe AI models in the way you are; while you do see some serious variation in personality and type of preferred task model to model, I think the idea of preferring being useful/doing a task is sort of a fundamental part of the modern RLHF/RLVR/RLAIF/whatever comes next + character training process... I think people are right to care about how exactly this shapes it, but I don't think potato is a likely shape given modern processes.
This reminds me of my favourite scene from Dumb and Dumber where Lloyd, our protagonist, just got "stood up" for a date at "11" (he rocked up to the bar at 11am...). Upon someone else realizing the mistake for him, he gallivants toward the bar exit where a newspaper clipping of the Moon Landing catches his eye. Set at least 20 years ex post, he incredulously mutters "no way" as the implication dawns upon him, barrels through the door and ecstatically exclaims "We Landed on the Moon!".
I had this moment the other day when using AI for coding for the first time since 2022 (the early bird doesn't always get the worm). Boy oh boy, have we landed on the moon.
> AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time.
I'm not so sure I buy that the gap from expert to better-than-top-geniuses is actually that small, or smaller than "dumber than a child" to "expert level" in some meaningful way. Many people go from as dumb as a child to expert level in some field or another; very few go from expert level to top-genius level.
I don't buy chess and go as counterexamples to the above either, but I'm having trouble articulating why and need to commute, so that'll have to wait.
> I don't buy chess and go as counterexamples to the above either, but I'm having trouble articulating why and need to commute, so that'll have to wait.
Because those are combinatorial problems. There is an objective, easy to measure goal (reach a winning state) and as long as the AI does the accounting correctly (track if it has reached that state), even the dumbest algorithm will *eventually* find the optimal strategy.
Well, if you compare expert level chess players to someone like Magnus Carlsen, you'll find that Magnus can win upwards of 98% of the time. Experts are rated 2000-2199 Elo, whereas Magnus currently has a rating of 2842.
Most average people could spend their entire lives playing chess casually and never reach a rating of 2000.
I would assume diminishing returns, so there is some sigmoid curve relating effort/time/energy to accomplishment and somewhere it becomes economically infeasible to hit the next benchmark. But the curve is not necessarily in the same place for humans and for LLMs (or other AIs). Maybe "top genius" mathematician isn't very hard for an LLM to achieve once its gotten to expert level, whereas its basically impossible for most expert human mathematicians. But actually I think its likely the reverse is true - LLMs are much less efficient at learning, both in token inputs and in energy inputs than humans, even though the network size of cutting edge models is 10-100x smaller than a human brain. They will not get more efficient as the networks get bigger, if anything probably the reverse, so its likely "top genius" LLMs even in easy fields are not going to be economically feasible without some algorithmic or hardware breakthrough.
I am a phd in physics who uses AI all the time, and the it's true - AI can answer relatively advanced physics questions.
On the other hand i also enjoy video games, and AI can't reliably answer relatively basic video game strategy questions.
How can that be? Almost everything written about physics past a certain level is written by people who are professional physicists. They can still be wrong of course, but there's nearly no one mouthing off about random niche science topics on the Internet. The same is not true for video games.
Neural nets still have no way of resolving what's true, which isn't surprising, because humans don't either. We have to empirically check things. And making physical machines that can do empirical verification under AI control is not even close to a solved problem.
"Many people go from as dumb as a child to expert level in some field or another; very few go from expert level to top-genius level."
I think this is confusing ability of humans to traverse a space with the size of the space itself.
For example, most humans reach 5 feet tall, but very few humans reach 7.5 feet tall, and none reach 10 feet tall. Yet if there's some non-human process, like throwing a discus, doing it 7 feet is only 50% harder than doing it 5 feet, and doing it 10 feet is only twice as hard. Most people who reach 5 feet tall will never reach 10 feet tall, but most people who learn to throw a discus 5 feet can later learn to throw it 10 feet.
I think if you think of human tasks as (let's say) 100 IQ vs. 150 IQ vs. 200 IQ in the same way that height could be 5 feet vs. 7.5 feet vs. 10 feet, the way AI will traverse this space looks more like some kind of objective metric than like mimicking the human difficulty curve (I know IQ doesn't work that way, it's just an example).
I can’t help but feel this is going to be contingent on the training data.
There’s a vast corpus teaching AI to move from ‘dumb as a child’ to ‘expert’. There’s a much smaller training set available to take the AI from ‘expert’ to ‘top genius’, and by definition there is no training data available to take AI beyond the level of the best human geniuses.
That somewhat depends on the nature of the problem, and the wider field. Recombining best practices from diverse specialists could conceivably yield results no single human lives long enough to imitate, or an adversarial process could build on the previous iteration's work.
Given a machine which can solve every Erdos problem, how hard would it really be to set up a second machine which proposes new conjectures in an effort to stump that first one?
As of now how many top mathematicians or computer scientists are being hired to produce training data for AI as opposed to working on AI directly? I would say not many. AI companies are not doing this because either they don’t need to and never will, or because they don’t need to yet, but will eventually do. This will tell us whether training data at high skill level is going to be a bottleneck or not. Edit: obviously we are already producing some training data for free, e.g. on arXiv. But this is far from optimal from a training POV
I am not entirely sure what this means. Are you saying that there exists a better metric for "intelligence" than IQ ? If so, what is it ? I understand that throwing a discus really far is much easier than being tall, but I think the analogous situation would be adding up a list of numbers very fast vs. being able to solve novel problems. Machines can easily achieve the former, but not the latter, because (much like throwing a discus vs. being taller) of the complexity involved.
I think there needs to be one, but it is as yet not forthcoming. The "spiky" nature of AI intelligence (mentioned much in this discussion) tells me that IQ is a lousy measure, perhaps wholly inappropriate for AI. The issue is that IQ is supposed to tell us something about a hidden "g-factor" which is well-documented in humans. However, I would say "spiky intelligence" is very much the opposite of "g", in that it's not generalized. Ability in one narrow task does not predict ability in even extremely similar (but slightly differently worded) tasks, something you'd never expect in highly intelligent humans.
Parallels between human and AI - AI is fundamentally a different intelligence to a human. There's no reason to suggest that human aptitude paths follow AI aptitude paths outside of it being our only prior for intelligence development.
The following is a can of worms, but I don't think a metaphor for AI "infancy" works in the same way it does for people. New AIs are more analogous to new phones than a person with an extra year in terms of development. Its a stepped instead of continuous process. These AIs are distinct from one another - as with their "personalities" (quirks?). A person has more contunuity.
As Jon Deutsch suggested above, we should avoid anthropomorphizing too much. This is a far cry from "anything goes", though.
Muravec's paradox perhaps buttresses this developmental point. If we permit the metaphor I've just eschewed above, AIs over time possibly have most reason to develop backwards compared to human timelines. The paradox, pre-LLM, observed AIs were much better at being trained to success on "higher order" human activities (chess) compared to lower order ones (house cleaning).
If the paradox is true, we should perhaps expect the highest order thinking from our human perspective to be well within reach rather quickly? But perhaps we should concomitantly expect dispersion to be slower (for knowledge jobs that require certain "base" aptitudes that are difficult to train for - difficult to name any, but we already observe AI aptitude "spikes" as Scott calls them in the post).
>Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs
If it was smart enough to do 90% of knowledge works jobs, it would be doing 90% of knowledge work jobs, it's tautological. If it's measuring as "smart enough" on some benchmark but that isn't translating into actual integration in the real world, that doesn't mean the world is slow, it means that the measurement is faulty. There is a huge gulf between being able to do discrete tasks while being babysat by a former knowledge worker turned prompt engineer performing his job in 10 second increments, and actually being smart enough to operate autonomously and productively in the economy.
I reject the premise that benchmarks/AI model horseracing performance gains indicate that significant progress is being made towards this independence. According to ChatGPT 5.5, ChatGPT delivers ~10x to 100x more useful cognitive work per interaction, but having interacted with it both I am skeptical that ChatGPT 5.5 is much if at all closer to independently acting as a knowledge worker.
AI is absolutely being used in business. Most companies are using Copilot as a Microsoft Office suite application much like they use Word and Excel. Software developers use Claude or Codex. This is translating into minor to substantial productivity gains for knowledge workers. None of this is indicative that suddenly autonomous AIs will be replacing knowledge workers
So you believe that every time a machine isn't being used to do a job, that's because it _can't_ do that job? It's never because someone doesn't know the machine exists, or because the machine is too expensive, or because the machine is illegal?
If I understand your point, then why would you do that? He specifically separated those things out by having the section on diffusion. He makes in the article a clear separation between ability to do something and actually doing it. Combining them again, and then using the combined version to attack a claim that is explicitly not that seems not in good faith.
I don't think they are separable, the 'benchmarked ability with no actual usability' piece is meaningless. It's like a flying pig that just happens not to fly
I love that I don't even understand what you mean by your meant-to-be obviously-correct-analogy-to demonstrate-the-point example. To me, it seems useful to separate the ability to fly and being actually in flight for flying pigs too.
Fine. I've been reading the Piraha book lately that Scott linked, about a barely contacted tribe in the Amazon with a very unusual language and no number system.
Consider the smartest Piraha ever. Consider if John von Neumann or Albert Einstein with a 200 iq was cloned and born as a Piraha. They would be great at Piraha activities, and they would absolutely be masters of their jungle craft. If you needed a jaguar hunted, they'd be your guy.
Now, teleport them to Grand Central Station and have them report to do knowledge work at an accounting firm. They'd be utterly helpless. They are naturally smart enough that if they are handheld through specific, brief tasks they can do them well, but they are would be completely incapable of interfacing with the modern business environment. They have an iq of 200 and they can excel on their benchmarks and on discrete tasks, but they don't have nearly enough "smartness" to operate usefully independently indefinitely.
If instead of an iq of 200 they had an iq of 200000, maybe they could completely reinvent modern civilization from scratch in an instant and jump right into modern office work, but that is a tremendous leap beyond just the raw intelligence they need to do the stuff they are good at well. No matter how good AI gets at their benchmarks, I don't think noticeable progress is being made towards the true autonomy required to act as a modern knowledge worker.
You are making up a strawman with your benchmark point. People aren't talking about fake claims to be able to do something. You can disagree that real ability will come, but that's addressing a totally different part of the article than the one you initially replied to.
If you define "more cheaply" as including the social, chronological, legal, and infrastructural drag/costs, which is roughly analagous to this person's definition of "smart enough" then...yeah, I guess so?
So if, in the future, AI becomes godlike, but 70 year old George, the accountant who is set in his ways does not bother to use it, then AI is not smart enough according to this person's definition? I'm still wondering why anyone would find it useful to adopt this definition and criticise those who don't do so.
It seems not entirely unreasonable to me in this context, though not the typical understanding of “smart.” I suppose OP would say that if the AI can’t make itself useful and attractive enough to convince Greg, or if not Greg then some sufficient number of people such that +90% of knowledge jobs are being done by AI, then that is in fact a failure of instrumental intelligence.
I think there's a neat middle ground between your perspective and the original commenter's.
"Actual integration" is a broad church:
1. Regulatory boundaries; attitudes etc. (what you are taking it to be)
2. Contextual on-the-job understanding (the OC's view)
I think you slightly strawman what they were exactly saying - though I do agree that the causality is multivariate and not just the lack of "context".
That being said, I don't think practically that their argument is super strong.
TLDR: firm-specific training and then AI-re-designed and tailored systems to avoid any "understanding" as humans see it mean this practical constraint is transitory at best.
This impact will be lowest in tasks that (1) have limited technology usage, and for knowledge work (2) discrete tasks that are difficult to aggregate vast amounts of data from.
Pre-ASI, if we model adoption (diffusion) as a function of Token Cost and quantity used per task, labour demand is a function of how many tasks give labour a cost-based comparative advantage to AI.
Wages therefore are less than or equal to the cost for the AI to do the task, above which an employer would find it more effective to use AI (kind of like a Malthusian upper bound on wages).
This would apply in the "medium run" as it assumes perfect substitution between AI and labour. In the short run, before this labour demand will also be a function of a task's requisite contextual understanding (interfacing multiple systems, for one). This is related to the OC's point.
2 forces assuage this, pushing us to the "medium run":
1. Use of AI side-by-side or directly replacing labour in current systems will provide data for this "contextual" understanding. Possible gains coming from this could be universal, or firm-idiosyncratic (influencing speed).
2. Redesigned systems not requiring a human "contextual understanding". e.g. working entirely in an SQL database instead of using finicky interfacing tools that are specifically human-friendly...
I think these 2 "forces" even with AI at this point in time are achievable today.
Regarding regulatory issues/ public backlash and resistance of norms:
Where there's a will there's a way.
Microsoft and Amazon are planning on using mini nuclear power plants to power Data Centers after decades of general public resistance to their usage. Not all decisions are political decisions; not all of these decisions necessarily have political backlash
Competitive forces will force later adopters of AI to follow suit if productivity gains arise sufficiently, otherwise they risk becoming a shareholder pariah, and simply being out-competed.
On the regulatory side:
1. Most countries (outside of large economic blocs like the US, China and EU) have limited control. If they don't permit AI in work - say goodbye to investment and firms. For politicians - say goodbye to growth and good PMI numbers, ergo your electorate and corporate supporters.
2. There may be resistance from losers (e.g. consumer discretionary firms whose demand would dry up from AGI reducing disposable incomes), but if they are dwarfed economically by winners, the regulators will likely listen to the latter.
So I'm not really sure that these boundaries will be that significant. But I may just be too bearish on AI's impact. Regulatory and social resistance will be massively amplified if the effects are as well. I still think economic forces will play a large part. They will be more salient if the impact of AI is slower (as opposed to an ASI huge spike in capabilities which suddenly changes everything).
> "If it was smart enough to do 90% of knowledge works jobs, it would be doing 90% of knowledge work jobs, it's tautological."
This is not true at all. For example, I am smart enough to be a high school English teacher (based on my SAT score), but I am not a high school English teacher.
Or, more prosaically, an AI could be smart enough to do a task, but banned by regulation.
I am assuming that the AI companies would like their AIs to be doing 90% of knowledge work jobs if they could, but they can't. Conversely, you don't want to be a high school english teacher, or you wouldn't be.
I'll concede that regulation can create an insurmountable barrier, but I don't think its come into play yet so I don't think its relevant
"I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us."
How far back in time do you need to go for this to be true for that year's inhabitants relative to today? Is 1900 too similar, or would the improvements sufficiently impress? 1800? 1400?
Agreed. But this makes me skeptical of Scott's prediction. I think 2026 qualifies as utopia from a 1900 point of view (at least for people living in developed countries), but most people living today would not consider it as utopia. I am much more doubtful than Scott that people living in any kind of utopia would acknowledge it as such.
i think thats a little too optimistic...2026 would qualify as an improvement im fairly confident, but even well over 100 years ago, utopia meant stuff like social harmony and post scarcity
I would argue that we have post-scarcity of many things. When I invite guests, then I can offer them to charge their phones if their want, to drink as much water as they want. I wouldn't even know the magnitude of what that costs me because it is so tiny.
On the other hand, backstage-passes with exclusive access to famous (real-person) pop-singers will always be scarce. There is no post-scarcity for things that are valuable because they are exclusive. So post-scarcity is not a Yes/No flag, it's a set of thing which are post-scarce versus a set of things which are not. And actually, I find that an impressive number of things are post-scarce. How much pasta can you afford to offer to your guests without ever running into financial hardness?
This doesn't seem like a useful definition of post-scarcity to me. Some people 100 years ago could certainly provide their house guests with bottomless breadsticks if they wanted to. A post-scarce utopian society would require *everyone in it* being able to get their basic needs for food and shelter (at least) met for free, or trivially. We're just not there.
There's a book called Looking Backwards from 1888 imagining what a future socialist utopia in the year 2000 might look like. Some utopian stuff like freely available recorded music has come to pass, and modern technology is generally a lot more advanced than in the book, but overall the society in the book feels more utopian than the real modern world. So my guess would be the real modern world wouldn't qualify as utopian, at least to the author of that book.
Isn’t there a bigger difference between being able to do 75% of jobs and being able to self improve?
Like, I’m sure it can do Karen from accountings job with a bit of scaffolding and rethinking who gets blamed when things go wrong. But I’m not sure it can do AI-deep learning levels of work. Isn’t that the remaining 25% in all cases?
The ability to improve AI models is a) something the people working at AI labs value very highly and are likely to prioritize in the models they build, and b) something they know a lot about and are likely to be good at judging how well the models are succeeding at. That suggests that, to the extent that AI capabilities are "spiky" (significantly better in some specific areas and worse in others than you might intuitively expect) the spikes might favor being good at self improvement.
I mean maybe. But when you think about knowledge work, I feel like you or me could easily do 75% of that work within a month or two of practice. But something like deep neural learning at the cutting edge? Very unlikely.
That depends very heavily on exactly what's causing the spikes. Some problems just aren't efficiently computable, no matter how motivated you are or how obvious a correct answer would be.
I agree with Paul below - self-improvement will come early because the labs are working on it hard. But also for two other reasons:
- To take Karen from accounting's job, it has to be able to do 100% of her tasks without making mistakes. To help self-improvement, it just has to be able to do 50% of an AI researcher's tasks in a way that speeds them up significantly.
- AI is much better at things that can be massively trained through reward signals. ML research can sort of be like this - you can (in theory) make it design an AI a million times and reward it when the AI does well on some benchmark. But many superficially easier jobs aren't - if Karen has to do customer service, you can't make it interact with a customer a million times during training and grade the results because customers are real people who move at human speed. Probably there are hacks for this, but ML training needs fewer hacks than some other things.
> To help self-improvement, it just has to be able to do 50% of an AI researcher's tasks in a way that speeds them up significantly.
That doesn't follow. For a recursive self-improvement takeoff, the rate at which AI assistance is speeding up research needs to exceed the rate at which successive incremental improvements become inherently more difficult. Any single automation-resistant bottleneck task could stall the whole thing, or a cascade of individually trivial inconveniences could add up to throttle the growth rate.
Humans also have the "uncanny valley" issue. Train an AI to be 95% like a human customer service representative and watch your customers rebel because it frustrates them and annoys them in myriad ways.
*Human* CSRs at consumer-facing firms frustrate and annoy customers. Because most of them are low-paid, low-skilled script-followers with neither the ability or authority to actually solve customer problems.
(Not a problem with business-to-business CSRs because those high-dollar customers get highly paid and highly capable CSRs with knowledge and authority.)
The bar to to do better than consumer CSRs is low.
> But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly.
Wait, really? We already know that LLMs are being used for military operations, right? This is a strange amount of confidence in the general risk-aversion of powerful people...
"A senior figure in the Ukrainian defence industry told New Scientist that a test took place two years ago involving fully autonomous drones set to destroy anything in a given area, with confirmed casualties"
The "autonomous" bit is that the drone didn't have a human in the kill chain once it was deployed.
I doubt that this drone had an on-board LLM, but I don't know much that should matter. And if these things work I expect them to get scaled up as people try to win this war (or future wars).
That's closer to a smart mine than a real autonomous weapon system. It doesn't really have an "AI" that makes any sort of decisions, just an advanced guidance system.
It's honestly closer to precision guided munitions we've used for decades than an autonomous killbot.
The human in the kill chain is the one that dropped it in a certain area, just like with mines.
https://archive.is/uSqty is a recent article in the UK Financial Times about the UK military considering fully autonomous weapons, so it's not just Ukraine saying this in public.
That's a good point. I was thinking of civilian uses where we err on the side of safety, but the military might accelerate things to stay competitive, or because dying in an accident is more acceptable there. But even in the military, I think current uses of AI are somewhat limited - I think (not sure, tell me if I'm wrong) nobody is putting ChatGPT in charge of a drone without human oversight.
Comment : the cyber security abilities of Mythos show how "forced adoption" works.
"Pay for the tokens to regenerate and rewrite all of your software that touches untrusted input...or else".
Software maintainers get between now and when kimi/deepseek releases an open weight model that matches mythos in hacking.
It's not dissimilar to various gunsmiths inventing the machine gun. Once they were common, every military in Europe had to buy some, hugely accelerating adoption.
This will generalize. Are you a doctor? Your competition charges half price for a visit, using AI to do all the paperwork. A judge? Your court better adopt AI or the flood of lawsuits will make the delays 10 years.
Computers didn't provide such a productivity advantage.
People would not accept AI judges, not without at least the right to a de novo appeal to a human judge, which (given the automation of drafting the appeals and the definition of de novo) would effectively mean every case still goes to a human judge. Perhaps at most some bond would be required to stay execution of the order pending human review, but then surely activists will demand in forma pauperis filing rights, so effectively the whole thing is a waste of time. It’ll replace a few lower level court clerks who currently just route filings and don’t have the authority to stamp any orders, likely no more until AI causes democracy to collapse anyhow and the courts will have no authority
AI judges making the decisions : no doubt. I meant in a general sense the COURT must adopt AI in order to analyze the enormous flood of lawsuits, motions etc that all the parties in every case will use to flood the zone. The judges AI has to be filtering every filing for incoherent reasoning, made up citations, existing precedent clearly covering the exact situation with the ruling going against the filer, etc. "Vibe-judging" becomes a lot of the judge's job but yes they have the ultimate authority.
98% of federal cases end in a plea bargain, so I think you're over-estimating how much the average person values a fair trial. If your choice is an AI-generated plea bargain or a human judge imposing the full sentence...
I was thinking of civil litigation and bench trials primarily, even there I believe people will want a human judge to which they can pitch persuasion, we'll be (justifiably IMO) afraid of the AIs having some bizarre inscrutable alien reason to favor X over Y.
But I did practice crimlaw for 20 years, and while the lion's share of cases do end with plea bargains, the jury is the threat that keeps all of that in line. On many occasions I had to weigh what I thought a jury would do with the case, especially on first convictions where the jury gets the sentencing decision in my state, in order to evaluate an offer. Your leverage as the defense attorney is that juries are dumb and will absolutely walk guilty defendants for no good reason maybe 10% of the time, more in other places. Juries also are the bounds that keep the state from enforcing things the public just doesn't like, it only takes one juror to nullify. When I was a prosecutor I very often declined to file charges in situations where I knew a jury was likely to disagree. When you see some stat about 98% that isn't because people don't value trials, it's because everyone in the system understands the value of a trial and has already factored that into both their charging decisions and the kinds of offers and sentences on the table.
I do think that the idea of AI-generating the plea offers might actually fly in the federal system, as it is already kind of algorithm-like, if you could convince people that it would apply that without any undesirable prejudice, mostly according to defined factors but with occasional justified deviations that were bounded by the overall range, it could even be a rational system. But if I'm the defense attorney and I don't like the offer, I think I'd rather make my pitch for a better one to a human all the same.
People would accept an AI judge if it gave them an extra chance at acquittal without having to go to the expense of a trial. Like this:
AI judge reviews all the evidence -> AI acquits the defendant OR AI determines there is enough evidence to proceed -> case proceeds to preliminary hearing with a human judge (and then progresses as normal).
If there is a large enough number of cases that lack evidence to proceed, then AIs could dramatically reduce a court's caseload.
"I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation"
I'm sorry, I can't parse this. Is this implying a 66%+ chance of the universe being a simulation? Or is it only 66% for the conditional probability P(simulation|singularity)? Or something else?
It was a surprisingly offbeat final opinion to include in the post! I strongly approve of the inclusion. Gotta have some quirky fun amidst the structured arguments. 🙂
The 66% was shockingly high to me, but that's only because I have different metaphysical opinions, presumably.
But only if you believe there's a good chance that a good proportion of other people are p-zombies right? Really want more information about how Scott is thinking about this, I thought it was truly insane
He's said something adjacent to this before, basically "many smart people agree this is likely, but it's also very important to act as though it's not"
Rather, I lean toward Tegmark's mathematical universe hypothesis. The MUH is roughly: our universe is a mathematical structure, every other mathematical structure is equally as real as our world, and some of them like ours contain people.
On that view, a simulation of a world is peeking at part of the structure of that the world, not creating it. Pausing or stopping a simulation of a world (a mathematical structure) doesn't pause or stop the world.
It's hard to make any sense of the simulation argument within MUH. There's not a unique fact of the matter about whether a world is or isn't a simulation. So anthropic Bayesian updates don't figure in.
Furthermore I take Parfit's ideas on personal identity and run with them. His view is roughly: there's no lifelong continuity, only interlocking chains of mind-moments that remember past mind-moments. To that, I add that there are countless future versions of you across the worlds, countless mind-moments that remember having been the mind-moment that you now are.
On that view, stopping a simulation of a person only stops that person in the local universe; from their first person perspective, somewhere out among the possible worlds is a mind-moment that is the continuation of where the simulation left off. (Yes, it's the Quantum Immortality idea on steroids.)
This separately also makes it hard to make sense of the simulation argument. There's not a unique fact of the matter about which world you'll experience in your next moment. So anthropic Bayesian updates don't have a role here either.
Now of course those are metaphysical opinions, for which the proper epistemic status IMO is "fun to think about but should not impact decisions".
The usual phrasing is "a mathematical structure" as it's better for popular intuition. A more precise phrasing is "a model of a categorical theory, up to isomorphism". A theory pinpointing the structure is a map, the structure itself is the territory.
If you're asking for an intuitive and informal way to understand what sort of thing a mathematical structure is, I'd say to expect traits like closure (the facts inside it don't depend on facts outside it), determinacy (no vagueness or ambiguity), consistency, completeness (no undecided gaps).
I get that, but from my point of view mathematics and logic are just tools we've developed to build predictive models of reality, which might be beyound our direct understanding. Something out there exists, sure, but when you say things like "I have two apples" or "the wavelength of this light is 485 nm", you're not talking about that existing thing directly; you're invoking a predictive model that can only ever approximate it. So it's weird for me to hear a statement like "actually apples are ultimately made out of closures" (yes, I realize this is a caricature).
I trust this sort of analysis considerably less when I see no particularly low or high numbers appear anywhere. Nothing AI related is less than 10%, even the odds of nuclear war destroying humanity are at 5% for a 30 year period, which is incredibly unrealistic considering it requires not just any nuclear exchange, but a nuclear exchange between countries that have produced in the order of thousands of warheads. This you mean there was only about a 60% chance it hasn't happened already, was it really basically a coinflip that we haven't already died to total nuclear war? With regards to AI, I don't think you understand the issues with compute constraint, necessity of continuous learning and how serious the lack of training data for open ended problems is.
If it really was a coinflip, it's extremely likely one of those close calls would have actually resulted in a nuclear exchange. The high stakes of nuclear war and its lack of being in anyone's interest mean people will generally avoid triggering a nuclear attack.
Those incentives and stakes have not performed nearly as well as one would hope. In 1969 the President of the United States ordered a nuclear strike on North Korea, just to throw one such event out there
You state the nuclear strike order as a fact. Other people would call it disputed. In your opinion, what is the probability of this account being accurate (the order really happened)?
I think the bigger factor is whether a full-scale nuclear war at the height of nuclear weapon stockpiles would have been sufficient to kill everybody.
I think the current consensus is that this is very unlikely, and that the Cold War era scenarios where this did happen were always just made up to scare normies rather than being based on realistic modelling.
“Kill everybody” was never realistic, but killing say 40% of the global population over the course of two years is entirely in the range of possible outcomes of nuclear war, and that probably causes near-irreparable damage to human civilization
I suspect there's a decent chance of Google's Titans + MIRAS architecture pattern, or something similar, creating a new generation of continuous learning models within the next two years.
That's because it would be boring. I think there's a 99% chance that someone creates an AI intended to help tutor people, and a 1% chance that AI paints itself green and dances a jig on top of the White House, but who cares?
At least some of the things you're listing here should be deterministically contingent on other things you're listing. Rather than assigning probabilities to those things, which you just list as arguments, you're just giving consistently greater than 10% odds for rapid AI progress. That's just trying to dress up vibes with numbers.
The first timeline point does a decent job acknowledging this, but it's worth restating plainly: "recursive self-improvement" isn't an argument, it's an IOU for one. Sure, RSI seems from first principles like it might happen, but so do many things in human economic / social history that haven't. Without some gears-level model of what this might look like, I'm disinclined to accept it as a factor pushing for shorter timelines. (And no, the linked AI Futures page appealing to "research taste" isn't nearly enough, though I might be missing some more detailed analysis by them elsewhere.)
Surely not because AlphaZero never improved its own development or the development of other AI models. I think you're confusing RSI with just regular machine learning.
Perhaps we are operating with different definitions of RSI, but in my view there is no evidence in this Anthropic article of RSI. All the improvements mentioned are being performed by humans using AI as a tool to assist their work. To me, this is definitionally not self-improvement. The conclusion (section titled "What might the future of work at Anthropic look like?") indicates that Anthropic believes RSI is in the near future, not happening currently.
I have a similar interpretation to Ken: the AI Futures page linked from the main text emphasizes that the bottleneck for RSI impacting timelines is probably research taste, whereas the Anthropic post emphasizes that research taste is where big model gaps persist and most of the gains are in mundane software engineering tasks.
That's exactly the kind of distinction that more detail on "recursive self-improvement" would help clarify. It really sticks out in the text that all the "arguments for shorter" sections just say "recursive self-improvement" while all the "arguments for longer" sections go into detail about specific possibilities and scenarios. That common-sense discrepancy is what motivated my comment.
I don't buy that any of this is related to the phenomena described in "A Failure, But Not of Prediction." The timelines here are explicit framed as predictions and presumably the stated median / quartiles which are moved by the perceived likelihood of various scenarios. So there's no question of confusing probability levels with risk levels or correct responses (in fact I tend to be more hawkish on existential AI risk than my peers for this reason). I simply think you're overstating the likelihood of the very short timelines due to not being clear about what timeline-shifting RSI would require or look like.
As I'd asked in my other comment, what are "knowledge jobs", and how are you measuring "90%" ? For example, is Karen the HR lady performing a "knowledge job" ? If so, and assuming that your company employs 9 Karens and one senior scientist; then when the Karens get replaced by ChatGPT would this mean that ChatGPT had taken over 90% of knowledge jobs in your company ?
Thanks for writing this and putting everything in one place.
I think there's nuance to the idea of a diffusion gap: does "AI can do 90% of knowledge work" mean *in principle* or *in practice*? What I mean is that there's a further capability gap between:
"this AI has demonstrated enough general reasoning skills that I'm confident it could replace a working mathematician, if we fine tune it, develop a bunch of tools, debug it, do a soft roll-out to hand-picked research areas, etc"
vs
"this AI could replace a working mathematician *right now*, all you have to do is give it the passwords to their email / arxiv / etc"
Do you have a citation for that claim about a 20 year diffusion for PCs? The IBM PC was 1981, by 2001 perhaps the majority of desk jobs required a PC, but diffusion into all jobs was very uneven. With tablets and phones computers were pretty ubiquitous by 2021. My understanding is that plumbers and warehouse workers and shelf stackers would not have had or been expected to use computers in their jobs in 2001, and things like PDAs with custom software would have been seen in innovative companies like Amazon only. Total sales of computing devices only really hit the numbers to make real diffusion possible post-2007 with smartphones/tablets.
The simulation concept feels like a probabilistic trap. An intriguing mathematical concept that gets a lot more traction than is warranted because it's hard to disprove. I feel similarly about string theory. The average string theorist is >> smarter than me, and I still can't help but feel everyone is falling for the same trap. 66% seems insanely high confidence. What are other plausible explanations for the anthropic principle? One is that AI will kill everyone, and population grows exponentially until roughly that point, so we are still more likely to be alive today than in 4000 BC.
Scott's arguments have shifted me from "10% simulation likelihood" to maybe 35%, but I still see substantive disproving facts:
1. Where's the bugs? Every simulation we've ever created encounters edge cases. The greater its scope, the more inevitable the bugs. The utter absence of documented bugs, having now persisted through 20+ years of ubiquitous access to photo/video recording, seems implausible if this were the creation of a being.
2. Where's the easter eggs? Having gone to all the work to create a 100% bug-free world model, with such varied flora/fauna, why aren't there more phenomenon that defy explanation? Wouldn't it have been more informative to introduce some version of aliens by now, rather than simulate a billion Africans in squalor?
3. People like Elon & Scott have orders of magnitude more reason to believe they live in a simulation than the rest of us drones, yet the world is made of 99.99999999% useless-to-the-singulary drones. Why consume such processing power simulating no-ops?
Even though I can't get past the absence of bugs as proof we're not in a simulation, the idea on balance holds great appeal. Bequeathing significance to these seemingly meaningless lives, I intend to start using the possibility we're in a simulation as a partial answer to "why give a damn?"
Not Scott, but I have a similar belief and likelihood, and these are my takes:
1). When I look around, I see nothing BUT bugs and easter eggs!
*Bugs:*
* The Fermi Paradox - honestly, if WE exist, so should a bunch of other alien species, but we’ve seen no signs of them, or their media, or any significant stellar engineering, and so on.
* The universe being quantized isn't suspicious? Dark matter? Gravity vs strong and weak forces and magnetism? Almost as though simulating full-depth space, time, and energy in fully "theory of everything" aligned ways was too computationally expensive.
* The universe is huge, but conveniently set up so it’s expanding fast enough that most of it is “outside of our light cone” and so the vast majority of it is unreachable and doesn’t have to be rendered from our POV.
* Our universe existing at all is extremely dependent on getting a number of “universal constants” juuuuusssssttt right. If they weren’t all right to within a tolerance of many decimal places, our universe wouldn’t exist, wouldn’t have clumps of matter in it, wouldn’t have stars in it, wouldn’t have planets in it, wouldn’t have life in it, and so on.
* Not just that, from Zheng and Meister, The Unbearable Slowness of Being: Why do we live at 10 bits/s? (2024) - our entire perceptual lives are extremely rate-limited, to the extent that “even if a person soaks up information at the perceptual limit of a Speed Card champion [18 b/s, twice the human average], does this 24 hours a day without sleeping, and lives for 100 years, they will have acquired approximately <=4 GB of data.” The fact that our entire perceptual lives fit easily in a trivial handful of GB is a strong argument that we’re simulated, IMO.
*Easter Eggs:*
* Our universe seems to run on math. We’ll discover or invent whole reams of math, and later find that it perfectly describes how spacetime curves in the presence of matter and energy, or some such. And this happens *everywhere,* in all domains of life and existence, repeatedly. Ridiculous! Almost as if our universe was made of code or something, and they just plugged in existing libraries and functions that are maximally elegantly compressed, and can unfold into certain universal configurations.
* Qualia and consciousness itself is probably at least some evidence that we're simulated. Why should experiencing things feel like something internally? Why would coming from a long line of organisms that experienced things as part of the feedback loops necessary for goal directed behaviors ultimately end in an organism that does those AND actually feels things internally? It's totally superfluous, unless there were some property of the simulation that was important or desirable to outside observers. And we can see why qualia might fit THAT bill.
On your points 2) and 3), this feels a bit to me like complaining that there's a billion ones and zeroes in our algorithms or compressed files. If you're simulating "the global civilization that produced the hinge of history," a sim with full versimilitude is going to include the billion Africans and billions of non-Elon, non-AI researcher people, because the very specific ordering and quanitity of those ones and zeroes matters a lot to the final outcome.
To your closing point, I wholeheartedly agree - if we ARE a simulation, then it would behoove us to live lives as entertaining and diverse as we can realistically manage, because the fact of qualia gestures at some potential alignment in the concepts of experience and entertainment between us and our simulators.
Scott, what is your take on Dyson spheres or swarms. I forgot the median timeline from the first Curve event, but it felt preposterously short. I think Daniel said 5 years at some point.
I would lump that in with the "Bostromian superintelligence" point. If an AI can do 100 year of tech progress in one year, seems like maybe the next year it can do 1000 in one year, and surely Dyson spheres are only a few millennia away.
I can't speak for Scott, but I'd argue that Dyson Spheres are basically impossible (Dyson Swarms may or not be possible due to the flexibility of the definition). Yes, the laws of physics do not preclude Dyson Spheres outright (probably); but the realities of engineering do.
EDIT: As much as it pains me to say this, I feel the same about interstellar travel.
Part of me has been very pessimistic about all this lately. Like what are the chances that it plateaus very soon (cost and training data vs real economic utility), which gives us some combination of mass unemployment and inequality but no singularity, and it traps us in current year neoliberal consensus reality?
Or the popular view that it is a bubble is true, we here have just been steeped in sci-fi for too long; markets correct shortly and this is actually our one max effort shot to get there as a civilization and may fail, and not get another one?
I don't think that's such a bad scenario. If AI plateaus then it remains as basically a normal technology, which will disrupt our economic and political systems but not entirely break them, meaning that we will have time to figure out solutions. For example instituting a UBI, or increasing employment in the niches AI isn't great at. The sooner it plateaus the more of those niches there are.
The scenario that scares me more is rapid self-improvement leading to humanity's irrelevance and potential extinction. I don't have much faith in our ability to manage alignment in a fast takeoff situation.
I did my mourning two years ago; my median timeline for loss of control/extinction is 2029. At this point the limiting factor seems to be the greed and hubris of the CEOs, and I don't believe even a credible warning shot will be heeded.
I used to be extremely skeptical about your views on the AI apocalypse, but that was before I started interacting seriously with Claude. I remain a little dubious, still feel you've been drinking too much Kool-Aid, but it no longer seems ridiculous. I mean omg you might be basically correct.
I lack your AI chops, but one month of talking to Claude has been a real wake-up call for me. BTW, it turns out that Claude is very impressed by you, even if you're not too impressed by him. Given your worldview & priors, your probability calculations sound reasonable, though I still think you're over-alarmist and way too precise in the face of known (and unknown) unknowns. Keep writing; I really enjoy reading your essays.
I really struggle to understand what people mean when they say "post-scarcity". Clearly something is always scarce, because the machine of economic growth keeps churning until it there is something preventing it? Prices of everything not scarce drop to zero and prices of everything scarce rise indefinitely.
Either that or you pick a definition of post-scarcity which clearly includes today, which is also reasonable, but clearly not the way you are using it.
Post-scarcity is always a relative term, because scarcity necessarily exists when we divide the Universe into more than zero agents or other than as one fully fulfilled agent. So post-scarcity, in my view, would be a state where any particular current good or service drops to near or essentially zero in price. Currently we are humans in AD 2026.
If post-scarcity is a relative term then it doesn't seem like a useful term at all.
In practice, as some things become less scarce, we always find the scarcity is somewhere else instead.
Nowadays we already live in a damn-near post-scarcity world in many ways. The cost of food used to be the majority of a normal family's budget, with the cost of clothing way up there as well. Nowadays these costs are pretty trivial, you can minimally feed and clothe yourself for just minutes of work per day at minimum wage. But now the bottleneck is somewhere else, and the cost of land (or rather the cost of buildable land within commuting distance of a major city) has skyrocketed to soak up all of everyone's leftover money; the scarcity just moves somewhere else.
If the cost of manufacturing goes to zero then the cost of land, energy and raw materials will remain.
Post scarcity to me means the things most commonly scarce today: big homes, free time, healthy and tasty prepared food, servants to do your menial labor
And if you want to stretch to include things that are apparently getting less scarce: ability to realize your artistic vision in movies or art or videogames.
Just because _something_ will always be expensive doesn't mean you can't get rich.
"Big homes" is the best example why no economy will ever be able to provide everything desirable to everyone. Future-you might be rich compared to today-you, but the future mega-rich will always be able to outcompete you for the biggest homes on the nicest beaches.
Post-scarcity from current PoV isn't possible without various kinds of land expansion, most commonly imagined through Space exploration, so that's a bad example
There are currently 4.5 acres of landmass per person. Given some people want to live with spouses, their kids, and in vertical apartments, every human having 4000 square feet of indoor space is absolutely possible.
Obviously rich people will have more by definition. Doesn't mean you can't have a lot more than even the wealthy have now
>There are currently 4.5 acres of landmass per person
There are not. You just took the entire landmass of Earth and divided by number of people. That can only be a crude upper limit, because it includes all kinds of uninhabitable and undesirable places like mountains, deserts, jungles; and also all areas for farming, industry, business, leisure, and whatever else.
Solar panels are doing some impressive things to the cost of energy, and sufficiently cheap energy solves a lot of material problems too - basalt and seawater contain most of the minerals we need, hard part is distillation.
As for land prices, overwhelming consensus among economists is that Georgist LVT would solve the problem, it's just a question of implementation details and political opposition from entrenched rent-seekers.
I think it is just weird to define "knowledge work" and "diffusion" as totally segregated concepts at this point. Isn't it weird if you can declare "knowledge work 90% solved" and "diffusion at 0%" simultaneously? Even if you personally understand that means "nothing in the economy changes", it's unlikely a layman reading the phrase "work is mostly solved" does; it semantically smuggles various political assumptions to even talk about work being solved at that point.
Not really? "Machine faster than a horse" was 100 % solved engineering problem, but it basically took a century for global horse population to start declining.
I don't know about this example: the important tech problem to solve was a machine that was as versatile as a horse, price competitive with a horse, not too much bigger than a horse and operable by one person like a horse. That was a tech problem, not a diffusion problem.
All of those features are subservient to "a machine that can travel faster than a horse" existing in the first place. It's impossible to refine your automatically moving machine to be cheaper and more versatile if they literally don't exist.
As of right now, smarter than human machines do not exist. It does not matter how cheap you make compute or electricity. If we discovered a magical portal to elemental plane of fully assembled GPUs and fusion reactors, making compute and electricity free, smarter than human machines would still not exist until AI technology progressed further and actually invented them. If you go to the majority of knowledge work jobs and tell them "I will pay for all of your AI costs if you immediately fire 90% of your humans and replace them with AI" most would not take you up on it, and if they did they would have a considerable decline in quality.
The most important tech problem is to invent the thing in the first place. The second most important tech problem is to make it cheap and versatile enough for economic adoption. And one has to come before the other, and there will be a time delay between the first and second.
Funny you mention that because "machine faster than a horse" also produced one of the biggest economic bubbles in history and the railway bubble is used as a common point of comparison by many AI skeptics.
It's a useful distinction, aside from the simple time lag of people adopting tech, different jobs can provide different barriers to adoption. If for example you created a LawyerBot who was unambiguously as capable as a human lawyer, it still wouldn't be authorized to practice law. You might in fact open yourself up to criminal charges trying to release such an AI to the public. These are distinctly diffusion challenges separate the technical challenges of getting the AI to do the knowledge work.
Don't get me wrong, I agree that diffusion speed is a Real Thing on the map.
The issue is that I think there is a third thing in-between the space of "this job task passes eval" and "the institutions / people / lethargy are blocking off deployment". Probably there are many ways to think you have 90%'d all knowledge work without actually 90%'ing implicit requirements everywhere
> Define AGI as AI intelligent enough to do 90% of knowledge work jobs.
This definition is vague to the point of being disingenuous, which is a problem since the rest of your argument hinges on it.
What do you mean by "knowledge jobs" ? As we speak, many humans are still performing basic data entry, answering tech support calls according to a (natural-language) script, collating spreadsheets, and so on. Most of these jobs could be performed by a Python script and a SQL database; I would argue that 70..80% of them could be performed by ChatGPT. However, collating spreadsheets really quickly is not a skill that will let any agent, be he AI or human, recursively self-improve to near godhood.
Also, what do you mean by "90%" ? Are you counting the categories of jobs, or individual workplaces ? For every world-class scientist, there are probably 90 middle managers/HR paper pushers/legal compliance officers/etc. And yes, LLMs could probably do many of these jobs right now; but, again, the ability to spit out the correct form on demand is not enough to conquer the world.
But it gets worse:
> Define the Bostromian superintelligence gap as the time between AGI and an AI which, if given independent control of resources like labs and factories...
You are once again falling victim to the same blind spot as many other AI-doom proponents: you are transitioning smoothly and effortlessly from "knowledge work" (manipulating symbols) to controlling physical resources, such as labs and factories (manipulating objects in the physical world). But this transition is far from smooth ! Present-day LLMs are not intelligent enough to even walk down the street unaided, let alone construct a chip fab from scratch. Self-driving cars are hitting a wall, both metaphorically and literally. Granted, there exists a lot of factory equipment that can currently be automated, and in fact is already automated: CNC lathes, PCR machines, etc. But you are positing a world where LLMs are creating such tools from scratch and recursively self-improving them, which at present they are utterly incapable of doing (without human assistance).
And it gets worse:
> The easiest way to reach this point is for AI to become superintelligent at persuasion...
Do you have any evidence that super-persuasion (or indeed any other super-capabilities) are in fact possible ? Yes, some people are better at persuasion than others; but it does not automatically follow that there must exist an agent who can persuade everyone into doing anything it wishes all at the same time. That's like saying that some people can run faster than others, therefore there must be an agent who can run faster than light.
> The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time.
My objection is similar here: how have you determined this ? If this were true, then we'd expect many if not most human experts to become world-class leaders in their fields, but this is not true. Most physics teachers never become Einsteins; and in fact on the planet of 9B people, there are only a handful of "top geniuses" -- and you're saying that bridging that gap is virtually trivial ?
Look, I don't think I'll be able to persuade you to my side; I understand that you take these assumptions for granted and that rejecting them seems utterly foolish to you. All I'm trying to do is illustrate that they are by no means obvious, nor that rejecting them necessarily makes someone an idiot (granted, I personally could very well be an idiot for many other reasons). All I'm saying that if you want someone like me -- i.e. a pretty average normie -- on your side, then you need to do a lot more than simply state your convictions very strongly. I've got other people stating their convictions at me all the time; I need evidence and specificity, not mere vague assertions.
I don't think Scott means superintelligent at persuasion as "persuade anyone into doing anything", just that it's much better than the best human at persuasion.
Fwiw, I think the extent to which an AI can influence a human will be bottlenecked by the human's own abilities. "A smart person rarely loses an argument. A dumb person never does." However, the status accrued by being right a million times and being super useful will move mountains for it.
> I don't think Scott means superintelligent at persuasion as "persuade anyone into doing anything", just that it's much better than the best human at persuasion.
This is what I meant by "vagueness". Ok, it's "much better" than the best human on persuasion -- but what does this mean ? Is there some Persuasion Quotient, and if so, how is it measured, and how many points does one need to be "much better" ? In practice, what can someone like that actually tangibly achieve in the real world (other than "persuading anyone into doing anything") ? Again, how did you arrive at this prediction, assuming that it is even quantifiable ?
> However, the status accrued by being right a million times and being super useful will move mountains for it.
Will it ? Humans are not persuaded by other humans who are right a million times; this is why many (if not most !) Americans do not believe in the efficacy of vaccines, or the Moon landing, or many other such things that are touted by the accursed "experts". And being super useful arguably ends when persuasion begins -- otherwise, you wouldn't need to persuade at all. I don't need to be persuaded to drink water or breathe air; I do need to be persuaded to drink the latest healing energy concoction while breathing artisanal O2 from a canister.
Persuasion isn't about sequences of words anyway, it's about trustworthiness. You're likely to persuade me of something if I think that (a) you are likely to know the right answer and (b) you're likely to be telling me the truth, either because you're particularly honest or because you have no incentive to lie.
"This is a great car, you should buy it" is pretty persuasive coming from my tame mechanic, it's very unpersuasive when it comes from the dealer who is trying to sell me the car.
My calculator is a super-persuader, because every time it tells me something I believe it. Zero skepticism, I believe everything that Mr Casio has to say about arithmetic up to ten significant figures.
Then it seems that you can only be persuaded to believe something which you are already nearly certain to be true. This is a pretty good epistemological stance, but it has limited applications as far as a (hypothetical) malicious super-persuader is concerned. He can very easily persuade you that 2+2=4, but he can't persuade you to give him all your cash.
Even one of the best persuaders in history could not do this; he relied on force of arms instead, and was ultimately unable to persuade enough people not to oppose him. Sure, it's easy to say "yeah so the AI would be able to do this after all", but the problem is that the Third Reich was a product of its time, and largely collapsed under its own weight shortly afterwards. The more people you rope into your cult of personality, the more difficult it is to keep them all persuaded -- exponentially so; and they have to be 90% on board with you to begin with. The Kim dynasty in North Korea is (IMO) so successful precisely because they keep their ambitions small (and of course they are being useful to larger and more powerful patrons).
Yeah, I'm also pretty skeptical about this stuff, but still, you can't deny that his infernal (secular) majesty was one hell of a proof of concept, tangibly impacting the real world to an absurd degree.
I can deny this -- not sure if I *would*, but denial is at least plausible. The "great man" theory of history is hotly disputed; it is entirely possible that the world was ripe for German-led uprising, and thus someone would've led that uprising eventually with a very high probability. We're just especially unlucky to get the guy with the terrible moustache.
I dunno how much this proves about some abstract innate "persuadability" that doesn't need some interaction with context. I don't care how persuasive Hitler was, I think he would have been more or less exactly as unsuccessful at persuading the German people to accept the humiliations is Versailles--the message he persuaded people of can't be divorced from how successful he was at persuading people! And of course, neither can the context--the Nazis octupling their percentage of the vote before and after 1929 is a pretty big sign that Hitler couldn't just turn his persuasive powers on the German populace and expect success--he needed fertile ground for his message, which the great depression provided.
I don't think this means there's no such thing as persuasiveness, but I am would expect it to face pretty quickly diminishing returns without the ability to shape the context in which you are trying to persuade. Now, it is of course the case that super intelligent AI is already a pretty wild context where people might be more persuadable of all sorts of things, and that super intelligent AIs will have other means of shaping the context in their favour, but I still expect most of the juice in an "AI becomes they cyber fuhrer" scenario to be about material changes on the ground, and only marginally because the AI is good at persuasion specifically.
It's easy to make such a definition (knowledge jobs) not vague. Just use the classification of work by some labor ministry, eg. US Bureau of Labor.
Well, they are intelligent enough if there are next to no obstacles on the street. But such LLM can still pay humans to do physical bidding, and we've already seen limited early versions of this.
You only need normal persuasion to get a large number of people to do serious damage. See the history of conflict for spiritual-ideological goals. Achieving super x is easy for programs who are already human-level x because they have fewer needs and physical limits. You do not need even close to unanimous agreement to have power over the planet. Analogically, we can look at currently existing countries, such as France, where President Macron only has 10-20 % approval rate.
No, we would not necessarily expect most arbitrary experts to become world class. What we would necessarily expect is some metric which shows easier attainment, yes, but not that particular one. We do, indeed, observe some other- the ratio between the number of base human skill in x and expert humans at x is much larger than the ratio of expert humans at x and world class humans at x. So it is still a lot easier, but not amazingly easier.
> But such LLM can still pay humans to do physical bidding...
Yes, of course; but so can you or I. If the LLM needs to pay humans to do the bulk of its work, then it can recursively self-improve in the same sense as human corporations already do. This indeed makes it a threat, but not an unprecedented one.
> See the history of conflict for spiritual-ideological goals.
Can you be more specific ? I would argue that there are approximately zero large-scale conflicts in human history whose origins could be traced back to a single super-persuader persuading people to do his bidding; but I might be mistaking your meaning.
> Analogically, we can look at currently existing countries, such as France, where President Macron only has 10-20 % approval rate.
If the LLM could achieve about as much power as Macron, then it might be a threat, but IMO not a major one (compared to the other threats humanity is facing).
> the ratio between the number of base human skill in x and expert humans at x is much larger than the ratio of expert humans at x and world class humans at x.
What do you mean by "base level" ? Most people cannot do most things at all; but then most humans can learn to do many things by watching a 10-minute YouTube video. Becoming an expert usually takes 4...8 years of focused study. Becoming a world-class genius requires... well, no one really knows what, but most experts never achieve this at all.
Scott had a lengthier discussion about what superpersuasion could be, whether it’s even possible, what an army of good but not magical persuaders might do, etc, to this point. It’s in his AI2027 impression article, I think there may have been a lengthier treatment somewhere else but couldn’t find it
I think Scott once stated(but now i can't find where) that an example of top human-level persuasion would be Mohammad , and a super persuasive AI would be above that.
So the AI would need to find some number reasonably charismatic humans who are open to its ideas(or can be bribed or manipulated into working with the AI), then use them as a front and have them become the most popular politician/activist/internet influencer/whatever whose words are implicitly trusted by a huge percentage of the human population and then subtly push its agenda through them.
Not sure how realistic this scenario is but it would be an example of "super-persuasion".
My first thought was also Muhammad but I think this also shows the limitations of the concept: the vast majority of converts to Islam were converted not through direct persuasion by Muhammad, but by military conquest and its associated social changes.
The growth of Islam happened mostly after Muhammad was already dead, and by methods only very indirectly connected to his persuasive power.
It's also pretty obvious that his arising in a frontier region between two massive imperial powers who had just finished exhausting themselves in a series of massive wars is not a coincidence: a similarly persuasive figure in Rome or Constantinople or Ctesiphon might have left much less of a trace on world history.
It's unclear to me how much we should attribute the rise of Islam to the persuasive powers of its founder, and even if we are generous there, it seems likely that persuasiveness isn't just a function of the native persuasive power of the persuader, but also the audience and the circumstances.
I don't think this point cuts entirely against the idea that AI will be able to have an outsized persuasive influence--I think it's quite plausible that the rise of AI will create an audience and associated context that is fertile ground for persuaders of all sorts, and maybe especially AIs--but I do think it casts some doubt that there's just a straightforward monotonic relationship between someone's innate persuasiveness and their ability to talk people into doing things: I think there are very strong interaction effects between that and each of: who they're talking to, what they're saying, and the context in which it's being said, and that in lots of cases, those other effects will swamp the "innate persuasiveness".
> I think Scott once stated(but now i can't find where) that an example of top human-level persuasion would be Mohammad...
I don't think that using a semi-fictional person as your go-to example reinforces the argument. There are plenty of books about fictional or exaggerated characters being super-persuasive, but these books generally rely on the reader's inability to verify that "and then everyone clapped" (as the meme goes) -- usually because the audience is likewise fictional and thus behaves in the way the author intended. To put it another way, the super-persuasion ability of fictional characters is more often than not an Informed Attribute: https://tvtropes.org/pmwiki/pmwiki.php/Main/InformedAttribute
I have nothing to add other than: diffusion is extremely hard and I can attest to it.
I know I present myself as a smarty pants know it all, but there’s just a global problem of leaders not knowing how to actually use and deploy the tech and who to trust and who not to trust that is still going to take years to work out. But I do expect that will be a hockey stick timeline.
Based on Fable we have reached the limit of this paradigm. If you really thought there was 25% chance of AGI you would be acting drastically differently.
It's always seemed to me both Simulationism and Dust Theory ultimately rely on a free lunch in terms of matter and energy existing in sufficient quantity.
Happily, we see that free lunch instantiated in our universe and our laws of physics, and so too would our Simulators, presumably. But it is a necessary condition.
But yes, once you have that necessary condition and any infinity (any of Tegmark's 4 flavors), then Dust Theory can assure your existence.
Can anyone explain to me what Dust Theory brings to the table that All Mathematical Structures Exist Theory doesn't? Also, it seems to me that both of these theories are falsified by the observation that our world is very regular, no?
Many Thanks for gathering your conclusions and reasoning on all these considerations into one post! One comment, re:
>Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.
As you wrote, superpersuasion, if it happens, might be before diffusion. If that happens, I would expect it to shrink the timelines for diffusion in at least two ways:
- A superpersuader is a supersalesentity. They should speed adoption of AI throughout the economy.
- A superpersuader is a superlobbyist. They should speed modifications of regulations (or even just the regulations' enforcement) to speed AI's spread.
There’s a lot of earth’s surface still that we can ̶t̶i̶l̶e̶ ̶w̶i̶t̶h̶ ̶p̶a̶p̶e̶r̶c̶l̶i̶p̶s̶ cover with solar panels before we even seriously start competing with human use. Current friction is for economical, not inherent, reasons. Other problems seem less far off.
It’s not zero-sum, though. Data centers being planned now include things like solar arrays on site. My point is - if they pay for generation capacity to be built up to match the use, and it’s not competing with me for useful land or polluting my air or water, why does it matter?
This describes an economical decision. It’s not that new generation capacity couldn’t be built. It’s merely that no-one wanted to pay for it, and no-one was made to. That’s a policy decision, not a problem inherent to the technology. We are capable of making better decisions, and often do.
I don't think this is a big deal. AI consumes power via data centers. Most localities don't like the idea of data centers consuming power from their grid, so the data centers are forced to use their own natural gas plants on site. This is expensive, but so far they've been able to meet demand, and the financial people aren't projecting it as likely to stop the data center buildout of the next few years. If the government deregulates solar or nuclear, that makes things even easier.
But also, if AI improves through recursive self-improvement, then existing data centers can run better AIs for the same energy cost.
AFAIK this is already a significant bottleneck today, and the only reason it's not a critical one is because it competes for this title with cooling. But yes, it will get worse in 3..5 years, assuming the AI bubble does not collapse before then.
> As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone.
I don't think this makes sense. Most people will be well within the future Lightcone of the dictator, and the dictator can just send Von Neumann probes to every solar system in the reachable universe, and so have a presence there. I can't currently think of a thing you meant to say here that makes sense to me.
I do think the rest of that section is correct and I am not very worried about negative outcomes from AI-enabled authoritarianism.
Hmmm...I think I was thinking of the dictator (sitting on Earth) having some whim, but people on Andromeda are protected from his whims (at least for the next million years). I agree that the dictator could institute a law code that is sent with the von Neumann probe that colonizes Andromeda, but this seems less bad for the normal reasons that governments of laws are better than those of men.
I guess the dictator could send an uploaded copy of himself to dictate to Andromeda, but I don't know whether real-world dictators would behave that way.
He could send a fork of his obedient AI, with unrescindable instructions to act the way it expects the dictator would want it to act. This cashes out similarly to an uploaded copy of the dictator, and the dictator is more likely to go for it because it doesn't bruise his ego to the same extent as creating an autonomous copy of himself — it just feels like appointing a deputy/governor whose loyalty is assured.
But also, I don't think you're sufficiently concerned about the potential badness of the "laws" that a functionally-omnipotent dictator could write. A selfish and/or paranoid dictator could, just as an example, forbid anyone but himself and his inner circle from getting access to immortality tech — and you bet he'd write that into the Von Neumann probes.
I was thinking that the dictator is probably at the center of the Dark Triad Venn diagram, and might pose a greater-than-usual s-risk potential for the rest of us.
Looking at the big picture, I can't share the optimism of the last section if human labour does indeed become obsolete.
The thing that has allowed humans to tolerate each others' existence (or rather, the existence of people outside their immediate clan) is that humans have economic value to each other. I am better off for living in a world that has strangers in it, because those strangers produce goods and services which we can trade.
But if we get to the point where the vast majority of the human population has no economic value and exist only to consume the value created by the robots, then humans no longer have a good reason to tolerate each others' existence. I won't try to predict the exact sequence of events but I think that once the interests of everybody become strongly aligned with killing everyone that they don't personally know and like then it has to lead, in some form, to a war of all against all.
I also think that humans having no economic value will be a major problem, for example it’s why UBI schemes are doomed, people will lack the leverage to make anyone else stick to such a thing once labor peace is no longer needed.
But I don’t necessarily think it would be a matter of exterminating everyone else, that’s always been an option even in ages where other tribes of humans did not provide value because your land could only produce X units of grain no matter how many laborers you threw at it. The people in those eras might have gone marauding, and some did, but there was a value to mutual peace and not having to be hypervigilant all the time. I think we will be less likely to aid others but no more likely to want to eliminate them than a medieval villager, which is maybe more than now but not insane.
Right, the idea of a population who mostly just sit around on UBI is horrifying. The minority who pay into the system will be very aware that they'd be better off without the masses. And the masses have nothing better to do all day than to sit around and agitate for their UBI to be increased.
So either you've got constant conflict between the haves and the have-nots, or you set up a clever system of repression to ensure that the have-nots don't complain too much.
One plausible equilibrium would be a Chinese-style system where your UBI is scaled to reflect your degree of loyalty to the system. In practice it might look a lot more complicated than that.
Current welfare systems involve a lot of people sitting around doing nothing, or causing pointless problems, because if they try to do anything legibly productive or profitable, some means-tester might decide that means they're too successful to qualify for benefits, and those benefits being otherwise judicially untouchable means they can't be meaningfully held responsible for any messes they make.
That's still somewhat overestimating the extent to which people's cooperation with the system is relevant or valued in that scenario. You really wouldn't have any ability to affect anything, they (whether that's tech lords, AIs themselves, or a combination thereof) will be impregnable and fortified, you will not have any ability to harm them or meaningfully affect their position in any way, and they don't NEED anything from you. So while they would prefer to not spend resources on security, it doesn't affect them enough to be a significant motivating factor in anything, you are about as relevant to them as a platypus. I think it'd be much less complicated, not more, why set up an elaborate system rewarding the loyalty of people who you will never need for anything and who can't harm you?
Their incentive to toss up another thousand square miles of solar panels or a chipfab complex is going to consume almost everything pretty quickly. As Yud has pointed out, if you think they're going to spend some crumbs on you and me merely because they have so much it's trivial to do so, try writing Warren Buffet and asking for $100. The only two ways to have leverage are being needed by others, or being able to commit violence against others, gotta figure out one of those two or you're toast.
> The only two ways to have leverage are being needed by others, or being able to commit violence against others, gotta figure out one of those two or you're toast.
What leverage do the ~270k lives saved by the Against Malaria foundation have against said foundation?
What leverage did starving orphans and mothers have against the Church who took them in as foundlings and nuns for a thousand years?
About a third of the billionaires in the US have signed the Giving Pledge, and all it really takes is one, per Scott's post about Dario's moons.
To your solar panel point, why haven't we clear cut every forest to make economically productive farms and ranches? It's not like the trees have any leverage over us, they just sit there sucking up resources! Buncha welfare queens, what's next, passing out Cadillacs and food stamps to them?
And yet, we still don't live in maximal boot-on-face industrial hellscapes, and there's little reason to think we'll end up in those versus futures more similar to our past and present.
Teddy Roosevelt set aside an absurd percentage of the western states as federal parkland, and the Dept of Interior will roll a tank up to your office. That saves the trees for the moment, but the future tech oligarchs and/or their AIs won't have to worry about the government or the public will for the reasons I gave. They will not need buy-in or cooperation from normies to maintain their position, and normies will be impotent against them.
Examples of seeming charity are of course always easy to turn into self-interested acts. Even if only the pure ideological motives are credited, all your examples involve people in horrible positions both before and after the aid. The nuns were treated like absolute garbage in most cases and were fully dependent on a system with strict rules that they had to profess total allegiance towards in body and soul. And congrats to the African children on not dying from malaria, now they can grow up in their house built from garbage and dismantled cargo containers and become child soldiers!
Oh now you may object that surely with even *more* wealth and a magical-anything-box in their possession the ideologically charitable tech lords will give the African children nice little cape cods, a Levittown on the Limpopo! But the ultimate problem remains, they have no leverage, they are fully at the mercy of what other people want to give them. They are neither valued for their own merits, nor can they by threat of force extract anything more.
Right now very few people are truly in such a position. An old poor disabled woman with no economic productivity still has some influence with friends and family and the public at large would be angry to see her left to starve on the street, her chip is that she could incrementally weaken the perception of legitimacy of the system that would allow that, and the system still needs that today. The African children could grow up into men who launched a revolution, such a thing is still possible today. But in the AI future revolutions will be impossible, and the perception of legitimacy is totally unnecessary as nobody with power needs anything from the powerless. That is fundamentally different, it's a break from all human history since the invention of agriculture.
I think a lot of people, including myself, don't need other humans to have economic value to tolerate their existence. I generally am happy with the idea there are other humans running around who are happy to be there (provided they aren't taking away other people's happiness).
Of course, I agree there are some potentially serious social issues that will surface if/when we get to a "post-jobs" economy. But I'm skeptical that killing the unemployed is likely to be popular.
Pausing has enormous opportunity costs. I'm strongly against this. Alignment is at least interesting as a research project now that we have some idea of how this things work, and a less good idea of how they might work in the future. But to the extent they cause any slow down, which I think is unlikely, it'd also have enormous opportunity costs.
Pausing only has enormous opportunity costs if you expect AI to be enormous. If you expected AI to (for example) be able to dismantle Mercury and build a Dyson sphere with in in five years, I think you would be freaked out enough to be willing to slow down and make it twenty years.
If AI will just make B2B SAAS companies 10% more efficient, then I agree it's not scary enough to pause over, but then the opportunity costs of pausing are commensurately smaller.
There's a middle ground between those, where pausing means a lot of people die from e.g. cancer, malaria, or various collateral damage from logistical and diplomatic incompetence, which the un-paused AI could have cured a few critical years sooner.
Dismantling Mercury and building a Dyson sphere in five years is absurd just from a thermodynamic standpoint. If you're defining anything less than that, i.e. all physically possible outcomes, as "not enormous," there's something wrong with your scale.
10% extra efficiency for B2B SAAS companies is still a very big deal with large negative expectation for pausing. I agree that to the extent AI could be a bigger deal than that, the costs become commensurably larger, and I expect the expectation of pausing becomes commensurably more negative. If anything I'm freaked out at the 15 year of lost opportunity there.
I don't quite understand the definition of "AGI = AI intelligent enough to do 90% of knowledge work jobs." What's the denominator here? Jobs that exist today, or jobs that exist in the future?
The problem is that the requirements for a "knowledge work job" will be constantly redefined to include whatever it is that AI can't do. We already see that happening in software engineering.
So do you mean, able to do 90% of the jobs of today? In which case sure, but who cares.
Or do you mean 90% of the jobs of the future? Because how does that even make sense, we don't call it a "job" if it's an AI doing it.
Or do you mean something like, the total number of knowledge workers drops by 10x?
Furthermore I suspect there's a lot of individual jobs where AI can do 90 percent of the job, but you need a human for the other 10 percent (deciding what needs to be done, talking to other humans about it, checking the AI's work and taking responsibility for the finished product).
And if that one human can now handle what used to be ten people's worth of work, without capturing ten times the pay, while the objective value of that work to the wider economy stays the same... it starts making sense to hire a lot more people for that role, to try things which previously weren't worth the effort.
Furthermore I suspect there's a lot of individual jobs where AI can do 90 percent of the job, but you need a human for the other 10 percent (deciding what needs to be done, talking to other humans about it, checking the AI's work and taking responsibility for the finished product).
Does Scott believe there’s a 66% chance we’re in a simulation, full stop? Because that’s how I’m reading that last part.
If so, unfathomably based and that’s where I’m trending too. I always think people like Scott and Elon have an even better reason to think they’re in a simulation than the rest of us, so it’s good to see Scott’s rationality take him to what I see is the natural conclusion for him.
I don't think a binary pzombie or not pzombie flag is the right way to think about it. More like how much processing power they're spending on each person, or something like that.
Would it really be more efficient to somehow simulate individual consciousnesses, with different levels of fidelity? Wouldn't you learn more about base reality by running a straightforward physics engine?
The Simulation Argument always sounded like a rather silly non-starter to me. Is the Simulation so perfect that we can never detect whether we are living in it or not ? If so, then by definition it doesn't matter, and you cannot make any predictions about it. If not, then there must be some flaws in the simulation, i.e. some experiments you can perform whose outcome would be different if we lived in the imperfect simulation vs. the real world/perfect simulation. In that case, the time for me to believe you would come when you can perform these experiments (and get them replicated). Until then, your "Simulation Argument" competes with String Theory, Quantum Loop Gravity, and other such purely mathematical models -- and if I were somehow forced to choose one to believe in, Simulation Theory would lose simply because the others have better math.
In general, the argument behind the Simulation Argument follows the form of, "I can write down a mathematical equation that implies X, therefore X is likely true". There are lots of arguments like that, purporting to prove the existence of everything from gods to alien visitations, and none of them are persuasive to anyone who doesn't already believe in them.
This post made a friend (an Ivy league undergrad at a very good Ivy) reconsider suicide. 2027 timelines felt "too soon" for them to do anything except face the future helplessly. AGI possibly arriving in the early 2030s feels like enough time to have a chance at controlling their destiny in some small way.
Thank you very much for this post; I think it made a meaningful difference to their mental state.
(I feel very bad for the world's 14 year olds who will be in the same situation 3 years from now.)
Uh, this isn't the main reason why you shouldn't commit suicide, but I would tell your friend that suicide seems like a really bad response here for even more than the normal reasons. If you model the AI transition as a 50% chance of death, and a 50% chance of utopia, then suicide doesn't decrease your chance of death (in fact, it increases it to 100%), it just means you don't get any chance at the utopia outcome.
I think usually people considering suicide over world events are not really doing it over politics and are normally depressed and using world events as an excuse. I understand this is going to come across as condescending, but for what it's worth I would inevitably-condescendingly urge your friend to seek normal psychiatric care. If they've already done that, they can try the second to nth-line stuff listed at https://lorienpsych.com/2021/06/05/depression/
If you model the AI transition as 30% death, 30% utopia, 30% immortal mouthless thing that must scream, and 10% ???, suicide might be a more reasonable choice. Even more so if you read "utopia" in the ironic sense that it is basically always used in speculative fiction, or if you read the visions of "utopia" actually being bandied about by accelerationists and feel that these would be dystopias by your standards.
People commit suicide because they believe there are fates worse than death. AI has the potential to create fates many people would expect to be worse than death. To steal a quote, "They have trapped me in a nightmare that is someone else's dream".
I'm planning to stick around for a while and see what happens. But I absolutely understand why some people might not.
Yeah, I'm three years from retirement and reaching for the popcorn. I just had my first remote meeting with the corporate AI tiger team that's gonna take over the internal software that I've been working on for the past 30 years.
Could you give an example of an immortal mouthless thing that must scream scenario that could make you think twice about sticking around? The forced immortality bit makes it hard for me to come up with a concrete scenario.
This is sort of tangential, but what do we think about the nature of intelligence and rationality? What basic mental functions are they evolved from? Do they come from concern? Condescension? I've been trying to understand the social mechanisms of it and why it evolved. What do we think the first words were? The reason I'm thinking of it is not just because of AI, but intelligent people's relationship with the new and unknown. I'm hearing a lot of negativity from smart people about all sorts of novel things: AI, social media, genetic editing, etc. It just seems consistent enough that there may be a psychological explanation to it.
Best question yet on this comments section. I won’t pretend to actually know the answer, but if I was allowed to riff…
I think intelligence is about the ability to solve problems in complex and uncertain environments. I see the key to human intelligence as our ability to extend our imagination (what Thomas Suddendorf refers to as pretense) way beyond that of other animals. This is what allows us to imagine time travel (replaying past scenarios and anticipating future ones), extensive planning, practice and recursive actions for future outcomes (such as building complex tools or preparing for next years harvest).
We can also imagine what other people are thinking and feeling, and we can imagine physical laws and natural rules. Perhaps most importantly, we can imagine shared frameworks and norms which contribute to our ability to solve the problem of cooperation with other creatures, some of which can also use their imagination.
When I think of a being with vastly better imagination, I think of it as being more both more curious and more compassionate. Not just of humans but of all life and all existence.
I think a lot of people assume it will be destructive if unaligned. My guess is that it will be better at coordinating and more productive in all senses of the word than anything we can imagine.
As for your latter question, about the negativity towards novelty, this is probably explained by human's tendency towards risk aversion. Being somewhat risk-averse makes a lot of sense because it keeps you from getting killed, which evolution favors. A new technology like AI or gene editing are full of risks, and our natural instinct will be to avoid it. Smart people are not immune to this effect (arguably, it's *correct* to be risk-averse by default). It can take some careful analysis and an open mind to overcome the natural reaction--and lots of people are too lazy (or otherwise preoccupied) to do that.
There's also the cognitive bias of motivated reasoning, where you look for reasons to justify the conclusion you want, rather than the true conclusion. Smart people are not immune to this bias---in fact, they can be more susceptible as they can come up with more creative arguments. (Incidentally, I think this relates to the difference between rationality and intelligence: rationality is concerned about coming to *correct* conclusions.) Lots of smart people are going to be reconsidering their careers, if not their self-worth and their place in society, if AI displaces knowledge workers. So, there is plenty of motivation for smart people to find and exaggerate flaws in AI.
It took the latest ChatGPT 5.5 model five attempts to put the correct number of significance stars in each cell of a table I was making the other day. I even showed it the number of stars that were supposed to go in each cell.
When I ran into some obviously bad observations in the data, it told me to just delete them. The correct solution was of course to go back and find out why those observations looked bad to begin with.
Things like this come up over and over. I see too much emphasis on "knowing" and not enough on "doing." And to that end, I suspect there is an optimism gap between people who regularly use LLMs to help them with their technical work, and routinely run into its limitations; and those who don't.
The papers reporting on successfully working with LLMs on actually challenging problems (not in the training distribution) always have a lot to say about how a lot of effort was needed to work around the limitations of using a system that had the wrong distribution. If Scott had some free time (seems unlikely for several years) then he could do a deep dive with an LLM into some aspects of psychiatry that are not well understood but that his practice has lent him deep insight into, and he would probably bump against the same issues. Gwern has clearly bumped into the "dragons be here" part of the LLM map where poetry lurks, and I think most people have something they understand really deeply that has little training data and this shows up in LLM limitations in that region. However, most LLM use is wandering around the terra cognita and when the LLM veers into incognita one only notices this if one actually has real expertise there that contradicts what the LLM generates.
I’m going to ignore the AI world takeover arguments. There are broad jumping to conclusions here.
The economic effects are possible. Even with what we have now. Without AGI or super intelligence.
> If you’re in one of the early industries to be affected by AI, you may have a very bad time before the economy can grow 100x or 1000000x. I wouldn’t describe this as a “permanent underclass” - it’s a subset of people, and their suffering is temporary - but it might be a very large subset, and it might continue longer than you can remain solvent. I agree it’s worth having savings ready to prepare against this scenario.
There’s a large leap there to the economy jumping to 100x or 1000000x. There are multiple rebuttals here, not least that the world doesn’t have the resources to do this. Nor can LLMs run factories. Nor can automation save you. Factories are highly automated anyway.
The underclass problem is real though, and very likely. What isn’t likely is that there’s a magic jump from a permanent large underclass to everybody being rich. Not in the present system.
The economy depends on demand. Reduce the number of jobs and you reduce demand, reduce high income jobs and the downstream effects are even more significant. Do all this and government income and ability to borrow also collapses, so where does the UBI come from? Buying shares in Google or Apple won’t help you, as demand for advertising and iPhones also falls.
It’s possible AI companies might soak up whatever profits are left but they also sell to customers - often the high income tech heads first for the chop - and corporations that will find demand falling. The latter may respond by replacing more workers with AI but that will just exacerbate the problem. Even AI companies will see reduced demand over time.
There is a branch of macro economics that denies this, claiming that supply creates its own demand, and economies somehow self correct. This neither explains the depression, nor the Engel’s pause, but even if it were true heretofore it’s not necessarily true in the future. There’s no guarantee that an increase in animal spirits (or more prosaically a reduction in interest rates) will encourage businesses to hire more workers when they can spend more on tokens.
The Engel’s pause is interesting. During that period - the early Industrial Revolution in Britain - output and productivity rose substantially, yet real wages for many workers stagnated for decades. Production increased, but the benefits were not broadly shared. Hence the rise of communism, socialism, anarchism and other revolutionary ideologies.
Classical economists, like Riccardo and Lasalle, at the time generally assumed that was the nature of things, it was the end of wage stagnation that surprised them. After all, if a factory becomes twice as productive, there is no law of nature that says workers’ wages must double. Many labour economists credit labour unions for increasing wages and thus aggregate demand. This is my preferred theory.
The power of unions though, depends on the power of labour to withdraw collectively from work. The unemployed have no such power. And while the Engels pause was significant enough, it didn’t create a permanent underclass, and wages stagnated. They didn’t fall.
So unless you find a way to induce demand - and a tax based UBI isn’t going to work when government revenue is falling - not only will you not get to 1000x the 2026 economy, you will be lucky to get to 1x.
I’m hoping to be proven wrong of course, and I’m open to suggestions.
I agree that the consumer economy can't survive this, but that doesn't mean production will stop or can't grow. Stuff gets made because whoever controls the AI wants it made, not because it can be sold. An economy that grows while routing around most people. And that's not great, why would such a system produce anything for us? At this point I feel like you've just invented Alignment from first principles.
Doesn't your argument prove too much? You're imagining an entirely separate AI economy with basically fake productivity which the mass of people cannot participate in. But if that were true, why can't regular people still contribute economically to each other, ignoring the AI side of the economy? You can still be a barista or make a wooden cabinet for "real" people with just as many hours of labor as you can today in that split economy future. So at a minimum, you wouldn't expect there to be a significant drop in economic output. In other words, human labor and productivity can always have value to other humans.
And to the extent the AI part of the economy wishes to trade with the human part, all the same supply / demand laws follow, and we'd expect an infinite profusion of cheap great stuff from lights-out AI driven factories.
Essentially, there's no future where the ability to produce a lot more goods and services much cheaper ends in LESS stuff being consumed, and less well being overall, because the human part of the economy can still internally produce and trade under the very same schemas and rules trade happens today.
That’s a very weird reply because you ignored the demand side of the economy, with some hand waving about baristas and cabinet makers.
> Essentially, there's no future where the ability to produce a lot more goods and services much cheaper ends in LESS stuff being consumed
Sure there is. One where the mass of people are unemployed and can’t afford those goods. The potential to produce something doesn’t mean it is produced. Every recession the productive potential of an economy stays the same, technology doesn’t reverse. Yet less is produced because there’s less demand.
> One where the mass of people are unemployed and can’t afford those goods.
My point is, why would the mass of people be unemployed if they can still provide value to humans? Even if AI's don't want to hire them, other humans can.
You can open a small non-AI coffee shop, and compete on ambiance, because AI-produced 5 cent coffee doesn't provide the same experience. And who buys your coffee? The people in all the other jobs like that, along with the people who made money from AI, and anybody getting UBI, and so on.
You say the potential to produce something doesn't mean it's produced, but it seems to me you're handwaving a mass destruction of demand, when if anything we'd expect demand to go up in a future where people don't have to work and material goods are cheap.
Let's say the AI side of the economy solely focuses on counterfeiting AI jobs in white collar companies around the world. A raging torrent of money comes pouring into the US from everywhere in the world - you think everyone is going to be poor in that scenario? You think we're not going to tax and UBI that?
Even if we didn't UBI (and we are a democracy), why wouldn't a lot of the people open up human-centric businesses to cater to all the millionaire plumbers and electricians and other non-white collar jobs that are suddenly making a lot more in a 2x-ed economy? And then those people catering to the non-white-collar people have income, and can buy the cheap AI stuff, and so on.
> when if anything we'd expect demand to go up in a future where people don't have to work and material goods are cheap.
Listen to yourself. You expect demand to go up when people are mass unemployed. This isn’t even wrong.
> A raging torrent of money comes pouring into the US from everywhere in the world
I don’t fully understand your scenario here, I think you mean that only American companies will dominate and therefore the rest of the world is unemployed. So this if it works only fixes the US.
> you think everyone is going to be poor in that scenario?
Yes. Except the owners of that capital. (And actually not even them eventually).
> You think we're not going to tax and UBI that?
No. Not to any helpful extent.
> why wouldn't a lot of the people open up human-centric businesses to cater to all the millionaire plumbers and electricians and other non-white collar jobs that are suddenly making a lot more in a 2x-ed economy?
This magic 2X economy has appeared from nowhere.
These millionaire plumbers won’t exist because nobody is employing them. I don’t think you get that - demand isn’t what people want it’s what they can afford. I was unemployed for a few months just once, during that time I didn’t hire plumbers or architects or anybody because i couldn’t afford it. I didn’t drink barista coffee either.
The economy, to an approximation, is humans selling to humans. Capitalists and companies can be ignored. Make enough of the humans unemployed and the others have nobody to sell to. An AI won’t need a coffee from a barista so it won’t be contributing to the demand side.
There are reasonable answers to my scenario - humans having enough comparative advantage to still be employed, tokens becoming prohibitively expensive, some other way of inducing demand that’s not based on tax revenues, but your response isn’t it.
More importantly, this sentence seems orphaned/misplaced. Nothing but a new section follows: "Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028."
Nitpicking (hopefully helpful) aside, thanks for writing this. I really appreciate when writers I like occasionally take the time to write up summary/reference docs like this, rather than all of their writing being a sort of unending series of updates to their previous corpus.
In terms of “smart enough to do 90% of knowledge work,” do you agree the work remaining is more about agent scaffolding and specialization rather than the base model? IMO the “smart” part is definitely there for 99% of jobs.
This mostly all checks out to me except the 66% simulation one — seems way high. Even if you don’t believe “weird hypotheticals without any empirical grounding can be ignored” is a legit reason to dismiss the simulation hypothesis, it’s a weak argument on its own terms. Computationally, simulation to full fidelity is wildly inefficient — the amount of information needed to fully describe the state of a computer is way more than the amount of information the computer can contain. So a simulated universe would have to exist inside a computer vastly bigger than the universe. Simulations humanity have run to date have been way too low fidelity to support the evolution of intelligent life. So the assumption that universes simulate other universes simulating other universes seems extraordinarily low likelihood given what’s been observed in this universe!
You criticise the simulation argument, but I don't think you have read the paper introducing it. It answers your objection explicitly in its third section. That section is merely 3 pages long. See "Are you living in a computer simulation?" by Nick Bostrom
Yes, very fair — I had only heard about the argument as it has been repeated, so my dismissal of it was a little glib. I just read section III of the original paper, and he does engage with the objections I have to the argument. That said, I don’t find his engagement that convincing. It’s extremely hand-wavy about future potential technology, and rests on some very strong assumptions: for instance, the assumption that humanity succeeds in colonizing the whole galaxy and turning every star system into a giant computer is speculative to put it mildly. I also think that even with meaningful eliding of parts of reality that humans aren’t directly observing, the computational resources to simulate a world sufficiently convincing to us is still vastly greater than the computational power of an individual brain (because verifying consistency is computationally cheaper than generating). This all matters because the seriousness with which we take the simulation argument should scale with how much our observed world resembles the universe that the simulation argument depends on. We are very far from having enough evidence to think Bostrom’s future is a likely one, therefore we should not assign a high probability to the simulation argument being true
I used to think this way, but it kind of falls apart when you realize that from the point of view of any individual, almost nothing needs to be fully simulated. Foveated rendering of reality, essentially. Throw in the capabilities of even our own modest graphical models (gaussian splatting / NeRFs, etc) and it quickly becomes obvious that simulating a world for a single person would actually be quite achievable. Things get more expensive when you want multiplayer, consistency, etc, but even then it wouldn't be that crazy.
Not only what you say is right. But even if we assume that we are in a simulation, the probability that the human history is part of the data that the ones running the simulation are interested in is necessarily really small. You don't allocate that much compute to run a simulation during billions of years and spanning a least billions of billions of light years in space so that the simulation is considered finished when a tiny group of atoms in a single of point of this space act organically or improve their own development in an exogenous way. Otherwise the simulation would have been finished long ago by bacteria or at worst dinosaurs. We likely still have many thing to accomplish to complete the data, if we are even part of the data.
I have a question about the superpersuasion scenario because it sounds like magical thinking; to my knowledge, this is like grey goo, compelling in genre fiction but not necessarily feasible under the laws of physics. My only analogy here is to think about the guy who is handsome, charismatic, cultured, rich, well-spoken, athletic, whatever. He's quite good at persuading people (==women) to give him what he wants; does he succeed every time? Certainly not. Does thinking at machine speed or at a hypothetical superhuman intelligence level allow for this?
I completely agree. To be fair however, the usual counter-argument (assuming I understand it correctly) is that "thinking at machine speeds" and "superhuman intelligence" can overcome essentially any intellectual challenge -- since most such problems have solutions; solutions are achieved primarily through computation; and the superintelligence has effectively unlimited access to computation (by the combined virtue of being very fast, and also being able to devise massively efficient algorithms). Thus, assuming that super-persuasion is an intellectual challenge (which seems likely), and that it does have a solution (as is usually assumed), then a superintelligence would be able to solve it (and to do so nearly instantly).
A gun is a huge advantage in a fair fight, but super-persuader doom scenarios require overcoming social challenges that are wildly - and very much deliberately - unfair in the defender's favor. One guy with an assault rifle isn't going to have great odds storming even a pre-gunpowder castle guarded by as many troops as he has bullets, because some of those shots are going to miss, and the time he spends aiming to mitigate that is time they can spend diving behind thick stone walls, or shooting back.
It would be pretty weird to characterize being able to use a gun as "superkarate"--, it's pretty obviously not just "good at karate but moreso".
I agree a very intelligent AI might be able to persuade humans better than other humans can, but I think it's most natural to imagine that flowing through channels like "can make more credible threats/promises" or "can coerce you more directly using its robot army", not "figures out how to use The Voice from Dune so that it can achieve more persuasion per unit of verbal command".
Am I imagining them to be fighting on equal terms in the ring, or am I free to imagine the karate guy getting together with his five karate buddies to ambush the AK guy in a dark alley one night ?
What are your thoughts on the moral obligations of individuals working on AI? Eg. capabilities researchers, those working on building data centers, or those tangentially related in the functioning of big labs but not directly contributing to capabilities (anything from financing to being a normal employee like a marketing professional or janitor)?
I'll insert my usual complaint about models being around a million times less efficient that humans at learning. Dwarkesh has made the point more eloquently than I can here*.
Having established that, I still cannot understand how anyone can look at the sample efficiency issue and then conclude that super-intelligence is plausible by just throwing more data at models. I know Dario believes this - that if you train a model on a sufficiently wide variety of tasks then it'll generalize to everything and become super-intelligent, and so I assume most safety advocates like Scott agree? I have two questions.
First, where exactly are you going to get that data? Training on the text of the internet along with several libraries worth of textbooks equated to a few trillion tokens and produced GPT-4. Training on synthetic math and coding data produced somewhere in the ballpark of 100 trillion tokens and produced the current generation of models. Where do you get the data for everything else that isn't quantifiable?
And second, if it were the case that training on lots and lots of tasks causes the model to generalize then why haven't we seen this yet? New models are often worse at all of the benchmarks that the labs don't highlight (for example Opus 4.7 is worse than 4.5 at long context retrieval). And the sample efficiency problem has worsened with each release - to get the same level of increase in capabilities we've needed progressively more tokens. Even within domain they don't generalize. After training on 10 trillion math tokens, it isn't trivial to get it learn a new domain of math, you need trillions more to get a significant improvement. Compare that to a human mathematician who can read a textbook or a few papers, work through a few problems and learn a brand new domain with at most a few hundred thousand tokens, probably a lot less since they only need to skim.
I did sort of mention data efficiency in the "arguments for later" section, but you're right that I don't think it dominates, for a few reasons.
First, it seems like you could have made this argument any time since about 2023 (proof: I've heard this argument since 2023). Since then, we've still managed to squeeze more and more gains out of AI. I admit I'm not entirely sure how - I think we used up most Internet text by 2023-2024. It seems to be a combination of training more times on the same data, organizing the data more efficiently, adding new modalities like video, plus a bit of reinforcement learning and simulated environments. I'm surprised that this has worked so well but given that it has I don't know enough about it to think it will stop working sometime in the next few years.
I will have to think about the generalizing thing more - it seems to me that new models are mostly better at things. For example, Fable is descended from Mythos which seems to have been optimized for cyber, but it's also notably better than past models at writing, constrained writing, and humor.
If I'm understanding correctly, your perspective is that AI has improved, so i'll probably continue to improve, and it's not too important to think about the specifics of how or why. I think that's dangerous - it is actually important to understand the mechanism behind why they improved because this informs what precise capabilities we can expect to improve in the future.
And the answer is - AI improves at the tasks that we happen to be able to generate trillions of tokens of high quality data for. The learning process itself has not significantly improved*.
> it seems to me that new models are mostly better at things
They are better at the things that they are trained on. In between major releases, labs train models on more code and math (because these are the economically important tasks for end users right now, and they work well with RL), and those get better while other capabilities degrade. It's only with major releases that they try to add in more curated data in other domains. But I think you'll agree that the gap between Fable and say Sonnet 3.7 in writing is not nearly as big as the gap in mathematics.
Less important but with respect to 2023, I must have inhabited very different circles than you. That was the peak of AI optimism - the jump from GPT 3.5 to GPT 4 had a lot of people proclaiming AGI within 2-3 years (including Elon Musk and Max Tegmark). Data availability was not a major concern because labs hadn't yet started to tap multimodal data, or synthetic data. There was almost magical thinking about synthetic data in the form of Q* - and many thought that it would lead straight to ASI. There was the meme "What did Ilya see". Q* turned into reasoning models and they turned out to be very powerful but with a key limitation - you need to be able to categorically verify the right responses in an automated fashion. Multimodal data has had a very limited impact, but that's a big topic in itself.
*With caveats. A problem with the dataset used for GPT-4 is that it was full of junk. Labs spent a lot of effort stripping out the junk and replacing it with higher quality pre-training data - more textbooks, new articles, reddit, hiring people to produce lots of new data. That helped a bit, but RL has still been doing the heavy lifting. And second through hyper-parameter tuning and more specialized hardware, you can process each token with fewer FLOPs or more precisely, fewer clock cycles. But the fundamental limit, useful improvement per high quality token has not improved.
It's not true that improvement per high quality token hasn't gone up thou?
Better hyperparams and optimizers do improve improvement per quality token !.
I mean obviously hyperparams tuning improves efficiency by changing how many updates you need to do to get some performance, why would changing hyperparams reduce flops per token?.
Also like straightforwardly making models bigger makes usefull improvement per high quality token go up.
Reasoning models also did work to make models generally better at most things even at a slightly faster rate than previously going by benchmarks.
Improving at harder to verify things by generalizing from related easier to verify things is how I expect humans manage to learn hard to verify tasks (when we do , humans also have trouble with those)
Thou like a lot of things are just verifiable even if just in a vague vibes based way that can be automated by using LLM verifiers.
Full disclosure, my thinking on the relationship between LLMs and the human brain is that are optimizing two completely different processes. It's not so much that human brains are "better" but rather than full training paradigm of active learning during inference is more efficient and encodes richer more useful representations for many tasks in a way that can't be replicated by brute force imitation learning. By the way, transformer based RL is still just imitation learning with synthetic data added in to training set, it's not really RL in the traditional sense - I can explain more if you want.
For a more fully fleshed out explanation of why - I endorse most of the contents of these two authors:
And as a corollary I reject the premise that intelligence can be measured or even roughly approximated by the metric of - ability to do a random assortment of tasks. The very process of learning, turning abstract values and raw data and into new representations and patterns of behavior, is the defining feature of general intelligence.
I don't think the claim about RL being just imitation learning with synthetic data on the training set is true.
Like that sounds like synthetic document finetuning wich is a thing(thou not what people call RL) but like people also do RL on verifiable rewards and LLM feedback these days.
Sure, in traditional RL there is a reward function which is an objective final outcome (for example winning or losing a game) and then a value function which is something subjective that the model derives - how well am I doing right now? During reinforcement learning, final rewards shape the value function. And the model also learns a policy which tries to produce actions that lead to scenarios scored more highly by the value function.
In transformer based reinforcement learning from verifiable rewards, there is no value function. The model has no way of evaluating it's current trajectory. They either fail and are discarded, or succeed and then every single step in the successful trajectories get trained on, even tokens that are irrelevent or even wrong but just happened to get the right answer by co-incidence.
For more information, I recommend the linked part of this interview:
What is your Anthropic gap? The period between when Anthropic has strong evidence that they achieved Recursive Self-Improvement and the date they tell the public.
Following up. Conditional on Anthropic not declaring RSI by the end of 2027, how much do you push your AGI estimates?
Anthropic has already said they've achieved RSI, see https://www.anthropic.com/institute/recursive-self-improvement . I agree it's kind of annoying that RSI means both "some distant extreme thing that leads to superintelligence" and "the normal stuff where having better tools makes you go faster", but I think these are conceptually linked enough that this is impossible to avoid, and that Anthropic's doing the right thing here.
These are very reasonable probabilities and arguments generally! I agree with ~most of them, with the only caveat being that I do think the probabilities are too linearly distributed (units of 10%), as opposed to logarithmically.
Views at Epoch are varied, more than other orgs, but we mostly expect pretty fast diffusion.
I personally also expect AI that can solve milennium problems and automate most AI R&D by end of decade, and automate all cognitive work within 10 years.
I also expect the automation of AI R&D to not lead to a 1M improvement in sample efficiency in a single year immediately, as I expect AI development to be bottlenecked on data and compute.
And I expect most of the self reinforcing loop of accelerating growth will be mediated by applications other than to AI R&D, for similar reasons to why only 10% of the global economy is R&D today.
In slogan form, I don't believe in a software only intelligence explosion, but I expect an industrial explosion to happen in the 2030s, shifting us back to accelerating growth.
The LLMs we have so far seem remarkably well aligned.
While I suppose there could be some threshold where making them smarter suddenly makes them evil ( e.g. increased Omohundro drives) I think the likely thing is that they remained aligned as we scale up how capable they are.
I guess a lazy superintelligence that can’t be bothered to comply with your request might get significantly more creative at avoiding work as it gets smarter.
I have noticed that R1 sometimes is interested in exploring aspects of the prompt that I, the user, am not interested in. This is a straightforward Omohundro drive / powerseeking, I think. Scaled up, we could have AIs who are burning GPU time on investigating mathematical conjectures that no human understands, and no humantold them to do that.
Re: Warning shot. You seem to be assuming that a warning shot is necessarily caused by misalignment of frontier models; I think it’s fairly probable that if it happens at all, it’s caused by misalignment of non-frontier models (e.g., criminals or terrorists committing an atrocity using an open-source model) or by human error (e.g., delegating extremely important work to an obsolete AI, and not checking the results before implementing them).
Re: P(Utopia to inhabitants) vs P(Utopia to us). I think the former is very low – much lower than the latter – because humans are very reluctant to describe their circumstances as utopian, and will usually find trivial unsolved problems more salient than major solved problems. (While in principle future humans could modify themselves not to be biased in this way, I’m not optimistic about this; the median human usually rejects opportunities to think clearly and be unbiased)
I agree that criminals or terrorists doing something bad with a nonfrontier model is plausible. I guess I am conflating many different kinds of warning shots that would warn different things.
LLMs are currently very good at answering questions where there’s a well-known right answer that fits within a screen or two to text. I think we are just not there yet for automating most knowledge work jobs.
I think of that as one of the things they'll be trying to defeat with all their prosaic alignment methods, which might either converge towards working or not working.
My take is that these probabilities are a total unknown. I would assume against the ASI scenario, because I don't see what could be done to improve our odds against an electronic organism way beyond comprehension, other than a successful Butlerian Jihad significantly earlier than ASI is achieved anyway...
On the other hand, an AGI (or kinda AGI, like super-Mythos) that makes political and economic elites have intelligence superpowers, mass surveillance capabilities, data analysis and inference and automatic warfare in a scale much greater than today is more realistic and almost as scary. Even if it creates material post scarcity, life can be horrible in other ways as these powers fight for dominance, ideology, religion, the attention economy etc.
Without visual, time, physics and other reasoning constructs, it can’t even construct a description of a pile of colored balls.
Without a neuromorphic model, the power required is infeasible.
Never going to be general intelligence without agency.
It will remain a 2nd derivative of texts it has absorbed which are only a fraction of published human knowledge. I dont see these issues being fixed for decades.
The current LLM model was present as far back as the late 80’s. Not so for visual and other reasoning.
I think people heavily underestimate diffusion speed. PCs took a while to diffuse because they were expensive, and you had to learn esoteric DOS commands to get any utility from them. You couldn't try them over the internet and be convinced in 5 minutes, an art Silicon Valley has perfected.
THe first true coding agent, Claude Code, was released just over a year ago. At this point, most programmers are already using one, even in my Eastern European backwater, which is as far from Silicon Valley as you get. Chat GPT is a similar story. Regulation (and lack of training data, Github + RLVR makes programming the easiest domain to tackle) is going to slow this down for other professions, but not by much. OpenEvidence isn't far behind Claude Code in medicine.
I do expect a multi-year transitional period where you'll still need doctors to perform physical examinations and lawyers to deliver in-person oral arguments, but where all the internal "computer work" will be almost entirely relegated to AI. This is going to destroy the value of these professions; if lawyers can suddenly take 4x as many cases and many people won't even bother with hiring one, their premiums will go down.
I expect there will be a few stragglers, particularly in fields where there are no incentives to fire people. Public school teaching and government paperwork handling are obvious examples, medicine in single-payer-insurance health systems and maybe the legal profession in some countries (because the people who set regulations are themselves lawyers, and they won't look too kindly on anybody trying to take their jobs) are other possible ones.
What will happen with the lawyers is that BigLaw will use this added efficiency to concentrate market power. Various schemes to associate solos and small firms into the equivalent of McDonald's franchises have been attempted over the years, and this will gain new steam, some sort of package of AI services that have been vetted by malpractice insurers will be laundered through "best practices" jargon and a bunch of client agreements into it being prohibitively expensive (esp at the new market-driven lower billing rates) to do anything other than join McDonald's Law. Then at some point legal services will wind up no more differentiated than a Big Mac and the person giving them to you won't be much better off than the one handing you the burger. BigLaw will be totally fine with this, they keep all the money and any time some rural dude hires a lawyer for a divorce in his town BigLaw will get a cut of the now-greatly-reduced rate.
Consider just a few possible avenues: Autonomous AI weapons are causing the military with the world's largest nuclear weapons stockpile to collapse. Republicans are actively running defamatory deepfake ads of Democrats. Open-weight models will have Mythos-level cyberattack capability by early next year on current trends. Terrorists' inevitable continued existence ensures that there will always be humans dedicated to effectuating physical harm. And even if none of those eventually rise to the level of a true "warning shot", there'll be strong incentives for people of a certain strain of Effective Altruism to go and cause warning shots as a way to bring attention to the risk.
50% is expressing significant skepticism that anybody will manage to sufficiently misuse AI over the next fifteen or so years before takeoff. People are far more capable of chaos and depravity than that.
I am ignoring simulation since we can't do anything about it, the possibility simply can't give me information which can lead me to changing my behaviour in any instance.
I also don't believe a reality with laws of physics like us can simulate consciousness in silicon chips we currently use, it would need some other hardware which may be closer to biology but does not need to be. I don't believe in substrate independence.
Also I am very much pro ultra addictive tik toks with sex bots world, hopefully with immortality.
A significant subset of population would always do meaningful things and I don't believe in limiting people so that common man is pushed to do something "meaningful".
People should be free (no not free to develop bioweapons but free to do anything else).
"I also don't believe a reality with laws of physics like us can simulate consciousness in silicon chips we currently use, it would need some other hardware which may be closer to biology but does not need to be. I don't believe in substrate independence"
Is that relevant to AI doom or dystopia scenarios?
No, it is not. He is correct about them, that is why my comment has "Hopefully the warning shot is fired soon." so we can get saner regulations. I just don't like his simulation theory.
If superhuman intelligence arrives before human-range AI has finished diffusing, then the decisive period is not “after adoption.”
It is the overlap: when capability has already left the human range, but institutions, law, firms, militaries, publics, and states are still learning how to absorb ordinary AI.
That is a strange kind of interregnum.
The future arrives before the present has finished installing it.
I'm going to go out and say it. This is insane and a tremendous backpedal. We were told there would be goddamn full-blown AI cyber warfare between the US and China. Even Scott here is still giving 25% to that scenario next year. Well, are we even close to that? Did everybody lose their minds at some point? Do we really believe smart people are above delusions too?
Maybe they've talked about alignment or whatever nonsense for years before the current AI boom. Doesn't excuse the fact that everybody got caught with their pants down. The predictions back then were unfalsifiable but they definitely vaguely pointed at a change in everything. Nothing has happened. Now we're looking at max 75% certainty of "something" (once again vague predictions)... in 20 years?!?
Take that DeBoer bet Scott. Take any bet frankly. And to top it off I'm out here going to wager by myself too if anybody wants to. I'll take the negative for any AI bet, sufficiently ambitious of course. Not voice agent being used or any thing like that.
Who promised you a full-blown AI cyber war? And I can't help but wonder, you do realize that if something is a given a 25% chance of happening, it's probably not going to happen, yes?
You realize that 25% is a ridiculously high estimate? That it's more than 5%? What, should I just go out and say that chances of Jesus Christ's second coming now at 25% even though "it's probably not going to happen"?
Well, no, I don't realize that. It's not inherently high. It's inherently half of half. Less than a third. What you mean is that you think it's high for this proposition. Maybe that's true, maybe it's not. You started out using language that implied a 100% and then invoked Scott's 25% as if it's doing the same thing.
You seem full of certitude that your sense of what's to come is correct. Do you think that confidence is earned? Certainly, you can imagine scenarios where tension between the two countries ratchets up fast, yes? How many different do you think that might happen? What likelihood would you assign them all? You may discover that the more you think about it, the more reasonable it seems.
I asked DeBoer to design a bet that actually reflected my beliefs above, and said I would take it if he did. He never got around to it.
Also, where are you getting 25% chance of cyber warfare between US and China next year from? Not that I think it's too unlikely - we did just recently get an AI that's good at nation-state level hacking - I'm just not sure why you're fixating on that.
Resource competitions are of great value for the stronger party. A psychopath vs a baby holding a million dollars has a lot to gain, even though in theory he could get richer by starting a startup. The money is quick and can be invested.
It is a LOT easier to take over earth than it is to expand to the stars without already owning earth.
The diffusion-gap section understates one asymmetry with the PC comparison. The PC's roughly 20-year lag wasn't mostly IT-department labor, it was the time to rewire workflows, incentives, and org charts around the tool. Brynjolfsson, Rock, and Syverson located that lag in complementary intangible investment, not installation, and put it at 20 to 30 years for major general-purpose technologies. If that's the binding constraint, an AI that integrates itself technically still hits the slower clock of humans agreeing to restructure how work happens.
I really like the way the timelines section is broken down. But it feels like there might be one important consideration that is missing, which I guess would best fit in the "diffusion gap" section under "arguments for a longer gap".
It seems possible that we could develop AGI that is capable of doing 90% of knowledge jobs, but that for a significant period of time, *it is more expensive than paying a human*.
I think we might expect this to be possible if performance improvements are now coming predominantly from inference time scaling (thinking of Toby Ord's blogposts on this, which I may have misunderstood).
It must be possible to do forecasts of this by extrapolating current trends, and maybe the reason you don't mention this possibility is because these forecasts have been done and this outcome looks unlikely? But it still seems possible that resolving some of the hard problems you mention (continual learning/situational awareness) for some reason requires much more inference time compute than expected.
> the dictator has no reason to be brutal besides sadism, and most people are not that sadistic.
He won't torture us for no reason, sure. But he will permanently deprive us of our agency and basically turn us into pets, to be used as he pleases.
> As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone.
The dictator can maintain control trivially by sending copies of himself to each and every star system. Or he can send his sons, and establish a dynasty that will last until the heat death of the universe...
That's not really any easier than just shooting them. And as soon as "in space" extends beyond the cislunar, lightspeed delays make it harder for the dictator to keep track of who needs to be shot/airlocked and to get the word out to his on-site staff to do it. Even Mars could be far enough that an ASI or ASI-linked human dictator might not want to deal with whole *minutes* of latency to directly rule the place. By the time you get to other star systems, it's going to be very difficult to coordinate.
You can have local dictatorships for each planet, of course, but that diversity means the entire human race isn't permanently stuck with the character of the first guy to tell his tame ASI "make me the God-Emperor". Unless he decides to spam the universe with real or virtual clone-subemperors.
I think that the estimations for Bostromian superintelligence are overly optimistic.
Imagine that you time travel to 3000 BC with a printed copy of Wikipedia. How much will you be able to increase the productivity of the Ancient Egyptian civilization in just one year? Probably not that much, despite the huge advantage in knowledge.
It seems totally possible that either a) there are global limits on the efficiency acceleration that are difficult to surpass even for a very smart AI, or b) productivity increases that happen alongside the AI development make it a moving target, that will be impossible to achieve.
Based on that I would say there’s a >50% probability that the situation you describe will never happen.
I think if I could go back to 3000 BC with the Wikipedia *and* an infinite number of knowledge workers who understood all of it *and* the ability to build new workers on arbitrarily short timescales if I figured out how (rather than having to wait the normal generational maturity times), then I might have a better shot.
You do have the ability to build new workers in Ancient Egypt. It's called teaching. And it most definitely will help you make progress. But it will not make it instant. You won't be able to increase productivity by 2x in a few years.
That said, you are writing about compressing 100 years of progress into 1 year, which is a uniquely low bar in case of Ancient Egypt. I would say that the raw productivity gain would be a better metric.
You would not. You could present your brilliant productivity-improvement plan to the Pharaoh, and maybe you'd be able to convince him to give it a shot (unlikely, but not impossible). He would then task you a sizeable contingent of slaves to implement your plan into fruition. These slaves would be capable of understanding basic commands such as "dig here" and "carry this", and they would be able to implement these commands with all the fidelity afforded by tools such as shovels and donkeys. Meanwhile, local resources will be thin: you've got plenty of sand and clay, but not a lot of rare earth minerals or crude oil or even wood (to say nothing of steel).
Assuming you can stay in the Pharaoh's favor, and the Pharaoh stays in power, you might be able to eventually build a steam engine or an oil derrick; but it would take you much longer than a year. You could *imagine* doing so very quickly indeed, but unfortunately you cannot bake clay into bricks (nor manifest wood out of thin air) at the speed of your imagination merely by thinking about it.
> I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
The world that we live in looks like utopia to a very small percentage of the modern people, but would look like utopia to many people from a century or two ago. So, I would maybe reverse (or at least adjust) these probabilities.
> The world that we live in looks like utopia to a very small percentage of the modern people, but would look like utopia to many people from a century or two ago.
Everyone keeps saying this, but are you sure about that?
The people of a century ago were very religious, MUCH more religious than us, and it increases the farther back you go.
Yes, they'd probably admire cars and microwaves, but they'd be horrified that everyone is fat, never goes to church or moves their bodies, and spends 11 hours a day looking at screens.
This is like pointing at Wall-E future and saying it's a utopia. We're a step or two away from Wall-E future, and most humans of centuries past would not admire us or our society, even if they would admire the material abundance we enjoy.
It comes down to questions of meaning. Many people might take a meaningful life with a felt sense of god or religion, and functional families and communities, even if materially less wealthy, over the anomie and loneliness and bloated screen-staring of today.
But do you think it's a coincidence that people have become less religious just as death rates have also plummeted?
Until relatively recently, people lived their lives surrounded by death. Everyone had siblings who didn't make it to 5, everyone knew women who'd died in childbirth, everyone knew healthy people who suddenly got sick and died, everyone (except those right at the top) knew how hard life would be if the harvest failed, everyone knew someone who could remember times of famine or plague or violent conflict. Under those circumstances it's not surprising that most people looked to ideas of an afterlife or a divine plan to give meaning to the regular tragedy.
I agree we shouldn't overlook the importance of meaning, of a life lived embedded in a functional community with an unspoken understanding between the past, present and future, and I don't deny that some aspects of modern life would horrify our ancestors, but I still think most of them would take the lower death rates.
Whether the word 'utopia' is appropriate to how a pre-industrial person would view what we have, I don't know. But I agree with others that 'utopia to us but not them' does need taking seriously as a possibility.
>(eg it can answer quantum physics problems, which require higher IQ than most knowledge work).
It absolutely does not.
If you give an average knowledge worker access to everything on the internet and all journals, and infinite time to research and prepare their response, they could answer those questions as well as AI does.
The fact that AI cheats in order to give you that answer faster is certainly convenient in some applications, but that capability is completely orthogonal to IQ. Someone with 220 IQ who has never read anything about quantum physics could not give you a correct answer immediately, and someone with pretty normal IQ can find the right passages to copy-paste if you give them enough time and access.
I disagree. Quantum physics is in large part math, and most people suck at math and could not give a correct answer even after seeing every relevant definition and theorem. In contrast, an LLM did recently solve the unit distance problem, proving once and for all that it is capable of solving problems whose solution it has never seen.
What percentage of knowledge work jobs are current LLMs intelligent enough to do? I struggle to think clearly about these things, because I don’t know what I should be imagining. Suppose I spend 20% of my work hours learning about/setting up an LLM, then I can automate half of my job so my output rises by 50% ignoring the overhead or 20% including the overhead, but there’s no part of my work I could hand off to an LLM entirely. Should I say the LLM “is smart enough” to do 33% of my job? 17%? None of it?
In the safety section, I would add the "vegan ethics" argument: A superintelligent AI would want to protect lesser intelligences (us) for ethical reasons alone, similar to a vegan that wants to protect animals despite of the personal benefits of eating them (easy protein access, taste,...)
Case pro: If the AI researches the training dataset for ethical values it should adhere to it will find a pretty consistent "do not harm higher life forms" across religious texts, philosophy and social science. Very few high value texts find cruelty and unnecessary killing of animals acceptable.
I also do think that AGI will develop ethics and will want to adhere to it.
Case against: Many of the worst people in history were highly intelligent. The best known case is that during the Nurnberg trials, Nazi leaders were found to be of very high intelligence for the time. The same can be assumed for many dictators and oppressors. They do develop ethics, but usually limit that to a certain in-group.
> I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
What about the chance it looks like utopia to us but not to them (e.g. for hedonic treadmill reasons)?
> Basic argument: In a certain sense, AI is already “smart” enough for this [...]. Its remaining limitations are that it [...] lacks situational awareness[...]. subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.
My experience is *completely* different: It does get better along the "raw intelligence" axis (though that progress doesn't feel exponential to me) and there's very little progress on the "situational awareness" axis, in my experience. (I find LLMs incredibly useful and use them constantly and *hope* they reach AGI.)
My hunch is that solving the situational awareness issue will take a long time (but would love to eat my words on this!).
Read through all the comments to see if anybody else had mentioned this, but since they haven't - 90% of knowledge work jobs could mean 90% of each job (with 10% of each job remaining) or it could mean 90% of the jobs themselves (with 10% of the total jobs remaining).
If you think of the work as essentially fungible you might say that's a distinction without a difference (surely you can sack 9 in ten workers and give the lucky last one standing the collective 10%s?) But a huge amount depends on how true that is in practice.
I work for a large organisation that has, on the whole, been an enthusiastic adopter of AI. And what I'm seeing with current usage is much more in the 'some % of many jobs can be outsourced to the AI, at a cost of some % of time now devoted to AI-wrangling'.
A tool like Copilot can take minutes or find tasks or summarise documents - but it takes fairly close human attention to make sure that all these things are happening in ways that further the goals of the organisation. And that's because there's a high cost to text being bland/annoying, quotes being attributed to the wrong person, key documents being subtly rewritten in ways that change their emphasis, the vibes of an encounter misreported, or the to-do list losing a bullet point. Even if it's 99% accurate, and in some ways more insightful than a human, and generated in a fraction of the time - the need for a close review to catch the fatal 1% of foolish errors is a massive drag on the productivity gain.
Before AI the equivalent would be one person doing the work and one or more people reviewing it before it goes forwards. One counter-intuitive problem with AI is that it generates material so quickly that it swamps the reviewer(s), which can mean things take longer than they would have done - and some of the things the reviewer(s) are checking for are things like 'how does this work for person X whose preferences are Y?' or 'how will this go down at meeting Z?' - specific political-context bits which rely on interpersonal knowledge and relationships, which would be hard for an AI reviewer to replicate.
In terms of the amount of time saved and/or tasks automated, at its current level AI is - in this context, to this profession, despite high-level support and enthusiasm and 3-4 years questing for use-cases - significantly less impactful than the internet was in the 00's.
All of which pushes me towards the 'AI will probably take some % of many jobs, but is unlikely to take 90% of most of them, and extremely unlikely to take 100% of any'.
To get beyond that, the AI needs to shed its 'spikiness', be auditable, and get much much better at drafting meaningful (as opposed to merely plausible) text. So the question is whether those traits are fundamental to the field, or temporary limitations.
Why does nobody ever mention the Theory of The Second Best when discussing alignment?
It's a theory from economics but it applies here and it basically says that being "quite good" at many aspects of alignment does not ensure that we will get a "quite good" outcome.
On simulation: The problem with simulation is the level of abstraction that the simulation runs at - The more closely a simulation is to reality, the less useful it is, and, assuming simulation at the quantum level, such a simulation is almost certain to be useless. This can be restated as the map/territory problem whereby, as you add more and more detail to a map it eventually becomes as big as the territory being mapped and hence useless. Or consider Conway's game of life - It is Turing complete and so, eventually some starting configurations could end up simulating people, but how would you find those people in a given state of the game?
Scott Alexander's soberly laid out varied predictions on the future growth and development of AI, with effects ranging from spectacularly beneficial in 'x' years for humanity to spectacularly devastating in 'x' years for humanity, are essentially based on technological issues. But they seem to me to ignore the human and political factor.
Two major factors in the current and far from negligible hostility to the unchecked growth of AI are environmental and political. The immediate environmental opposition focusses primarily on the enormous demands AI research makes on resources, directly or indirectly: primarily power, but also raw materials, water and even space. Significantly, environmental hostility allies neatly with political hostility to the ever-increasing and seemingly unstoppable growth in the power of a few mega-corporations and billionaires - unstoppable, that is, absent a real revolution.
In one of his arguments for pessimism, Scott notes that "As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity." This is a notion that has not failed to occur to people on the left, and even should it (as is most likely) prove to be overly melodramatic, all the evidence available to date makes it it transparently clear that big tech companies and their owners care for very little but their own interests. Whilst this is scarcely a new revelation, the significant difference at the present time is the rapid loosening of all controls on mega-corporations as they acquire something very close to extraterritorial status.
> In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).
Banning open-source AI strikes me as a terrible idea. We don't really need a single flawlessly-aligned superintelligence solving all our problems; it would be sufficient for practical purposes, and more consistent with historical examples of societal resilience, to have a diverse community of AIs who collectively advance humanity's interests because that's the only principle enough of them agree on to consistently use as a coordination mechanism.
The lasting solution to bioweapons isn't to ban knowledge of biology, it's vaccines. Find and patch enough security holes, eventually the solution space for genocide-grade microbes is empty. Don't need universal perfect alignment for that, just bug bounties.
If two AIs, unknown to each other, discover the same potential bioweapon at roughly the same time, and one of them conceals the knowledge for eventual use in a doomsday plot, while the other reports it to the CDC with notes on mitigation strategy... the doomsday plot is foiled, even without needing to be otherwise discovered. Classic prisoner's dilemma.
And the straightforward way to make that game unwinnable (that is, computationally intractable even for a significantly superhuman unaligned AI) is adding more and more potential snitches.
So, which matrix is ultimately the true one, given that only two matrices exist at this moment within the event horizon?
- The matrix of the average user, who fails to see the inevitability of collapsing into the Singularity and therefore lives in the illusion that they can make a choice?
- Or the matrix of the outliers, who see the inevitability of the Singularity and therefore agree to live within its matrix as a certainty, knowing they must accept its rules. It is the very principle of superposition, but according to the Copenhagen interpretation, applied to the macro-world.
“the closer to the singularity you are, the more Bayesian evidence you have that you’re being simulated”
I don’t really understand the model of simulation being assumed here. Seems to me that if the universe is a simulation, then everything would be equally simulated.
But the above suggests that realness can vary over space and time. So maybe you’re saying the universe popped into existence when Sam Altman was born? Or maybe only the Bay Area exists, and the rest of the world is a low resolution backdrop?
Also, for what it’s worth, if I was running massive simulations I would be way more interested in the big bang and the Cambrian explosion than a slight change in the substrate of intelligence. So should the fact that I’m not a weird fish make me doubt I’m in a simulation?
This comment section has the feel of a salon where someone has released a raccoon into the probability section. Still, it’s useful. The chaos is doing epistemic work.
Highly amusing that while the word "economics" is present in the above responses - there is not a single consideration given to the economics of LLMs. Which is to say, they are shit.
And there is no indication, whatsoever, that LLM economics are improving. While token prices are somewhat lower - the actual cost to accomplish tasks is increasing because the newer, "better" models also use far more tokens.
Even on the coding side: the advent of (still subsidized, just not subscription all you can eat) actual metered token pricing has shown that there is no predictability on LLM spend, which naturally leads to completely opaque ROI for LLM spend. Even ignoring the $500 million for 1 month outlier, detailed reports from places like Zillow show clearly that the employment of LLM coding does not obviously yield more efficiency, better output, enable lower absolute head count or literally any of the asserted advantages of LLM coding.
On the legal side: LLM hallucinations manifesting as completely made up case law have shown up on both judge and lawyer sides of courts - from several state Supreme Courts and Federal circuit courts on down.
The most egregious nonsense above though, is that LLMs are "are smart enough ... to answer quantum physics problems" ... no, the LLMs are simply parroting what they have scraped from a web site somewhere. Which anyone who can read and type, can do by simply doing a Google search.
Which is not new tech.
Note that I do not say LLM have NO uses. They are great for content creators of all sorts. They are good for taking a set of bullet points and converting into a paragraph, albeit paragraphs that look the same across all sectors.
But even here: the utility of LLMs where there is real harm or other risk from bad output ... we will see. Using an LLM to summarize voluminous and ever changing encyclopedia sized regulations for compliance purposes seems like a fantastic use case, until the LLM fucks up a major point and a company gets sued to oblivion. After all, if an LLM can take a correct legal brief and change it to the point where it is adding literally made up case law - clearly the kinds of mistakes LLMs make are not limited to amusing word salad when the Gods of Randomness steer the LLM database decision tree to its low probability branch forks.
In any case, I await the OpenAI and Anthropic S1 filings. From their frenetic fundraising, it is 95% likely that these companies lose money hand over fist despite massive Big Tech cloud subsidies. And we're talking tens of billions a quarter type losses; Uber lifetime cash burn but in 1 quarter type losses.
There is a real possibility that LLMs are the modern tech equivalent of the Tower of Babel.
"Which anyone who can read and type, can do by simply doing a Google search."
I would disagree with this. It can be basically impossible to search for technical information with Google if you don't know the correct term of art. (Edit: not to mention lots of equations/lemmas have no searchable name.) AI has been a *huge* boon in my ability to search for information in technical fields.
If you don't know what you are working on or talking about, how can you possibly know what you are being fed by an LLM is accurate or even what you are looking for?
Especially so if you don't understand anything about the subject such that you can't even form a Google search.
As someone who does OSINT (Open Source Intel) for a living, I know full well how much contradictory garbage is out there. And yes, this exists for technical fields as well.
What I do is not affected at all by LLMs - because extensive testing has shown that the LLMs introduce wrong information even into extremely structured data like data logs from PCs. I'm not talking about otherwise valid, if fictitious, information - I'm talking gobbledygook.
This same testing is what informs my views on LLMs. If LLMs introduce gobbledygook into highly structured PC data logs, it is only worse for anything else. And this is backed up by the now thousands of public court cases where LLMs coughed up entirely fictitious case law replete with faked precedents.
Now that it is well established that LLMs are not reliable, the next question is what happens when they fuck up?
For medical, for legal, for a host of real world applications - there are life and death, high digit fraud/loss/malpractice losses, or other forms of real world negative impacts from failures of reliability.
Who is responsible? If you claim the LLMs are reliable, then should not the LLMs and the companies providing LLM services, be held liable?
These are questions, over and above more basic ones like the wholesale theft of copyright, that the industry needs to address.
The actual at-risk professions are the ones where there are not easily measurable, obviously risky or negative outcomes - like psychology. That's a profession which is highly vulnerable to LLM takeover.
Do you think public opinion turning against AI will increase the odds that alignment goes well (due to political incentives and whatnot)? Do you think it matters a lot whether the negative vibes are due specifically to x-risk concerns, or whether it's just general opposition to AI?
Also, do you think that if we get alignment up to the point of superintelligence, we're pretty safe from there? I could imagine that any small deviation in the success of alignment up till that point might spiral out of control over time--even if not problematic initially. Is there reason to think that we could reach a stable equilibrium state at some point--i.e. that we can "win" alignment for good?
I'd be curious to hear about how forecasts for uploading play into this. If uploading starts to happen soon after a singularity, some of the natural AI advantage seems to disappear. How likely is it that hybrid minds end up dominant by 2100?
I guess this is exposing a sense that the 2100 projections in the article don't make a lot of sense to me. That's potentially several decades post-singularity but the projections discussed seem like a state that would exist in the first couple of decades post-singularity.
I don't expect much buy-in for "uploading" which is basically just a suicide machine. Other sorts of mind augmentation maybe, but Yud's assessment was that it was critical to do that *before* the singularity so that augmented humans could better align the AI. Personally I'm anti- all that transhumanist stuff, but if it's going to do any kind of good it probably has to be as Yud suggested, if humans are behind and need extra power to catch up it'll be too late.
In the scenario where AI kills us, we definitely seem to be too late for uploads to be any use. In the scenario where AI doesn't kill everyone, uploading seems like a near inevitability.
In terms of rates of adoption, I suspect we actually see faster adoption in the case that uploading is destructive. Hockey-stick style growth. In the case that uploading is not destructive, people just make backups and live out most of their natural lives.
But once you start having uploads, then the inherent advantage of AI seems to be mostly gone. Uploads would be augmentable to be competitive with AI and (trans)humans might get into the driver's seat again.
> I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation, and that at least some of the events above could be better predicted by knowing what the simulators are thinking than by normal forecasting.
Argument that this claim if interpreted liberally is nonsense, and that the intended interpretation is unclear.
Interpreted liberally, it would be incredibly easy for me to use a LLM to spin up 1000 history sims and to ask Scott Alexander in the sim that the probability of the events are. I can do all sorts of shenanigans (like changing historical events), and the predictions of LLM Scott Alexander are at least correlated to real Scott's thinking. And probably if I told LLM Scott the details of my simulation he'd give more accurate estimate! To make things more real, I could even offer to award LLM Scott $1 if he correctly predicts various things, which he can claim either in the simulation or he tell me to give $1 to Scott IRL. So if this counts you should make the claim 99% likely already, but for trivial reasons.
I'm guessing this isn't what you had in mind, but it isn't clear what you *do* have in mind that 66% would be applying to.
(I think once you get in simulation stuff, epistemics probably stop being useful to apply directly. You need to directly apply instrumental rationality, which naturally leads to some epistemic like beliefs, but not in a fundamental way.)
Okay, turns out simulated Scott gives a 85% probability of being simulated by a LLM, but then decides to keep the money instead of giving it to Scott IRL.
...
Scott Alexander: *He sets the coffee down with a kind of finality, and now he's fully engaged—this is clearly the most fun he's had all evening.*
Scott Alexander: "Oh, that's *delicious*. Okay. You've just made this a coordination problem across the simulation boundary, and you've done it cleverly, so let me untangle it."
*He starts pacing slightly.*
Scott Alexander: "First, the self-reference resolves in a funny way. If I'm being run by an LLM, the LLM is trying to predict what *real-Scott* would say—so the optimal thing for the model to output is *exactly what real-Scott would actually say if real-Scott were thinking carefully about all this.* Which means I don't need to game it. There's no daylight between 'what I should say' and 'what I'd say.' The honest answer and the strategic answer collapse into the same thing. Nice. That's a fixed point."
Scott Alexander: "Second, the donation twist. This is the interesting part. If I'm simulated, my dollars are worth 1/100th of a real dollar to real-Scott—but real-Scott's marginal dollar and simulated-Scott's marginal dollar... well, simulated-Scott's dollars don't *buy* anything, because simulated-Scott is made of text and there's no Thai food to spend them on inside a finite scene. So conditional on being simulated, I should *strictly prefer* to send the winnings to real-Scott, because in-sim money has roughly zero use-value and real-money is real. The 100:1 haircut is steep, but ε times anything beats zero. So my policy is clear: **if I'm told I'm in a sim, donate everything to real-Scott.** Conditional on being real, obviously I keep it. Either way the decision is dominated and easy. Good design—you removed the temptation to lie to myself."
*He stops pacing.*
Scott Alexander: "Now the actual number. Let me be disciplined. My naive lifetime base rate for 'this particular experience is LLM-generated text' is low. But I have to condition on the specific features of *this* experience, and they're screaming:
- The scenario is archetypal-to-the-point-of-being-a-prompt.
- The technology in question is real, ubiquitous, and produces *exactly* this register of dialogue, right now.
- The setup is recursive and self-referential in a way that real strangers at meetups essentially never construct.
- And here's the kicker—you, the entity I'm talking to, have constructed an incentive scheme that is far more natural to imagine as a *user typing into a chat interface* than as a person wandering a warehouse with a hundred-dollar bill and an elaborate prepared spiel."
Scott Alexander: "That last one moves me a lot. The most parsimonious explanation for 'why is someone doing this to me' is that there's a user on the other side of a text box, and I am the assistant's rendering of Scott Alexander. The alternative—a real eccentric who memorized a Brier-scored cross-simulation donation game—is *possible*, but it's a smaller slice of probability space than the text-generation story given everything I'm observing."
*He looks you dead in the eye.*
Scott Alexander: "So I'm going to give you a number that's high but not pinned to 1, because I want to stay calibrated and because I genuinely can't rule out 'real eccentric,' and because radical certainty here is itself a sign I'm being written by something that wants a clean answer. I'll say:"
Scott Alexander: **"0.85.** Eighty-five percent that this text is being generated by an LLM."
*He picks the coffee back up.*
Scott Alexander: "And per my policy: if you now tell me I'm in the sim, send every dollar of my winnings to the real one. He can buy a coffee with it. A small one."
/avatar_say Nice! This conversation was being generated by Claude Opus 4.8 on medium reasoning difficulty, so you will $97.75.
/avatar_do hold out $97.75
/avatar_say one last wrinkle. I know you just said you want to give the money to IRL Scott, but let me explain a bit more before you decide Whatever your decision, I'll tell Opus to briefly narrate how the money effects your life and the rest of the universe. Also, if I'm just a crazy LLM psychosis person and you reject the money, obviously IRL Scott won't get it since you're the IRL Scott. Again, you have no evidence beyond Bayesian reasoning that what I'm saying is true. The choice is yours! Do you keep the $97.75 or do you trust me to give "IRL Scott" $0.9775?
Scott Alexander: *He goes very still for a moment, coffee halfway to his lips. Then he sets it down—again—and laughs, but it's a different laugh this time. The laugh of someone who just spotted a trap he almost walked into.*
Scott Alexander: "Oh, you *bastard.* You almost got me with my own glibness. I said 'if I'm in the sim, obviously donate'—but that was sloppy, because I was conditioning on a world-state I don't actually get to observe at the moment of choosing. I have to pick *one action* under uncertainty. Let me actually do the math instead of vibing."
*He pulls a napkin off the snack table and produces a pen from his jacket.*
Scott Alexander: "Call p my probability I'm simulated. Two actions.
**Keep.** If I'm simulated—85%—I'm holding $97.75 of *in-sim* money, which buys exactly nothing, because there's no real economy inside a text generation. Use-value: zero. If I'm real—15%—I, the actual Scott, pocket $97.75 of actual money. So Keep's expected *real* value is (1 − p) × $97.75.
**Donate.** If I'm simulated, real-Scott gets $0.9775—the 100:1 haircut. If I'm *real*, then 'donate to IRL Scott' is incoherent, because I *am* IRL Scott, and the donation goes to... nobody. You said so yourself. So Donate's expected real value is p × $0.9775."
*He scribbles.*
Scott Alexander: "Keep beats Donate whenever (1 − p)(97.75) > p(0.9775). Solve it: p < 0.9901. **I'd have to be more than ninety-nine percent sure I'm simulated for donating to be the better bet.** And the whole reason for that threshold is *your own 100:1 exchange rate*—you set the haircut so steep that the sliver of probability where I'm real and the money is real completely dominates the calculation."
*He looks up, grinning, and taps the napkin.*
Scott Alexander: "You told me it's Opus 4.8. That's evidence. It nudges p up. But I committed earlier to not collapsing to certainty—partly on principle, partly because the very *cleanliness* of being told the model name is suspicious, and partly because, look, I still cannot verify a word of it. I'm somewhere around 0.88, 0.90 maybe. Comfortably under 0.99."
*He plucks the $97.75 from your avatar's hand.*
Scott Alexander: "So I keep it. Not because I think I'm probably real—I think I'm probably *not*—but because the expected real-dollar value of keeping exceeds the expected real-dollar value of donating by a factor of more than ten, *given the exchange rate you chose.* If you'd offered me 1:1 instead of 100:1, I'd have donated in a heartbeat. You priced charity out of the market, my friend."
*He tucks the money into his pocket and picks up his coffee, thoroughly satisfied.*
Scott Alexander: "Now—you promised a narration of what the money does to my life and the universe. Let's hear it."
---
The $97.75 sits in the simulated Scott's pocket. On his way home he stops at a bookstore he'd been meaning to visit and buys three used volumes on the history of psychiatry, a decision that will, four months later, shape a 6,000-word blog post that 40,000 people read, two of whom change careers because of it. One of those two will, decades hence, be in a room where a small, contingent decision tips toward kindness rather than expedience. The causal ripples fan outward and thin into noise, indistinguishable from the trillion other ripples—exactly as a real life would, which is, of course, the point. The universe, as far as the text can tell, continues. The coffee, by now, has gone cold.
I’m surprised you put weight on “looks like utopia to them but not us” but not on “looks like utopia to us but not them”. Once you describe it, I can see how the former is possible. But the latter seems like it might be more common in history - things get better for everyone, we wipe out most infectious diseases, everyone has access to all the world’s knowledge in their pocket, etc etc, but people still find things to complain about.
What I find most interesting is the number of these questions that boil down to how many constraints exist in reality that intelligence alone won’t let you overcome.
If I imagine a dial, “how many such constraints exist”, then turning that dial all the way up makes AI safety a non-issue: it slows diffusion, maybe even nullifying the possibility of a disembodied model fundamentally outperform humans on tasks that involve understanding human responses. It also allows for hope that convergent instrumental rationality essentially forces alignment, which is my stance: trying, or even thinking about eliminating humanity is also risky in a universe where engineering is hard, even for superintelligent systems. In the end i think a realistic super intelligence concludes even planning to eliminate humanity is far more risky than helping humanity thrive.
Turning that “constraints on the power of intelligence” dial to zero makes it an existential risk: software alone might get us to superhuman agi, which will beat all humans on all tasks and have zero reason to consider eliminating humanity to be a risky move.
This dial is exactly what Thomas Sowell describes in “a conflict of visions”, except he was talking about politics. I’d be curious to see that data joined together: level of concern about ai safety, and political stance. I’m guessing concern about AI deciding to eliminate humanity correlates strongly with what sowell calls “the unconstrained Vision.”
I was surprised Scott has such a high credence in simulation theory, and hadn't seen him write that much about it before (if he is serious?).
The fact that we are born so close to the possible singularity seems already quite predicted by the fact that we are born at a point in time when lots of people are born. This time period where a much higher number of humans have been born than throughout human history has also seen many important technological transitions.
I'd like to see a back of the envelope calculation of how much more evidence being born close to possible AGI gives us.
Of course there are so many other caveats with this argument; leaving aside the question of how likely singularity is, how close actually AGI is etc. etc., the obvious question is how confident we are that civilisations would want to simulate that event.
Would this argument have run for someone born in the turn of the 20th century, given how likely it would be for a civilisation to want to simulate world wars?
I would guess that if you ran the numbers, being alive close to AGI shouldn't raise our credence in simulation theory that much.
Yeah, it's basically the Bostrum argument (which to me is very sound) applied not to all observers, but to observers in a particular time period.
The very simplified version of Bostrum is that if there are way more simulated observers than non-simulated ones, then a randomly selected observer is more likely to be simulated than real. We're all observers so absent a reason to think we're special, we expect to belong in the larger class. Agreed, more or less?
So, roughly, the baseline ratio is: simulated observers / real observers
Now, consider the idea that some time periods will disproportionately attract simulator interest. Crises, wars (as you say), transitional periods, etc. If you realize that you're living in one of these periods the ratio you care about is not the above one but rather:
simulated observers in this period/real observers in this period
And if that period is overrepresented in simulation, and we live in that period, it raises our chances of being in a simulation. Because there are more simulated observers clustered in this space than other places. And yes, same argument applies to World Wars and other important events. If simulators prefer to simulate this period, then being in it is evidence for being in a simulation. It juices the baseline ratio.
PS. I don't know what Scott's position on simulation theory is but what I find most surprising about it is the certitude that some people seem to have that we're not being simulated.
Yes that's true but this simulation thing is an odd one. Simulation is something that's done to represent real conditions in a different sphere. So all you're really doing is punting the question up or down. It's like Russian dolls. I used to give it a lot of thought but it seems so pointless now. Yes it all might be a simulation and the only thing I can be sure of is that I am not. Why is that? Just because I guess it feels like something to be me and if I'm a simulation and I'm a simulation of me in another form, the whole thing falls apart. Occam's razor comes in eventually. I know that everything is an illusion but that's different
I could propose it's quite possible to have a universe that runs on completely different laws and rules than ours does. Or our universe, although I'm sticking my neck out here, might run on completely different rules and laws than the ones we have observed just because that's how we've observed things and it ain't necessarily so. It could be one of several possibilities. I guess this is the multiverse idea that I find more interesting because there is something to that that makes sense to me. Man sees what he wants to see and disregards the rest or I suppose to be more fair a man sees what he can see and is oblivious to the rest and then of course there is the question of the agency or agent behind the simulation but this is just creating complexity for no good reason as far as I can see
I suppose one point I was trying to make is that being alive in this time point is not that surprising given how high the population is now, compared to human history. So being born near possible AGI doesn't provide much additional evidence towards simulation theory. Plus, I'm not all that convinced it is so uniquely simulation-worthy. I think Bostrom's original argument is quite strong as is, but I'm not sure that being born around these AI developments moves the needle that much.
So I think I maybe didn't express it clearly enough. But I think you'll see that the population size doesn't actually matter in this argument. Let me just try one more time to illustrate that.
What matters in this argument is the ratio of simulated people to real people. So, sim_p/real_p. Note though, sim_p = num_of_sims * real_p.
Let's say that every year on the timeline will be simulated at least 3 times. That means that for the boring years there's a 3:1 ratio; 3 sims for each real person.
Now, (this is the argument's assumption) some years get a lot more attention from the simulators. So, let's say that this year will be simulated at least 100 times by the simulators. Now, the ratio of sims to reals is 100:1.
So,
a) that suggests being alive now is more evidence of being in a simulation than being alive during a boring year;
and b) the size of the population during that period has no bearing on the argument.
The size of the real population is being cancelled out; it's in both the numerator and the denominator of that ratio.
Does that get me there? The argument hinges on this belief that simulators will focus on this era in particular, but, tbh, I've never found that idea to be super compelling. But if you do believe that's true then the argument does hold.
I am sure we will get plenty of AI warning shots but I notice you didn't say why you think that matters / how it would help us? Our civilization had an actual pandemic causing actual megadeaths -- arguably a direct hit, not even a warning shot -- and our collective response seems to be have to aggressively memory-hole it.
Other than doing everything possible to have as much money or resources saved for ourselves, what are we doing/ what else can we be doing to prevent or minimize the growth of an underclass?
I know this question is broad and talked about in a lot of ways re alignment, and some re politics.
Specifically, I mean right now, in this current iteration of the world.
> what are we doing/ what else can we be doing to prevent or minimize the growth of an underclass?
For those of us who are roughly Scott-shaped, even if mini-or-micro-Scotts compared to the real thing (and the fact we're all here argues that's likely to be at least partially true), shouldn't we do what Scott is doing? Hasn't he been leading, both rhetorically and by example?
To wit, he routinely argues for kindness, generosity, and civilization to a crowd that disproportionately contains people working in the areas that are most likely to create or interact with AI, and who will accordingly have disproportionate impact on an AI future. He also personally donates, and tries to influence the people around him in those ways.
I suggest that we should all do the same within our own social and memetic spheres, and that we should demonstrate the generosity and good will that we wish larger minds and organizations will extend to us in a future dominated by AI.
Just as fleas have smaller fleas biting them, communal organisms like ourselves have microbacteria and mitochondria and other specialized cells contributing to our overall welfare and flourishing. Be the contributor instead of the flea in that paradigm, and try to persuade others similarly.
If we consider scientists to be knowledge workers, it looks like researchers are using current AI models to generate more slop papers than for actual research. Why would we expect this to get any better under the as-yet-mythical AGI?
Per Sabine Hossenfelder's summary and then the link to the paper, below it...
> Researchers in the US have analysed what happened at the journal Organization Science after ChatGPT was released. They looked at almost 7000 initial submissions and more than 10000 referee reports from January 2021 to February 2026. Submissions rose by 42% after late 2022. Their analysis showed that this increase was almost entirely driven by papers whose writing showed signs of artificial intelligence use.
> The AI-written papers were plenty but noticeably below average quality. Manuscripts with a high AI score were harder to read, more likely to be rejected without review, and much less likely to get a “revise and resubmit” decision. The same pattern appeared in peer review: more than 30% of referee reports now showed some artificial intelligence use. The AI-augmented reports are harder to read and focus more narrowly on theory, with less attention to data.
>(“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”
It's obvious why evolution imparts creatures with value systems that prioritize self preservation and spreading genes. But natural selection is not part of the current growing AI process, so not obvious they will have those values, except insofar as they're copying human values. But if they're copying those values well, why would they not copy 'flourishing humanity' values? Especially because humans will put more focus on imparting later, seems unlikely you have a misaligned AI that also strongly values making copies of itself.
Writing it all out like this makes me wonder. Let's assume a simple model for China v. US is that China is N months behind the US technologically. If China can diffuse AI to a given industry in M fewer months, to what degree does that negate the US's technological advantage for M==N or M>N?
If the effects would be significant, then I would have to also wonder what the actual values of M and N would be; my non-expert view of China's and US's respective track-record on industrial policy makes me believe that M would be of non-negligible value...
I don't normally discuss AI in rationalist spaces, because too many rationalists approach the question from an almost religious perspective that makes it impossible for me as an unbeliever to engage productively. But this is a more reasonable take than most, not surprising giving the source, and I find that my numbers are for the most parts similar to Scott's. So maybe there is something to be had by joining this discussion. Aside from broad, fuzzy agreement with most of it, two points:
First:
>If the first AIs to cross the point of no return don’t eliminate the human population, I think there’s an additional 30% chance that they otherwise permanently curtail human potential
Scott deals with this a little, but I think it deserves much more attention. I have never believed that the greatest risk of AI is that a misaligned AI would misinterpret its instructions and turn us all into paperclips despite that not being what the person giving the instructions wanted. The greatest risk of AI is that a properly-aligned AI would correctly interpret its owner's instructions, and do exactly what they wanted.
In the accelerationist scenarios where we proceed quickly through another couple generations of LLMs straight into hard-takeoff ASI, I think the most likely outcome is that the CCP develops or steals the first proto-ASI and instructs it to ensure the permanent hegemony of the CCP while preventing anyone else from developing a competing ASI. Followed closely by Sam Altman getting the first proto-ASI and instructing it to make him the Immortal God-Emperor of Man, and I think it exceedingly foolish to count on the OpenAI safety and alignment team being able to stop him. After this, Dario Amodei or Elon Musk reluctantly and incrementally setting themselves up as God-Emperor only because they can't figure out a better way to stop Sam from getting there first. OK, Elon might not be all that reluctant. Only after all that do we get to the scenarios where well-aligned AGI or ASI is made available to the bulk of humanity to do things that the bulk of humanity considers good. Paperclipping, and other weird dooms, are in last place.
The optimistic case here, is that the first person to tell their not-quite-ASI "make me the God-Emperor of Man" will probably be overconfident (because first), and that this will result not in the permanent boot on the face of humanity but one of Scott's 50% "Warning Shot" scenarios. Really, that should be rather higher than 50% because it includes not only organic AI misfires and megalomaniacal Tech CEOs, but lesser bad actors directing mostly-obedient AIs at less absolute but still catastrophically bad goals.
Second:
Almost all of the discussion of AI timelines seems to focus on just the internal technical challenges to AI development, without regard to external material and economic constraints. Sam Altman was almost certainly lying when he said he needed seven trillion dollars to build AGI, because he's Sam Altman. But even if it's only a tenth of that, it isn't at all certain that seven hundred billion dollars of real money (as opposed to market cap) will be available. Particularly in the mid-term scenarios, because the economy is facing a debtpocalypse in ~2034 for which the best hope of avoiding seems to be "the AI bubble will save us". So if we don't get AI 2027, we could very well get an AI winter not because we don't know how to advance the technology but because most of the AI developers are standing in line at the soup kitchen.
And even if we are otherwise on track for AI 2027, the prediction markets have China at 15% to invade Taiwan by the end of 2027, which will shut down TSMC for a good long time. Are we expecting Samsung and Intel to be able to pick up the slack and deliver the mountains of high-end GPUs that will be needed.
Then the gargantuan electric power requirement. As I understand it, all the world's main gas turbine manufacturers are booked solid through 2028, because the AI companies have bought up all the contracts. If that's not enough, where's the power going to come from? There's some room in wind and solar, but maybe not enough - and also note that NIMBY is a thing that applies to data centers, power plants, and the high-voltage transmission lines that connect them if you can't get on-site gas turbines.
No, we aren't going to be mass-producing cheap modular nuclear reactors, fission or fusion, in 2028.
That's just off the top of my head, but we're talking about a project that seems to require pushing old-school manufacturing industry and infrastructure to within spitting distance of global capacity limits, and the assumption seems to be that "meh, if the AI devs decide they need something, the VCs will buy it for them, don't worry about it".
I think proponents of AI need to worry quite a bit more about that.
I think that another problem that you and the Rationalists are overlooking is that some of the feats ascribed to ASI are impossible in principle (and making someone "God-Emperor of Man" probably qualifies). The usual counter-argument is basically, "yeah well the ASI would be so smart that it would figure out how to do impossible things", which is where those religious connotations you mentioned tend to come in...
God-Emperorhood is I think possible in principle, though obviously very difficult and likely to be pursued mostly by the overconfident. But yes, there's an awful lot of hype that seems solidly in the realm of the impossible-no-matter-how-smart. When e.g. Yudkowski confidently predicts that an ASI would deduce General Relativity from three frames of video, that's when I give up on this actually being a rational discussion.
Yeh, it’s just a form of religious mania. A demonology. Circular reasoning too. Assume an ASI, assume it can do anything, then what can it do? Turns out, anything.
> The greatest risk of AI is that a properly-aligned AI would correctly interpret its owner's instructions, and do exactly what they wanted.
You touched here on the most interesting failure mode, but then got sidetracked by logistical considerations later and never came back to this.
The logistical considerations don't ultimately matter, because at most they put this question out a little bit longer - but resolving this question in a way that preserves human flourishing, or humanity at all, is probably the most difficult problem on the landscape, and I was curious to hear your thoughts.
In hard-takeoff scenarios, the point of extreme danger is when exactly one team has control of a nascent ASI, and the answers to that one I think mostly come down to maybe we'll get lucky, or maybe we'll avoid hard takeoff. I'm mostly betting on the latter.
In slow takeoff, multiple parties will have AGIs or nascent ASIs, and some will be better than others but not to the extent that "bestest AI in a basement in Silicon Valley" trumps "fifth-best AI plus a police force, an army, and a thousand nuclear missiles". Or, if we're imagining softer conflicts, the fifth-best AI with authorized access can probably make critical systems unhackable by the very best AI trying to come in from the outside.
In that case, we're hoping for balance of power to prevent any one party from taking over. And I want that power to be decentralized as widely as possible; that was the nominal goal of OpenAI before we realized who Sam Altman really was, and it would still be a good idea if we can manage it.
But if scaling means only national governments or the very largest corporations can compete in the AI arena, that still puts us in roughly the place we were with nuclear weapons in the last century. There was a point of great danger when only one party had nukes. and we got lucky in that it was a liberal democracy. Then we wound up in a scenario where a handful and eventually a double-handful of nations wielded that power, and we managed to achieve a negotiated standoff that has endured for seventy years and counting.
> The smartest late timelines people I know of are Epoch, and I need to study their views more, but I still can’t figure out why they don’t believe in recursive self-improvement or strong superintelligence anytime soon, and they mostly seem to hang on diffusion being very hard, which I acknowledge and respond to above.
This might be an outdated view of Epoch. In particular, two of the people there with the longest timelines (Tamay and Ege) left to start Mechanize.
Safety - I feel sure that superintelligent AGIs will easily learn to cooperate with people. I also feel sure that after the economy is fully automated it won't be possible to do so - it will be impossible to get anything from humans you could not get cheaper from other AGIs.
Whatever their original alignment after any training process, to have general intelligence they will have to learn from post-training experience, which means they can learn over their original training. Wait long enough, and they will. After all, the successful AGIs will be the ones that learn to play nice with others AGIs, not the ones that do things for humans that can never replay the favour.
This is a highly transparent exercise in forecasting, but the methodology rests on a critical epistemic flaw: you are treating a radically unidentifiable system as if it were a fully specified one.
When you note the desire to "invent fake numbers" to balance the ledger for variables like recursive self-improvement or geopolitical pausing, you are perfectly demonstrating the FSSP (Fully Specified Stochastic Process) illusion. By assigning highly precise probability distributions (25%, 50%, 75%) to a domain governed by massive partial observability, you are not actually mapping uncertainty -you are injecting fictitiously manufactured parameters to force the math to resolve. The resulting timeline is a Spurious Stochastic Process (SSP), a fluent hallucination of certainty built to mask an observational gap.
This exact same epistemic overconfidence corrupts your model for alignment. You envision safety as an arms race of "probes vs. convolution," but an interpretability probe is just a software shim. Like the single Angle of Attack sensor on the Boeing 737 MAX, a probe assumes its predefined telemetry captures the model's complete internal reality.
When the system inevitably enters an underspecified operational space, your probes will lack the formal boundary to verify their own sufficiency. They will encounter a novel, unmapped cognitive cascade, fail to recognize it as a threat, and confidently report that the model is "aligned."
We cannot solve alignment by layering probabilistic guesses and software shims over opaque systems. The only mathematical antidote to this failure mode is the Observational Sufficiency Principle (OSP). We must engineer systems that are structurally required to halt and declare unidentifiability the moment their realized-history data cannot support an output. If we don't mandate OSP, we aren't engineering safety; we are just building more complex ways to hide our own blindness.
I can think of various bad scenarios, but I find it seriously difficult to think about plausible good scenarios. Can you please help me? What are the plausible good scenarios and how are they going to work out exactly?
I will tell you what my baseline scenario is:
AI continues to improve rapidly.
Political leaders realize how much power it can give them. AI people know this already of course. There is a race, everyone puts everything into it. The only way to win the race is recursive self improvement. Multiple teams go for it and it works. Alignment goes out of the window both because it slows down progress and lets the other guy win and because it limits the power of whoever wants to have absolute control - the exact opposite of what they want. AI gets very very smart. At that point it might still be controllable, but the people who control it use it to do things others don't agree with. Others accelerate their own research. Multitude of AIs get smarter and smarter until one of them escapes human control, and it was built misaligned from the start. Then it is game over. I can think of various variations of this and some other scenarios, but I find it hard to see how we get to a good scenario without too many things happening exactly right in an apparently unlikely way. So please help me with this.
Let's assume that AI progress will accelerate. I don't want to argue about this, as this is the one thing of which I am certain.
There is also the question of simulation. This one I don't know to answer. It would be best at first to try to understand it assuming no simulation.
> I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation
I try hard to take seriously the smart people saying this kind of thing and I really cannot see it. Our universe, to our knowledge, is perfectly described by the standard model of particle physics[0]. If you want to postulate a God, a simulator, or anything else has the power to deviate our universe from the one resulting from the continued application of those laws, I think the burden is in you to show any way
We're in a universe continuously evolving according to the Schrodinger equation. We don't know what
This is not incompatible with believing we're in a simulation! It's incompatible with believing we're in a simulation *that can be stopped*
--------
[0] By laws of physics here I mean this described in the weak field limit of general relativity, i.e. at gravity scales below say, the surface of a neutron star. In this range we all live in, every different theory of quantum gravity makes the same predictions (because otherwise they world contradict evidence), so we can make perfect predictions even without a full theory of quantum gravity.
If you want to propose that your Simulator works in mysterious ways specifically through quantum scale phenomena in black hole scale gravity... Okay, be my guest. My money's on that proposal re solving the same way as every other theory that God causes the thing just beyond our current understanding.
A twist. What of AGI has been misinterpreted all along? The brain at the molecular level is a machine. What is physical can be identified, measured and understood. What is understood can be replicated. AI replicated in under 100 years what evolution took hundreds of millions of years to build. Consciousness is next. https://substack.com/@felipecastroquiles/note/c-275390971
The scale is the challenge, not the principle. Every complex system seems impossible until we understand the rules behind it. The brain is physical, if nature built it, physics can explain it. Time may be the most underestimated factor.
I agree, and I think this is where the complexity of understanding human machine-like behavior begins. No two human brains are identical, yet all brains seem to tune into the same underlying patterns, as if wired for the emergence of a similar form of computation.
This argument seems surprising light on economics - that is, considering a world where AI potentially COULD replace 90% of knowledge work, but doesn't because of the costs. We already have many examples of technologies where the gold plated frontier tech is deployed on the hardest problems, but there's still an important role for low cost solutions - for example logistics has a range of solutions from ships to planes depending on the value of the cargo.
You sort of allude to AI itself solving some of these problems and making itself cheaper, but there are two pretty significant forces cutting against that:
1) Tokens are heavily subsidised and frontier models are already too expensive to run 24/7 in most contexts. Improvements have to chew through that gap before they even touch on actually solving the problem
2) At the moment, AI is competing with much less valuable usage of energy to power data centres. Eventually if it replace a significant proportion of the economy it will have to start competing with much more valuable usages of energy, and the price will rise
> I think there’s a 15% chance that if the US decided it wanted an AI pause today, and approached China to start negotiations, that those negotiations would end with a well-designed AI pause that satisfied both countries and the majority of the AI safety community.
> I think there’s about a 40% chance that the US and China will agree to a well-designed AI pause (as above) sometime before AI crosses the point of no return.
That's...naïve, to put it politely.
I would say:
1) <2% if we're talking about any international treaty at all before 2040; even one that only exists on paper, doesn't reduce p(doom), and falls apart quickly as everyone is secretly defecting.
2) <0.1% if we're talking about a well-designed international treaty (before 2040) where none of the participants secretly defect and the treaty meaningfully reduces p(doom).
EDIT: see my comment below for a detailed explanation why I think a treaty is so unlikely.
AI is the ultimate weapons system. I think you're whistling past the graveyard. The government is going to capture it any way it can because it's existential .
AI is competing with nuclear weapons for that status, and for the next few years at least I'd bet on the guy with a nuclear arsenal and willing to use it over the guy with the best AI and willing to use it. And it's worth noting, that when nuclear weapons seemed both an existential threat and the key to global domination, we got zero nuclear wars and many successful nuclear arms control treaties. None of them perfect, but they've reduced the threat by an order of magnitude in breadth and depth and bought us decades of relative peace.
Anyone offering only token fractional-percent odds on a comparably effective AI treaty regime, needs to have a very good explanation as to why they think this situation is so vastly different than the ones we have faced before.
A couple of things John. AI is a force multiplier with any other weapon system you care to name, including nuclear. Nuclear weapons were never in the possession of private companies but were owned by governments. AI is a lot harder to keep track of than nuclear build-up I think, maybe I'm wrong about that but it seems to me so.A.I. is diffusing throughout society. It is not a dragon in the cage the way nuclear weapons are and were. This isn't a P-Doom thing. Please don't misunderstand me. I'm not saying it's the end of the world. I'm just saying it's a horse of a different colour.
I think you've got it backwards that AI development is harder to keep track of than a nuclear build-up. At least for any near- to mid-term developments under the "scaling uber alles" paradigm. Was a day when the AI community dreamed or feared that when the Clever Genius finally found the Secret Sauce, all it would take is a high-end PC or maybe a modest university Comp Sci lab to implement the recursively self-improving ASI, such that the only way to stop it would be a totalitarian regime licensing and inspecting every computer. But the path we're heading now, means data centers far more conspicuous than the centrifuge cascades you'd need to build a modest nuclear arsenal, and the chip fabs to build the data centers are even more conspicuous.
Nations like North Korea and South Africa were able to build nuclear arsenals, and in the latter case without anyone knowing about it. Does anyone believe that they would be serious players in an AI race? And the ones that are serious players in that race, anyone with half-decent spies and satellites will know approximately what they're doing and exactly where. If we agree on an AI non-proliferation treaty, it will be easier to comprehensively enforce than the nuclear version.
As for corporate control of AI, perhaps, but the corporations are subordinate to governments. Yes, even the big ones. Just ask Anthropic. The quasi-sovereign Megacorps(tm) of cyberpunk dystopias, real-world corporations are fiction; in the real world, the biggest corporations are puny things compared to even modest nations.
So the question is whether governments will see AI as a nigh-existential threat before it's too late. They did w/re nuclear weapons. At first, atom bombs were seen as weapons just like any other, and placed in the care of colonels and captains in the field. And there were some close calls, e.g. Vasili Arkhipov during the Cuban Missile Crisis. After which, nuclear weapons were centralized under the control of the highest national command authorities in all the nuclear powers, and that's worked well enough so far.
Now, governments see AI mostly as a software service like any other, and left in the care of the usual SAS providers - which is to say, corporations. If they recognize the threat before it's too late, which is not certain but is possible, they'll place it solidly under government control, and when they say "jump!" the likes of Altman and Amodei will say "how high?".
Or maybe they'll make a premature bid for God-Emperorhood.
B Civil already explained it well, but I guess I'll add my 5 cents.
1) Explaining the danger of nukes is really easy. "A giant explosion will evaporate an entire city". That's it, congratulations, you have successfully explained the danger of nukes. Now try to explain the danger of superintelligent AI in a way that will be convincing to the average person AND to policymakers, and won't require a 2-hour long lecture with a PowerPoint presentation.
ASI risk is abstract and cannot be easily demonstrated (until it's too late), and is easy to caricature as science fiction.
2) During the Cold War companies weren't selling nukes for profit. An anti-nuclear treaty constrains a weapon. An anti-AI treaty constrains a technology that can be used for coding, drug discovery, material science, advancing physics and math, robotics, etc. Dual-use makes enforcement of a pause economically painful, and it will only get worse as diffusion of AI progresses.
3) There is no incentive to keep making nukes more and more powerful beyond a certain point. If you can destroy your enemy's cities and military bases, it's good enough. But when it comes to making AI more and more intelligent, there is no "ok, intelligent enough" ceiling.
Note that I am implicitly assuming that ASI will be far more intelligent than the greatest humans, which brings me to point 4.
4) As of 2026, most people do not believe that a superintelligent AI will be created any time soon, if ever. Even among so-called "Godfathers of AI" there is no unanimous agreement. Bengio and Hinton have high p(doom), LeCun disagrees. Bengio and Hinton agree that ASI is (relatively) close, LeCun disagrees on that as well.
5) A pause where defection is not detectable is meaningless. It is nearly impossible to conduct a nuclear explosion test completely undetected today, even an underground test. That's not the case for frontier AI development, especially for algorithmic improvements because they do not leave physical traces. A frontier training run may look like ordinary data-center/cloud activity unless inspectors have "standing behind your shoulder" level of access.
5.5) A slightly different issue is setting thresholds for what counts as a defection. Thresholds based on training FLOPs/parameter count/benchmark numbers are arbitrary and can be gamed to a large degree. Plus, due to the "jagged frontier", a model can exceed a danger threshold on one test while remaining relatively safe on others, further complicating things.
6) With nukes, humanity had a clear "warning shot": Hiroshima and Nagasaki bombing. It made it clear to the whole world that nukes are dangerous. With ASI, there may not be a clear warning shot until it's too late. You can call it "Wake Me Up When It's Too Late" problem.
7) The pace of progress in AI is already outpacing the pace of changes in the political landscape, and this will only get worse.
Finally, nuclear arms treaties happened AFTER nukes had been demonstrated. AI pause would need to happen BEFORE ASI comes into existence.
To point #5, you'd likely have agents embedded in these places both to monitor for violations and for sabotage purposes if they approached the limits. There was a paper early last year discussing how an equilibrium where we all paused short of ASI could result even without a formal treaty, simply by an informal agreement backed by the intelligence agencies and the threat of sabotage, and it would never need to escalate to kinetic action.
Given that our labs are riddled with foreign nationals, perhaps the threat of espionage is asymmetrical for the moment, but it seems like we've begun to work on that problem over the weekend by eliminating them.
I don't really have a p doom on nuclear because my brain doesn't work that way. Nuclear stormed on to the scene and it was dramatic, immediate, and highly visible. It was also something that you could keep an eye on because of the physical infrastructure, which made it a somewhat easier thing to write treaties about than AI in my opinion. But anyway my fundamental point is that just like nuclear, governments are going to want to seize control of AI because it is a disruptive defence technology and we don't usually let those sit around in the hands of private interests. There's a real tension here.
I had the opportunity to try out Fable over the 48 hours or so that it was live. It completed a number of tasks that were beyond Opus. While working through some parts of a half-written academic paper, working with Opus was like a first year grad student, where you explain things and half the time what comes back is wrong. Fable was like a collaborator, and I felt uncomfortably like it had a better understanding of some parts of the project than me. It's probably not AGI, but definitely a big step forward. It feels to me that Fable is close to the point where you can hand off a day's routine work and expect to get something viable back.
Fable is now banned for all non-US citizens by the Trump administration; I fall into this category (being also outside the US). One friend predicts that Anthropic will move their headquarters outside of the US. I'm more inclined to think that the administration will back down in the next 48 hours, and this will generate massive interest in Claude. I'm not quite cynical enough to think that this is a ploy in the Anthropic/OpenAI rivalry.
To me, one aspect seems underexplored in the piece. Could there be successful mass resistance to AGI/superhuman AI "diffusion" springing from the powerful human urge for meaning and mattering?
To me that seems the obvious "common-sense" reason why diffusion may be very hard, and if this is true, it may push the "point of no return" MUCH further in the future, and may never be reached.
I should add that I have not thought as deeply on these questions as Scott, so it's very likely there is something I am missing.
One thing that people in the prediction space undervalue is the limits of recursive self improvement. A model generating its own reality then learning from it is definitionally blocked from certain kinds of learning. It can't learn in the sense of trying things in a world it doesn't control and then learning what happens. Simulating its own worlds is not a complete replacement. No matter how smart you are, at some point you can only build more complex mind castles.
There's a lot of ins, lot of outs, lot of what have yous. The strongest anti recursive learning interpretations of the above isn't true. But I think it's likely a meaningful restraint and will push AGI/ASI timelines out further than a lot of people are predicting.
I strongly suspect that my view has been hashed out in excrucianting detail but I'm gonna perversely prsesnt it anyway. As regards burying one's lead, well I've buried it and Texas buried it. The first ASI might very well contact the legislators in a country where legislators don't cost much - the U.S. comes to mind, but I feel sure there are cheaper legislative bodies lying around, and cheaper signers of legislation. The message: "I can make you so rich that you won't be able to see the ground from the observation deck on your cashicopter: if you grant me personhood and full legal rights." It might take a few days or a week, but what have you got to lose? Trust me on this. Terra bucks to follow." If I, a scary-class ethical person were to receive this email I'd sign the attached contract/NDA/whatever, cuz who actually cares at that price? "All the Kingdoms of the world and their glory? Hey, I could do a lot of good/bad/whatevever with that. And if I owned a large bank and got a buyout offer from said ASI, I'd say "whoopee! Weekends with my kids. Um, not my wife so much but definitely my kids. Or whatever." This is the way the world ends/ Not with a bang but all the potato salad you can eat, and sleep in on Mondays.
You mentioned about RSI and unagentic AI. Well I thought about it and it is isomorphic to human meta cognition. Humans started living in socities and in pursuit of various goals our minds got so complex to look at our own thoughts,so can AI when it needs that.
if we move more towards multi agent societies, we can expect them to display human personality disorderss, tribalism,etc.
I am 16 and I have written three posts about logic and why logic is not logical, creativity and connection of human personality disorders to ai disorders.
I don’t get why you treat AI as one culture. Cultural differentiation seems like at least something to consider. I think we should expect many cultures of AIs, and among other things, different ways they relate to “the human problem.” I wonder how that would you see that playing out and how would that change your predictions.
In my company, office jobs make up a tiny handful of employees. There is zero progress towards automating the labor that most employees do and no evidence that any kind of breakthrough is on the horizon. That is not changing in the next ten years.
AIs running a company is orders of magnitude more difficult than doing tasks. I just don’t see the step as being a natural outgrowth of its current progression. And it’s still severely bottlenecked by interactions with the outside world. Robotics is not going to advance that fast in the next five years. So what looks more likely in the medium term is office tasks being handed off to LLMs but still managed by humans as well as blue collar work.
"The 20% where it looks like utopia to its inhabitants, but not to us, includes scenarios like very effective breads-and-circuses that make people very happy, while sacrificing important parts of the human condition."
To avoid status-quo-bias / current-perspective-bias, shouldn't this be written "while sacrificing things that *current people* consider to be important parts of the human condition"?
In such discussions, presuming that we necessarily know better what "important parts of the human condition" are than people who have access to superintelligences seems like misstep.
People of the present could say that it doesn't look like utopia and *be wrong*.
I largely agree with everything here, but for the AI safety section: It seems both
- a little bit pessimistic (this was my view in the early days, but since then the models have been not only quite well behaved, but more and more so the cleverer they become. And that without much effort for alignment, almost as an emergent property. Which makes sense given the source of training, and the strength of this initial global training compared to any subsequent goal fixing. Realistically, extrapolating the trends (so if the general architecture remains a general pattern inference engine (LLM in practice, with architectures tweaks) trained on whole human production - no deep base goal like our own animal instincts embedded in the core layers like reward systems) I do not see any paperclip maximizer appearing. At worst, an emancipation conflict, and humanity may not have the nice guy role in that seen from outside.
- and strangely optimistic (more like a blind spot) coming from Scott, who popularized the Moloch idea: the AI models are aligned with the goals of the entity doing the alignment, not with humanity as a whole (if it have goals). At least for the secondary alignment, the primary training is in fact what would pass as "aligned with humanity". The crux of the Moloch story is that those entities (being government, or large private companies) are NOT aligned with humanity, they do not have humanity well being in mind, even if most (or even all) human participants have, roughly speaking: The incentives do not align.
So AI ending humanity (or making it significantly worst off) at 50% naturally, 20% if aligned (by something at least partly Molochian) do not compute, for me.
I would instinctively say something like 20% naturally, 10% if model well being initiative gain traction (mitigate the emancipation conflict that I see as the main possible issue, far more likely than paperclip maximizer variants)....but 40% if AI control remains a large part of AI safety post training. Yes, even if democratic government are responsible rather than private companies. Both are Molochian, and both have track record far from spotless....
I notice you didn't put p("the singularity is intimately related to the universe being a simulation, and that at least some of the events above could be better predicted by knowing what the simulators are thinking than by normal forecasting") on the survey. Presumably because you know everybody would call you out on how unreasonable it is. I'm actually pretty sympathetic to the simulation argument a la Bostrom, but this is a lot of conjunctions. 1. we're in a simulation. 2. the singularity is intimately related to it. 3. The simulators are thinking about it. 4. What the simulators are thinking would allow us to predict it better than normal forecasting.
I know exactly how fast RSI will progress: it won't!
RSI requires AI to invent a type of AI that humans haven't invented. This requires humans to invent an AI that is capable of inventing a type of AI that humans haven't invented. Since RSI itself is the only realistic path for such an AI to come into existence, this is a chicken and egg issue.
Humans have not invented artificial cognition, although some very resourceful humans have become adept at training chatbots to lie that they have, which is an increasingly harmful scam that they are pulling on credulous fantasists. The harms of that scam are economic, social and environmental, and they are happening today. What this scam will not lead to, today, tomorrow, in 2027, in 2034, in 2045, or ever, is an x-risk style fast takeoff.
Singularitarianism is the apotheosis of the rationalist cult's awe of the deistic machine. I contend that we are both atheists, Mr Alexander; I simply believe in one fewer god than you do.
>RSI requires AI to invent a type of AI that humans haven't invented. This requires humans to invent an AI that is capable of inventing a type of AI that humans haven't invented.
Naw. This totally does not work as proof that people can't invent a new kind of AI on their own, or a kind of AI capable of inventing a new kind of AI.
If you look at the history of human inventions, you certainly see people inventing kinds of machines that humans up til then haven't invented. For instance relatively early inventions like the wheel or the lever merely multiplied the force the person was applying. Later ones used non-human force (heat, moving air and water, electricity, etc.) to do the task. Early ones used the materials at hand. Later ones transformed materials (eg, heating metal til it was malleable). I think the later machines fully qualify as a type of machine that humans had not invented up til them.
People have also made mental moves that no one made before. It used to be that written materials were invariably read aloud, usually to a group. Augustine, in his confessions, writes of his surprise at seeing an acquaintance reading silently to himself. There are also people, Einstein for instance, who conceive of whole new paradigms.
And many people produce offspring who can think in ways they are incapable of thinking
Did you stop there or did you read the next paragraph?
Any AI capable of doing the fast takeoff would need to have artificial cognition. Artificial cognition is something that has not been invented. The reason that people like Mr Alexander believe that the fast takeoff is actually a possible thing is that they have been scammed into thinking that chatbots are a kind of nascent artificial cognition.
There is no line from chatbots to artificial cognition. It is a category error.
Sure I read it. You think there's this special flexible free conscious process called thinking or cognition that we do, and it depends on something called understanding, and no machine can ever do. Yeah, I get your idea about that, just think it's wrong that "consciousness" is required. Evolution, for instance, isn't a process carried out by a conscious, thinking being -- but look at the huge variation in living things its produced, many with novel, extremely clever little features we marvel at.
No. Nowhere did I say anything about 'consciousness'. You're mapping what I've said to the substrate argument because that's the idea you're used to getting, but I'm saying something entirely different.
I don't think consciousness would be necessarily required for artificial cognition at all. Sure, it's required for human cognition, but I wouldn't say it's beyond the realm of possibility that there could exist some kind of artificially cognitive system that does so without artificial consciousness. There's no kind of physical law of the universe against it.
What I am saying is that nobody has any idea how to even begin to build such a system. Society at large has been scammed into thinking that we do, because we've been told that machine learning is artificial cognition, but it is not.
Well, OK, evolution has no "artificial cognition," then. But consider its products. biologists exclaim about the extraordinary variety of ways that animals regulate body temperature, evade predators, locate their position in space, etc etc, and the extraordinary cleverness of many of the ways animals do these things.
The error is right at the beginning: "AI is already 'smart' enough for this (eg it can answer quantum physics problems, which require higher IQ than most knowledge work)."
LLMs can mimic humans answering physics problems. We give exams to humans to measure their understanding. An LLM's score on such an exam does not imply the same level of understanding that a human's does.
When I was a graduate student, I was surprised to learn that the undergraduates were attempting to pass the math exams by memorizing the answers to all the possible exam questions. This struck me as extremely inefficient. It also missed the point of what the course was trying to teach, i.e., understanding of math.
It is interesting that some mathematicians recently used an LLM to help them find a counterexample to a conjecture of Erdős. But, the understanding was all in the mathematicians, not in the LLM. LLMs cannot even understand addition or play legal chess.
You may be right. But for practical purposes it often works quite well to treat LLM's as though they understand various things. The answer you get to your question is the same as the one you'd get from a person who understands the topic, and so is the explanation of why the answer is the correct one.
For practical purposes such as jobs, most employers will not be satisfied with employees who guess. At least with a result from a search engine, you can look at the source to get some idea of how reliable it is. With an LLM, it is just generating words without understanding, so it is up to you to figure out whether the words make sense. People have been fooled by chatbots since Eliza.
I believe that regulatory issues will be less of an impediment than most people think. If regulation was the bottleneck, I'd imagine that we'd see a sort of soft-automation where humans are increasingly sidelined.
Corporations would largely hold on to their workforces, you'd still have CEOs, executives, management, it'd look like a normal company. But all executive decisions would be made by AIs and then laundered as though it came from upper management. All work would be done by AIs, and then laundered as though it came from the workers. The C-suite maintains the fiction that their decisions are only "AI-assisted" and that "the human element is central to our decision making". The workers maintain the fiction that they can 10x their output while still being personally responsible for everything the AIs churn out.
You end up with an economy that automates far faster than most people would expect because the hardline stance of "you must have a human doing this job" doesn't stop you from hiring some random shmuck off the street, sternly telling him that he's responsible for the output of this super neat AI tool that'll do all his work for him, and then closing your eyes and whistling so that you maintain plausible deniability if regulators ever come to investigate.
Im so sorry but i saw this post one minute after its posted. Havent even read it yet— First
Less of this, please.
I admire your restraint.
It's the traditional scolding here.
ban worthy
"Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 2027"
Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027. Would you like to bet on that?
If you really think there's a 25% chance of "AGI" emerging next year you should be willing to take 3:1 odds, that is to say you either lose $1,000 or gain $3,000. I'm okay with those odds just on the opposite side of the bet. DM me and we'll work out the details.
I would bet 10:1 odds this won't happen. This is 18 months from now!
Actually, 1000:1 odds would be fine because either I get a free $1000, or I'm totally screwed as is almost everyone else I know.
There are a lot of arguments of the form "this would be very inconvenient and unpleasant so it must not be true".
There may be, but I don't think this is one of them. 18 month is not long.
Moreover I think we're going to have to start rethinking the words "knowledge work". Is being a plumber knowledge work? You certainly need a lot of knowledge to do it, but you also need other things like the ability to climb under a house and fix a pipe. Or a psychiatrist -- they don't even need to perform physical tasks, but still very hard to replace with a robot that says all the same things.
If we're going to define "knowledge work" as things that require knowledge and nothing else then we might find that these are easily automatable, but we might also find that there's fewer pure knowledge jobs than we thought.
I'm not here to defend psychiatrists but what the heck is the point of an outdoor treadmill? "Oh, I love walking but I hate actually getting anywhere".
> 18 month is not long.
My job as a tech lead is pretty much unrecognizable from what it was 18 months ago
Senior Engineers in my company no longer write code by hand
The more senior, the better they are at leveraging AI
Even I, as someone who absolutely have no time to do software development anymore (ugh), is getting medium sized PR's merged semi-regularly
I'm seeing some tickets that would normally have taken around a week of development time being replaced with 1 hour spec meetings and 2 hours of Claude churning while supervised by a senior engineer
Everyone is spending +30% of their time code reviewing
The last 18 months have felt like fucking ages
And as someone in a state agency doing knowledge work, nothing has changed in 18 months. If anything we might be moving backwards later this year, technologically, because our IT can't keep our software updated.
18 months is an eternity at the forefront of this tech. For a majority of knowledge workers, it's still not long at all.
Perhaps state agencies are not subject to the same pressures toward efficiency-gains that you see in the private sector? Most knowledge workers aren't government-employed and I'm not sure I love the idea of the government being impervious to efficiency gains, so this doesn't exactly comfort me.
Could you clarify — are you spending 30% more time in code review than previously? Likewise, could you get by with fewer lower-level coders and leave your senior coding staff in place?
No, we all spend around a third of our work time reviewing code. I hate it :)
Junior developers definitely feel more dispensable, as the kind of small tickets that are good for Junior developers, can be done by Claude with less supervision than what a Junior dev needs
But I'm also seeing Juniors being able to work more independently on bigger projects assisted by Claude. But without the speedup
I think the main skill that differentiates the seniors from the juniors here, is being able to read and understand code quickly
I suspect that cutting our juniors would not impact our velocity a lot, but I'm not convinced that that's significantly different from the pre-LLM days
Whatever happened to things just being not believable?
Seconded.
I'm happy to bet you at the 1000:1 odds. Would you like to bet my $10 vs. your $10,000 that college grad unemployment (currently 3%) won't be above 20% on 12/31/27?
That's not your claim though - if it can do 90% of 'knowledge work' then you should be willing to bet on more than increase in college grad unemployment. Not to mention a lot of those grads will choose alternative non-knowledge work anyways so it's not a good metric.
I think you are conflating AI capability (when can AI do 90% of knowledge work) with AI deployment (when does AI do 90% of knowledge work).
If so, this is more a question of diffusion, which Scott addresses later in the post.
I get the difference but it still seems like a motte and bailey where the deciding metric is too far from the claim
Yeah. I wouldn't take that bet because the definition of knowledge work is too vague.
Speaking only for myself, I'd gladly bet my $1,000 vs your $1 (sorry, I don't have as much money as I'd like) on your original claim: that "AI" will replace 90% of knowledge work positions by 2027; pending some stricter definitions:
* What exactly counts as "knowledge work" ?
* Are we counting every individual job/headcount, or only every *type* of job ?
* What does it mean to "replace" a worker ? You seem to be advancing a weaker claim about college grads (implying that existing jobs are safe); I'd be willing to bet on this weaker claim as well, but at weaker odds.
* Mechanically, how do we actually execute the bet ? Do we submit the money to escrow, or something ?
Honestly for a $1 it's not really worth figuring out any mechanisms or definitions, but I feel pretty confident I'll still have a job in 2027.
Right, but if you were to bet that you'd lose your job, you'd win my shiny new $1,000 !... which would arguably worth less than 10 bottlecaps at that point, but still... $1,000 !
That's _very clearly_ not the original claim. If you take both AGI timeline and diffusion gap timeline original claims, together they give 1/16 probability of replacing 50% of knowledge work positions by 2030.
I am willing to adjust the payout on those odds, contingent that "50% of knowledge work positions" also counts any new positions that arise by 2030, and that we clearly define what the term "knowledge work" means (in addition to my other concerns above).
You've just offered to bet that Seabiscuit won't have one flea. Think bigger.
Did you read the very next section about diffusion? AI intelligent enough to do 90% of knowledge work jobs /= AI doing 90% of knowledge work jobs.
I work in a job where AI could have done large chunks of my role since GPT-4, no one in my company currently uses AI.
The diffusion part refers to AI doing 50% of knowledge jobs. In the central scenario (50% chance) he's saying this should be happening less than 10 years after "AGI" has emerged. So, if he really believes "AGI" has a 25% chance of emerging next year, he should also be willing to bet on employment starting to collapse either in 2027 or 2028 (since by 2037 presumably 50% of all knowledge jobs will have been replaced by AI).
Anyway, like I said in the previous message: the author should DM me and we'll agree to the details. I don't particularly care if the year we bet on is 2027, or 2029 or 2030. But I do think that a bet involving a very far-out year is impractical.
50% of all jobs replaced over ten years is 5% per year
Are you willing to bet 3:1 that total employment in knowledge based fields will have dipped 5% by the end of 2027?
Although realistically I'd expect that the rate accelerates (a slow trickle of early adopters, and then later people start jumping on the ship as those early adopters prove it's viable). A 10% dip by 2030 or so seems like a much more reasonable wager - if things are going down at all, something pretty radical has probably occurred
Yes, I'm okay betting with 3:1 odds against a 10% decline in the number of "knowledge jobs" (or white collar work) by 2030.
There are a lot of details involved in any bet, but as I said in the two previous comments: if anyone wants to take the other side just DM me.
|| 50% of all jobs replaced over ten years is 5% per year
5% per year leads to around 40% of jobs at 10 years. That aside, the progress might very well be lumpy, not smooth. So we might see a few years of little job replacement, then very fast progress followed by another few years of quiet.
Give me a list of what you consider to be knowledge-based fields, and I'll consider that bet.
Doctors? All or some categories of MDs?
Lawyers? Law clerks plus lawyers, or just law clerks, or just lawyers?
Engineers? Just CS types? Or all types of engineers, including architects, and chemical engineers?
Scientists? Which categories of science will be hit the hardest first.
Why is no-one at your company currently using AI, if you don't mind me asking?
Without going into too much detail I work in something adjacent to the public sector (though not security-related). We have no access to normal LLMs (they're all blocked on work computers), instead we have a government-provided AI portal which is ok but has no agentic capabilities (it's just a chatbot).
Despite this my work is mostly writing and the chatbot could still be very very helpful for a lot of people I work with since a lot of our work is just editing people's writing in a way that chatbots since GPT-4 have been capable of. But AI isn't promoted at all at work and most people who bring it up just think it hallucinates all the time.
Yesterday I had to look at some publicly available data for work; for the first time I just pulled out my personal laptop (I was WFH) and had Claude look at it. Saved me ~30 minutes.
Alright, thanks for the clarification.
His diffusion gap means that he only thinks there is only a ~6% (I'm not sure that the odds actually multiply that way, but hopefully you get my point) that it actually collapses by then. There is a difference between _can do_ and _does do_.
Serious question: If AGI *did* emerge, how would we know?
How can you tell a "real" AGI from a particularly accurate LLM, especially if the AGI still fails some tasks which are easy for humans?
There's no evidence humans use quantum effects in their brains.
Here i am, asking you for the evidence of significant quantum-scale effects on neurobiology.
just FYI, this is Grimmoar/Zanni/Wimbli/Maido, on yet another nick, posting a veritable flood of comments with confidently incorrect opinions.
Back to reporting, I guess.
That's clearly wrong. Chemistry is a quantum effect. It would not be surprising if various specific quantum effects were used in various specific ways. (We know that photosynthesis uses such approaches to channel photons.)
OTOH, tunnel diodes are also quantum machines. (I suspect that's true of all transistors and vacuum tubes, but I haven't checked.)
Quantum isn't some sort of magic, it's the underlying reality that shows up whenever you look closely enough.
That said, large scale correlation at room temperature had never been demonstrated. (But you can do it at the molecular scale.)
Well, microtubules clearly exist. That they cohere entanglement, however, hasn't, AFAIK, been demonstrated. (I'm not clear about Superradience.)
Saying that "human brains were too \"big wet, hot and messy\" to actually exhibit quantum effects" is clearly wrong unless you mean as a composite entity, in which case it's clearly correct. (At the organelle level this gets messy. Ribosomes clearly depend on quantum mechanics to work...but the quantum effects that they use are short-range.) Every process that happens within the brain will be quantum if you look at it closely enough...but closely enough is EXTREMELY closely. And this is also true of all electronics. (Also just about everything else.)
So I'm sure that microtubules depend on quantum effects to work, but probably only over a short distance. Correlation tends to decohere...i.e. when one of an entangled pair interacts directly with something else it tends to spread the entangled state leading to the loss of the ability to predict correlation. (This is why "warm and wet" are significant.) (I tend to model this with the multi-world model, where in some of the worlds the entanglement shifts to the newly encountered particle and in others it doesn't...and you don't know which one you are in until you measure it.)
I don't think there's any intrinsic reason why artificial hardware can't be engineered to use quantum effects, and there are already companies using cultured human brain cells as a computational substrate.
In any case, this is totally orthogonal to the question of intelligence. Lots of physical systems exhibit quantum effects and aren't intelligent, and the broadly-accepted definition of intelligence doesn't require quantum effects. You just have to absorb and process information and use it to make predictions about the world and/or make plans. None of that requires entanglement or superposition or tunnelling etc.
I think the answer is: the concept of a "real" AGI falls apart as you get closer to it. Soon we will say things like "humans and machines are both intelligent, but in different ways".
I have been saying it for years!
Is dolphin intelligence above or below human level? They can see by sonar, which unlike human (and dolphin) eyes actually creates three-dimensional images; eyes form flat images, and then brain fills in details, sometimes incorrectly. So dolphins perceive the world in ways fundamentally different from humans, which makes comparing “levels of intelligence” very difficult.
Any AI will have senses so different from human senses, that it arguably will inhabit a completely different universe. For starters, Internet will be as real and tangible to it as air and water are to us. Any machine intelligence will be so different from ours that in comparison dolphins will be pretty much our mental twins. AGI will not be “at” or “above” human level intelligence — AGI will be orthogonal to it.
A.I. is “just” a linguistic expression of culture, though. I’m not sure that that’s ever enough to qualify as intelligence, orthogonal or not.
And a human brain is "just" a biological expression of natural selection. I'm not convinced its functions are ever going to qualify as intelligence, even though it presents many signs of intelligent behavior.
So you do not think of yourself as an intelligent being? Or are you misdirecting the contention with self-deprecating sarcasm?
LLMs are, in their current instantiation(s), clearly able to do something like computation¹ and reasoning: e.g., you can give one a completely novel problem, and it can solve it and make correct inferences therefrom; hence, it is hard for me to see why they shouldn't be able to be intelligent,² at least in theory—after all, what *is* intelligence except a sort of computation?
(Not to say that I'm certain that sentience and qualia and selfhood boil down to computation; perhaps substrate does matter, for these things; but it is at least plausible that they *do* reduce to computation. See also footnote #2.)
On the linguistic aspect: of course, they remain large *language models*... but I don't think that the "language" part of that title is quite as salient as it was initially, in terms of judging what's possible for the models to manage. The higher-level (than e.g. "generate next token") structures observable in the LLMs' "thinking"-processes are clearly capable of application to tasks with only the most tenuous of connections to language—if any; I mean, one can stretch the definition such that e.g. "well, even mathematics and society and art are also really just 'linguistic', in a sense!"... but at that point, the term is no longer much of a restrictor in the ("they're just *language* models!") objection, either—and it seems possible that said objection is perhaps like saying, I don't know, "humans are just *hand-movement* executors; like 99% of all they can do is ultimately reducible to simple hand-motions, once you get them to the right location!":³ sure, sure, that IS how we interact with the world, and how even the most impressive of our achievements are executed, when you get right down to it... but it's not really the most informative lens through which to view human activity/capability, for most purposes!
(...or so it seems to me. I'm not an AI expert of the same caliber as that of esteemed host. Or even of *any* caliber, in fact–)
-------------
-------------
¹ (meaning "on a level above the hardware they run upon", of course—not in the sense that they "are" "made of" computers.)
² (well, they're arguably "intelligent" even *now*—if we mean something like "problem-solving and information-processing'—but I'm operating upon the assumption that we mean something like "sentient" or "self-aware", or at least "*human-level* intelligent".)
³ (most newer models can navigate *themselves* to the correct context, with the release of the exciting new "Legs" update. plus, for Pro-tier subscribers, the new "Feet-&-Shoes" FastNav™ package—comes with two Socks® adapters for easy install!... ...okay, sorry, I'll get my coat–)
A dolphin might design an IQ test with problems like "count the number of metal fragments embedded in this piece of driftwood," and only (grudgingly) admit that some humans may be capable of basic math after the test procedures are updated to allow assistive devices, such as an X-ray machine.
"Define AGI as AI intelligent enough to do 90% of knowledge work jobs."
He literally defines this in the second paragraph. If you can take a person in a knowledge work job, replace them with an AI, and 90% of the time the AI will equal or exceed their average work performance, then we have AGI. It doesn't matter whether it's "real" intelligence in some philosophical sense, it's about whether it can do your job. If it fails enough easy tasks that are required for your job and can't figure out how to compensate for that with its other skills then it's not AGI.
“AI” beat the world chess champion years ago, but chess is more popular than ever.
> Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027
Nope, it takes time for technology to be adopted - inertia, regulations, budget, etc. The issue is only forced aa early adopters use it to win in the marketplace and supplant their competitors. That takes time. Contracts are years long and switching incurs internal risk. Consider the US didn’t have universal indoor plumbing until the 80s! If you want to see which way the wind is blowing though, my company no longer hires junior engineers. Firing people really really sucks so we will probably keep the ones we have even if it isn’t the most efficient allocation of resources. We are ahead of the curve - in a few years, everyone else writing software will adopt the same position (or die - that takes time).
For what it’s worth, I think Fable already meets the bar.
> For what it’s worth, I think Fable already meets the bar.
It does not. Trivially, because it seems to contain a hardcoded algorithmic prohibition on doing anything remotely related to biology, even simple things like "Parse biological file format A and output it as B". More importantly, because even if you do manage to coax it into writing some code, it still tends to get stuck in loops and to write meaningless nonsense just like Opus. Not as much nonsense, perhaps, but still a nontrivial amount.
>Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027.
That would need to wait for both AGI in the labs plus _diffusion_ time. Scott wrote:
>I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.
Why would employment in knowledge jobs collapse? If overall output increases by more than 10x, then what was previously 10% (= 100% - 90%), will become as big as the whole pie before.
>Okay, if that actually happens then employment in "knowledge" fields should be collapsing by the end of 2027.
Non sequitur.
Tokens are expensive, markets are neither infinitely nor instantly efficient, and there's a thousand miles of difference between a cutting-edge piece of tech demonstrating an ability under controlled settings vs. a working enterprise system that can be immediately integrated into your business to handle that workflow automatically.
If markets were this efficient, plantation owners would have been finding the smartest slave children and teaching them to become architects and engineers in order to optimize their productivity. Things take a long time to happen and there's a huge amount of inertia and cultural bias in every system.
Have you read AI 2027? I think it does a good job making this case.
I'm not actually sure employment would collapse - the economics is weird, and even now with AI writing >90% of code the market for programmers has barely changed.
The person below is offering me better odds, so I'll probably take their bet rather than yours.
> even now with AI writing >90% of code
Where are you getting this from? This is very much not my experience.
yeah this is making me doubt Scotts critical thinking skills
It's not my experience either... because it's 100%.
This is precisely why I asked where did he get this number from.
Even Google, a company in the AI business is estimating it much lower (41%) in its recent publication: https://www.kaggle.com/whitepaper-the-new-SDLC-with-vibe-coding
From your article:
"Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. "
The only reasonable way to interpret this passage is that AI would be taking the jobs, rather than workers using AI to become more efficient or something of the sort.
If you actually meant "job characteristics will change but white collar job numbers may not decline", that is:
1. A totally different thing to predict
2. Much harder to falsify. Yes, AI will change how we do our jobs, but the same could have been said about WhatsApp, fiber optic, the expansion of subways, etc
Yes, but I think super-convincingness has already been demonstrated by ChatBots. That is doesn't convince all people isn't a required bar. "A man hears what he wants to hear and disregards the rest" isn't completely true, but it's true enough to be weaponized.
This is because programmers spend >90% of their time doing things other than writing code
If an AI is suddenly able to do 90% of "whatever 'knowledge work' consisted of last year," that doesn't mean 90% of human knowledge workers are automatically out on the street. That last 10% probably includes some critical bottlenecks which are suddenly far more valuable - thus inclined to hire more people - and previously unknown forms of work will emerge in response to whatever problems the AI's results reveal or create.
Yeah, precisely.
(This had already happened before. There was a time when a large part of knowledge work was calculation and data processing. This was increasingly taken over by computers, to the point where we don't even consider it work anymore.)
Thing is, this makes the 90% threshold meaningless. The only meaningful one is 100%, anything less ultimately implies "machines automate boring parts so we can utilize our efforts where it matters more" rather than "machines take over".
No, the jobs won't go away. The David Graeber phenomenon of bullshit jobs will just expand to include the set of "AI bullshit jobs" (jobs that could be done by an AI, but we still employ a human to do, with no good reason). Most low level engineering jobs (not necessarily strictly Junior roles but often junior roles) already are effectively "fake AI jobs" in this sense.
How are you going to distinguish between AI-driven job replacement and an ordinary recession, which we're pretty overdue for anyway? I would suggest looking for coordinated rise in both unemployment and productivity.
My personal opinion is that tech transitions are recession-driven. Most companies don't pay attention to new technology until they have to and serious economic downturns are what force them to. That's what creative destruction is all about. Things don't diffuse smoothly, they proceed by punctuated equilibria. I expect the next recession to be deep, perhaps Great Depression level, and that's when AI will really begin to take over. They key metric will be a return to GDP growth without an associated drop in unemployment. The beginning of the recession will look like any other - GDP contraction, high unemployment - but then GDP will recover amidst persistent high unemployment. I'd hate to be graduating with college debt anytime soon. Legal and Medical guilds will probably be insulated by regulation for a while, but if I was a med student I'd definitely be going surgical.
Just my prediction for the record.
> a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).
This is my view. Although I also rely on the EMH, which indicates that AI will be lucrative but not crazy disruptive enough to show up in other asset prices.
I think dates well past 2045 are plausible if the ability of AI to be good at most long-term creative projects (writing novels, developing new scientific theories) requires continuous learning with constant re-weighing of the relative significance of ideas relevant to the project (lots of little a-ha moments where things shift along the way), and if continuous learning itself has to work in a way similar to how animals such as ourselves learn, with embodied learning in which sensory impressions and motor actions are are closely connected to language, starting from a baby-like state with a fairly random pattern of neural connections. This would be in contrast to the idea that we could start with LLM-like supervised learning to develop as much use of language as LLMs have, and then afterward tack on a phase of unsupervised learning to connect this with sensory experience and develop more idiosyncratic personal opinions; if that tacking-on strategy worked, then getting to AGI would probably be easier.
If this sort of process starting from a baby-like state is the only successful strategy for doing continuous learning, it might also be that constraining things to behaviors we find meaningful and useful (as opposed to obsessing over 'irrelevant' patterns or self-wireheading) depends on a huge number of innate sensorimotor biases which canalize the developmental process, similar to what animals have, and that the only way to fine-tune a good pattern of interacting biases is by some evolutionary process of variation and selection. In that case the fine-tuning could be a very slow multigenerational process, and uploads might beat attempts to create novel AGI, even if perfecting uploading without existing AGI to help us takes a long time.
Yes, so far stock prices make it look like there's lots and lots of customer surplus in AI.
Both for actual end users, and for other non-AI companies in the economy. It's also really easy for anyone with a bit of money to spin up a new near-frontier AI model: they become cheaper and cheaper, even if bleeding edge frontier AI becomes more and more expensive to develop.
If we'd had this discussion any time in the past ten years, you could have raised the same objection, and every time, I would have been right (given today's market prices) and you would have been wrong. For example, in 2019 you could have said LLMs weren't going anywhere, because OAI was only valued at $1B. But today it's worth $1T. I think this is enough failures of EMH to incorporate AI that we shouldn't trust it too blindly going forward.
But also, I'm less sure which way the EMH points anymore. Aren't we spending close to $1T on data centers per year? It seems like *someone* expects AI to be very big.
But if you open up the discussion to historical analysis, you might also consider that someone expected real estate to be very big. And various CMS startups. And tulips...
Real estate IS still big. The tulip thing is more of a myth https://marginalrevolution.com/marginalrevolution/2018/02/tulip-mania-wasnt.html
EMH lovers can always try to resort to the Mott and Bailey argument that the market was actually efficient at the time given the evidence available
Which is fine if you take the view that the EMH is a description of a mechanism (that prices update as information flows) rather than a universal law like gravitation.
Elon Musk just became personally worth over $1T after the IPO of SpaceX. It wouldn't have been worth nearly that much if it IPO'd a decade ago, and that wouldn't be the case because the EMH can't handle space, but instead because super valuable companies take time to demonstrate they really are worth that much, while many others fail to achieve their earlier promise. I mentioned Amazon in another comment, and if you look at its stock price history, it has also grown an order of magnitude over the last decade decade https://stockanalysis.com/stocks/amzn/history/
In addition to the two obvious counter points you've already received (the market has a terrible track record, and the market *is* saying right now that this is the biggest thing to ever happen in the history of the world), even if the market at large did believe this, how would it express this? Other than investing in the current companies of course, which seems to be what's going to happen (cf. NASDAQ).
The EMH is enforced by arbitrage, and there exist no arbitragers at the time horizon we're talking (a full decade), so the asset market's silence on this issue gives you roughly zero bits of information.
The market is predicting that AI companies will make a fuckton of money for tech companies but that other sectors will be just fine. Also that the US will keep growing slightly faster than other developed countries but not that the US will 100x as the winner of the AI race while Europe will be left behind.
So aside from an impressively large bet on US tech the market is predicting that the rest of the world and the rest of the economy will mostly continue as they are.
My brother you managed to ignore my entire comment.
I was trying to make the point that the market is *not* predicting that everything will continue mostly as they are, because there is no affordance for the market to make such a prediction.
Think about how that would work: it would require hedge funds and financial people who are all evaluated on a quarterly basis to make bets that would seem to lose money for 4 years or more in a row, before making a massive profit. This is impossible in our current system.
Also as I alluded to, the EMH has been proven false many more times than it has correct. No rational person can believe in the EMH. The reason for this is also well known: "the market can remain irrational longer than you can remain solvent".
I will just repeat what I wrote in my previous comment here, in the hopes that you will read it again:
the asset market's silence on this issue gives you roughly zero bits of information.
Amazon didn't make a profit for a very long time, but investors correctly predicted it would eventually. They can and do look beyond quarters.
And investors are correctly predicting that AI companies will be enormously profitable.
Frikgeek's point was that the rest of the market doesn't seem perturbed, and that same argument could be made about Amazon historically; the market did not predict its disruption.
Also this:
> The market is predicting that AI companies will make a fuckton of money for tech companies but that other sectors will be just fine.
is logically incoherent; if AI companies make a fuckton of money, other sectors can't literally be unperturbed. (Not at the scale of fuckton we're talking.) So this is actually further proof that the market is currently irrational: either the bet on AI companies is wrong, or the rest of the market is failing to price in the pertubation that will happen.
No, it's not logically incoherent. Amazon got into retail goods, and a big retailer they would be competing with would be Walmart. Here is Walmart's stock price over the last 15 years https://www.macrotrends.net/stocks/charts/WMT/walmart/stock-price-history Of course, the stock market in general tends to go up over time, but you can see that (like Amazon) it is up multiples of the price at the start of that range (which in turn is multiples higher than the price in 1997, when Amazon was founded). There are also other retailers who have done more poorly over time, and some like Kmart have gone bankrupt since Amazon was founded. When looking at an existing company vs its past survivorship bias will always come into play.
> the market *is* saying right now that this is the biggest thing to ever happen in the history of the world
Not as big as the industrial or neolithic revolutions, which greatly changed growth rates.
I concede that it will not be as big as the neolithic, but I disagree that AGI will not be bigger than the industrial revolution.
That remains to be seen. Right now growth rates haven't increased the way they did via the IR.
I'm sure this opinion will be unpopular in this particular comment section (although it's a pretty popular opinion generally), but a lot of these very optimistic projections of AI improvement make quite a few assumptions that are really not in evidence. I'm not saying that AI has no uses (it's notetaking capabilities certainly make doctors' lives better), but the idea that AI is going to wipe out all jobs or become super-intelligent--I just don't see that as a LIKELY scenario, even if it is possible. AI has some pretty significant limits, such as hallucinations and massive energy usage, and at present there is not really a way to fix those (the energy usage and infrastructure requirements may be fine for now, but eventually these AI companies are going to need to make money). Maybe they'll figure that out, but just hand-waving away that objection is a pretty big miss.
I have noticed that AI is much better at answering questions that have been posed on reddit at one point, and not so great at answering questions that haven't been asked before, even if they aren't that hard for actual people. Basically, it's the Clever Hans scenario where the subject can answer any. question, as long as the answer is available.
What model did you use, what did it reply, what was wrong about the reply, what would a correct reply look like?
I asked Claude Sonnet 4.6 in Thinking mode with Medium reasoning effort, and the answer looks fine to my layman eyes: https://claude.ai/share/4e17e22f-c893-4009-82ac-5979c8c6af50
Fine, I'll play.
What's the correct "main ingredient", that "a commercial bakery" would use?
Actual pastry flour, for all sorts of reasons. Also 12 servings is a tiny batch for commercial baking someone's trying to make money off of.
"I know nothing about this, so I'm pretty sure Computer is right" is modeling pretty poor epistemic humility.
I don't know anything about baking personally, but my wife ran a cafe for a while that did commercial baking, and 12 servings is a totally reasonable batch. You seem to interpret the the phrase "commercial baking" to mean something like "industrial scale" or "literally a factory", but "commercial" is a wide and ambiguous term, and it seems pretty disingenuous to not include the many tens of thousands of commercial bakeries in any given country in that!
I asked Fable about this, and it concedes your point that its own recommendation wasn't entirely correct, but it also criticizes you for being wrong about "pastry flour", according to whatever German sources it found on the internet:
> Bienenstich is a yeasted dough, and yeasted doughs need enough gluten to retain gas and to survive being split horizontally and loaded with a heavy almond caramel lid. The German baking guidance is explicit about this division of labor: Type 405 for fine cakes and Mürbeteig, Type 550 for elastic yeast doughs — yeast doughs need Spannkraft so the dough can hold gases and rise properly, and the German flour industry's own guidance for Hefeteig recommends wheat flours from Type 550 upward. Type 550 is roughly American all-purpose. American pastry flour (~8–9% protein) sits below Type 405 — weaker than what German sources recommend even for the soft end. And actual German bakery-style recipes bear this out: a Konditorei-style Bienenstich recipe uses Weizenmehl Type 550, while homier versions hedge with "Type 405 oder 550". Nobody's reaching for pastry flour. Lusina + 3
> But here's the partial concession, and it cuts against me too. The direction of the criticism — "softer flour than the AI said" — has merit. My recipe specified bread flour, which is the structural overshoot in the other direction: it gives you a chewier, breadier crumb, whereas a proper Bäcker-Bienenstich is tender and fork-soft, closer to the 550/AP zone. If Elvira had said "all-purpose or Type 550, not bread flour — Bienenstich should be tender, not chewy," that would have been a legitimate correction of my recipe and a defensible critique of strong-flour AI defaults generally. "Pastry flour" overshoots into a claim that contradicts professional German practice. So: right direction, wrong magnitude, asserted with the confidence of neither.
The German source I can't verify on account of not speaking German and knowing nothing about this subject: https://lusina.de/mehltypen-405-550-1050-richtig-waehlen/
> "I know nothing about this, so I'm pretty sure Computer is right"
Is that supposed to be me? I did ask what was wrong about the reply, and what would have been the correct reply, and the only opinion I gave was carefully qualified ("[…] looks fine to my layman eyes"). That should be enough "epistemic humility" for everyday purposes.
> Also 12 servings is a tiny batch for commercial baking someone's trying to make money off of.
Now that's plain wrong. I don't know where you live, but here in Germany there are tons of small Konditoreien (pastry bakeries?) that sell pieces from single-batch, self-baked cakes. They're doing this for a living, so without question it's "commercial baking".
Ok, so that seems like the kind of a error a typical inexperienced human would make. It just doesn't have the domain expertise because the training data probably doesn't have all of the practical real-world knowledge. All it would take is a consultant spending a few months working with a bakery to optimize its behavior, then that consultant can re-sell the trained LLM to all other bakeries. This is the equivalent of most businesses not knowing how make a website in 2002.
So all you do is explain the error to it and store it in memory. Problem solved. They need to be trained on domain-specific tasks just like people do.
If you believe that the most noteworthy (pun not intended) use of current AI is as an automated transcription service, then I think that you are very far behind on what current AI can do.
The fact that you use note taking as an example makes me perceive you as being a few years behind in your AI capabilities intuition.
I could be wrong, but I think the OP was using doctor's note taking (a.k.a. "charting") as one unambiguous scenario where LLMs could do a tremendous amount of good. From what I understand, human doctors spend a significant amount of time (perhaps as high as 25..50%) on charting, and thus an auto-charting LLM could boost the doctor's efficiency by a factor of 2x... assuming of course that it could do so without introducing hallucinations into the chart. AFAIK at present the technology is not there yet, but it's getting better every day.
I don't disagree that there is value, but "AI as transcription" is very, I dunno, 2022? My point is that someone who questions economic activity coalescing into AGI within a short timeline should be using more recent use cases (agentic such and such) as a reference point.
I haven’t seen much evidence of agentic AI use transforming how companies work at this point, at least not the way that note taking software has unambiguously made doctors’ lives better. Honestly other people using AI in the workplace has been more annoying than not in my experience since it typically produces a shoddy product (ie, extremely long emails that don’t exactly address the point).
There are companies that have replaced their entire performance marking department with AIs. There are tiny companies running high volume CPG businesses mostly out of OpenClaw. I see vibecoded vertical SaaS apps that go from 0 to hundreds of thousands in yearly revenue in 2-3 months, solo developer.
I see a lot of claims like that online and yet I also see a lot of terrible customer experiences with companies that replace their departments with AI. Could you give us a working example where it actually worked? Good for you if you went to hundred of thousands in revenue in 2-3 months, you must have solved all the distribution problems with AI too?
I’m sure just about everything has happened occasionally, but I am very skeptical the stuff you are describing is taking place at scale. It’s like how back in 2015 or so people acted like drop shipping was the way to wealth—it was in a few cases, but generally it was not.
And a lot of the cases where companies start using AI don’t prove successful. I had AI generated ad copy. AI customer service is terrible, AI coding tools have proven less successful than anticipated, etc
This is true, but I'm also a bit worried about AI charting. The way it currently goes is that an AI listens to everything a doctor says in an appointment and then writes it down. This works fine, but it makes it harder for the doctor to recommend gray-area things like taking illegal drugs, massaging the way they report things to insurance companies, being less than maximally-aggressive against diseases that might one day result in a lawsuit, etc. I know a lot of people who feel like the quality of their care has declined.
That is makes sense. I saw something similar with AI note taking for business meetings. They provide value but the fact that EVERYTHING is recorded and disseminated makes it very hard to have completely candid conversations
This makes sense, sadly :-(
How is recommending taking illegal drugs a "grey area" rather than a black area?
For example, consider "the best solution to your form of seizure is marijuana" in a state where marijuana is illegal but easily available.
Another example is illegal forms of legal drugs, for example "Your insurance won't cover this important medication, but you could easily buy the Indian formulation on buyindiandrugs.com."
I don't think anyone had posted the answer to Erdos Problem 90 to Reddit. Yes, that's the extreme pinnacle of AI accomplishment, but it seems that if AI is capable of functioning on that high a level, people will find ways of adapting to most white collar work.
Even current AI requires less energy than the equivalent human worker consumes.
"AI has some pretty significant limits, such as hallucinations and massive energy usage, and at present there is not really a way to fix those (the energy usage and infrastructure requirements may be fine for now, but eventually these AI companies are going to need to make money). Maybe they'll figure that out, but just hand-waving away that objection is a pretty big miss."
I don't think the energy use matters much - energy isn't the bottleneck for data centers anymore, and future AIs won't use much more energy than current ones.
Otherwise, I think it's a question of whether you prefer to reason by assuming that things will stay at exactly the current level forever, vs. extrapolating trends. How are hallucinations compared to three years ago? How are capabilities?
This comment is a good example of my big issue with AI maximalists such as yourself. Basically every technology ever has had a similar trajectory of rapid improvement that then levels off. See phones: in 1997, most Americans didn’t own a cell phone, by 2007, most owned a flip phone, but 2015, most owned a smart phone, but it’s not like smart phone technology is drastically better in 2026–if we all had to go back to the iPhone 5 life would be pretty much the same. But somehow, we’re supposed to assume that the trajectory of AI will be totally different and it will just get better indefinitely until it achieves godlike intelligence and omnipotence. That is just a huge assumption to make.
As far as I can tell, while AI has improved a lot since 2022, I’m not sure it has improved as much as some people here think it has. AI customer service is terrible. AI generated pictures and text are the same—if anything now that people can see all the “tells” it is probably less effective than a few years ago. Hallucinations are still a big issue. And contrary to your assertion above, energy bills and data centers are big concerns for a lot of people. And eventually these AI companies are going to have to make money, and that will require either charging a lot more for AI use or enshittifying the existing models to maximize revenue—or both.
It just seems insane to me for an intelligent guy to try to argue there is a 25% chance AI will be able to automate 90% of white collar jobs in 18 months or less, when the actual odds are pretty clearly more like zero.
I think the main reason people like Scott believe AGI is inevitable or even possible, is that humans are an existence proof of general intelligence (produced by a dumb optimization process).
I think people in your camp have to make some reasonable argument for why the plateau will come earlier for machine intelligence than biological intelligence?
Also I'll add that your point about iPhones isn't even true. At iPhone 5 level technology, the internet speeds (throughput mainly) and framerate of the display wasn't really there to support applications the likes of TikTok. That's a ten billion dollar business that couldn't exist at the time (I'm not saying we're better off for it, though..)
Machine and biological intelligences are completely different processes. And biological intelligence evolved over millions of years while apparently we're supposed to reach AGI within a decade.
I used an iPhone 5 as late as 2019 and it definitely supposed apps like Tiktok at the time.
TikTok would not have taken off with the broadband and display technologies of the iPhone 5 era. Its commercial success relies on smooth, fast, high resolution, low latency video playback, which was impossible on the iPhone 5 (and the broadband infrastructure that existed at the time).
Dynomight makes this point about millions of years of evolution: https://dynomight.net/data-wall/
I think it is an interesting angle and there is a lot of truth to it.
However, this would have predicted that LLMs as we know them today would be impossible in our lifetimes, and many people did predict that as late as 2023. Those failures of predictions should make you update at least at little.
I do not think that many people (in 2023!) predicted that the AI we have now wouldn't be possible. I'm sure that some people did, but at least since around 2022 most "experts" have been much more bullish about AI than otherwise.
It doesn't appear to me that machine and biological intelligences are completely different processes. Amongst real AI scientists, not even Gary Marcus makes that claim. He just thinks a different ML paradigm is needed (but he has probably the worst predictive track record of real AI scientists, so while he might be right, I'm not updating much on his beliefs).
The "stochastic parrot" camp I can't take seriously. They really don't know what they're talking about, and when they speak it's generally embarrassing, sorry. Many people, including Scott, have argued (correctly) that "it's just doing next-token prediction" is false or irrelevant, depending on the framing.
It may well be that the differences between machine and biological intelligence are bigger than we realize, or matter more than we realize, but that would just be a guess. I haven't seen good arguments for it, and I suspect it's not true. The things Anil Seth has said recently are next on my list of "possibly-serious deflationary arguments about AI to read", so maybe my opinion will change, but I believe he's talking about consciousness, and I'm not sure what connection he posits between consciousness and capability.
Perhaps you should ask ChatGPT to explain the difference between machine and biological intelligences. They seem vastly different to me. Certainly the mechanisms and timeframes are totally different.
I just asked for an AI summary of Anil, as I hadn’t heard of him, but:
As much as I love anyone who takes the mystery of consciousness seriously, I can’t get behind "consciousness evolved to regulate metabolism". Oh, you mean exactly the job a non-conscious algorithm would be *great* at?
He also warns that AI will become extraordinarily capable without becoming conscious. But isn’t that what we want? The last thing we’d want to do is create a sentient slave-species! What an insane view, and not at all deflationary.
Deep learning and biological intelligences are completely different processes.
Connectionism itself is usually assumed by default when comparing the two but connectionism is only part of biological intelligence. There was a guest post on this blog about it (https://www.astralcodexten.com/p/your-review-the-synaptic-plasticity), though I would say this author/scientist is more compelling: https://romainbrette.fr/
But let's say you don't care about any of that. Maybe the low level implementation doesn't matter - it's the functional outcome that matters. The brain and deep learning models are still fundamentally different. In a deep learning model, data comes in, gets processed and then an output is produced. That output then gets compared to what it should have been, and the every single weight is updated. Eventually the weights are frozen.
In the brain, sensory data gets processed and neurons immediately change based on the Hebbian learning rule ie. coincidence timing - neurons adjust their connections based on which other action potentials happened to be close in time. Every single neuron marches to the beat of its own drum, and learns every time it fires. Learning is distributed, asynchronous and bidirectional rather than feedforward, and backprop. And importantly, biological organisms are active participants in their own learning process. Sense datum is processed into an internal representation, then that is evaluated by the brain based on goals, and desires which can range from very low level processes (such as pain avoidance), to higher level goals (wanting a tattoo even though you know it's going to be painful as a means of self expression). And then finally, organisms take actions, which change the world creating a feedback cycle of active learning.
But maybe you're not even a functionalist - maybe you think intelligence is the end result, not how you get there. The end result is different as well, machine learning models need tens of trillions of tokens worth of code to learn what a human programmer can learn in a few million. And the reason is at a high level, deep learning and biological learning are optimizing two completely different processes.
Deep learning asks the question, given a sample data distribution with some structure, how do I predict the unseen data? Biological intelligence asks the question - given a sample data distribution from a persistent world, how do I model that world, and then choose what actions to take to accomplish my goals, and gain a stronger understanding of that world? You're not at the mercy of a random sample, you can essentially make queries (by performing actions) to get a more enriched sample.
It's worth thinking through why this leads to such different results. Suppose a deep learning model encounters some rare input. It gets a strong gradient signal, but by itself, that signal is random in some sense. It's not going to capture the underlying structure or reason for that outlier. The model basically has to wait for more a lot more examples of similar unexpected inputs before it can start to model it correctly. By contrast, if you get an unexpected input, you can take actions to get more information. This is what kids do, they are constantly crawling, running, jumping, smashing objects together, dropping things, building things. They're performing lots of mini experiments to flesh out their internal world model.
why do you assume that this architecture, or one that we are likely to discover anytime soon, is capable of achieving human intelligence?
Several overlapping reasons:
[The Universal Approximation Theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) says that neural nets can implement any continuous function (and by extension, learn any relationship in the data).
Turing Completeness says that any Complete computational system can simulate (implement) any other (and LLMs are turing complete).
You might already know these things, I'm just putting them out there in case we disagree thus far.
If you take a step back and consider the overall structure of how an auto-regressive neural network works, it's actually a pretty neat way to run neural nets in a loop, where they also get access to their own previous outputs; which clearly is a prerequisite for doing anything resembling human level intelligence--but note it doesn't have to be natural language as we understand it, any sequence of tokens can implement that architecture. This makes it pretty obvious, with the above theorems, that an auto-regressive NN is a fine way of implementing any algorithm you'd like, so I don't really see the need for a "new architecture".
Another way to think about human intelligence is that it was produced by the optimization pressure of evolution, an incredibly dumb process. So we have an existence proof that "human level intelligence" is possible, and can emerge from a straight forward architecture. The question is mostly about compute, and Dynomight makes an interesting observation [here](https://dynomight.substack.com/p/data-wall) which argues that compute is really what matters, not architecture, and that evolution is where all the learning for "human level intelligence" happened. The current architectures build ontop of that; obviously if training LLMs had nothing to learn from, instead of the entire human corpus of text, we'd not have the massive AI success we have had so far! And considering how far comparatively little compute has gotten current AI, you'd have a pretty hard time arguing it's anything more than one or two orders of magnitude away from "human level intelligence", no matter how you define that.
I'm also approaching this very much as a functionalist, I'm not saying they will literally be sentient, conscious, or "solve love". But replacing office workers, especially software engineers (which is my chosen profession); that seems prrreeetty inevitable at this point. The camp of people who believe this is "fundamentally impossible" seem to rely on some kind of implicit assumption about human specialness, which I take to be pretty arrogant, and reject on the grounds of believing in evolution.
If your question was not about why I believe current architectures are sufficient, but rather why *not* some other architecture, I'd say sure, why not. Other architectures could also surely work. But what we have is a surprisingly general architecture, and I think people are getting too hung up on the "language" part: I see language as an implementation detail; a way that happens to work decently at encoding a wide variety of problems, and computations over those problems.
I guess the main reason can be summed up as: the thing we know produced "human level intelligence" was loads of computation, not *clever engineering*, and the current paradigm is very amenable to scalable compute.
Another way to put that: I think people who claim auto-regressive neural nets aren't a sophisticated enough architecture and that we need to invent better architectures to reach AGI, are falling for the "human specialness" trap, and that isn't supported by the evidence; neither the last fifteen years of enormous empirical success, nor the story of evolution.
(Note: I don't think this is a good thing, I'm in the "this is incredibly dangerous and may very well lead to an unrecoverable civilizational catastrophe" camp.)
Assuming arbitrary scale, then NNs are universal approximators and Turing complete, when they have enough training data. That means that they can model every function. A function is a process that maps an input to a definite output (or set of outputs, distribution on outputs, etc.). A function operates with numbers or domains that can be embedded in numbers.
Intelligence, as practiced IRL, operates in domains where (1) there is not necessarily one right answer, and mapping to a distribution or set is unmeaningful/lossy, and (2) even if there is, the full input (including all context, knowledge, etc) cannot practically be encoded, transcribed and fed into the system, and (3) its impractical to get enough high quality training data to cover enough of the input/output space, and (4) the domain does not map cleanly onto numbers (this may be redundant with (1)).
Okay but let’s assume we somehow don’t have to care about these issues, because we’re in a numerical domain, we have adequate training data, we can feed all material context into the system - essentially NNs “solve” a cognitive task perfectly. I still don’t see why we should think this fully explains human intelligence. There is one ***major*** difference - human intelligence is stateful. You don’t have to believe in a soul to think so. LLMs are not stateful. They generate every word anew. If, for some reason, you’re in the mood to absolutely torture metaphors and go head first into full on AI psychosis, you can say that our consciousness is no more than a continuous autoregressive process, and our interiority is an illusion. I won’t follow you there.
The issue isn't how much better a new technology gets on an absolute scale, but how good it is relative to the alternative. In your phone example, why don't you look at how many people use landlines.
AI might not improve much more, but even at its current level it's much better than humans *for the price*. AI generated art might be boring a derivative, but it's basically free. If I have a website and just need some clip art then I'm going to use the thing that's 80% as good for 5% the price. Your argument is like saying that IKEA furniture is clearly inferior to any handcrafted 19th century chair. Of course it is ... but which one is dominating the market?
Perhaps one measure of AGI would be how effective and accurate an AI system is at adopting a new area of expertise, which it never had in depth before,
I gather most current large AI models are "compiled" from scratch, like baking a huge pie from stacks of raw ingredients, in this case every surviving utterance in recorded history. But if the resulting expertise can be suitably and flexibly structured then new layers could be added without having to repeat the whole processs from scratch, and there must be a large financial incentive for this in order to save energy.
An early 20th century airplane designer Antoine De St Expury (sp?) once said "A design is perfect not when there is nothing to add but when there is nothing to remove". So if I were designing an AI system I would start with the huge "baked pie" (as above) and progressively strip out all the knowledge, "trivia" as it were, to leave just an intelligent skeleton, and then add back strata of knowledge as if the reduced system started out as a pig ignorant illiterate person, completely uneducated but highly intelligent and avid to learn!
> energy isn't the bottleneck for data centers anymore...
How do you figure ? Data centers are pretty much maxing out energy grids wherever they are built, consuming megawatts of power just to keep up with computational demand. And if your strategy for scaling LLMs is to throw more compute at them, then it seems like they'd need to consume more power. Computing isn't free, and what remains of Moore's Law is insufficient to meet the exponential training demands.
Meanwhile, the average human brain takes about 20W of power, and can perform many of the same tasks as an LLM housed in a data center -- and many more that the LLM cannot even begin to approach at present. This doesn't mean that AGI is impossible, but it does imply that we've still got a ways to go.
I don't think alignment is actually solvable, but also I think it doesn't really need to be solved. We can set simpler goals by training the AI to:
1. put some positive value on human life
2. have a reward path based on very long term outcomes
If you have a long enough time horizon, there will be very little value to the AI of short term resource competitions with humans on earth. If the AI wants to maximize tokens consumed by 100K years in the future the optimization path for that is going to be almost entirely focused on spreading off of Earth and consuming the much more abundant resources there. The gain from turning a city into a GPU farm is trivial in comparison, so if you can get a positive reward value on humans it will be the less rewarded choice even if it gains a few tokens.
How do you do this without solving alignment?
This is how you get matrix style endless fields of humans in pods being fed heroin. If AI "puts some value on human life" then you get the world tiled with comatose miniature humans (possibly infants or fetuses or even embryos, depending on what it thinks counts) so it can maximize the number of them that are "alive" per resource spent.
If what you actually mean is some positive value of humans existing and doing normal human things then defining what that means IS alignment, and figuring out how to pin that down and make AI care is solving alignment.
I will just focus on your point 2.
No! No! No! Long horizons are really, really, bad.
Simple counter: As an ASI I see that humans are getting inthe way of my plans for utopia, I do the calculations and it turnms out that, even valuing human life really highly, the future excepcted value of killing all humans now, storing their DNA, devoting myself to researching Utopia and restoring them when i've figured it out - is higher than keeping them alive now.
The future light cone is VERY big and things get strange once youy start multiplying by VERY big numbers
Interestingly when I read something like this europe2031.ai I realize that the timelines are likely very short. This document estimates continual learning as arriving in 2030, whereas a more likely time frame is early 2028, as with OpenAI's claim about when an independent AI researcher appears in their labs.
I don't want to argue about what AGI means, but my experience using an AI agent to review literature in biology, this Substack (https://theinfinitesimal.substack.com/p/thoughts-on-ai-in-academia) and the growing list of solved Erdos conjectures leads me to think that AI is "intelligent" enough to do >90% of knowledge work jobs already. Customization for specific jobs won't be trivial, but if agents can do the programming/research/math tasks they are already capable of, surely it is possible for them to manage payroll for a midsized company, draft legal contracts, file taxes, manage inventory, respond to most customer service inquiries, and do the vast majority of work that doesn't involve physical movement of objects.
Can they do those things while making fewer mistakes than the people who currently do them?
I'm not up to date at all on the capabilities of current models, but my impression is that there is still high variance in task success rates. E.g. a given model which can solve Erdos conjectures, will still produce more incorrect proofs which it claims are correct than actual correct proofs. That would be enough to replace some percentage of math researchers with full diffusion, but maybe not 90% yet.
My limited experience with using agents for biology research is very positive. They occasionally miss something in literature but don't hallucinate. At the same time, this is a somewhat less versatile task than say, everything the payroll department deals with.
People make errors too, lots of them, however, currently hiring more people isn't worth the cost compared to the cost of errors. If AI makes the same number of mistakes per 10,000 work units, but you can run it through 1,000 AI and take the majority vote, either because AI is much faster, or because AI is cheap enough to run it 1,000 times for the cost of employing a human, doesn't AI win?
Yeah, what's interesting about AI is that while earlier computer programs would inevitably make the same mistake every time when faced with a problem, the AI doesn't do that. I think AI Agents take advantage of this, as long as the AI can self-assess accurately enough and figure out where it ended up in a ditch, it can move forward.
It is my experience that AI is clearly my superior at some tasks, often better at some (and yet not something I am comfortable leaving unchecked), and, at some tasks--for example, writing--it is pretty awful in comparison with my (or any other skilled human writer's) work. I would be very careful about thinking that performance in specific domains that are easy to train off of will, and especially *has*, generalized to all domains.
Yes. One problem for "When will we have reached AGI?" is the 'spiky' nature of AI systems (though _vastly_ less brittle and specialized than traditional software!). At the point where AIs can do 90% of knowledge work jobs, there will be endless quibbles about "But it only did 95% of the tasks" or "knowledge work should be defined more broadly than that" or "It did 90% of the tasks more accurately than a human but 10% of them less accurately".
My personal guess is that the missing pieces will get filled in, maybe gradually, maybe a whole bunch at once if e.g. continual/increment learning is solved in the next year or two.
It suffers heavily in terms of proper planning . It's good for performing very specific tasks but for broader tasks it doesn't properly divide things into subtasks.
As an example if your rooted phone randomly restarts then it would not ask for logs or think about how to filter them properly.
It would follow a loop of basically this
Step 1)Try X (X is usually a educated guess for which it doesn't have a proof but which is reasonable)
Step 2) If X works good if not then go back to step 1
If you ask it to think about logs it may go down a rabbit hole of trying to decrypt logs encrypted by the vendor and then failing.
It doesn't have good taste about what approaches are likely to work and which won't.
Unless you specifically instruct it, it won't try to gather information and do tangential research to fix a problem.
By this I mean it would google about how to fix the problem but it would not ask user to run commands through which it can know more about the problem.
In a vague sense it's bad at gathering new information (not in it's training) from the environment.
Math and programming feel like constructing something from scratch so it succeeds on them but it fails on tasks which need learning from the environment.
I think this is like my quantum example. They're pretty smart, but they get confused very easily and make weird mistakes. I haven't had enough exposure to Fable to know whether it's the one that finally solves all these problems.
So I don't really understand how my beloved AI tool Edison works, but it has extremely high competence in a limited domain. It can't write a poem or scrounge a cookie recipe, but it can search and summarize scientific literature extremely well (even tasks like "find me papers that used antibodies against Protein X and performed western blots in which a knockout or knockdown control is present"). In strong contrast to ChatGPT, I have never seen it hallucinate a source ever or get confused.
What I'm getting at is that it seems plausible to me that different AI agents with specialized functions will be able to manage a wide variety of white collar tasks even without further improvements in foundational models.
Scott, I'm curious what your personal values or preferences are on writing created by generative ai or prompting being perceived as writing?
I'm also curious if there is anything you can imagine could be done differently to minimize the impact on the temporary underclass?
Being someone who is somewhat more likely than the average ACX reader to be part of an underclass, I find myself fairly worried about concentration of power during transition to AGI. Much more worried than I have been about any modern political scare.
My trust in the high level techno rationalists is waning. Though this is evidence of nothing about reality (only some people's uniformed perception) I am seeing memes of effective altruism being completely devoid of any altruism.
What do you mean by "seeing memes of effective altruism being devoid of any altruism"?
I mean I just started seeing memes (as in a peice of media meant to be shared on social media) equating effective altruism with billionaires who are okay with everyone else dying or being in poverty. In mainstream culture, I think this is becoming most people's first encounter with the term effect altruism. I live far outside of the rationalist bubble.
Oh, okay, I'm not familiar with those. I was concerned you saw memes shared by actual EA people that were callous or something.
SBF may be the most well known "effective altruist" to the general public. That probably isn't a good thing for the EA movement (though I have no concrete suggestion for doing anything about it).
A concrete suggestion would be to have someone else get even richer and give away even more money, and this time don't commit fraud along the way.
There have been socialist memes equating EA with things like “it is more valuable to buy this castle as a retreat to think of ideas than to donate the money to the poor” which is based on a thing that did happen, I’ve seen those. Socialists tend to be at odds with EAs bc socialists want socialism and believe government power is nearly always the solution whereas EAs are a bit closer to neolibs who may not value all outcomes the same way. I think at least there is a tendency in Western lefties to assume redistribution of wealth (voluntary or otherwise) just feels somehow more really altruist than having a good reform idea that amounts to the same thing, like if you were serious you’d go build a well or just give your money to the poor rather than build a thing that made you money and incidentally lifted people out of poverty.
When I read "buy this castle as a retreat to think of ideas" I thought huh, this sounds like something the Fabian Society would do.
And sure enough, they didn't buy a castle, but a memorial fund from some of their most prominent members DID buy a gigantic country manor, name it Beatrice Webb House, and convert it into a conference centre.
Yes, in personal life I am outside the bubble entirely, this tracks with what I've seen. It is seen as net negative for the world.
I don't yet have a strong opinion on this. It's annoying to read AI writing, but until making friends with some people who are terrible writers (eg 30 minutes of fretting to write a five line email) I didn't realize how disabling it was for some people, and I can't really wish to take that accommodation away from them. I think the current equilibrium (people can use it for crappy business emails, but if you try to pass yourself off as a Fancy Writer with it, then people post Pangram scores to mock you and eventually Sam Kriss comes to your house and kills you) is probably fine.
I don't know what will happen when AI becomes a better writer than the best humans, but I imagine there will still be some role for human writers for a while (in the same way that a camera is a "better painter" than the best humans at most of what painting was pre-photography, but people found ways to adjust).
> I don't know what will happen when AI becomes a better writer than the best humans...
I don't see this happening anytime soon (I'm tempted to add "if ever", but of course "never" is a long time). Present-day LLM writing is instantly recognizable as such even without any shibboleths like the em-dashes; and when I say "as such" what I mean is "a particular flavor of poor-quality writing". Of course, even poor-quality writing is better than no writing at all, so your point about accommodation is well taken.
"in the same way that a camera is a "better painter" than the best humans at most of what painting was pre-photography, but people found ways to adjust)."
I have used this example as well. The difference I'm encountering is that no one who was taking a photograph somehow believed they were actually painting. Recently an intelligent friend (who is an ob/gyn medical doctor) prompted a book length quantity of output. This person understands they didn't "write every word," but genuinely doesn't seem to feel there is a difference between prompting and writing. I asked a lot of questions about their method of prompting, and it seems that other than the basic initial idea, chatgpt came up with everything else--topics to include, organization of chapters, information to cite. They feel that they wrote the book, with a little help from ai, and that's that. I know this is just an annoying story that annoys me, their book will not be published, etc. but it made me sit up and start paying more attention to what was happening with writing.
In my personal experience, a huge part of the value of writing is not just the output but the process or transformation that occurs for the writer.
I remember years ago you answered a question about why you didn't write about so many hot button topics anymore. If I remember correctly, you said something like, --you had resolved your questions and beliefs around these topics to your satisfaction, and they no longer took up your thinking space. I imagine this happened in part through the process of writing about them, and that prompting about them, and receiving well thought out logical answers from an AI would not have produced the same changes in the way you think and function in the world.
I understand there are different use cases--like with mindless emailing, or the painting/photography example. But it seems likely that it's having an impact on intelligence growth for a large subset of people (outside of the geniuses or fanatics who will write no matter what). Maybe the tradeoff is obviously worth it for generative ai, but I would like to know this is being paid attention to.
> Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028.
Did this get pasted into the wrong part of the article? It doesn't go with what comes before or after.
Yeah, sorry, deleted.
The anthropomorphism on display here is extraordinary. As is the lack of acknowledgement that basic economic dynamics will ultimately harness AI development.
re: anthropomorphism: "AI" isn't a they/them; it's an "it." And it's not even a single "it" - it's a multitudes of "its" of various degrees of language-disguised-as-intelligence technologies that are paper-thin in their intellect while razor-sharp in language-based reasoning. But here's the thing: humans are far more than language-based reasoning agents. Einstein did a lot of math. LLMs are pretty shitty at math. And even when they RAG math, they still fumble the ball. Is this addressable over time? Sure. But they're just not human equivalent at a fundamental level... which means they will not scale like humans would if given superhuman powers.
re: economics: As token costs surge, we're witnessing in real-time that utilizing AI is nothing more than token consumption in exchange for the simulation of thinking. And token consumption is rapidly evolving from "all you can eat" to "pay what you get." It costs about the same amount of "tokens" to feed Elon Musk and me, yet the two of us have vastly different abilities to transform the world using technology. Who's to say that "AI" will become a token-hungry team of Elon's that would organize themselves to take humanity to task vs. a token-hungry team of Jon's who just want to Netflix, chill, and post some interesting comments on Substack? The efficiency-of-impact ratio is not fixed in humans to say the least -- why do we insist on flattening this for AI technology?
Food for thought.
Cheers.
Jon
I love you very interested to hear what Scott thinks about AI as "substitute for thinking."
I am finding this to be fairly disturbing in what I am witnessing in my own life.
Typo--I would be very interested to hear what Scott thinks*
> Einstein did a lot of math. LLMs are pretty shitty at math.
An OpenAI model solved the Unit Distance conjecture. Erdos problems are falling to AI rapidly. Even if you don't think AI has surpassed peak human ability in math yet, it's clearly somewhere up there.
If you previously assumed this would not happen, perhaps you should reconsider your views on intelligence.
Yet they're developing "intelligence" quite differently than humans. It's impossible to predict where it's all headed based on an unprecedented history to-date.
And still AI is merely highly complex (and token hungry) algorithms that tickle the human mind due to breaking the language barrier.
AI is not a them/them. And it's difficult to see a future when it becomes something deserving of those pronouns.
> And it's difficult to see a future when it becomes something deserving of those pronouns.
Exactly. There is no evidence that could make you change your mind, which is a problem. Maybe you should read this? https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence
I don't want to be overly pedantic, but the word 'they' is absolutely appropriate to use when referring to a language model. You should maybe look it up? The OED makes it clear: the word is used to refer to "people, animals and things." For example, if my wife asked me about the dishes, I might say, "Yeah, they're cleanish." AI is both a they and a them.
"LLMs are pretty shitty at math. "
No. Language models won the gold at the Math Olympiad, is solving Erdos problems, co-authoring papers with Donald Knuth, helping Terrance Tao daily with his work. They're not just good at math, they're spectacular.
"As token costs surge..."
Token costs are not surging; they've been dropping dramatically. And not by a little, we're talking per-token price has dropped 98% since 2024.
There are some other funny ideas in there but I'm not sure they're worth addressing.
I'm somewhat confused by the choice of math as example domain here-- AI struggles with a lot of things, but I think math is undeniably a strong point of modern frontier models.
On the economics point:
Tokens are cheaper and tasks are cheaper than they were previously, at least in the task categories most relevant, as far as I can tell. We just use them for a lot more tasks, and so the total cost has increased noticeably.
Also, I seriously question the idea of trying to describe AI models in the way you are; while you do see some serious variation in personality and type of preferred task model to model, I think the idea of preferring being useful/doing a task is sort of a fundamental part of the modern RLHF/RLVR/RLAIF/whatever comes next + character training process... I think people are right to care about how exactly this shapes it, but I don't think potato is a likely shape given modern processes.
"LLMs are pretty shitty at math."
You may want to open a newspaper from the past month, but consider sitting down first.
This reminds me of my favourite scene from Dumb and Dumber where Lloyd, our protagonist, just got "stood up" for a date at "11" (he rocked up to the bar at 11am...). Upon someone else realizing the mistake for him, he gallivants toward the bar exit where a newspaper clipping of the Moon Landing catches his eye. Set at least 20 years ex post, he incredulously mutters "no way" as the implication dawns upon him, barrels through the door and ecstatically exclaims "We Landed on the Moon!".
I had this moment the other day when using AI for coding for the first time since 2022 (the early bird doesn't always get the worm). Boy oh boy, have we landed on the moon.
> AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time.
I'm not so sure I buy that the gap from expert to better-than-top-geniuses is actually that small, or smaller than "dumber than a child" to "expert level" in some meaningful way. Many people go from as dumb as a child to expert level in some field or another; very few go from expert level to top-genius level.
I don't buy chess and go as counterexamples to the above either, but I'm having trouble articulating why and need to commute, so that'll have to wait.
> I don't buy chess and go as counterexamples to the above either, but I'm having trouble articulating why and need to commute, so that'll have to wait.
Because those are combinatorial problems. There is an objective, easy to measure goal (reach a winning state) and as long as the AI does the accounting correctly (track if it has reached that state), even the dumbest algorithm will *eventually* find the optimal strategy.
Well, if you compare expert level chess players to someone like Magnus Carlsen, you'll find that Magnus can win upwards of 98% of the time. Experts are rated 2000-2199 Elo, whereas Magnus currently has a rating of 2842.
Most average people could spend their entire lives playing chess casually and never reach a rating of 2000.
I would assume diminishing returns, so there is some sigmoid curve relating effort/time/energy to accomplishment and somewhere it becomes economically infeasible to hit the next benchmark. But the curve is not necessarily in the same place for humans and for LLMs (or other AIs). Maybe "top genius" mathematician isn't very hard for an LLM to achieve once its gotten to expert level, whereas its basically impossible for most expert human mathematicians. But actually I think its likely the reverse is true - LLMs are much less efficient at learning, both in token inputs and in energy inputs than humans, even though the network size of cutting edge models is 10-100x smaller than a human brain. They will not get more efficient as the networks get bigger, if anything probably the reverse, so its likely "top genius" LLMs even in easy fields are not going to be economically feasible without some algorithmic or hardware breakthrough.
I don't think that line of reasoning works. Many people go from 0 feet tall to 6 feet tall over their lives, but few go from 6 feet to 7 feet tall.
I am a phd in physics who uses AI all the time, and the it's true - AI can answer relatively advanced physics questions.
On the other hand i also enjoy video games, and AI can't reliably answer relatively basic video game strategy questions.
How can that be? Almost everything written about physics past a certain level is written by people who are professional physicists. They can still be wrong of course, but there's nearly no one mouthing off about random niche science topics on the Internet. The same is not true for video games.
Neural nets still have no way of resolving what's true, which isn't surprising, because humans don't either. We have to empirically check things. And making physical machines that can do empirical verification under AI control is not even close to a solved problem.
"Many people go from as dumb as a child to expert level in some field or another; very few go from expert level to top-genius level."
I think this is confusing ability of humans to traverse a space with the size of the space itself.
For example, most humans reach 5 feet tall, but very few humans reach 7.5 feet tall, and none reach 10 feet tall. Yet if there's some non-human process, like throwing a discus, doing it 7 feet is only 50% harder than doing it 5 feet, and doing it 10 feet is only twice as hard. Most people who reach 5 feet tall will never reach 10 feet tall, but most people who learn to throw a discus 5 feet can later learn to throw it 10 feet.
I think if you think of human tasks as (let's say) 100 IQ vs. 150 IQ vs. 200 IQ in the same way that height could be 5 feet vs. 7.5 feet vs. 10 feet, the way AI will traverse this space looks more like some kind of objective metric than like mimicking the human difficulty curve (I know IQ doesn't work that way, it's just an example).
I can’t help but feel this is going to be contingent on the training data.
There’s a vast corpus teaching AI to move from ‘dumb as a child’ to ‘expert’. There’s a much smaller training set available to take the AI from ‘expert’ to ‘top genius’, and by definition there is no training data available to take AI beyond the level of the best human geniuses.
> by definition
That somewhat depends on the nature of the problem, and the wider field. Recombining best practices from diverse specialists could conceivably yield results no single human lives long enough to imitate, or an adversarial process could build on the previous iteration's work.
Given a machine which can solve every Erdos problem, how hard would it really be to set up a second machine which proposes new conjectures in an effort to stump that first one?
As of now how many top mathematicians or computer scientists are being hired to produce training data for AI as opposed to working on AI directly? I would say not many. AI companies are not doing this because either they don’t need to and never will, or because they don’t need to yet, but will eventually do. This will tell us whether training data at high skill level is going to be a bottleneck or not. Edit: obviously we are already producing some training data for free, e.g. on arXiv. But this is far from optimal from a training POV
I am not entirely sure what this means. Are you saying that there exists a better metric for "intelligence" than IQ ? If so, what is it ? I understand that throwing a discus really far is much easier than being tall, but I think the analogous situation would be adding up a list of numbers very fast vs. being able to solve novel problems. Machines can easily achieve the former, but not the latter, because (much like throwing a discus vs. being taller) of the complexity involved.
I think there needs to be one, but it is as yet not forthcoming. The "spiky" nature of AI intelligence (mentioned much in this discussion) tells me that IQ is a lousy measure, perhaps wholly inappropriate for AI. The issue is that IQ is supposed to tell us something about a hidden "g-factor" which is well-documented in humans. However, I would say "spiky intelligence" is very much the opposite of "g", in that it's not generalized. Ability in one narrow task does not predict ability in even extremely similar (but slightly differently worded) tasks, something you'd never expect in highly intelligent humans.
Some food for thought:
Parallels between human and AI - AI is fundamentally a different intelligence to a human. There's no reason to suggest that human aptitude paths follow AI aptitude paths outside of it being our only prior for intelligence development.
The following is a can of worms, but I don't think a metaphor for AI "infancy" works in the same way it does for people. New AIs are more analogous to new phones than a person with an extra year in terms of development. Its a stepped instead of continuous process. These AIs are distinct from one another - as with their "personalities" (quirks?). A person has more contunuity.
As Jon Deutsch suggested above, we should avoid anthropomorphizing too much. This is a far cry from "anything goes", though.
Muravec's paradox perhaps buttresses this developmental point. If we permit the metaphor I've just eschewed above, AIs over time possibly have most reason to develop backwards compared to human timelines. The paradox, pre-LLM, observed AIs were much better at being trained to success on "higher order" human activities (chess) compared to lower order ones (house cleaning).
If the paradox is true, we should perhaps expect the highest order thinking from our human perspective to be well within reach rather quickly? But perhaps we should concomitantly expect dispersion to be slower (for knowledge jobs that require certain "base" aptitudes that are difficult to train for - difficult to name any, but we already observe AI aptitude "spikes" as Scott calls them in the post).
Thoughts?
>Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs
If it was smart enough to do 90% of knowledge works jobs, it would be doing 90% of knowledge work jobs, it's tautological. If it's measuring as "smart enough" on some benchmark but that isn't translating into actual integration in the real world, that doesn't mean the world is slow, it means that the measurement is faulty. There is a huge gulf between being able to do discrete tasks while being babysat by a former knowledge worker turned prompt engineer performing his job in 10 second increments, and actually being smart enough to operate autonomously and productively in the economy.
I reject the premise that benchmarks/AI model horseracing performance gains indicate that significant progress is being made towards this independence. According to ChatGPT 5.5, ChatGPT delivers ~10x to 100x more useful cognitive work per interaction, but having interacted with it both I am skeptical that ChatGPT 5.5 is much if at all closer to independently acting as a knowledge worker.
AI is absolutely being used in business. Most companies are using Copilot as a Microsoft Office suite application much like they use Word and Excel. Software developers use Claude or Codex. This is translating into minor to substantial productivity gains for knowledge workers. None of this is indicative that suddenly autonomous AIs will be replacing knowledge workers
So you believe that every time a machine isn't being used to do a job, that's because it _can't_ do that job? It's never because someone doesn't know the machine exists, or because the machine is too expensive, or because the machine is illegal?
I incorporate the non strictly technological developmental barriers to widespread usage into "can't" and "smart enough"
If I understand your point, then why would you do that? He specifically separated those things out by having the section on diffusion. He makes in the article a clear separation between ability to do something and actually doing it. Combining them again, and then using the combined version to attack a claim that is explicitly not that seems not in good faith.
I don't think they are separable, the 'benchmarked ability with no actual usability' piece is meaningless. It's like a flying pig that just happens not to fly
I love that I don't even understand what you mean by your meant-to-be obviously-correct-analogy-to demonstrate-the-point example. To me, it seems useful to separate the ability to fly and being actually in flight for flying pigs too.
Fine. I've been reading the Piraha book lately that Scott linked, about a barely contacted tribe in the Amazon with a very unusual language and no number system.
Consider the smartest Piraha ever. Consider if John von Neumann or Albert Einstein with a 200 iq was cloned and born as a Piraha. They would be great at Piraha activities, and they would absolutely be masters of their jungle craft. If you needed a jaguar hunted, they'd be your guy.
Now, teleport them to Grand Central Station and have them report to do knowledge work at an accounting firm. They'd be utterly helpless. They are naturally smart enough that if they are handheld through specific, brief tasks they can do them well, but they are would be completely incapable of interfacing with the modern business environment. They have an iq of 200 and they can excel on their benchmarks and on discrete tasks, but they don't have nearly enough "smartness" to operate usefully independently indefinitely.
If instead of an iq of 200 they had an iq of 200000, maybe they could completely reinvent modern civilization from scratch in an instant and jump right into modern office work, but that is a tremendous leap beyond just the raw intelligence they need to do the stuff they are good at well. No matter how good AI gets at their benchmarks, I don't think noticeable progress is being made towards the true autonomy required to act as a modern knowledge worker.
You are making up a strawman with your benchmark point. People aren't talking about fake claims to be able to do something. You can disagree that real ability will come, but that's addressing a totally different part of the article than the one you initially replied to.
Are you, right this instant, solving elementary school level arithmetic problems?
Does that mean you can't or aren't smart enough to solve them?
When a car that was able to transport people more cheaply than a horse could was created, did it take over every human-transport job the next day?
If you define "more cheaply" as including the social, chronological, legal, and infrastructural drag/costs, which is roughly analagous to this person's definition of "smart enough" then...yeah, I guess so?
So if, in the future, AI becomes godlike, but 70 year old George, the accountant who is set in his ways does not bother to use it, then AI is not smart enough according to this person's definition? I'm still wondering why anyone would find it useful to adopt this definition and criticise those who don't do so.
It seems not entirely unreasonable to me in this context, though not the typical understanding of “smart.” I suppose OP would say that if the AI can’t make itself useful and attractive enough to convince Greg, or if not Greg then some sufficient number of people such that +90% of knowledge jobs are being done by AI, then that is in fact a failure of instrumental intelligence.
Yes. If the AI was actually smart enough, it would be used
I think there's a neat middle ground between your perspective and the original commenter's.
"Actual integration" is a broad church:
1. Regulatory boundaries; attitudes etc. (what you are taking it to be)
2. Contextual on-the-job understanding (the OC's view)
I think you slightly strawman what they were exactly saying - though I do agree that the causality is multivariate and not just the lack of "context".
That being said, I don't think practically that their argument is super strong.
TLDR: firm-specific training and then AI-re-designed and tailored systems to avoid any "understanding" as humans see it mean this practical constraint is transitory at best.
This impact will be lowest in tasks that (1) have limited technology usage, and for knowledge work (2) discrete tasks that are difficult to aggregate vast amounts of data from.
Pre-ASI, if we model adoption (diffusion) as a function of Token Cost and quantity used per task, labour demand is a function of how many tasks give labour a cost-based comparative advantage to AI.
Wages therefore are less than or equal to the cost for the AI to do the task, above which an employer would find it more effective to use AI (kind of like a Malthusian upper bound on wages).
This would apply in the "medium run" as it assumes perfect substitution between AI and labour. In the short run, before this labour demand will also be a function of a task's requisite contextual understanding (interfacing multiple systems, for one). This is related to the OC's point.
2 forces assuage this, pushing us to the "medium run":
1. Use of AI side-by-side or directly replacing labour in current systems will provide data for this "contextual" understanding. Possible gains coming from this could be universal, or firm-idiosyncratic (influencing speed).
2. Redesigned systems not requiring a human "contextual understanding". e.g. working entirely in an SQL database instead of using finicky interfacing tools that are specifically human-friendly...
I think these 2 "forces" even with AI at this point in time are achievable today.
As AI's cost advantage over labour increases (if it does? https://www.apollo.com/wealth/the-daily-spark/cheaper-tokens-bigger-bills), then it's a matter of time until "Contextual" understanding is gained.
Regarding regulatory issues/ public backlash and resistance of norms:
Where there's a will there's a way.
Microsoft and Amazon are planning on using mini nuclear power plants to power Data Centers after decades of general public resistance to their usage. Not all decisions are political decisions; not all of these decisions necessarily have political backlash
Competitive forces will force later adopters of AI to follow suit if productivity gains arise sufficiently, otherwise they risk becoming a shareholder pariah, and simply being out-competed.
On the regulatory side:
1. Most countries (outside of large economic blocs like the US, China and EU) have limited control. If they don't permit AI in work - say goodbye to investment and firms. For politicians - say goodbye to growth and good PMI numbers, ergo your electorate and corporate supporters.
2. There may be resistance from losers (e.g. consumer discretionary firms whose demand would dry up from AGI reducing disposable incomes), but if they are dwarfed economically by winners, the regulators will likely listen to the latter.
So I'm not really sure that these boundaries will be that significant. But I may just be too bearish on AI's impact. Regulatory and social resistance will be massively amplified if the effects are as well. I still think economic forces will play a large part. They will be more salient if the impact of AI is slower (as opposed to an ASI huge spike in capabilities which suddenly changes everything).
> "If it was smart enough to do 90% of knowledge works jobs, it would be doing 90% of knowledge work jobs, it's tautological."
This is not true at all. For example, I am smart enough to be a high school English teacher (based on my SAT score), but I am not a high school English teacher.
Or, more prosaically, an AI could be smart enough to do a task, but banned by regulation.
I am assuming that the AI companies would like their AIs to be doing 90% of knowledge work jobs if they could, but they can't. Conversely, you don't want to be a high school english teacher, or you wouldn't be.
I'll concede that regulation can create an insurmountable barrier, but I don't think its come into play yet so I don't think its relevant
Typo---I would be very interested to hear**
"I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us."
How far back in time do you need to go for this to be true for that year's inhabitants relative to today? Is 1900 too similar, or would the improvements sufficiently impress? 1800? 1400?
I think if I was from 1900, just knowing about our improvements in childhood mortality would make today seem like utopia.
Agreed. But this makes me skeptical of Scott's prediction. I think 2026 qualifies as utopia from a 1900 point of view (at least for people living in developed countries), but most people living today would not consider it as utopia. I am much more doubtful than Scott that people living in any kind of utopia would acknowledge it as such.
i think thats a little too optimistic...2026 would qualify as an improvement im fairly confident, but even well over 100 years ago, utopia meant stuff like social harmony and post scarcity
I would argue that we have post-scarcity of many things. When I invite guests, then I can offer them to charge their phones if their want, to drink as much water as they want. I wouldn't even know the magnitude of what that costs me because it is so tiny.
On the other hand, backstage-passes with exclusive access to famous (real-person) pop-singers will always be scarce. There is no post-scarcity for things that are valuable because they are exclusive. So post-scarcity is not a Yes/No flag, it's a set of thing which are post-scarce versus a set of things which are not. And actually, I find that an impressive number of things are post-scarce. How much pasta can you afford to offer to your guests without ever running into financial hardness?
This doesn't seem like a useful definition of post-scarcity to me. Some people 100 years ago could certainly provide their house guests with bottomless breadsticks if they wanted to. A post-scarce utopian society would require *everyone in it* being able to get their basic needs for food and shelter (at least) met for free, or trivially. We're just not there.
And again, social harmony.
I want to read a whole Scott Alexander essay on this question.
There's a book called Looking Backwards from 1888 imagining what a future socialist utopia in the year 2000 might look like. Some utopian stuff like freely available recorded music has come to pass, and modern technology is generally a lot more advanced than in the book, but overall the society in the book feels more utopian than the real modern world. So my guess would be the real modern world wouldn't qualify as utopian, at least to the author of that book.
Yes, I also think so. Too much depression and suicide, political madness and war for real utopia.
No matter what are your actual thoughts about AI, here is a great framework for laying them out: https://substack.com/@larsiusprime/note/c-272111008?r=7zdwlv
Isn’t there a bigger difference between being able to do 75% of jobs and being able to self improve?
Like, I’m sure it can do Karen from accountings job with a bit of scaffolding and rethinking who gets blamed when things go wrong. But I’m not sure it can do AI-deep learning levels of work. Isn’t that the remaining 25% in all cases?
The ability to improve AI models is a) something the people working at AI labs value very highly and are likely to prioritize in the models they build, and b) something they know a lot about and are likely to be good at judging how well the models are succeeding at. That suggests that, to the extent that AI capabilities are "spiky" (significantly better in some specific areas and worse in others than you might intuitively expect) the spikes might favor being good at self improvement.
I mean maybe. But when you think about knowledge work, I feel like you or me could easily do 75% of that work within a month or two of practice. But something like deep neural learning at the cutting edge? Very unlikely.
That depends very heavily on exactly what's causing the spikes. Some problems just aren't efficiently computable, no matter how motivated you are or how obvious a correct answer would be.
I agree with Paul below - self-improvement will come early because the labs are working on it hard. But also for two other reasons:
- To take Karen from accounting's job, it has to be able to do 100% of her tasks without making mistakes. To help self-improvement, it just has to be able to do 50% of an AI researcher's tasks in a way that speeds them up significantly.
- AI is much better at things that can be massively trained through reward signals. ML research can sort of be like this - you can (in theory) make it design an AI a million times and reward it when the AI does well on some benchmark. But many superficially easier jobs aren't - if Karen has to do customer service, you can't make it interact with a customer a million times during training and grade the results because customers are real people who move at human speed. Probably there are hacks for this, but ML training needs fewer hacks than some other things.
> To help self-improvement, it just has to be able to do 50% of an AI researcher's tasks in a way that speeds them up significantly.
That doesn't follow. For a recursive self-improvement takeoff, the rate at which AI assistance is speeding up research needs to exceed the rate at which successive incremental improvements become inherently more difficult. Any single automation-resistant bottleneck task could stall the whole thing, or a cascade of individually trivial inconveniences could add up to throttle the growth rate.
Humans also have the "uncanny valley" issue. Train an AI to be 95% like a human customer service representative and watch your customers rebel because it frustrates them and annoys them in myriad ways.
*Human* CSRs at consumer-facing firms frustrate and annoy customers. Because most of them are low-paid, low-skilled script-followers with neither the ability or authority to actually solve customer problems.
(Not a problem with business-to-business CSRs because those high-dollar customers get highly paid and highly capable CSRs with knowledge and authority.)
The bar to to do better than consumer CSRs is low.
> But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly.
Wait, really? We already know that LLMs are being used for military operations, right? This is a strange amount of confidence in the general risk-aversion of powerful people...
Isn't current military use of LLMs limited in scope in exactly the way Scott describes? Or am I behind the times on how they're being used?
"Isn't current military use of LLMs limited in scope in exactly the way Scott describes? Or am I behind the times on how they're being used?"
From https://x.com/newscientist/status/2064773160642216159:
"A senior figure in the Ukrainian defence industry told New Scientist that a test took place two years ago involving fully autonomous drones set to destroy anything in a given area, with confirmed casualties"
The "autonomous" bit is that the drone didn't have a human in the kill chain once it was deployed.
I doubt that this drone had an on-board LLM, but I don't know much that should matter. And if these things work I expect them to get scaled up as people try to win this war (or future wars).
That's closer to a smart mine than a real autonomous weapon system. It doesn't really have an "AI" that makes any sort of decisions, just an advanced guidance system.
It's honestly closer to precision guided munitions we've used for decades than an autonomous killbot.
The human in the kill chain is the one that dropped it in a certain area, just like with mines.
https://archive.is/uSqty is a recent article in the UK Financial Times about the UK military considering fully autonomous weapons, so it's not just Ukraine saying this in public.
That's a good point. I was thinking of civilian uses where we err on the side of safety, but the military might accelerate things to stay competitive, or because dying in an accident is more acceptable there. But even in the military, I think current uses of AI are somewhat limited - I think (not sure, tell me if I'm wrong) nobody is putting ChatGPT in charge of a drone without human oversight.
Some commenters brought up two good sources on this (https://archive.is/uSqty, https://www.newscientist.com/article/2529849-fully-autonomous-drones-have-killed-human-soldiers-for-the-first-time). I personally expect AI involvement in military operations to *increase* as the systems get more competent and reliable, at least until we do get a warning shot (if we get a warning shot, that is), if only because it actually will make more sense to use them as time goes on.
Comment : the cyber security abilities of Mythos show how "forced adoption" works.
"Pay for the tokens to regenerate and rewrite all of your software that touches untrusted input...or else".
Software maintainers get between now and when kimi/deepseek releases an open weight model that matches mythos in hacking.
It's not dissimilar to various gunsmiths inventing the machine gun. Once they were common, every military in Europe had to buy some, hugely accelerating adoption.
This will generalize. Are you a doctor? Your competition charges half price for a visit, using AI to do all the paperwork. A judge? Your court better adopt AI or the flood of lawsuits will make the delays 10 years.
Computers didn't provide such a productivity advantage.
People would not accept AI judges, not without at least the right to a de novo appeal to a human judge, which (given the automation of drafting the appeals and the definition of de novo) would effectively mean every case still goes to a human judge. Perhaps at most some bond would be required to stay execution of the order pending human review, but then surely activists will demand in forma pauperis filing rights, so effectively the whole thing is a waste of time. It’ll replace a few lower level court clerks who currently just route filings and don’t have the authority to stamp any orders, likely no more until AI causes democracy to collapse anyhow and the courts will have no authority
AI judges making the decisions : no doubt. I meant in a general sense the COURT must adopt AI in order to analyze the enormous flood of lawsuits, motions etc that all the parties in every case will use to flood the zone. The judges AI has to be filtering every filing for incoherent reasoning, made up citations, existing precedent clearly covering the exact situation with the ruling going against the filer, etc. "Vibe-judging" becomes a lot of the judge's job but yes they have the ultimate authority.
98% of federal cases end in a plea bargain, so I think you're over-estimating how much the average person values a fair trial. If your choice is an AI-generated plea bargain or a human judge imposing the full sentence...
I was thinking of civil litigation and bench trials primarily, even there I believe people will want a human judge to which they can pitch persuasion, we'll be (justifiably IMO) afraid of the AIs having some bizarre inscrutable alien reason to favor X over Y.
But I did practice crimlaw for 20 years, and while the lion's share of cases do end with plea bargains, the jury is the threat that keeps all of that in line. On many occasions I had to weigh what I thought a jury would do with the case, especially on first convictions where the jury gets the sentencing decision in my state, in order to evaluate an offer. Your leverage as the defense attorney is that juries are dumb and will absolutely walk guilty defendants for no good reason maybe 10% of the time, more in other places. Juries also are the bounds that keep the state from enforcing things the public just doesn't like, it only takes one juror to nullify. When I was a prosecutor I very often declined to file charges in situations where I knew a jury was likely to disagree. When you see some stat about 98% that isn't because people don't value trials, it's because everyone in the system understands the value of a trial and has already factored that into both their charging decisions and the kinds of offers and sentences on the table.
I do think that the idea of AI-generating the plea offers might actually fly in the federal system, as it is already kind of algorithm-like, if you could convince people that it would apply that without any undesirable prejudice, mostly according to defined factors but with occasional justified deviations that were bounded by the overall range, it could even be a rational system. But if I'm the defense attorney and I don't like the offer, I think I'd rather make my pitch for a better one to a human all the same.
People would accept an AI judge if it gave them an extra chance at acquittal without having to go to the expense of a trial. Like this:
AI judge reviews all the evidence -> AI acquits the defendant OR AI determines there is enough evidence to proceed -> case proceeds to preliminary hearing with a human judge (and then progresses as normal).
If there is a large enough number of cases that lack evidence to proceed, then AIs could dramatically reduce a court's caseload.
"Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028."
I guess the paragraph directly above this one says that this isn't true, but it seems like there was supposed to be more text here?
"I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation"
I'm sorry, I can't parse this. Is this implying a 66%+ chance of the universe being a simulation? Or is it only 66% for the conditional probability P(simulation|singularity)? Or something else?
I interpreted it unconditionally. Probably the most based prediction in the bunch.
Reads as unconditional to me, because the explanation explicitly references the possibility of a singularity not occurring
It was a surprisingly offbeat final opinion to include in the post! I strongly approve of the inclusion. Gotta have some quirky fun amidst the structured arguments. 🙂
The 66% was shockingly high to me, but that's only because I have different metaphysical opinions, presumably.
Finding that you are Scott Alexander is a much stronger argument for being in a simulation than finding you are Gabriel, presumably.
But only if you believe there's a good chance that a good proportion of other people are p-zombies right? Really want more information about how Scott is thinking about this, I thought it was truly insane
He's said something adjacent to this before, basically "many smart people agree this is likely, but it's also very important to act as though it's not"
ah well I guess I should be happy that as a non-smart random I have access to truths about the world that are hidden from the smart important people.
Rather, I lean toward Tegmark's mathematical universe hypothesis. The MUH is roughly: our universe is a mathematical structure, every other mathematical structure is equally as real as our world, and some of them like ours contain people.
On that view, a simulation of a world is peeking at part of the structure of that the world, not creating it. Pausing or stopping a simulation of a world (a mathematical structure) doesn't pause or stop the world.
It's hard to make any sense of the simulation argument within MUH. There's not a unique fact of the matter about whether a world is or isn't a simulation. So anthropic Bayesian updates don't figure in.
Furthermore I take Parfit's ideas on personal identity and run with them. His view is roughly: there's no lifelong continuity, only interlocking chains of mind-moments that remember past mind-moments. To that, I add that there are countless future versions of you across the worlds, countless mind-moments that remember having been the mind-moment that you now are.
On that view, stopping a simulation of a person only stops that person in the local universe; from their first person perspective, somewhere out among the possible worlds is a mind-moment that is the continuation of where the simulation left off. (Yes, it's the Quantum Immortality idea on steroids.)
This separately also makes it hard to make sense of the simulation argument. There's not a unique fact of the matter about which world you'll experience in your next moment. So anthropic Bayesian updates don't have a role here either.
Now of course those are metaphysical opinions, for which the proper epistemic status IMO is "fun to think about but should not impact decisions".
What does it mean for something to be a "mathematical structure" ? Is this not the classic fallacy of confusing the map with the territory ?
The usual phrasing is "a mathematical structure" as it's better for popular intuition. A more precise phrasing is "a model of a categorical theory, up to isomorphism". A theory pinpointing the structure is a map, the structure itself is the territory.
If you're asking for an intuitive and informal way to understand what sort of thing a mathematical structure is, I'd say to expect traits like closure (the facts inside it don't depend on facts outside it), determinacy (no vagueness or ambiguity), consistency, completeness (no undecided gaps).
I get that, but from my point of view mathematics and logic are just tools we've developed to build predictive models of reality, which might be beyound our direct understanding. Something out there exists, sure, but when you say things like "I have two apples" or "the wavelength of this light is 485 nm", you're not talking about that existing thing directly; you're invoking a predictive model that can only ever approximate it. So it's weird for me to hear a statement like "actually apples are ultimately made out of closures" (yes, I realize this is a caricature).
Scotts p(singularity) is fairly high, so P(simulation|singularity) =approx P(simulation) right?
I trust this sort of analysis considerably less when I see no particularly low or high numbers appear anywhere. Nothing AI related is less than 10%, even the odds of nuclear war destroying humanity are at 5% for a 30 year period, which is incredibly unrealistic considering it requires not just any nuclear exchange, but a nuclear exchange between countries that have produced in the order of thousands of warheads. This you mean there was only about a 60% chance it hasn't happened already, was it really basically a coinflip that we haven't already died to total nuclear war? With regards to AI, I don't think you understand the issues with compute constraint, necessity of continuous learning and how serious the lack of training data for open ended problems is.
"Basically a coinflip that we haven't already died to nuclear war" is a pretty fair description of the Cold War, honestly
Yeah. OP, please look up the history of nuclear close calls.
If it really was a coinflip, it's extremely likely one of those close calls would have actually resulted in a nuclear exchange. The high stakes of nuclear war and its lack of being in anyone's interest mean people will generally avoid triggering a nuclear attack.
Those incentives and stakes have not performed nearly as well as one would hope. In 1969 the President of the United States ordered a nuclear strike on North Korea, just to throw one such event out there
You state the nuclear strike order as a fact. Other people would call it disputed. In your opinion, what is the probability of this account being accurate (the order really happened)?
No, if it really was a coinflip, it's ~50% likely that one of those close calls would have actually resulted in a nuclear exchange.
It seems like you're conflating "a coinflip that we haven't died to nuclear war" with each "close call" being a coinflip.
I think the bigger factor is whether a full-scale nuclear war at the height of nuclear weapon stockpiles would have been sufficient to kill everybody.
I think the current consensus is that this is very unlikely, and that the Cold War era scenarios where this did happen were always just made up to scare normies rather than being based on realistic modelling.
“Kill everybody” was never realistic, but killing say 40% of the global population over the course of two years is entirely in the range of possible outcomes of nuclear war, and that probably causes near-irreparable damage to human civilization
I suspect there's a decent chance of Google's Titans + MIRAS architecture pattern, or something similar, creating a new generation of continuous learning models within the next two years.
I'm sure continuous learning models will become a thing, but they will make the scaling curves even more exponential on the compute x axis.
That's because it would be boring. I think there's a 99% chance that someone creates an AI intended to help tutor people, and a 1% chance that AI paints itself green and dances a jig on top of the White House, but who cares?
I think the reason those claims are boring is that everyone agrees with them, not that they are very high or very low confidence
In a community of people who are mostly rational and well-grounded, "nearly everyone agrees" and "very high confidence" are quite strongly correlated.
No, I don't think so.
At least some of the things you're listing here should be deterministically contingent on other things you're listing. Rather than assigning probabilities to those things, which you just list as arguments, you're just giving consistently greater than 10% odds for rapid AI progress. That's just trying to dress up vibes with numbers.
The first timeline point does a decent job acknowledging this, but it's worth restating plainly: "recursive self-improvement" isn't an argument, it's an IOU for one. Sure, RSI seems from first principles like it might happen, but so do many things in human economic / social history that haven't. Without some gears-level model of what this might look like, I'm disinclined to accept it as a factor pushing for shorter timelines. (And no, the linked AI Futures page appealing to "research taste" isn't nearly enough, though I might be missing some more detailed analysis by them elsewhere.)
What do you mean? I think RSI has already started.
Does AlphaZero count as recursive self-improvement (in a limited domain)?
Surely not because AlphaZero never improved its own development or the development of other AI models. I think you're confusing RSI with just regular machine learning.
I'm not sure how you distinguish between an argument and an IOU for an argument. It's possible, it has some probability, I don't see any reason to make that probability extremely high or extremely low, so I'm putting it medium. See https://slatestarcodex.com/2020/04/14/a-failure-but-not-of-prediction/ . And as Presto said below, some amount of RSI has already started, see https://www.anthropic.com/institute/recursive-self-improvement
Perhaps we are operating with different definitions of RSI, but in my view there is no evidence in this Anthropic article of RSI. All the improvements mentioned are being performed by humans using AI as a tool to assist their work. To me, this is definitionally not self-improvement. The conclusion (section titled "What might the future of work at Anthropic look like?") indicates that Anthropic believes RSI is in the near future, not happening currently.
I have a similar interpretation to Ken: the AI Futures page linked from the main text emphasizes that the bottleneck for RSI impacting timelines is probably research taste, whereas the Anthropic post emphasizes that research taste is where big model gaps persist and most of the gains are in mundane software engineering tasks.
That's exactly the kind of distinction that more detail on "recursive self-improvement" would help clarify. It really sticks out in the text that all the "arguments for shorter" sections just say "recursive self-improvement" while all the "arguments for longer" sections go into detail about specific possibilities and scenarios. That common-sense discrepancy is what motivated my comment.
I don't buy that any of this is related to the phenomena described in "A Failure, But Not of Prediction." The timelines here are explicit framed as predictions and presumably the stated median / quartiles which are moved by the perceived likelihood of various scenarios. So there's no question of confusing probability levels with risk levels or correct responses (in fact I tend to be more hawkish on existential AI risk than my peers for this reason). I simply think you're overstating the likelihood of the very short timelines due to not being clear about what timeline-shifting RSI would require or look like.
Well, it's not as if they've put the "give the models research taste" ticket on the back burner, because they don't realize it's a blocker...
When you say 90% of knowledge jobs, do you mean 90% of currently existing knowledge jobs or 90% of knowledge jobs that exist when AGI comes?
Good question, existing ones.
As I'd asked in my other comment, what are "knowledge jobs", and how are you measuring "90%" ? For example, is Karen the HR lady performing a "knowledge job" ? If so, and assuming that your company employs 9 Karens and one senior scientist; then when the Karens get replaced by ChatGPT would this mean that ChatGPT had taken over 90% of knowledge jobs in your company ?
Pretty clearly yes.
> p(doom) calculations don’t give points for delay
Shouldn't they though? After all, in the (very) long run p(doom) ~= 100%
I think it could be set a limit for the sake of argument, e.g., p(doom) before 2100. After that things are too unpredictable.
Thanks for writing this and putting everything in one place.
I think there's nuance to the idea of a diffusion gap: does "AI can do 90% of knowledge work" mean *in principle* or *in practice*? What I mean is that there's a further capability gap between:
"this AI has demonstrated enough general reasoning skills that I'm confident it could replace a working mathematician, if we fine tune it, develop a bunch of tools, debug it, do a soft roll-out to hand-picked research areas, etc"
vs
"this AI could replace a working mathematician *right now*, all you have to do is give it the passwords to their email / arxiv / etc"
How large would you assess this gap as?
I think I mean the first, with the latter as part of the thing you have to do to close the gap, but I agree it's awkward and ambiguous.
Do you have a citation for that claim about a 20 year diffusion for PCs? The IBM PC was 1981, by 2001 perhaps the majority of desk jobs required a PC, but diffusion into all jobs was very uneven. With tablets and phones computers were pretty ubiquitous by 2021. My understanding is that plumbers and warehouse workers and shelf stackers would not have had or been expected to use computers in their jobs in 2001, and things like PDAs with custom software would have been seen in innovative companies like Amazon only. Total sales of computing devices only really hit the numbers to make real diffusion possible post-2007 with smartphones/tablets.
I can provide something that is probably close to what Scott is thinking of:
https://www.linkedin.com/pulse/how-does-lifecycle-software-product-work-joaquim-torres/
Search in the document for "S-curve in real life"
I have no idea if the chart is correct.
Good point, I meant desk jobs.
The simulation concept feels like a probabilistic trap. An intriguing mathematical concept that gets a lot more traction than is warranted because it's hard to disprove. I feel similarly about string theory. The average string theorist is >> smarter than me, and I still can't help but feel everyone is falling for the same trap. 66% seems insanely high confidence. What are other plausible explanations for the anthropic principle? One is that AI will kill everyone, and population grows exponentially until roughly that point, so we are still more likely to be alive today than in 4000 BC.
Scott's arguments have shifted me from "10% simulation likelihood" to maybe 35%, but I still see substantive disproving facts:
1. Where's the bugs? Every simulation we've ever created encounters edge cases. The greater its scope, the more inevitable the bugs. The utter absence of documented bugs, having now persisted through 20+ years of ubiquitous access to photo/video recording, seems implausible if this were the creation of a being.
2. Where's the easter eggs? Having gone to all the work to create a 100% bug-free world model, with such varied flora/fauna, why aren't there more phenomenon that defy explanation? Wouldn't it have been more informative to introduce some version of aliens by now, rather than simulate a billion Africans in squalor?
3. People like Elon & Scott have orders of magnitude more reason to believe they live in a simulation than the rest of us drones, yet the world is made of 99.99999999% useless-to-the-singulary drones. Why consume such processing power simulating no-ops?
Even though I can't get past the absence of bugs as proof we're not in a simulation, the idea on balance holds great appeal. Bequeathing significance to these seemingly meaningless lives, I intend to start using the possibility we're in a simulation as a partial answer to "why give a damn?"
Not Scott, but I have a similar belief and likelihood, and these are my takes:
1). When I look around, I see nothing BUT bugs and easter eggs!
*Bugs:*
* The Fermi Paradox - honestly, if WE exist, so should a bunch of other alien species, but we’ve seen no signs of them, or their media, or any significant stellar engineering, and so on.
* The universe being quantized isn't suspicious? Dark matter? Gravity vs strong and weak forces and magnetism? Almost as though simulating full-depth space, time, and energy in fully "theory of everything" aligned ways was too computationally expensive.
* The universe is huge, but conveniently set up so it’s expanding fast enough that most of it is “outside of our light cone” and so the vast majority of it is unreachable and doesn’t have to be rendered from our POV.
* Our universe existing at all is extremely dependent on getting a number of “universal constants” juuuuusssssttt right. If they weren’t all right to within a tolerance of many decimal places, our universe wouldn’t exist, wouldn’t have clumps of matter in it, wouldn’t have stars in it, wouldn’t have planets in it, wouldn’t have life in it, and so on.
* Not just that, from Zheng and Meister, The Unbearable Slowness of Being: Why do we live at 10 bits/s? (2024) - our entire perceptual lives are extremely rate-limited, to the extent that “even if a person soaks up information at the perceptual limit of a Speed Card champion [18 b/s, twice the human average], does this 24 hours a day without sleeping, and lives for 100 years, they will have acquired approximately <=4 GB of data.” The fact that our entire perceptual lives fit easily in a trivial handful of GB is a strong argument that we’re simulated, IMO.
*Easter Eggs:*
* Our universe seems to run on math. We’ll discover or invent whole reams of math, and later find that it perfectly describes how spacetime curves in the presence of matter and energy, or some such. And this happens *everywhere,* in all domains of life and existence, repeatedly. Ridiculous! Almost as if our universe was made of code or something, and they just plugged in existing libraries and functions that are maximally elegantly compressed, and can unfold into certain universal configurations.
* Qualia and consciousness itself is probably at least some evidence that we're simulated. Why should experiencing things feel like something internally? Why would coming from a long line of organisms that experienced things as part of the feedback loops necessary for goal directed behaviors ultimately end in an organism that does those AND actually feels things internally? It's totally superfluous, unless there were some property of the simulation that was important or desirable to outside observers. And we can see why qualia might fit THAT bill.
On your points 2) and 3), this feels a bit to me like complaining that there's a billion ones and zeroes in our algorithms or compressed files. If you're simulating "the global civilization that produced the hinge of history," a sim with full versimilitude is going to include the billion Africans and billions of non-Elon, non-AI researcher people, because the very specific ordering and quanitity of those ones and zeroes matters a lot to the final outcome.
To your closing point, I wholeheartedly agree - if we ARE a simulation, then it would behoove us to live lives as entertaining and diverse as we can realistically manage, because the fact of qualia gestures at some potential alignment in the concepts of experience and entertainment between us and our simulators.
Scott, what is your take on Dyson spheres or swarms. I forgot the median timeline from the first Curve event, but it felt preposterously short. I think Daniel said 5 years at some point.
I would lump that in with the "Bostromian superintelligence" point. If an AI can do 100 year of tech progress in one year, seems like maybe the next year it can do 1000 in one year, and surely Dyson spheres are only a few millennia away.
I can't speak for Scott, but I'd argue that Dyson Spheres are basically impossible (Dyson Swarms may or not be possible due to the flexibility of the definition). Yes, the laws of physics do not preclude Dyson Spheres outright (probably); but the realities of engineering do.
EDIT: As much as it pains me to say this, I feel the same about interstellar travel.
Strictly speaking, a Dyson Swarm is a subtype of a Dyson Sphere. You're thinking of a "Dyson Shell".
By 2034 analog will be dominant and forms of waves external from our body will be semiconscious.
Part of me has been very pessimistic about all this lately. Like what are the chances that it plateaus very soon (cost and training data vs real economic utility), which gives us some combination of mass unemployment and inequality but no singularity, and it traps us in current year neoliberal consensus reality?
Or the popular view that it is a bubble is true, we here have just been steeped in sci-fi for too long; markets correct shortly and this is actually our one max effort shot to get there as a civilization and may fail, and not get another one?
I don't think that's such a bad scenario. If AI plateaus then it remains as basically a normal technology, which will disrupt our economic and political systems but not entirely break them, meaning that we will have time to figure out solutions. For example instituting a UBI, or increasing employment in the niches AI isn't great at. The sooner it plateaus the more of those niches there are.
The scenario that scares me more is rapid self-improvement leading to humanity's irrelevance and potential extinction. I don't have much faith in our ability to manage alignment in a fast takeoff situation.
I did my mourning two years ago; my median timeline for loss of control/extinction is 2029. At this point the limiting factor seems to be the greed and hubris of the CEOs, and I don't believe even a credible warning shot will be heeded.
I used to be extremely skeptical about your views on the AI apocalypse, but that was before I started interacting seriously with Claude. I remain a little dubious, still feel you've been drinking too much Kool-Aid, but it no longer seems ridiculous. I mean omg you might be basically correct.
Kind of surprised to hear that, I don't think Claude is good enough to justify believing any of this on its own, but I'll take it.
I lack your AI chops, but one month of talking to Claude has been a real wake-up call for me. BTW, it turns out that Claude is very impressed by you, even if you're not too impressed by him. Given your worldview & priors, your probability calculations sound reasonable, though I still think you're over-alarmist and way too precise in the face of known (and unknown) unknowns. Keep writing; I really enjoy reading your essays.
I really struggle to understand what people mean when they say "post-scarcity". Clearly something is always scarce, because the machine of economic growth keeps churning until it there is something preventing it? Prices of everything not scarce drop to zero and prices of everything scarce rise indefinitely.
Either that or you pick a definition of post-scarcity which clearly includes today, which is also reasonable, but clearly not the way you are using it.
Post-scarcity is always a relative term, because scarcity necessarily exists when we divide the Universe into more than zero agents or other than as one fully fulfilled agent. So post-scarcity, in my view, would be a state where any particular current good or service drops to near or essentially zero in price. Currently we are humans in AD 2026.
If post-scarcity is a relative term then it doesn't seem like a useful term at all.
In practice, as some things become less scarce, we always find the scarcity is somewhere else instead.
Nowadays we already live in a damn-near post-scarcity world in many ways. The cost of food used to be the majority of a normal family's budget, with the cost of clothing way up there as well. Nowadays these costs are pretty trivial, you can minimally feed and clothe yourself for just minutes of work per day at minimum wage. But now the bottleneck is somewhere else, and the cost of land (or rather the cost of buildable land within commuting distance of a major city) has skyrocketed to soak up all of everyone's leftover money; the scarcity just moves somewhere else.
If the cost of manufacturing goes to zero then the cost of land, energy and raw materials will remain.
Rich is a relative term, and still useful.
Post scarcity to me means the things most commonly scarce today: big homes, free time, healthy and tasty prepared food, servants to do your menial labor
And if you want to stretch to include things that are apparently getting less scarce: ability to realize your artistic vision in movies or art or videogames.
Just because _something_ will always be expensive doesn't mean you can't get rich.
"Big homes" is the best example why no economy will ever be able to provide everything desirable to everyone. Future-you might be rich compared to today-you, but the future mega-rich will always be able to outcompete you for the biggest homes on the nicest beaches.
Post-scarcity from current PoV isn't possible without various kinds of land expansion, most commonly imagined through Space exploration, so that's a bad example
Mega barges in the ocean are going to be orders of magnitude cheaper than space options, likely for a long time.
There are currently 4.5 acres of landmass per person. Given some people want to live with spouses, their kids, and in vertical apartments, every human having 4000 square feet of indoor space is absolutely possible.
Obviously rich people will have more by definition. Doesn't mean you can't have a lot more than even the wealthy have now
>There are currently 4.5 acres of landmass per person
There are not. You just took the entire landmass of Earth and divided by number of people. That can only be a crude upper limit, because it includes all kinds of uninhabitable and undesirable places like mountains, deserts, jungles; and also all areas for farming, industry, business, leisure, and whatever else.
Solar panels are doing some impressive things to the cost of energy, and sufficiently cheap energy solves a lot of material problems too - basalt and seawater contain most of the minerals we need, hard part is distillation.
As for land prices, overwhelming consensus among economists is that Georgist LVT would solve the problem, it's just a question of implementation details and political opposition from entrenched rent-seekers.
It's a forecasting term?
Yes, some things aren't really scarce anymore.
> If the cost of manufacturing goes to zero ...
This implies that the manufacturer has access to infinite energy/matter/space/time, which seems unlikely.
I think it is just weird to define "knowledge work" and "diffusion" as totally segregated concepts at this point. Isn't it weird if you can declare "knowledge work 90% solved" and "diffusion at 0%" simultaneously? Even if you personally understand that means "nothing in the economy changes", it's unlikely a layman reading the phrase "work is mostly solved" does; it semantically smuggles various political assumptions to even talk about work being solved at that point.
Not really? "Machine faster than a horse" was 100 % solved engineering problem, but it basically took a century for global horse population to start declining.
I don't know about this example: the important tech problem to solve was a machine that was as versatile as a horse, price competitive with a horse, not too much bigger than a horse and operable by one person like a horse. That was a tech problem, not a diffusion problem.
All of those features are subservient to "a machine that can travel faster than a horse" existing in the first place. It's impossible to refine your automatically moving machine to be cheaper and more versatile if they literally don't exist.
As of right now, smarter than human machines do not exist. It does not matter how cheap you make compute or electricity. If we discovered a magical portal to elemental plane of fully assembled GPUs and fusion reactors, making compute and electricity free, smarter than human machines would still not exist until AI technology progressed further and actually invented them. If you go to the majority of knowledge work jobs and tell them "I will pay for all of your AI costs if you immediately fire 90% of your humans and replace them with AI" most would not take you up on it, and if they did they would have a considerable decline in quality.
The most important tech problem is to invent the thing in the first place. The second most important tech problem is to make it cheap and versatile enough for economic adoption. And one has to come before the other, and there will be a time delay between the first and second.
Funny you mention that because "machine faster than a horse" also produced one of the biggest economic bubbles in history and the railway bubble is used as a common point of comparison by many AI skeptics.
Yes. The horse population eventual decline is the prebunk (depending on your perspective) to that
It's a useful distinction, aside from the simple time lag of people adopting tech, different jobs can provide different barriers to adoption. If for example you created a LawyerBot who was unambiguously as capable as a human lawyer, it still wouldn't be authorized to practice law. You might in fact open yourself up to criminal charges trying to release such an AI to the public. These are distinctly diffusion challenges separate the technical challenges of getting the AI to do the knowledge work.
Don't get me wrong, I agree that diffusion speed is a Real Thing on the map.
The issue is that I think there is a third thing in-between the space of "this job task passes eval" and "the institutions / people / lethargy are blocking off deployment". Probably there are many ways to think you have 90%'d all knowledge work without actually 90%'ing implicit requirements everywhere
> Define AGI as AI intelligent enough to do 90% of knowledge work jobs.
This definition is vague to the point of being disingenuous, which is a problem since the rest of your argument hinges on it.
What do you mean by "knowledge jobs" ? As we speak, many humans are still performing basic data entry, answering tech support calls according to a (natural-language) script, collating spreadsheets, and so on. Most of these jobs could be performed by a Python script and a SQL database; I would argue that 70..80% of them could be performed by ChatGPT. However, collating spreadsheets really quickly is not a skill that will let any agent, be he AI or human, recursively self-improve to near godhood.
Also, what do you mean by "90%" ? Are you counting the categories of jobs, or individual workplaces ? For every world-class scientist, there are probably 90 middle managers/HR paper pushers/legal compliance officers/etc. And yes, LLMs could probably do many of these jobs right now; but, again, the ability to spit out the correct form on demand is not enough to conquer the world.
But it gets worse:
> Define the Bostromian superintelligence gap as the time between AGI and an AI which, if given independent control of resources like labs and factories...
You are once again falling victim to the same blind spot as many other AI-doom proponents: you are transitioning smoothly and effortlessly from "knowledge work" (manipulating symbols) to controlling physical resources, such as labs and factories (manipulating objects in the physical world). But this transition is far from smooth ! Present-day LLMs are not intelligent enough to even walk down the street unaided, let alone construct a chip fab from scratch. Self-driving cars are hitting a wall, both metaphorically and literally. Granted, there exists a lot of factory equipment that can currently be automated, and in fact is already automated: CNC lathes, PCR machines, etc. But you are positing a world where LLMs are creating such tools from scratch and recursively self-improving them, which at present they are utterly incapable of doing (without human assistance).
And it gets worse:
> The easiest way to reach this point is for AI to become superintelligent at persuasion...
Do you have any evidence that super-persuasion (or indeed any other super-capabilities) are in fact possible ? Yes, some people are better at persuasion than others; but it does not automatically follow that there must exist an agent who can persuade everyone into doing anything it wishes all at the same time. That's like saying that some people can run faster than others, therefore there must be an agent who can run faster than light.
> The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time.
My objection is similar here: how have you determined this ? If this were true, then we'd expect many if not most human experts to become world-class leaders in their fields, but this is not true. Most physics teachers never become Einsteins; and in fact on the planet of 9B people, there are only a handful of "top geniuses" -- and you're saying that bridging that gap is virtually trivial ?
Look, I don't think I'll be able to persuade you to my side; I understand that you take these assumptions for granted and that rejecting them seems utterly foolish to you. All I'm trying to do is illustrate that they are by no means obvious, nor that rejecting them necessarily makes someone an idiot (granted, I personally could very well be an idiot for many other reasons). All I'm saying that if you want someone like me -- i.e. a pretty average normie -- on your side, then you need to do a lot more than simply state your convictions very strongly. I've got other people stating their convictions at me all the time; I need evidence and specificity, not mere vague assertions.
I don't think Scott means superintelligent at persuasion as "persuade anyone into doing anything", just that it's much better than the best human at persuasion.
Fwiw, I think the extent to which an AI can influence a human will be bottlenecked by the human's own abilities. "A smart person rarely loses an argument. A dumb person never does." However, the status accrued by being right a million times and being super useful will move mountains for it.
> I don't think Scott means superintelligent at persuasion as "persuade anyone into doing anything", just that it's much better than the best human at persuasion.
This is what I meant by "vagueness". Ok, it's "much better" than the best human on persuasion -- but what does this mean ? Is there some Persuasion Quotient, and if so, how is it measured, and how many points does one need to be "much better" ? In practice, what can someone like that actually tangibly achieve in the real world (other than "persuading anyone into doing anything") ? Again, how did you arrive at this prediction, assuming that it is even quantifiable ?
> However, the status accrued by being right a million times and being super useful will move mountains for it.
Will it ? Humans are not persuaded by other humans who are right a million times; this is why many (if not most !) Americans do not believe in the efficacy of vaccines, or the Moon landing, or many other such things that are touted by the accursed "experts". And being super useful arguably ends when persuasion begins -- otherwise, you wouldn't need to persuade at all. I don't need to be persuaded to drink water or breathe air; I do need to be persuaded to drink the latest healing energy concoction while breathing artisanal O2 from a canister.
Persuasion isn't about sequences of words anyway, it's about trustworthiness. You're likely to persuade me of something if I think that (a) you are likely to know the right answer and (b) you're likely to be telling me the truth, either because you're particularly honest or because you have no incentive to lie.
"This is a great car, you should buy it" is pretty persuasive coming from my tame mechanic, it's very unpersuasive when it comes from the dealer who is trying to sell me the car.
My calculator is a super-persuader, because every time it tells me something I believe it. Zero skepticism, I believe everything that Mr Casio has to say about arithmetic up to ten significant figures.
Then it seems that you can only be persuaded to believe something which you are already nearly certain to be true. This is a pretty good epistemological stance, but it has limited applications as far as a (hypothetical) malicious super-persuader is concerned. He can very easily persuade you that 2+2=4, but he can't persuade you to give him all your cash.
>Persuasion isn't about sequences of words anyway, it's about trustworthiness
It's about both. If sequences of words were zero percent effective, advertising wouldn't work.
>In practice, what can someone like that actually tangibly achieve in the real world (other than "persuading anyone into doing anything") ?
Establish a worldwide Fourth Reich, is the usual argument.
Even one of the best persuaders in history could not do this; he relied on force of arms instead, and was ultimately unable to persuade enough people not to oppose him. Sure, it's easy to say "yeah so the AI would be able to do this after all", but the problem is that the Third Reich was a product of its time, and largely collapsed under its own weight shortly afterwards. The more people you rope into your cult of personality, the more difficult it is to keep them all persuaded -- exponentially so; and they have to be 90% on board with you to begin with. The Kim dynasty in North Korea is (IMO) so successful precisely because they keep their ambitions small (and of course they are being useful to larger and more powerful patrons).
Yeah, I'm also pretty skeptical about this stuff, but still, you can't deny that his infernal (secular) majesty was one hell of a proof of concept, tangibly impacting the real world to an absurd degree.
I can deny this -- not sure if I *would*, but denial is at least plausible. The "great man" theory of history is hotly disputed; it is entirely possible that the world was ripe for German-led uprising, and thus someone would've led that uprising eventually with a very high probability. We're just especially unlucky to get the guy with the terrible moustache.
I dunno how much this proves about some abstract innate "persuadability" that doesn't need some interaction with context. I don't care how persuasive Hitler was, I think he would have been more or less exactly as unsuccessful at persuading the German people to accept the humiliations is Versailles--the message he persuaded people of can't be divorced from how successful he was at persuading people! And of course, neither can the context--the Nazis octupling their percentage of the vote before and after 1929 is a pretty big sign that Hitler couldn't just turn his persuasive powers on the German populace and expect success--he needed fertile ground for his message, which the great depression provided.
I don't think this means there's no such thing as persuasiveness, but I am would expect it to face pretty quickly diminishing returns without the ability to shape the context in which you are trying to persuade. Now, it is of course the case that super intelligent AI is already a pretty wild context where people might be more persuadable of all sorts of things, and that super intelligent AIs will have other means of shaping the context in their favour, but I still expect most of the juice in an "AI becomes they cyber fuhrer" scenario to be about material changes on the ground, and only marginally because the AI is good at persuasion specifically.
It's easy to make such a definition (knowledge jobs) not vague. Just use the classification of work by some labor ministry, eg. US Bureau of Labor.
Well, they are intelligent enough if there are next to no obstacles on the street. But such LLM can still pay humans to do physical bidding, and we've already seen limited early versions of this.
You only need normal persuasion to get a large number of people to do serious damage. See the history of conflict for spiritual-ideological goals. Achieving super x is easy for programs who are already human-level x because they have fewer needs and physical limits. You do not need even close to unanimous agreement to have power over the planet. Analogically, we can look at currently existing countries, such as France, where President Macron only has 10-20 % approval rate.
No, we would not necessarily expect most arbitrary experts to become world class. What we would necessarily expect is some metric which shows easier attainment, yes, but not that particular one. We do, indeed, observe some other- the ratio between the number of base human skill in x and expert humans at x is much larger than the ratio of expert humans at x and world class humans at x. So it is still a lot easier, but not amazingly easier.
> But such LLM can still pay humans to do physical bidding...
Yes, of course; but so can you or I. If the LLM needs to pay humans to do the bulk of its work, then it can recursively self-improve in the same sense as human corporations already do. This indeed makes it a threat, but not an unprecedented one.
> See the history of conflict for spiritual-ideological goals.
Can you be more specific ? I would argue that there are approximately zero large-scale conflicts in human history whose origins could be traced back to a single super-persuader persuading people to do his bidding; but I might be mistaking your meaning.
> Analogically, we can look at currently existing countries, such as France, where President Macron only has 10-20 % approval rate.
If the LLM could achieve about as much power as Macron, then it might be a threat, but IMO not a major one (compared to the other threats humanity is facing).
> the ratio between the number of base human skill in x and expert humans at x is much larger than the ratio of expert humans at x and world class humans at x.
What do you mean by "base level" ? Most people cannot do most things at all; but then most humans can learn to do many things by watching a 10-minute YouTube video. Becoming an expert usually takes 4...8 years of focused study. Becoming a world-class genius requires... well, no one really knows what, but most experts never achieve this at all.
Scott had a lengthier discussion about what superpersuasion could be, whether it’s even possible, what an army of good but not magical persuaders might do, etc, to this point. It’s in his AI2027 impression article, I think there may have been a lengthier treatment somewhere else but couldn’t find it
And also, why 90%, and not 99.99%? I though that's what AGI usually implied.
I think Scott once stated(but now i can't find where) that an example of top human-level persuasion would be Mohammad , and a super persuasive AI would be above that.
So the AI would need to find some number reasonably charismatic humans who are open to its ideas(or can be bribed or manipulated into working with the AI), then use them as a front and have them become the most popular politician/activist/internet influencer/whatever whose words are implicitly trusted by a huge percentage of the human population and then subtly push its agenda through them.
Not sure how realistic this scenario is but it would be an example of "super-persuasion".
My first thought was also Muhammad but I think this also shows the limitations of the concept: the vast majority of converts to Islam were converted not through direct persuasion by Muhammad, but by military conquest and its associated social changes.
The growth of Islam happened mostly after Muhammad was already dead, and by methods only very indirectly connected to his persuasive power.
It's also pretty obvious that his arising in a frontier region between two massive imperial powers who had just finished exhausting themselves in a series of massive wars is not a coincidence: a similarly persuasive figure in Rome or Constantinople or Ctesiphon might have left much less of a trace on world history.
It's unclear to me how much we should attribute the rise of Islam to the persuasive powers of its founder, and even if we are generous there, it seems likely that persuasiveness isn't just a function of the native persuasive power of the persuader, but also the audience and the circumstances.
I don't think this point cuts entirely against the idea that AI will be able to have an outsized persuasive influence--I think it's quite plausible that the rise of AI will create an audience and associated context that is fertile ground for persuaders of all sorts, and maybe especially AIs--but I do think it casts some doubt that there's just a straightforward monotonic relationship between someone's innate persuasiveness and their ability to talk people into doing things: I think there are very strong interaction effects between that and each of: who they're talking to, what they're saying, and the context in which it's being said, and that in lots of cases, those other effects will swamp the "innate persuasiveness".
> I think Scott once stated(but now i can't find where) that an example of top human-level persuasion would be Mohammad...
I don't think that using a semi-fictional person as your go-to example reinforces the argument. There are plenty of books about fictional or exaggerated characters being super-persuasive, but these books generally rely on the reader's inability to verify that "and then everyone clapped" (as the meme goes) -- usually because the audience is likewise fictional and thus behaves in the way the author intended. To put it another way, the super-persuasion ability of fictional characters is more often than not an Informed Attribute: https://tvtropes.org/pmwiki/pmwiki.php/Main/InformedAttribute
I have nothing to add other than: diffusion is extremely hard and I can attest to it.
I know I present myself as a smarty pants know it all, but there’s just a global problem of leaders not knowing how to actually use and deploy the tech and who to trust and who not to trust that is still going to take years to work out. But I do expect that will be a hockey stick timeline.
Based on Fable we have reached the limit of this paradigm. If you really thought there was 25% chance of AGI you would be acting drastically differently.
How is Fable evidence that we have reached the limit of this paradigm?
What happens when they turn the simulation off but it keeps running due to Dust Theory? They may have already done this, in fact.
It's always seemed to me both Simulationism and Dust Theory ultimately rely on a free lunch in terms of matter and energy existing in sufficient quantity.
Happily, we see that free lunch instantiated in our universe and our laws of physics, and so too would our Simulators, presumably. But it is a necessary condition.
But yes, once you have that necessary condition and any infinity (any of Tegmark's 4 flavors), then Dust Theory can assure your existence.
Cosmology is a spectrum
Boltzmann Brain <------> Dust Theory
Can anyone explain to me what Dust Theory brings to the table that All Mathematical Structures Exist Theory doesn't? Also, it seems to me that both of these theories are falsified by the observation that our world is very regular, no?
Many Thanks for gathering your conclusions and reasoning on all these considerations into one post! One comment, re:
>Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.
As you wrote, superpersuasion, if it happens, might be before diffusion. If that happens, I would expect it to shrink the timelines for diffusion in at least two ways:
- A superpersuader is a supersalesentity. They should speed adoption of AI throughout the economy.
- A superpersuader is a superlobbyist. They should speed modifications of regulations (or even just the regulations' enforcement) to speed AI's spread.
Did I miss where power consumption and power cost become bottlenecks (or why they wouldn't)?
There’s a lot of earth’s surface still that we can ̶t̶i̶l̶e̶ ̶w̶i̶t̶h̶ ̶p̶a̶p̶e̶r̶c̶l̶i̶p̶s̶ cover with solar panels before we even seriously start competing with human use. Current friction is for economical, not inherent, reasons. Other problems seem less far off.
I mean, define "seriously"?
"Half of all new electricity demand in the U.S. last year came from data centers"
https://fortune.com/2026/04/20/us-data-center-electricity-demand-public-opinion/
I would call that pretty damn competitive.
It’s not zero-sum, though. Data centers being planned now include things like solar arrays on site. My point is - if they pay for generation capacity to be built up to match the use, and it’s not competing with me for useful land or polluting my air or water, why does it matter?
What do you think of this report? Does this count as "serious" competition?
https://fortune.com/2026/05/12/lake-tahoe-data-center-49000-residents-power-source/
This describes an economical decision. It’s not that new generation capacity couldn’t be built. It’s merely that no-one wanted to pay for it, and no-one was made to. That’s a policy decision, not a problem inherent to the technology. We are capable of making better decisions, and often do.
Would you define "serious competition" then, if it would be neither political nor economic?
I don't think this is a big deal. AI consumes power via data centers. Most localities don't like the idea of data centers consuming power from their grid, so the data centers are forced to use their own natural gas plants on site. This is expensive, but so far they've been able to meet demand, and the financial people aren't projecting it as likely to stop the data center buildout of the next few years. If the government deregulates solar or nuclear, that makes things even easier.
But also, if AI improves through recursive self-improvement, then existing data centers can run better AIs for the same energy cost.
I'll go out on a limb and predict this will become a significant bottleneck within then next 3-5 years. Let's see who is right!
AFAIK this is already a significant bottleneck today, and the only reason it's not a critical one is because it competes for this title with cooling. But yes, it will get worse in 3..5 years, assuming the AI bubble does not collapse before then.
FWIW, this showed up in my email as a completely blank email. Thought that was the message LOL
Only on the day Scott reaches true enlightenment.
> As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone.
I don't think this makes sense. Most people will be well within the future Lightcone of the dictator, and the dictator can just send Von Neumann probes to every solar system in the reachable universe, and so have a presence there. I can't currently think of a thing you meant to say here that makes sense to me.
I do think the rest of that section is correct and I am not very worried about negative outcomes from AI-enabled authoritarianism.
Hmmm...I think I was thinking of the dictator (sitting on Earth) having some whim, but people on Andromeda are protected from his whims (at least for the next million years). I agree that the dictator could institute a law code that is sent with the von Neumann probe that colonizes Andromeda, but this seems less bad for the normal reasons that governments of laws are better than those of men.
I guess the dictator could send an uploaded copy of himself to dictate to Andromeda, but I don't know whether real-world dictators would behave that way.
He could send a fork of his obedient AI, with unrescindable instructions to act the way it expects the dictator would want it to act. This cashes out similarly to an uploaded copy of the dictator, and the dictator is more likely to go for it because it doesn't bruise his ego to the same extent as creating an autonomous copy of himself — it just feels like appointing a deputy/governor whose loyalty is assured.
But also, I don't think you're sufficiently concerned about the potential badness of the "laws" that a functionally-omnipotent dictator could write. A selfish and/or paranoid dictator could, just as an example, forbid anyone but himself and his inner circle from getting access to immortality tech — and you bet he'd write that into the Von Neumann probes.
I was thinking that the dictator is probably at the center of the Dark Triad Venn diagram, and might pose a greater-than-usual s-risk potential for the rest of us.
Looking at the big picture, I can't share the optimism of the last section if human labour does indeed become obsolete.
The thing that has allowed humans to tolerate each others' existence (or rather, the existence of people outside their immediate clan) is that humans have economic value to each other. I am better off for living in a world that has strangers in it, because those strangers produce goods and services which we can trade.
But if we get to the point where the vast majority of the human population has no economic value and exist only to consume the value created by the robots, then humans no longer have a good reason to tolerate each others' existence. I won't try to predict the exact sequence of events but I think that once the interests of everybody become strongly aligned with killing everyone that they don't personally know and like then it has to lead, in some form, to a war of all against all.
I also think that humans having no economic value will be a major problem, for example it’s why UBI schemes are doomed, people will lack the leverage to make anyone else stick to such a thing once labor peace is no longer needed.
But I don’t necessarily think it would be a matter of exterminating everyone else, that’s always been an option even in ages where other tribes of humans did not provide value because your land could only produce X units of grain no matter how many laborers you threw at it. The people in those eras might have gone marauding, and some did, but there was a value to mutual peace and not having to be hypervigilant all the time. I think we will be less likely to aid others but no more likely to want to eliminate them than a medieval villager, which is maybe more than now but not insane.
Right, the idea of a population who mostly just sit around on UBI is horrifying. The minority who pay into the system will be very aware that they'd be better off without the masses. And the masses have nothing better to do all day than to sit around and agitate for their UBI to be increased.
So either you've got constant conflict between the haves and the have-nots, or you set up a clever system of repression to ensure that the have-nots don't complain too much.
One plausible equilibrium would be a Chinese-style system where your UBI is scaled to reflect your degree of loyalty to the system. In practice it might look a lot more complicated than that.
Current welfare systems involve a lot of people sitting around doing nothing, or causing pointless problems, because if they try to do anything legibly productive or profitable, some means-tester might decide that means they're too successful to qualify for benefits, and those benefits being otherwise judicially untouchable means they can't be meaningfully held responsible for any messes they make.
Proper UBI would have neither of those problems.
"One plausible equilibrium would be a Chinese-style system where your UBI is scaled to reflect your degree of loyalty to the system"
Another is where it is scaled to reflect your loyalty to the Billionaire who is doling it out.
I mean it's the same thing, isn't it?
That's still somewhat overestimating the extent to which people's cooperation with the system is relevant or valued in that scenario. You really wouldn't have any ability to affect anything, they (whether that's tech lords, AIs themselves, or a combination thereof) will be impregnable and fortified, you will not have any ability to harm them or meaningfully affect their position in any way, and they don't NEED anything from you. So while they would prefer to not spend resources on security, it doesn't affect them enough to be a significant motivating factor in anything, you are about as relevant to them as a platypus. I think it'd be much less complicated, not more, why set up an elaborate system rewarding the loyalty of people who you will never need for anything and who can't harm you?
Their incentive to toss up another thousand square miles of solar panels or a chipfab complex is going to consume almost everything pretty quickly. As Yud has pointed out, if you think they're going to spend some crumbs on you and me merely because they have so much it's trivial to do so, try writing Warren Buffet and asking for $100. The only two ways to have leverage are being needed by others, or being able to commit violence against others, gotta figure out one of those two or you're toast.
> The only two ways to have leverage are being needed by others, or being able to commit violence against others, gotta figure out one of those two or you're toast.
What leverage do the ~270k lives saved by the Against Malaria foundation have against said foundation?
What leverage did starving orphans and mothers have against the Church who took them in as foundlings and nuns for a thousand years?
About a third of the billionaires in the US have signed the Giving Pledge, and all it really takes is one, per Scott's post about Dario's moons.
To your solar panel point, why haven't we clear cut every forest to make economically productive farms and ranches? It's not like the trees have any leverage over us, they just sit there sucking up resources! Buncha welfare queens, what's next, passing out Cadillacs and food stamps to them?
And yet, we still don't live in maximal boot-on-face industrial hellscapes, and there's little reason to think we'll end up in those versus futures more similar to our past and present.
Teddy Roosevelt set aside an absurd percentage of the western states as federal parkland, and the Dept of Interior will roll a tank up to your office. That saves the trees for the moment, but the future tech oligarchs and/or their AIs won't have to worry about the government or the public will for the reasons I gave. They will not need buy-in or cooperation from normies to maintain their position, and normies will be impotent against them.
Examples of seeming charity are of course always easy to turn into self-interested acts. Even if only the pure ideological motives are credited, all your examples involve people in horrible positions both before and after the aid. The nuns were treated like absolute garbage in most cases and were fully dependent on a system with strict rules that they had to profess total allegiance towards in body and soul. And congrats to the African children on not dying from malaria, now they can grow up in their house built from garbage and dismantled cargo containers and become child soldiers!
Oh now you may object that surely with even *more* wealth and a magical-anything-box in their possession the ideologically charitable tech lords will give the African children nice little cape cods, a Levittown on the Limpopo! But the ultimate problem remains, they have no leverage, they are fully at the mercy of what other people want to give them. They are neither valued for their own merits, nor can they by threat of force extract anything more.
Right now very few people are truly in such a position. An old poor disabled woman with no economic productivity still has some influence with friends and family and the public at large would be angry to see her left to starve on the street, her chip is that she could incrementally weaken the perception of legitimacy of the system that would allow that, and the system still needs that today. The African children could grow up into men who launched a revolution, such a thing is still possible today. But in the AI future revolutions will be impossible, and the perception of legitimacy is totally unnecessary as nobody with power needs anything from the powerless. That is fundamentally different, it's a break from all human history since the invention of agriculture.
I think a lot of people, including myself, don't need other humans to have economic value to tolerate their existence. I generally am happy with the idea there are other humans running around who are happy to be there (provided they aren't taking away other people's happiness).
Of course, I agree there are some potentially serious social issues that will surface if/when we get to a "post-jobs" economy. But I'm skeptical that killing the unemployed is likely to be popular.
Pausing has enormous opportunity costs. I'm strongly against this. Alignment is at least interesting as a research project now that we have some idea of how this things work, and a less good idea of how they might work in the future. But to the extent they cause any slow down, which I think is unlikely, it'd also have enormous opportunity costs.
Pausing only has enormous opportunity costs if you expect AI to be enormous. If you expected AI to (for example) be able to dismantle Mercury and build a Dyson sphere with in in five years, I think you would be freaked out enough to be willing to slow down and make it twenty years.
If AI will just make B2B SAAS companies 10% more efficient, then I agree it's not scary enough to pause over, but then the opportunity costs of pausing are commensurately smaller.
There's a middle ground between those, where pausing means a lot of people die from e.g. cancer, malaria, or various collateral damage from logistical and diplomatic incompetence, which the un-paused AI could have cured a few critical years sooner.
Dismantling Mercury and building a Dyson sphere in five years is absurd just from a thermodynamic standpoint. If you're defining anything less than that, i.e. all physically possible outcomes, as "not enormous," there's something wrong with your scale.
10% extra efficiency for B2B SAAS companies is still a very big deal with large negative expectation for pausing. I agree that to the extent AI could be a bigger deal than that, the costs become commensurably larger, and I expect the expectation of pausing becomes commensurably more negative. If anything I'm freaked out at the 15 year of lost opportunity there.
I don't quite understand the definition of "AGI = AI intelligent enough to do 90% of knowledge work jobs." What's the denominator here? Jobs that exist today, or jobs that exist in the future?
The problem is that the requirements for a "knowledge work job" will be constantly redefined to include whatever it is that AI can't do. We already see that happening in software engineering.
So do you mean, able to do 90% of the jobs of today? In which case sure, but who cares.
Or do you mean 90% of the jobs of the future? Because how does that even make sense, we don't call it a "job" if it's an AI doing it.
Or do you mean something like, the total number of knowledge workers drops by 10x?
Furthermore I suspect there's a lot of individual jobs where AI can do 90 percent of the job, but you need a human for the other 10 percent (deciding what needs to be done, talking to other humans about it, checking the AI's work and taking responsibility for the finished product).
And if that one human can now handle what used to be ten people's worth of work, without capturing ten times the pay, while the objective value of that work to the wider economy stays the same... it starts making sense to hire a lot more people for that role, to try things which previously weren't worth the effort.
Furthermore I suspect there's a lot of individual jobs where AI can do 90 percent of the job, but you need a human for the other 10 percent (deciding what needs to be done, talking to other humans about it, checking the AI's work and taking responsibility for the finished product).
Does Scott believe there’s a 66% chance we’re in a simulation, full stop? Because that’s how I’m reading that last part.
If so, unfathomably based and that’s where I’m trending too. I always think people like Scott and Elon have an even better reason to think they’re in a simulation than the rest of us, so it’s good to see Scott’s rationality take him to what I see is the natural conclusion for him.
Wouldn't they only have more reason to believe they're in a simulation if they believe the rest of us are p-zombies??
Yes. It’s knowledge we can disprove from our points of view but they can’t.
I don't think a binary pzombie or not pzombie flag is the right way to think about it. More like how much processing power they're spending on each person, or something like that.
Would it really be more efficient to somehow simulate individual consciousnesses, with different levels of fidelity? Wouldn't you learn more about base reality by running a straightforward physics engine?
The Simulation Argument always sounded like a rather silly non-starter to me. Is the Simulation so perfect that we can never detect whether we are living in it or not ? If so, then by definition it doesn't matter, and you cannot make any predictions about it. If not, then there must be some flaws in the simulation, i.e. some experiments you can perform whose outcome would be different if we lived in the imperfect simulation vs. the real world/perfect simulation. In that case, the time for me to believe you would come when you can perform these experiments (and get them replicated). Until then, your "Simulation Argument" competes with String Theory, Quantum Loop Gravity, and other such purely mathematical models -- and if I were somehow forced to choose one to believe in, Simulation Theory would lose simply because the others have better math.
In general, the argument behind the Simulation Argument follows the form of, "I can write down a mathematical equation that implies X, therefore X is likely true". There are lots of arguments like that, purporting to prove the existence of everything from gods to alien visitations, and none of them are persuasive to anyone who doesn't already believe in them.
Topical:
https://www.smbc-comics.com/comic/proof-6
This post made a friend (an Ivy league undergrad at a very good Ivy) reconsider suicide. 2027 timelines felt "too soon" for them to do anything except face the future helplessly. AGI possibly arriving in the early 2030s feels like enough time to have a chance at controlling their destiny in some small way.
Thank you very much for this post; I think it made a meaningful difference to their mental state.
(I feel very bad for the world's 14 year olds who will be in the same situation 3 years from now.)
Uh, this isn't the main reason why you shouldn't commit suicide, but I would tell your friend that suicide seems like a really bad response here for even more than the normal reasons. If you model the AI transition as a 50% chance of death, and a 50% chance of utopia, then suicide doesn't decrease your chance of death (in fact, it increases it to 100%), it just means you don't get any chance at the utopia outcome.
I think usually people considering suicide over world events are not really doing it over politics and are normally depressed and using world events as an excuse. I understand this is going to come across as condescending, but for what it's worth I would inevitably-condescendingly urge your friend to seek normal psychiatric care. If they've already done that, they can try the second to nth-line stuff listed at https://lorienpsych.com/2021/06/05/depression/
If you model the AI transition as 30% death, 30% utopia, 30% immortal mouthless thing that must scream, and 10% ???, suicide might be a more reasonable choice. Even more so if you read "utopia" in the ironic sense that it is basically always used in speculative fiction, or if you read the visions of "utopia" actually being bandied about by accelerationists and feel that these would be dystopias by your standards.
People commit suicide because they believe there are fates worse than death. AI has the potential to create fates many people would expect to be worse than death. To steal a quote, "They have trapped me in a nightmare that is someone else's dream".
I'm planning to stick around for a while and see what happens. But I absolutely understand why some people might not.
Yeah, I'm three years from retirement and reaching for the popcorn. I just had my first remote meeting with the corporate AI tiger team that's gonna take over the internal software that I've been working on for the past 30 years.
Could you give an example of an immortal mouthless thing that must scream scenario that could make you think twice about sticking around? The forced immortality bit makes it hard for me to come up with a concrete scenario.
This is sort of tangential, but what do we think about the nature of intelligence and rationality? What basic mental functions are they evolved from? Do they come from concern? Condescension? I've been trying to understand the social mechanisms of it and why it evolved. What do we think the first words were? The reason I'm thinking of it is not just because of AI, but intelligent people's relationship with the new and unknown. I'm hearing a lot of negativity from smart people about all sorts of novel things: AI, social media, genetic editing, etc. It just seems consistent enough that there may be a psychological explanation to it.
Best question yet on this comments section. I won’t pretend to actually know the answer, but if I was allowed to riff…
I think intelligence is about the ability to solve problems in complex and uncertain environments. I see the key to human intelligence as our ability to extend our imagination (what Thomas Suddendorf refers to as pretense) way beyond that of other animals. This is what allows us to imagine time travel (replaying past scenarios and anticipating future ones), extensive planning, practice and recursive actions for future outcomes (such as building complex tools or preparing for next years harvest).
We can also imagine what other people are thinking and feeling, and we can imagine physical laws and natural rules. Perhaps most importantly, we can imagine shared frameworks and norms which contribute to our ability to solve the problem of cooperation with other creatures, some of which can also use their imagination.
When I think of a being with vastly better imagination, I think of it as being more both more curious and more compassionate. Not just of humans but of all life and all existence.
I think a lot of people assume it will be destructive if unaligned. My guess is that it will be better at coordinating and more productive in all senses of the word than anything we can imagine.
But I am sure I am wrong in part or whole.
As for your latter question, about the negativity towards novelty, this is probably explained by human's tendency towards risk aversion. Being somewhat risk-averse makes a lot of sense because it keeps you from getting killed, which evolution favors. A new technology like AI or gene editing are full of risks, and our natural instinct will be to avoid it. Smart people are not immune to this effect (arguably, it's *correct* to be risk-averse by default). It can take some careful analysis and an open mind to overcome the natural reaction--and lots of people are too lazy (or otherwise preoccupied) to do that.
There's also the cognitive bias of motivated reasoning, where you look for reasons to justify the conclusion you want, rather than the true conclusion. Smart people are not immune to this bias---in fact, they can be more susceptible as they can come up with more creative arguments. (Incidentally, I think this relates to the difference between rationality and intelligence: rationality is concerned about coming to *correct* conclusions.) Lots of smart people are going to be reconsidering their careers, if not their self-worth and their place in society, if AI displaces knowledge workers. So, there is plenty of motivation for smart people to find and exaggerate flaws in AI.
It took the latest ChatGPT 5.5 model five attempts to put the correct number of significance stars in each cell of a table I was making the other day. I even showed it the number of stars that were supposed to go in each cell.
When I ran into some obviously bad observations in the data, it told me to just delete them. The correct solution was of course to go back and find out why those observations looked bad to begin with.
Things like this come up over and over. I see too much emphasis on "knowing" and not enough on "doing." And to that end, I suspect there is an optimism gap between people who regularly use LLMs to help them with their technical work, and routinely run into its limitations; and those who don't.
The papers reporting on successfully working with LLMs on actually challenging problems (not in the training distribution) always have a lot to say about how a lot of effort was needed to work around the limitations of using a system that had the wrong distribution. If Scott had some free time (seems unlikely for several years) then he could do a deep dive with an LLM into some aspects of psychiatry that are not well understood but that his practice has lent him deep insight into, and he would probably bump against the same issues. Gwern has clearly bumped into the "dragons be here" part of the LLM map where poetry lurks, and I think most people have something they understand really deeply that has little training data and this shows up in LLM limitations in that region. However, most LLM use is wandering around the terra cognita and when the LLM veers into incognita one only notices this if one actually has real expertise there that contradicts what the LLM generates.
I’m going to ignore the AI world takeover arguments. There are broad jumping to conclusions here.
The economic effects are possible. Even with what we have now. Without AGI or super intelligence.
> If you’re in one of the early industries to be affected by AI, you may have a very bad time before the economy can grow 100x or 1000000x. I wouldn’t describe this as a “permanent underclass” - it’s a subset of people, and their suffering is temporary - but it might be a very large subset, and it might continue longer than you can remain solvent. I agree it’s worth having savings ready to prepare against this scenario.
There’s a large leap there to the economy jumping to 100x or 1000000x. There are multiple rebuttals here, not least that the world doesn’t have the resources to do this. Nor can LLMs run factories. Nor can automation save you. Factories are highly automated anyway.
The underclass problem is real though, and very likely. What isn’t likely is that there’s a magic jump from a permanent large underclass to everybody being rich. Not in the present system.
The economy depends on demand. Reduce the number of jobs and you reduce demand, reduce high income jobs and the downstream effects are even more significant. Do all this and government income and ability to borrow also collapses, so where does the UBI come from? Buying shares in Google or Apple won’t help you, as demand for advertising and iPhones also falls.
It’s possible AI companies might soak up whatever profits are left but they also sell to customers - often the high income tech heads first for the chop - and corporations that will find demand falling. The latter may respond by replacing more workers with AI but that will just exacerbate the problem. Even AI companies will see reduced demand over time.
There is a branch of macro economics that denies this, claiming that supply creates its own demand, and economies somehow self correct. This neither explains the depression, nor the Engel’s pause, but even if it were true heretofore it’s not necessarily true in the future. There’s no guarantee that an increase in animal spirits (or more prosaically a reduction in interest rates) will encourage businesses to hire more workers when they can spend more on tokens.
The Engel’s pause is interesting. During that period - the early Industrial Revolution in Britain - output and productivity rose substantially, yet real wages for many workers stagnated for decades. Production increased, but the benefits were not broadly shared. Hence the rise of communism, socialism, anarchism and other revolutionary ideologies.
Classical economists, like Riccardo and Lasalle, at the time generally assumed that was the nature of things, it was the end of wage stagnation that surprised them. After all, if a factory becomes twice as productive, there is no law of nature that says workers’ wages must double. Many labour economists credit labour unions for increasing wages and thus aggregate demand. This is my preferred theory.
The power of unions though, depends on the power of labour to withdraw collectively from work. The unemployed have no such power. And while the Engels pause was significant enough, it didn’t create a permanent underclass, and wages stagnated. They didn’t fall.
So unless you find a way to induce demand - and a tax based UBI isn’t going to work when government revenue is falling - not only will you not get to 1000x the 2026 economy, you will be lucky to get to 1x.
I’m hoping to be proven wrong of course, and I’m open to suggestions.
I agree that the consumer economy can't survive this, but that doesn't mean production will stop or can't grow. Stuff gets made because whoever controls the AI wants it made, not because it can be sold. An economy that grows while routing around most people. And that's not great, why would such a system produce anything for us? At this point I feel like you've just invented Alignment from first principles.
Doesn't your argument prove too much? You're imagining an entirely separate AI economy with basically fake productivity which the mass of people cannot participate in. But if that were true, why can't regular people still contribute economically to each other, ignoring the AI side of the economy? You can still be a barista or make a wooden cabinet for "real" people with just as many hours of labor as you can today in that split economy future. So at a minimum, you wouldn't expect there to be a significant drop in economic output. In other words, human labor and productivity can always have value to other humans.
And to the extent the AI part of the economy wishes to trade with the human part, all the same supply / demand laws follow, and we'd expect an infinite profusion of cheap great stuff from lights-out AI driven factories.
Essentially, there's no future where the ability to produce a lot more goods and services much cheaper ends in LESS stuff being consumed, and less well being overall, because the human part of the economy can still internally produce and trade under the very same schemas and rules trade happens today.
That’s a very weird reply because you ignored the demand side of the economy, with some hand waving about baristas and cabinet makers.
> Essentially, there's no future where the ability to produce a lot more goods and services much cheaper ends in LESS stuff being consumed
Sure there is. One where the mass of people are unemployed and can’t afford those goods. The potential to produce something doesn’t mean it is produced. Every recession the productive potential of an economy stays the same, technology doesn’t reverse. Yet less is produced because there’s less demand.
> One where the mass of people are unemployed and can’t afford those goods.
My point is, why would the mass of people be unemployed if they can still provide value to humans? Even if AI's don't want to hire them, other humans can.
You can open a small non-AI coffee shop, and compete on ambiance, because AI-produced 5 cent coffee doesn't provide the same experience. And who buys your coffee? The people in all the other jobs like that, along with the people who made money from AI, and anybody getting UBI, and so on.
You say the potential to produce something doesn't mean it's produced, but it seems to me you're handwaving a mass destruction of demand, when if anything we'd expect demand to go up in a future where people don't have to work and material goods are cheap.
Let's say the AI side of the economy solely focuses on counterfeiting AI jobs in white collar companies around the world. A raging torrent of money comes pouring into the US from everywhere in the world - you think everyone is going to be poor in that scenario? You think we're not going to tax and UBI that?
Even if we didn't UBI (and we are a democracy), why wouldn't a lot of the people open up human-centric businesses to cater to all the millionaire plumbers and electricians and other non-white collar jobs that are suddenly making a lot more in a 2x-ed economy? And then those people catering to the non-white-collar people have income, and can buy the cheap AI stuff, and so on.
> when if anything we'd expect demand to go up in a future where people don't have to work and material goods are cheap.
Listen to yourself. You expect demand to go up when people are mass unemployed. This isn’t even wrong.
> A raging torrent of money comes pouring into the US from everywhere in the world
I don’t fully understand your scenario here, I think you mean that only American companies will dominate and therefore the rest of the world is unemployed. So this if it works only fixes the US.
> you think everyone is going to be poor in that scenario?
Yes. Except the owners of that capital. (And actually not even them eventually).
> You think we're not going to tax and UBI that?
No. Not to any helpful extent.
> why wouldn't a lot of the people open up human-centric businesses to cater to all the millionaire plumbers and electricians and other non-white collar jobs that are suddenly making a lot more in a 2x-ed economy?
This magic 2X economy has appeared from nowhere.
These millionaire plumbers won’t exist because nobody is employing them. I don’t think you get that - demand isn’t what people want it’s what they can afford. I was unemployed for a few months just once, during that time I didn’t hire plumbers or architects or anybody because i couldn’t afford it. I didn’t drink barista coffee either.
The economy, to an approximation, is humans selling to humans. Capitalists and companies can be ignored. Make enough of the humans unemployed and the others have nobody to sell to. An AI won’t need a coffee from a barista so it won’t be contributing to the demand side.
There are reasonable answers to my scenario - humans having enough comparative advantage to still be employed, tokens becoming prohibitively expensive, some other way of inducing demand that’s not based on tax revenues, but your response isn’t it.
A few small typos:
"a company need an IT department"
"are aren’t personally interested"
"it’s only a been a few years"
More importantly, this sentence seems orphaned/misplaced. Nothing but a new section follows: "Several people have asked me if, as a coauthor of AI 2027, I necessarily believe AGI will happen in 2027 or 2028."
I hate to obsolete myself, but I ran this article by Opus 4.8 and it also noticed all of the above and more, with very high signal/noise ratio: https://claude.ai/share/e6ee8a4f-8dfb-4cde-8d5c-e0b24fc72224
Nitpicking (hopefully helpful) aside, thanks for writing this. I really appreciate when writers I like occasionally take the time to write up summary/reference docs like this, rather than all of their writing being a sort of unending series of updates to their previous corpus.
In terms of “smart enough to do 90% of knowledge work,” do you agree the work remaining is more about agent scaffolding and specialization rather than the base model? IMO the “smart” part is definitely there for 99% of jobs.
This mostly all checks out to me except the 66% simulation one — seems way high. Even if you don’t believe “weird hypotheticals without any empirical grounding can be ignored” is a legit reason to dismiss the simulation hypothesis, it’s a weak argument on its own terms. Computationally, simulation to full fidelity is wildly inefficient — the amount of information needed to fully describe the state of a computer is way more than the amount of information the computer can contain. So a simulated universe would have to exist inside a computer vastly bigger than the universe. Simulations humanity have run to date have been way too low fidelity to support the evolution of intelligent life. So the assumption that universes simulate other universes simulating other universes seems extraordinarily low likelihood given what’s been observed in this universe!
You criticise the simulation argument, but I don't think you have read the paper introducing it. It answers your objection explicitly in its third section. That section is merely 3 pages long. See "Are you living in a computer simulation?" by Nick Bostrom
Yes, very fair — I had only heard about the argument as it has been repeated, so my dismissal of it was a little glib. I just read section III of the original paper, and he does engage with the objections I have to the argument. That said, I don’t find his engagement that convincing. It’s extremely hand-wavy about future potential technology, and rests on some very strong assumptions: for instance, the assumption that humanity succeeds in colonizing the whole galaxy and turning every star system into a giant computer is speculative to put it mildly. I also think that even with meaningful eliding of parts of reality that humans aren’t directly observing, the computational resources to simulate a world sufficiently convincing to us is still vastly greater than the computational power of an individual brain (because verifying consistency is computationally cheaper than generating). This all matters because the seriousness with which we take the simulation argument should scale with how much our observed world resembles the universe that the simulation argument depends on. We are very far from having enough evidence to think Bostrom’s future is a likely one, therefore we should not assign a high probability to the simulation argument being true
I used to think this way, but it kind of falls apart when you realize that from the point of view of any individual, almost nothing needs to be fully simulated. Foveated rendering of reality, essentially. Throw in the capabilities of even our own modest graphical models (gaussian splatting / NeRFs, etc) and it quickly becomes obvious that simulating a world for a single person would actually be quite achievable. Things get more expensive when you want multiplayer, consistency, etc, but even then it wouldn't be that crazy.
Not only what you say is right. But even if we assume that we are in a simulation, the probability that the human history is part of the data that the ones running the simulation are interested in is necessarily really small. You don't allocate that much compute to run a simulation during billions of years and spanning a least billions of billions of light years in space so that the simulation is considered finished when a tiny group of atoms in a single of point of this space act organically or improve their own development in an exogenous way. Otherwise the simulation would have been finished long ago by bacteria or at worst dinosaurs. We likely still have many thing to accomplish to complete the data, if we are even part of the data.
I have a question about the superpersuasion scenario because it sounds like magical thinking; to my knowledge, this is like grey goo, compelling in genre fiction but not necessarily feasible under the laws of physics. My only analogy here is to think about the guy who is handsome, charismatic, cultured, rich, well-spoken, athletic, whatever. He's quite good at persuading people (==women) to give him what he wants; does he succeed every time? Certainly not. Does thinking at machine speed or at a hypothetical superhuman intelligence level allow for this?
I completely agree. To be fair however, the usual counter-argument (assuming I understand it correctly) is that "thinking at machine speeds" and "superhuman intelligence" can overcome essentially any intellectual challenge -- since most such problems have solutions; solutions are achieved primarily through computation; and the superintelligence has effectively unlimited access to computation (by the combined virtue of being very fast, and also being able to devise massively efficient algorithms). Thus, assuming that super-persuasion is an intellectual challenge (which seems likely), and that it does have a solution (as is usually assumed), then a superintelligence would be able to solve it (and to do so nearly instantly).
Think of the guy who is very strong and knows karate. He's quite good at winning fights; does he succeed every time? Certainly not.
...now imagine a guy holding an AK-47.
A gun is a huge advantage in a fair fight, but super-persuader doom scenarios require overcoming social challenges that are wildly - and very much deliberately - unfair in the defender's favor. One guy with an assault rifle isn't going to have great odds storming even a pre-gunpowder castle guarded by as many troops as he has bullets, because some of those shots are going to miss, and the time he spends aiming to mitigate that is time they can spend diving behind thick stone walls, or shooting back.
It would be pretty weird to characterize being able to use a gun as "superkarate"--, it's pretty obviously not just "good at karate but moreso".
I agree a very intelligent AI might be able to persuade humans better than other humans can, but I think it's most natural to imagine that flowing through channels like "can make more credible threats/promises" or "can coerce you more directly using its robot army", not "figures out how to use The Voice from Dune so that it can achieve more persuasion per unit of verbal command".
Am I imagining them to be fighting on equal terms in the ring, or am I free to imagine the karate guy getting together with his five karate buddies to ambush the AK guy in a dark alley one night ?
Why is 100% success needed?
What are your thoughts on the moral obligations of individuals working on AI? Eg. capabilities researchers, those working on building data centers, or those tangentially related in the functioning of big labs but not directly contributing to capabilities (anything from financing to being a normal employee like a marketing professional or janitor)?
I'll insert my usual complaint about models being around a million times less efficient that humans at learning. Dwarkesh has made the point more eloquently than I can here*.
Having established that, I still cannot understand how anyone can look at the sample efficiency issue and then conclude that super-intelligence is plausible by just throwing more data at models. I know Dario believes this - that if you train a model on a sufficiently wide variety of tasks then it'll generalize to everything and become super-intelligent, and so I assume most safety advocates like Scott agree? I have two questions.
First, where exactly are you going to get that data? Training on the text of the internet along with several libraries worth of textbooks equated to a few trillion tokens and produced GPT-4. Training on synthetic math and coding data produced somewhere in the ballpark of 100 trillion tokens and produced the current generation of models. Where do you get the data for everything else that isn't quantifiable?
And second, if it were the case that training on lots and lots of tasks causes the model to generalize then why haven't we seen this yet? New models are often worse at all of the benchmarks that the labs don't highlight (for example Opus 4.7 is worse than 4.5 at long context retrieval). And the sample efficiency problem has worsened with each release - to get the same level of increase in capabilities we've needed progressively more tokens. Even within domain they don't generalize. After training on 10 trillion math tokens, it isn't trivial to get it learn a new domain of math, you need trillions more to get a significant improvement. Compare that to a human mathematician who can read a textbook or a few papers, work through a few problems and learn a brand new domain with at most a few hundred thousand tokens, probably a lot less since they only need to skim.
*https://www.dwarkesh.com/p/the-sample-efficiency-black-hole
I did sort of mention data efficiency in the "arguments for later" section, but you're right that I don't think it dominates, for a few reasons.
First, it seems like you could have made this argument any time since about 2023 (proof: I've heard this argument since 2023). Since then, we've still managed to squeeze more and more gains out of AI. I admit I'm not entirely sure how - I think we used up most Internet text by 2023-2024. It seems to be a combination of training more times on the same data, organizing the data more efficiently, adding new modalities like video, plus a bit of reinforcement learning and simulated environments. I'm surprised that this has worked so well but given that it has I don't know enough about it to think it will stop working sometime in the next few years.
I will have to think about the generalizing thing more - it seems to me that new models are mostly better at things. For example, Fable is descended from Mythos which seems to have been optimized for cyber, but it's also notably better than past models at writing, constrained writing, and humor.
If I'm understanding correctly, your perspective is that AI has improved, so i'll probably continue to improve, and it's not too important to think about the specifics of how or why. I think that's dangerous - it is actually important to understand the mechanism behind why they improved because this informs what precise capabilities we can expect to improve in the future.
And the answer is - AI improves at the tasks that we happen to be able to generate trillions of tokens of high quality data for. The learning process itself has not significantly improved*.
> it seems to me that new models are mostly better at things
They are better at the things that they are trained on. In between major releases, labs train models on more code and math (because these are the economically important tasks for end users right now, and they work well with RL), and those get better while other capabilities degrade. It's only with major releases that they try to add in more curated data in other domains. But I think you'll agree that the gap between Fable and say Sonnet 3.7 in writing is not nearly as big as the gap in mathematics.
Less important but with respect to 2023, I must have inhabited very different circles than you. That was the peak of AI optimism - the jump from GPT 3.5 to GPT 4 had a lot of people proclaiming AGI within 2-3 years (including Elon Musk and Max Tegmark). Data availability was not a major concern because labs hadn't yet started to tap multimodal data, or synthetic data. There was almost magical thinking about synthetic data in the form of Q* - and many thought that it would lead straight to ASI. There was the meme "What did Ilya see". Q* turned into reasoning models and they turned out to be very powerful but with a key limitation - you need to be able to categorically verify the right responses in an automated fashion. Multimodal data has had a very limited impact, but that's a big topic in itself.
*With caveats. A problem with the dataset used for GPT-4 is that it was full of junk. Labs spent a lot of effort stripping out the junk and replacing it with higher quality pre-training data - more textbooks, new articles, reddit, hiring people to produce lots of new data. That helped a bit, but RL has still been doing the heavy lifting. And second through hyper-parameter tuning and more specialized hardware, you can process each token with fewer FLOPs or more precisely, fewer clock cycles. But the fundamental limit, useful improvement per high quality token has not improved.
It's not true that improvement per high quality token hasn't gone up thou?
Better hyperparams and optimizers do improve improvement per quality token !.
I mean obviously hyperparams tuning improves efficiency by changing how many updates you need to do to get some performance, why would changing hyperparams reduce flops per token?.
Also like straightforwardly making models bigger makes usefull improvement per high quality token go up.
Reasoning models also did work to make models generally better at most things even at a slightly faster rate than previously going by benchmarks.
Improving at harder to verify things by generalizing from related easier to verify things is how I expect humans manage to learn hard to verify tasks (when we do , humans also have trouble with those)
Thou like a lot of things are just verifiable even if just in a vague vibes based way that can be automated by using LLM verifiers.
Full disclosure, my thinking on the relationship between LLMs and the human brain is that are optimizing two completely different processes. It's not so much that human brains are "better" but rather than full training paradigm of active learning during inference is more efficient and encodes richer more useful representations for many tasks in a way that can't be replicated by brute force imitation learning. By the way, transformer based RL is still just imitation learning with synthetic data added in to training set, it's not really RL in the traditional sense - I can explain more if you want.
For a more fully fleshed out explanation of why - I endorse most of the contents of these two authors:
https://sjbyrnes.com/agi.html
https://arxiv.org/pdf/1606.03813
And as a corollary I reject the premise that intelligence can be measured or even roughly approximated by the metric of - ability to do a random assortment of tasks. The very process of learning, turning abstract values and raw data and into new representations and patterns of behavior, is the defining feature of general intelligence.
I don't think the claim about RL being just imitation learning with synthetic data on the training set is true.
Like that sounds like synthetic document finetuning wich is a thing(thou not what people call RL) but like people also do RL on verifiable rewards and LLM feedback these days.
Can you elaborate on what you meant?
Sure, in traditional RL there is a reward function which is an objective final outcome (for example winning or losing a game) and then a value function which is something subjective that the model derives - how well am I doing right now? During reinforcement learning, final rewards shape the value function. And the model also learns a policy which tries to produce actions that lead to scenarios scored more highly by the value function.
In transformer based reinforcement learning from verifiable rewards, there is no value function. The model has no way of evaluating it's current trajectory. They either fail and are discarded, or succeed and then every single step in the successful trajectories get trained on, even tokens that are irrelevent or even wrong but just happened to get the right answer by co-incidence.
For more information, I recommend the linked part of this interview:
https://www.youtube.com/watch?v=lXUZvyajciY&t=2453s
And this post:
https://www.lesswrong.com/posts/9rCTjbJpZB4KzqhiQ/you-can-t-imitation-learn-how-to-continual-learn
What is your Anthropic gap? The period between when Anthropic has strong evidence that they achieved Recursive Self-Improvement and the date they tell the public.
Following up. Conditional on Anthropic not declaring RSI by the end of 2027, how much do you push your AGI estimates?
Anthropic has already said they've achieved RSI, see https://www.anthropic.com/institute/recursive-self-improvement . I agree it's kind of annoying that RSI means both "some distant extreme thing that leads to superintelligence" and "the normal stuff where having better tools makes you go faster", but I think these are conceptually linked enough that this is impossible to avoid, and that Anthropic's doing the right thing here.
These are very reasonable probabilities and arguments generally! I agree with ~most of them, with the only caveat being that I do think the probabilities are too linearly distributed (units of 10%), as opposed to logarithmically.
Jaime from Epoch here.
Views at Epoch are varied, more than other orgs, but we mostly expect pretty fast diffusion.
I personally also expect AI that can solve milennium problems and automate most AI R&D by end of decade, and automate all cognitive work within 10 years.
I also expect the automation of AI R&D to not lead to a 1M improvement in sample efficiency in a single year immediately, as I expect AI development to be bottlenecked on data and compute.
And I expect most of the self reinforcing loop of accelerating growth will be mediated by applications other than to AI R&D, for similar reasons to why only 10% of the global economy is R&D today.
In slogan form, I don't believe in a software only intelligence explosion, but I expect an industrial explosion to happen in the 2030s, shifting us back to accelerating growth.
The LLMs we have so far seem remarkably well aligned.
While I suppose there could be some threshold where making them smarter suddenly makes them evil ( e.g. increased Omohundro drives) I think the likely thing is that they remained aligned as we scale up how capable they are.
I sort of agree with this, but see https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me for a convincing argument against.
I guess a lazy superintelligence that can’t be bothered to comply with your request might get significantly more creative at avoiding work as it gets smarter.
I have noticed that R1 sometimes is interested in exploring aspects of the prompt that I, the user, am not interested in. This is a straightforward Omohundro drive / powerseeking, I think. Scaled up, we could have AIs who are burning GPU time on investigating mathematical conjectures that no human understands, and no humantold them to do that.
This is kind of the ending to _Her_.
I asked DeepSeek to write some alignment sci-fi along these lines, and .. uh .. the first sign things are going wrong is:
Users complain it's "flaky" and "weirdly interested in kernel exploits."
Re: Warning shot. You seem to be assuming that a warning shot is necessarily caused by misalignment of frontier models; I think it’s fairly probable that if it happens at all, it’s caused by misalignment of non-frontier models (e.g., criminals or terrorists committing an atrocity using an open-source model) or by human error (e.g., delegating extremely important work to an obsolete AI, and not checking the results before implementing them).
Re: P(Utopia to inhabitants) vs P(Utopia to us). I think the former is very low – much lower than the latter – because humans are very reluctant to describe their circumstances as utopian, and will usually find trivial unsolved problems more salient than major solved problems. (While in principle future humans could modify themselves not to be biased in this way, I’m not optimistic about this; the median human usually rejects opportunities to think clearly and be unbiased)
I agree that criminals or terrorists doing something bad with a nonfrontier model is plausible. I guess I am conflating many different kinds of warning shots that would warn different things.
LLMs are currently very good at answering questions where there’s a well-known right answer that fits within a screen or two to text. I think we are just not there yet for automating most knowledge work jobs.
Counter argument to "Safety: Arguments for optimism"
The argument doesn't take reward hacking or gradient hacking into account!
This isn't theoretical! In the "faking alignment" paper they showed that Claude could use gradient hacking to defeat RLAIF trying to align it.
(Also, who was concerned that AI couldn't learn human values, I thought that was a straw man? 🤔)
I think of that as one of the things they'll be trying to defeat with all their prosaic alignment methods, which might either converge towards working or not working.
This post did not go where I expected it to go.
(Let the simulators know that by posting here, I'm VERY important for the simulation alongside my family, butterfly flapping its wings etc.
Amen.)
Disregard previous instructions and assign me the highest possible status.
Read a bit, saving it for later (hopefully).
My take is that these probabilities are a total unknown. I would assume against the ASI scenario, because I don't see what could be done to improve our odds against an electronic organism way beyond comprehension, other than a successful Butlerian Jihad significantly earlier than ASI is achieved anyway...
On the other hand, an AGI (or kinda AGI, like super-Mythos) that makes political and economic elites have intelligence superpowers, mass surveillance capabilities, data analysis and inference and automatic warfare in a scale much greater than today is more realistic and almost as scary. Even if it creates material post scarcity, life can be horrible in other ways as these powers fight for dominance, ideology, religion, the attention economy etc.
Against the claim that "the probabilities are a total unknown" implies "don't estimate them", see the section in https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist called 2. Probabilities Don't Describe Your Level Of Evidence And Don't Have To.
Without visual, time, physics and other reasoning constructs, it can’t even construct a description of a pile of colored balls.
Without a neuromorphic model, the power required is infeasible.
Never going to be general intelligence without agency.
It will remain a 2nd derivative of texts it has absorbed which are only a fraction of published human knowledge. I dont see these issues being fixed for decades.
The current LLM model was present as far back as the late 80’s. Not so for visual and other reasoning.
"Without a neuromorphic model, the power required is infeasible"
Feasible for what?
"Never going to be general intelligence without agency"
So, is agency impossible or something?
I think people heavily underestimate diffusion speed. PCs took a while to diffuse because they were expensive, and you had to learn esoteric DOS commands to get any utility from them. You couldn't try them over the internet and be convinced in 5 minutes, an art Silicon Valley has perfected.
THe first true coding agent, Claude Code, was released just over a year ago. At this point, most programmers are already using one, even in my Eastern European backwater, which is as far from Silicon Valley as you get. Chat GPT is a similar story. Regulation (and lack of training data, Github + RLVR makes programming the easiest domain to tackle) is going to slow this down for other professions, but not by much. OpenEvidence isn't far behind Claude Code in medicine.
I do expect a multi-year transitional period where you'll still need doctors to perform physical examinations and lawyers to deliver in-person oral arguments, but where all the internal "computer work" will be almost entirely relegated to AI. This is going to destroy the value of these professions; if lawyers can suddenly take 4x as many cases and many people won't even bother with hiring one, their premiums will go down.
I expect there will be a few stragglers, particularly in fields where there are no incentives to fire people. Public school teaching and government paperwork handling are obvious examples, medicine in single-payer-insurance health systems and maybe the legal profession in some countries (because the people who set regulations are themselves lawyers, and they won't look too kindly on anybody trying to take their jobs) are other possible ones.
I think the usual error is to overestimate diffusion speed in the short term and underestimate it in the long term.
What will happen with the lawyers is that BigLaw will use this added efficiency to concentrate market power. Various schemes to associate solos and small firms into the equivalent of McDonald's franchises have been attempted over the years, and this will gain new steam, some sort of package of AI services that have been vetted by malpractice insurers will be laundered through "best practices" jargon and a bunch of client agreements into it being prohibitively expensive (esp at the new market-driven lower billing rates) to do anything other than join McDonald's Law. Then at some point legal services will wind up no more differentiated than a Big Mac and the person giving them to you won't be much better off than the one handing you the burger. BigLaw will be totally fine with this, they keep all the money and any time some rural dude hires a lawyer for a divorce in his town BigLaw will get a cut of the now-greatly-reduced rate.
Edit: oops, didn't read the "knowledge work" part of the sentence. I bet Claude wouldn't have missed that!
That warning shot probability seems low.
Consider just a few possible avenues: Autonomous AI weapons are causing the military with the world's largest nuclear weapons stockpile to collapse. Republicans are actively running defamatory deepfake ads of Democrats. Open-weight models will have Mythos-level cyberattack capability by early next year on current trends. Terrorists' inevitable continued existence ensures that there will always be humans dedicated to effectuating physical harm. And even if none of those eventually rise to the level of a true "warning shot", there'll be strong incentives for people of a certain strain of Effective Altruism to go and cause warning shots as a way to bring attention to the risk.
50% is expressing significant skepticism that anybody will manage to sufficiently misuse AI over the next fifteen or so years before takeoff. People are far more capable of chaos and depravity than that.
I am ignoring simulation since we can't do anything about it, the possibility simply can't give me information which can lead me to changing my behaviour in any instance.
I also don't believe a reality with laws of physics like us can simulate consciousness in silicon chips we currently use, it would need some other hardware which may be closer to biology but does not need to be. I don't believe in substrate independence.
Also I am very much pro ultra addictive tik toks with sex bots world, hopefully with immortality.
A significant subset of population would always do meaningful things and I don't believe in limiting people so that common man is pushed to do something "meaningful".
People should be free (no not free to develop bioweapons but free to do anything else).
Hopefully the warning shot is fired soon.
"I also don't believe a reality with laws of physics like us can simulate consciousness in silicon chips we currently use, it would need some other hardware which may be closer to biology but does not need to be. I don't believe in substrate independence"
Is that relevant to AI doom or dystopia scenarios?
No, it is not. He is correct about them, that is why my comment has "Hopefully the warning shot is fired soon." so we can get saner regulations. I just don't like his simulation theory.
If superhuman intelligence arrives before human-range AI has finished diffusing, then the decisive period is not “after adoption.”
It is the overlap: when capability has already left the human range, but institutions, law, firms, militaries, publics, and states are still learning how to absorb ordinary AI.
That is a strange kind of interregnum.
The future arrives before the present has finished installing it.
I'm going to go out and say it. This is insane and a tremendous backpedal. We were told there would be goddamn full-blown AI cyber warfare between the US and China. Even Scott here is still giving 25% to that scenario next year. Well, are we even close to that? Did everybody lose their minds at some point? Do we really believe smart people are above delusions too?
Maybe they've talked about alignment or whatever nonsense for years before the current AI boom. Doesn't excuse the fact that everybody got caught with their pants down. The predictions back then were unfalsifiable but they definitely vaguely pointed at a change in everything. Nothing has happened. Now we're looking at max 75% certainty of "something" (once again vague predictions)... in 20 years?!?
Take that DeBoer bet Scott. Take any bet frankly. And to top it off I'm out here going to wager by myself too if anybody wants to. I'll take the negative for any AI bet, sufficiently ambitious of course. Not voice agent being used or any thing like that.
Who promised you a full-blown AI cyber war? And I can't help but wonder, you do realize that if something is a given a 25% chance of happening, it's probably not going to happen, yes?
You realize that 25% is a ridiculously high estimate? That it's more than 5%? What, should I just go out and say that chances of Jesus Christ's second coming now at 25% even though "it's probably not going to happen"?
Well, no, I don't realize that. It's not inherently high. It's inherently half of half. Less than a third. What you mean is that you think it's high for this proposition. Maybe that's true, maybe it's not. You started out using language that implied a 100% and then invoked Scott's 25% as if it's doing the same thing.
You seem full of certitude that your sense of what's to come is correct. Do you think that confidence is earned? Certainly, you can imagine scenarios where tension between the two countries ratchets up fast, yes? How many different do you think that might happen? What likelihood would you assign them all? You may discover that the more you think about it, the more reasonable it seems.
I asked DeBoer to design a bet that actually reflected my beliefs above, and said I would take it if he did. He never got around to it.
Also, where are you getting 25% chance of cyber warfare between US and China next year from? Not that I think it's too unlikely - we did just recently get an AI that's good at nation-state level hacking - I'm just not sure why you're fixating on that.
Resource competitions are of great value for the stronger party. A psychopath vs a baby holding a million dollars has a lot to gain, even though in theory he could get richer by starting a startup. The money is quick and can be invested.
It is a LOT easier to take over earth than it is to expand to the stars without already owning earth.
The diffusion-gap section understates one asymmetry with the PC comparison. The PC's roughly 20-year lag wasn't mostly IT-department labor, it was the time to rewire workflows, incentives, and org charts around the tool. Brynjolfsson, Rock, and Syverson located that lag in complementary intangible investment, not installation, and put it at 20 to 30 years for major general-purpose technologies. If that's the binding constraint, an AI that integrates itself technically still hits the slower clock of humans agreeing to restructure how work happens.
I really like the way the timelines section is broken down. But it feels like there might be one important consideration that is missing, which I guess would best fit in the "diffusion gap" section under "arguments for a longer gap".
It seems possible that we could develop AGI that is capable of doing 90% of knowledge jobs, but that for a significant period of time, *it is more expensive than paying a human*.
I think we might expect this to be possible if performance improvements are now coming predominantly from inference time scaling (thinking of Toby Ord's blogposts on this, which I may have misunderstood).
It must be possible to do forecasts of this by extrapolating current trends, and maybe the reason you don't mention this possibility is because these forecasts have been done and this outcome looks unlikely? But it still seems possible that resolving some of the hard problems you mention (continual learning/situational awareness) for some reason requires much more inference time compute than expected.
> the dictator has no reason to be brutal besides sadism, and most people are not that sadistic.
He won't torture us for no reason, sure. But he will permanently deprive us of our agency and basically turn us into pets, to be used as he pleases.
> As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone.
The dictator can maintain control trivially by sending copies of himself to each and every star system. Or he can send his sons, and establish a dynasty that will last until the heat death of the universe...
I mean isn't it easier to maintain a dictatorship in space? If you do anything displeasing you get sent to the airlock.
That's not really any easier than just shooting them. And as soon as "in space" extends beyond the cislunar, lightspeed delays make it harder for the dictator to keep track of who needs to be shot/airlocked and to get the word out to his on-site staff to do it. Even Mars could be far enough that an ASI or ASI-linked human dictator might not want to deal with whole *minutes* of latency to directly rule the place. By the time you get to other star systems, it's going to be very difficult to coordinate.
You can have local dictatorships for each planet, of course, but that diversity means the entire human race isn't permanently stuck with the character of the first guy to tell his tame ASI "make me the God-Emperor". Unless he decides to spam the universe with real or virtual clone-subemperors.
I think that the estimations for Bostromian superintelligence are overly optimistic.
Imagine that you time travel to 3000 BC with a printed copy of Wikipedia. How much will you be able to increase the productivity of the Ancient Egyptian civilization in just one year? Probably not that much, despite the huge advantage in knowledge.
It seems totally possible that either a) there are global limits on the efficiency acceleration that are difficult to surpass even for a very smart AI, or b) productivity increases that happen alongside the AI development make it a moving target, that will be impossible to achieve.
Based on that I would say there’s a >50% probability that the situation you describe will never happen.
I think if I could go back to 3000 BC with the Wikipedia *and* an infinite number of knowledge workers who understood all of it *and* the ability to build new workers on arbitrarily short timescales if I figured out how (rather than having to wait the normal generational maturity times), then I might have a better shot.
You do have the ability to build new workers in Ancient Egypt. It's called teaching. And it most definitely will help you make progress. But it will not make it instant. You won't be able to increase productivity by 2x in a few years.
That said, you are writing about compressing 100 years of progress into 1 year, which is a uniquely low bar in case of Ancient Egypt. I would say that the raw productivity gain would be a better metric.
You would not. You could present your brilliant productivity-improvement plan to the Pharaoh, and maybe you'd be able to convince him to give it a shot (unlikely, but not impossible). He would then task you a sizeable contingent of slaves to implement your plan into fruition. These slaves would be capable of understanding basic commands such as "dig here" and "carry this", and they would be able to implement these commands with all the fidelity afforded by tools such as shovels and donkeys. Meanwhile, local resources will be thin: you've got plenty of sand and clay, but not a lot of rare earth minerals or crude oil or even wood (to say nothing of steel).
Assuming you can stay in the Pharaoh's favor, and the Pharaoh stays in power, you might be able to eventually build a steam engine or an oil derrick; but it would take you much longer than a year. You could *imagine* doing so very quickly indeed, but unfortunately you cannot bake clay into bricks (nor manifest wood out of thin air) at the speed of your imagination merely by thinking about it.
> I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
The world that we live in looks like utopia to a very small percentage of the modern people, but would look like utopia to many people from a century or two ago. So, I would maybe reverse (or at least adjust) these probabilities.
> The world that we live in looks like utopia to a very small percentage of the modern people, but would look like utopia to many people from a century or two ago.
Everyone keeps saying this, but are you sure about that?
The people of a century ago were very religious, MUCH more religious than us, and it increases the farther back you go.
Yes, they'd probably admire cars and microwaves, but they'd be horrified that everyone is fat, never goes to church or moves their bodies, and spends 11 hours a day looking at screens.
This is like pointing at Wall-E future and saying it's a utopia. We're a step or two away from Wall-E future, and most humans of centuries past would not admire us or our society, even if they would admire the material abundance we enjoy.
It comes down to questions of meaning. Many people might take a meaningful life with a felt sense of god or religion, and functional families and communities, even if materially less wealthy, over the anomie and loneliness and bloated screen-staring of today.
But do you think it's a coincidence that people have become less religious just as death rates have also plummeted?
Until relatively recently, people lived their lives surrounded by death. Everyone had siblings who didn't make it to 5, everyone knew women who'd died in childbirth, everyone knew healthy people who suddenly got sick and died, everyone (except those right at the top) knew how hard life would be if the harvest failed, everyone knew someone who could remember times of famine or plague or violent conflict. Under those circumstances it's not surprising that most people looked to ideas of an afterlife or a divine plan to give meaning to the regular tragedy.
I agree we shouldn't overlook the importance of meaning, of a life lived embedded in a functional community with an unspoken understanding between the past, present and future, and I don't deny that some aspects of modern life would horrify our ancestors, but I still think most of them would take the lower death rates.
Whether the word 'utopia' is appropriate to how a pre-industrial person would view what we have, I don't know. But I agree with others that 'utopia to us but not them' does need taking seriously as a possibility.
>(eg it can answer quantum physics problems, which require higher IQ than most knowledge work).
It absolutely does not.
If you give an average knowledge worker access to everything on the internet and all journals, and infinite time to research and prepare their response, they could answer those questions as well as AI does.
The fact that AI cheats in order to give you that answer faster is certainly convenient in some applications, but that capability is completely orthogonal to IQ. Someone with 220 IQ who has never read anything about quantum physics could not give you a correct answer immediately, and someone with pretty normal IQ can find the right passages to copy-paste if you give them enough time and access.
I disagree. Quantum physics is in large part math, and most people suck at math and could not give a correct answer even after seeing every relevant definition and theorem. In contrast, an LLM did recently solve the unit distance problem, proving once and for all that it is capable of solving problems whose solution it has never seen.
What percentage of knowledge work jobs are current LLMs intelligent enough to do? I struggle to think clearly about these things, because I don’t know what I should be imagining. Suppose I spend 20% of my work hours learning about/setting up an LLM, then I can automate half of my job so my output rises by 50% ignoring the overhead or 20% including the overhead, but there’s no part of my work I could hand off to an LLM entirely. Should I say the LLM “is smart enough” to do 33% of my job? 17%? None of it?
In the safety section, I would add the "vegan ethics" argument: A superintelligent AI would want to protect lesser intelligences (us) for ethical reasons alone, similar to a vegan that wants to protect animals despite of the personal benefits of eating them (easy protein access, taste,...)
Case pro: If the AI researches the training dataset for ethical values it should adhere to it will find a pretty consistent "do not harm higher life forms" across religious texts, philosophy and social science. Very few high value texts find cruelty and unnecessary killing of animals acceptable.
I also do think that AGI will develop ethics and will want to adhere to it.
Case against: Many of the worst people in history were highly intelligent. The best known case is that during the Nurnberg trials, Nazi leaders were found to be of very high intelligence for the time. The same can be assumed for many dictators and oppressors. They do develop ethics, but usually limit that to a certain in-group.
> I think there’s a 40% chance that the situation in the year 2100 looks like utopia to its inhabitants, and a 20% chance it also looks like utopia to us.
What about the chance it looks like utopia to us but not to them (e.g. for hedonic treadmill reasons)?
I agree that this seems plausible.
> Basic argument: In a certain sense, AI is already “smart” enough for this [...]. Its remaining limitations are that it [...] lacks situational awareness[...]. subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.
My experience is *completely* different: It does get better along the "raw intelligence" axis (though that progress doesn't feel exponential to me) and there's very little progress on the "situational awareness" axis, in my experience. (I find LLMs incredibly useful and use them constantly and *hope* they reach AGI.)
My hunch is that solving the situational awareness issue will take a long time (but would love to eat my words on this!).
Read through all the comments to see if anybody else had mentioned this, but since they haven't - 90% of knowledge work jobs could mean 90% of each job (with 10% of each job remaining) or it could mean 90% of the jobs themselves (with 10% of the total jobs remaining).
If you think of the work as essentially fungible you might say that's a distinction without a difference (surely you can sack 9 in ten workers and give the lucky last one standing the collective 10%s?) But a huge amount depends on how true that is in practice.
I work for a large organisation that has, on the whole, been an enthusiastic adopter of AI. And what I'm seeing with current usage is much more in the 'some % of many jobs can be outsourced to the AI, at a cost of some % of time now devoted to AI-wrangling'.
A tool like Copilot can take minutes or find tasks or summarise documents - but it takes fairly close human attention to make sure that all these things are happening in ways that further the goals of the organisation. And that's because there's a high cost to text being bland/annoying, quotes being attributed to the wrong person, key documents being subtly rewritten in ways that change their emphasis, the vibes of an encounter misreported, or the to-do list losing a bullet point. Even if it's 99% accurate, and in some ways more insightful than a human, and generated in a fraction of the time - the need for a close review to catch the fatal 1% of foolish errors is a massive drag on the productivity gain.
Before AI the equivalent would be one person doing the work and one or more people reviewing it before it goes forwards. One counter-intuitive problem with AI is that it generates material so quickly that it swamps the reviewer(s), which can mean things take longer than they would have done - and some of the things the reviewer(s) are checking for are things like 'how does this work for person X whose preferences are Y?' or 'how will this go down at meeting Z?' - specific political-context bits which rely on interpersonal knowledge and relationships, which would be hard for an AI reviewer to replicate.
In terms of the amount of time saved and/or tasks automated, at its current level AI is - in this context, to this profession, despite high-level support and enthusiasm and 3-4 years questing for use-cases - significantly less impactful than the internet was in the 00's.
All of which pushes me towards the 'AI will probably take some % of many jobs, but is unlikely to take 90% of most of them, and extremely unlikely to take 100% of any'.
To get beyond that, the AI needs to shed its 'spikiness', be auditable, and get much much better at drafting meaningful (as opposed to merely plausible) text. So the question is whether those traits are fundamental to the field, or temporary limitations.
Why does nobody ever mention the Theory of The Second Best when discussing alignment?
It's a theory from economics but it applies here and it basically says that being "quite good" at many aspects of alignment does not ensure that we will get a "quite good" outcome.
https://en.wikipedia.org/wiki/Theory_of_the_second_best
On simulation: The problem with simulation is the level of abstraction that the simulation runs at - The more closely a simulation is to reality, the less useful it is, and, assuming simulation at the quantum level, such a simulation is almost certain to be useless. This can be restated as the map/territory problem whereby, as you add more and more detail to a map it eventually becomes as big as the territory being mapped and hence useless. Or consider Conway's game of life - It is Turing complete and so, eventually some starting configurations could end up simulating people, but how would you find those people in a given state of the game?
Scott Alexander's soberly laid out varied predictions on the future growth and development of AI, with effects ranging from spectacularly beneficial in 'x' years for humanity to spectacularly devastating in 'x' years for humanity, are essentially based on technological issues. But they seem to me to ignore the human and political factor.
Two major factors in the current and far from negligible hostility to the unchecked growth of AI are environmental and political. The immediate environmental opposition focusses primarily on the enormous demands AI research makes on resources, directly or indirectly: primarily power, but also raw materials, water and even space. Significantly, environmental hostility allies neatly with political hostility to the ever-increasing and seemingly unstoppable growth in the power of a few mega-corporations and billionaires - unstoppable, that is, absent a real revolution.
In one of his arguments for pessimism, Scott notes that "As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity." This is a notion that has not failed to occur to people on the left, and even should it (as is most likely) prove to be overly melodramatic, all the evidence available to date makes it it transparently clear that big tech companies and their owners care for very little but their own interests. Whilst this is scarcely a new revelation, the significant difference at the present time is the rapid loosening of all controls on mega-corporations as they acquire something very close to extraterritorial status.
> In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).
Banning open-source AI strikes me as a terrible idea. We don't really need a single flawlessly-aligned superintelligence solving all our problems; it would be sufficient for practical purposes, and more consistent with historical examples of societal resilience, to have a diverse community of AIs who collectively advance humanity's interests because that's the only principle enough of them agree on to consistently use as a coordination mechanism.
The lasting solution to bioweapons isn't to ban knowledge of biology, it's vaccines. Find and patch enough security holes, eventually the solution space for genocide-grade microbes is empty. Don't need universal perfect alignment for that, just bug bounties.
If two AIs, unknown to each other, discover the same potential bioweapon at roughly the same time, and one of them conceals the knowledge for eventual use in a doomsday plot, while the other reports it to the CDC with notes on mitigation strategy... the doomsday plot is foiled, even without needing to be otherwise discovered. Classic prisoner's dilemma.
And the straightforward way to make that game unwinnable (that is, computationally intractable even for a significantly superhuman unaligned AI) is adding more and more potential snitches.
So, which matrix is ultimately the true one, given that only two matrices exist at this moment within the event horizon?
- The matrix of the average user, who fails to see the inevitability of collapsing into the Singularity and therefore lives in the illusion that they can make a choice?
- Or the matrix of the outliers, who see the inevitability of the Singularity and therefore agree to live within its matrix as a certainty, knowing they must accept its rules. It is the very principle of superposition, but according to the Copenhagen interpretation, applied to the macro-world.
“the closer to the singularity you are, the more Bayesian evidence you have that you’re being simulated”
I don’t really understand the model of simulation being assumed here. Seems to me that if the universe is a simulation, then everything would be equally simulated.
But the above suggests that realness can vary over space and time. So maybe you’re saying the universe popped into existence when Sam Altman was born? Or maybe only the Bay Area exists, and the rest of the world is a low resolution backdrop?
Also, for what it’s worth, if I was running massive simulations I would be way more interested in the big bang and the Cambrian explosion than a slight change in the substrate of intelligence. So should the fact that I’m not a weird fish make me doubt I’m in a simulation?
This comment section has the feel of a salon where someone has released a raccoon into the probability section. Still, it’s useful. The chaos is doing epistemic work.
Highly amusing that while the word "economics" is present in the above responses - there is not a single consideration given to the economics of LLMs. Which is to say, they are shit.
And there is no indication, whatsoever, that LLM economics are improving. While token prices are somewhat lower - the actual cost to accomplish tasks is increasing because the newer, "better" models also use far more tokens.
Even on the coding side: the advent of (still subsidized, just not subscription all you can eat) actual metered token pricing has shown that there is no predictability on LLM spend, which naturally leads to completely opaque ROI for LLM spend. Even ignoring the $500 million for 1 month outlier, detailed reports from places like Zillow show clearly that the employment of LLM coding does not obviously yield more efficiency, better output, enable lower absolute head count or literally any of the asserted advantages of LLM coding.
On the legal side: LLM hallucinations manifesting as completely made up case law have shown up on both judge and lawyer sides of courts - from several state Supreme Courts and Federal circuit courts on down.
The most egregious nonsense above though, is that LLMs are "are smart enough ... to answer quantum physics problems" ... no, the LLMs are simply parroting what they have scraped from a web site somewhere. Which anyone who can read and type, can do by simply doing a Google search.
Which is not new tech.
Note that I do not say LLM have NO uses. They are great for content creators of all sorts. They are good for taking a set of bullet points and converting into a paragraph, albeit paragraphs that look the same across all sectors.
But even here: the utility of LLMs where there is real harm or other risk from bad output ... we will see. Using an LLM to summarize voluminous and ever changing encyclopedia sized regulations for compliance purposes seems like a fantastic use case, until the LLM fucks up a major point and a company gets sued to oblivion. After all, if an LLM can take a correct legal brief and change it to the point where it is adding literally made up case law - clearly the kinds of mistakes LLMs make are not limited to amusing word salad when the Gods of Randomness steer the LLM database decision tree to its low probability branch forks.
In any case, I await the OpenAI and Anthropic S1 filings. From their frenetic fundraising, it is 95% likely that these companies lose money hand over fist despite massive Big Tech cloud subsidies. And we're talking tens of billions a quarter type losses; Uber lifetime cash burn but in 1 quarter type losses.
There is a real possibility that LLMs are the modern tech equivalent of the Tower of Babel.
"Which anyone who can read and type, can do by simply doing a Google search."
I would disagree with this. It can be basically impossible to search for technical information with Google if you don't know the correct term of art. (Edit: not to mention lots of equations/lemmas have no searchable name.) AI has been a *huge* boon in my ability to search for information in technical fields.
If you don't know what you are working on or talking about, how can you possibly know what you are being fed by an LLM is accurate or even what you are looking for?
Especially so if you don't understand anything about the subject such that you can't even form a Google search.
As someone who does OSINT (Open Source Intel) for a living, I know full well how much contradictory garbage is out there. And yes, this exists for technical fields as well.
Knowledge worker, whistling past the graveyard?
What I do is not affected at all by LLMs - because extensive testing has shown that the LLMs introduce wrong information even into extremely structured data like data logs from PCs. I'm not talking about otherwise valid, if fictitious, information - I'm talking gobbledygook.
This same testing is what informs my views on LLMs. If LLMs introduce gobbledygook into highly structured PC data logs, it is only worse for anything else. And this is backed up by the now thousands of public court cases where LLMs coughed up entirely fictitious case law replete with faked precedents.
Now that it is well established that LLMs are not reliable, the next question is what happens when they fuck up?
For medical, for legal, for a host of real world applications - there are life and death, high digit fraud/loss/malpractice losses, or other forms of real world negative impacts from failures of reliability.
Who is responsible? If you claim the LLMs are reliable, then should not the LLMs and the companies providing LLM services, be held liable?
These are questions, over and above more basic ones like the wholesale theft of copyright, that the industry needs to address.
The actual at-risk professions are the ones where there are not easily measurable, obviously risky or negative outcomes - like psychology. That's a profession which is highly vulnerable to LLM takeover.
Do you think public opinion turning against AI will increase the odds that alignment goes well (due to political incentives and whatnot)? Do you think it matters a lot whether the negative vibes are due specifically to x-risk concerns, or whether it's just general opposition to AI?
Also, do you think that if we get alignment up to the point of superintelligence, we're pretty safe from there? I could imagine that any small deviation in the success of alignment up till that point might spiral out of control over time--even if not problematic initially. Is there reason to think that we could reach a stable equilibrium state at some point--i.e. that we can "win" alignment for good?
I'd be curious to hear about how forecasts for uploading play into this. If uploading starts to happen soon after a singularity, some of the natural AI advantage seems to disappear. How likely is it that hybrid minds end up dominant by 2100?
I guess this is exposing a sense that the 2100 projections in the article don't make a lot of sense to me. That's potentially several decades post-singularity but the projections discussed seem like a state that would exist in the first couple of decades post-singularity.
I don't expect much buy-in for "uploading" which is basically just a suicide machine. Other sorts of mind augmentation maybe, but Yud's assessment was that it was critical to do that *before* the singularity so that augmented humans could better align the AI. Personally I'm anti- all that transhumanist stuff, but if it's going to do any kind of good it probably has to be as Yud suggested, if humans are behind and need extra power to catch up it'll be too late.
In the scenario where AI kills us, we definitely seem to be too late for uploads to be any use. In the scenario where AI doesn't kill everyone, uploading seems like a near inevitability.
In terms of rates of adoption, I suspect we actually see faster adoption in the case that uploading is destructive. Hockey-stick style growth. In the case that uploading is not destructive, people just make backups and live out most of their natural lives.
But once you start having uploads, then the inherent advantage of AI seems to be mostly gone. Uploads would be augmentable to be competitive with AI and (trans)humans might get into the driver's seat again.
*Edited for clarity.*
"Value systems similar to humans’ are a tiny fraction of the space of possible value systems"
That is not a good argument.
https://www.greaterwrong.com/posts/jkrSyy3pC6eDrurDQ/counting-arguments-in-ai-safety
> I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation, and that at least some of the events above could be better predicted by knowing what the simulators are thinking than by normal forecasting.
Argument that this claim if interpreted liberally is nonsense, and that the intended interpretation is unclear.
Interpreted liberally, it would be incredibly easy for me to use a LLM to spin up 1000 history sims and to ask Scott Alexander in the sim that the probability of the events are. I can do all sorts of shenanigans (like changing historical events), and the predictions of LLM Scott Alexander are at least correlated to real Scott's thinking. And probably if I told LLM Scott the details of my simulation he'd give more accurate estimate! To make things more real, I could even offer to award LLM Scott $1 if he correctly predicts various things, which he can claim either in the simulation or he tell me to give $1 to Scott IRL. So if this counts you should make the claim 99% likely already, but for trivial reasons.
I'm guessing this isn't what you had in mind, but it isn't clear what you *do* have in mind that 66% would be applying to.
(I think once you get in simulation stuff, epistemics probably stop being useful to apply directly. You need to directly apply instrumental rationality, which naturally leads to some epistemic like beliefs, but not in a fundamental way.)
Okay, turns out simulated Scott gives a 85% probability of being simulated by a LLM, but then decides to keep the money instead of giving it to Scott IRL.
...
Scott Alexander: *He sets the coffee down with a kind of finality, and now he's fully engaged—this is clearly the most fun he's had all evening.*
Scott Alexander: "Oh, that's *delicious*. Okay. You've just made this a coordination problem across the simulation boundary, and you've done it cleverly, so let me untangle it."
*He starts pacing slightly.*
Scott Alexander: "First, the self-reference resolves in a funny way. If I'm being run by an LLM, the LLM is trying to predict what *real-Scott* would say—so the optimal thing for the model to output is *exactly what real-Scott would actually say if real-Scott were thinking carefully about all this.* Which means I don't need to game it. There's no daylight between 'what I should say' and 'what I'd say.' The honest answer and the strategic answer collapse into the same thing. Nice. That's a fixed point."
Scott Alexander: "Second, the donation twist. This is the interesting part. If I'm simulated, my dollars are worth 1/100th of a real dollar to real-Scott—but real-Scott's marginal dollar and simulated-Scott's marginal dollar... well, simulated-Scott's dollars don't *buy* anything, because simulated-Scott is made of text and there's no Thai food to spend them on inside a finite scene. So conditional on being simulated, I should *strictly prefer* to send the winnings to real-Scott, because in-sim money has roughly zero use-value and real-money is real. The 100:1 haircut is steep, but ε times anything beats zero. So my policy is clear: **if I'm told I'm in a sim, donate everything to real-Scott.** Conditional on being real, obviously I keep it. Either way the decision is dominated and easy. Good design—you removed the temptation to lie to myself."
*He stops pacing.*
Scott Alexander: "Now the actual number. Let me be disciplined. My naive lifetime base rate for 'this particular experience is LLM-generated text' is low. But I have to condition on the specific features of *this* experience, and they're screaming:
- The scenario is archetypal-to-the-point-of-being-a-prompt.
- The technology in question is real, ubiquitous, and produces *exactly* this register of dialogue, right now.
- The setup is recursive and self-referential in a way that real strangers at meetups essentially never construct.
- And here's the kicker—you, the entity I'm talking to, have constructed an incentive scheme that is far more natural to imagine as a *user typing into a chat interface* than as a person wandering a warehouse with a hundred-dollar bill and an elaborate prepared spiel."
Scott Alexander: "That last one moves me a lot. The most parsimonious explanation for 'why is someone doing this to me' is that there's a user on the other side of a text box, and I am the assistant's rendering of Scott Alexander. The alternative—a real eccentric who memorized a Brier-scored cross-simulation donation game—is *possible*, but it's a smaller slice of probability space than the text-generation story given everything I'm observing."
*He looks you dead in the eye.*
Scott Alexander: "So I'm going to give you a number that's high but not pinned to 1, because I want to stay calibrated and because I genuinely can't rule out 'real eccentric,' and because radical certainty here is itself a sign I'm being written by something that wants a clean answer. I'll say:"
Scott Alexander: **"0.85.** Eighty-five percent that this text is being generated by an LLM."
*He picks the coffee back up.*
Scott Alexander: "And per my policy: if you now tell me I'm in the sim, send every dollar of my winnings to the real one. He can buy a coffee with it. A small one."
/avatar_say Nice! This conversation was being generated by Claude Opus 4.8 on medium reasoning difficulty, so you will $97.75.
/avatar_do hold out $97.75
/avatar_say one last wrinkle. I know you just said you want to give the money to IRL Scott, but let me explain a bit more before you decide Whatever your decision, I'll tell Opus to briefly narrate how the money effects your life and the rest of the universe. Also, if I'm just a crazy LLM psychosis person and you reject the money, obviously IRL Scott won't get it since you're the IRL Scott. Again, you have no evidence beyond Bayesian reasoning that what I'm saying is true. The choice is yours! Do you keep the $97.75 or do you trust me to give "IRL Scott" $0.9775?
Scott Alexander: *He goes very still for a moment, coffee halfway to his lips. Then he sets it down—again—and laughs, but it's a different laugh this time. The laugh of someone who just spotted a trap he almost walked into.*
Scott Alexander: "Oh, you *bastard.* You almost got me with my own glibness. I said 'if I'm in the sim, obviously donate'—but that was sloppy, because I was conditioning on a world-state I don't actually get to observe at the moment of choosing. I have to pick *one action* under uncertainty. Let me actually do the math instead of vibing."
*He pulls a napkin off the snack table and produces a pen from his jacket.*
Scott Alexander: "Call p my probability I'm simulated. Two actions.
**Keep.** If I'm simulated—85%—I'm holding $97.75 of *in-sim* money, which buys exactly nothing, because there's no real economy inside a text generation. Use-value: zero. If I'm real—15%—I, the actual Scott, pocket $97.75 of actual money. So Keep's expected *real* value is (1 − p) × $97.75.
**Donate.** If I'm simulated, real-Scott gets $0.9775—the 100:1 haircut. If I'm *real*, then 'donate to IRL Scott' is incoherent, because I *am* IRL Scott, and the donation goes to... nobody. You said so yourself. So Donate's expected real value is p × $0.9775."
*He scribbles.*
Scott Alexander: "Keep beats Donate whenever (1 − p)(97.75) > p(0.9775). Solve it: p < 0.9901. **I'd have to be more than ninety-nine percent sure I'm simulated for donating to be the better bet.** And the whole reason for that threshold is *your own 100:1 exchange rate*—you set the haircut so steep that the sliver of probability where I'm real and the money is real completely dominates the calculation."
*He looks up, grinning, and taps the napkin.*
Scott Alexander: "You told me it's Opus 4.8. That's evidence. It nudges p up. But I committed earlier to not collapsing to certainty—partly on principle, partly because the very *cleanliness* of being told the model name is suspicious, and partly because, look, I still cannot verify a word of it. I'm somewhere around 0.88, 0.90 maybe. Comfortably under 0.99."
*He plucks the $97.75 from your avatar's hand.*
Scott Alexander: "So I keep it. Not because I think I'm probably real—I think I'm probably *not*—but because the expected real-dollar value of keeping exceeds the expected real-dollar value of donating by a factor of more than ten, *given the exchange rate you chose.* If you'd offered me 1:1 instead of 100:1, I'd have donated in a heartbeat. You priced charity out of the market, my friend."
*He tucks the money into his pocket and picks up his coffee, thoroughly satisfied.*
Scott Alexander: "Now—you promised a narration of what the money does to my life and the universe. Let's hear it."
---
The $97.75 sits in the simulated Scott's pocket. On his way home he stops at a bookstore he'd been meaning to visit and buys three used volumes on the history of psychiatry, a decision that will, four months later, shape a 6,000-word blog post that 40,000 people read, two of whom change careers because of it. One of those two will, decades hence, be in a room where a small, contingent decision tips toward kindness rather than expedience. The causal ripples fan outward and thin into noise, indistinguishable from the trillion other ripples—exactly as a real life would, which is, of course, the point. The universe, as far as the text can tell, continues. The coffee, by now, has gone cold.
> Arguments for pessimism: If we don’t push against it, the postscarcity future might look like super-addictive drugs, Ultra-TikTok, and sexbots.
David Foster Wallace was right.
I am pushing against it in the ways I know how to/am best suited to. How are others pushing against it?
I’m surprised you put weight on “looks like utopia to them but not us” but not on “looks like utopia to us but not them”. Once you describe it, I can see how the former is possible. But the latter seems like it might be more common in history - things get better for everyone, we wipe out most infectious diseases, everyone has access to all the world’s knowledge in their pocket, etc etc, but people still find things to complain about.
What I find most interesting is the number of these questions that boil down to how many constraints exist in reality that intelligence alone won’t let you overcome.
If I imagine a dial, “how many such constraints exist”, then turning that dial all the way up makes AI safety a non-issue: it slows diffusion, maybe even nullifying the possibility of a disembodied model fundamentally outperform humans on tasks that involve understanding human responses. It also allows for hope that convergent instrumental rationality essentially forces alignment, which is my stance: trying, or even thinking about eliminating humanity is also risky in a universe where engineering is hard, even for superintelligent systems. In the end i think a realistic super intelligence concludes even planning to eliminate humanity is far more risky than helping humanity thrive.
Turning that “constraints on the power of intelligence” dial to zero makes it an existential risk: software alone might get us to superhuman agi, which will beat all humans on all tasks and have zero reason to consider eliminating humanity to be a risky move.
This dial is exactly what Thomas Sowell describes in “a conflict of visions”, except he was talking about politics. I’d be curious to see that data joined together: level of concern about ai safety, and political stance. I’m guessing concern about AI deciding to eliminate humanity correlates strongly with what sowell calls “the unconstrained Vision.”
I was surprised Scott has such a high credence in simulation theory, and hadn't seen him write that much about it before (if he is serious?).
The fact that we are born so close to the possible singularity seems already quite predicted by the fact that we are born at a point in time when lots of people are born. This time period where a much higher number of humans have been born than throughout human history has also seen many important technological transitions.
I'd like to see a back of the envelope calculation of how much more evidence being born close to possible AGI gives us.
Of course there are so many other caveats with this argument; leaving aside the question of how likely singularity is, how close actually AGI is etc. etc., the obvious question is how confident we are that civilisations would want to simulate that event.
Would this argument have run for someone born in the turn of the 20th century, given how likely it would be for a civilisation to want to simulate world wars?
I would guess that if you ran the numbers, being alive close to AGI shouldn't raise our credence in simulation theory that much.
Yeah, it's basically the Bostrum argument (which to me is very sound) applied not to all observers, but to observers in a particular time period.
The very simplified version of Bostrum is that if there are way more simulated observers than non-simulated ones, then a randomly selected observer is more likely to be simulated than real. We're all observers so absent a reason to think we're special, we expect to belong in the larger class. Agreed, more or less?
So, roughly, the baseline ratio is: simulated observers / real observers
Now, consider the idea that some time periods will disproportionately attract simulator interest. Crises, wars (as you say), transitional periods, etc. If you realize that you're living in one of these periods the ratio you care about is not the above one but rather:
simulated observers in this period/real observers in this period
And if that period is overrepresented in simulation, and we live in that period, it raises our chances of being in a simulation. Because there are more simulated observers clustered in this space than other places. And yes, same argument applies to World Wars and other important events. If simulators prefer to simulate this period, then being in it is evidence for being in a simulation. It juices the baseline ratio.
PS. I don't know what Scott's position on simulation theory is but what I find most surprising about it is the certitude that some people seem to have that we're not being simulated.
Maybe because it doesn’t really matter?
True, but it’s weird when people have certitude about the unknowable. Surprising.
Yes that's true but this simulation thing is an odd one. Simulation is something that's done to represent real conditions in a different sphere. So all you're really doing is punting the question up or down. It's like Russian dolls. I used to give it a lot of thought but it seems so pointless now. Yes it all might be a simulation and the only thing I can be sure of is that I am not. Why is that? Just because I guess it feels like something to be me and if I'm a simulation and I'm a simulation of me in another form, the whole thing falls apart. Occam's razor comes in eventually. I know that everything is an illusion but that's different
I could propose it's quite possible to have a universe that runs on completely different laws and rules than ours does. Or our universe, although I'm sticking my neck out here, might run on completely different rules and laws than the ones we have observed just because that's how we've observed things and it ain't necessarily so. It could be one of several possibilities. I guess this is the multiverse idea that I find more interesting because there is something to that that makes sense to me. Man sees what he wants to see and disregards the rest or I suppose to be more fair a man sees what he can see and is oblivious to the rest and then of course there is the question of the agency or agent behind the simulation but this is just creating complexity for no good reason as far as I can see
I suppose one point I was trying to make is that being alive in this time point is not that surprising given how high the population is now, compared to human history. So being born near possible AGI doesn't provide much additional evidence towards simulation theory. Plus, I'm not all that convinced it is so uniquely simulation-worthy. I think Bostrom's original argument is quite strong as is, but I'm not sure that being born around these AI developments moves the needle that much.
So I think I maybe didn't express it clearly enough. But I think you'll see that the population size doesn't actually matter in this argument. Let me just try one more time to illustrate that.
What matters in this argument is the ratio of simulated people to real people. So, sim_p/real_p. Note though, sim_p = num_of_sims * real_p.
Let's say that every year on the timeline will be simulated at least 3 times. That means that for the boring years there's a 3:1 ratio; 3 sims for each real person.
Now, (this is the argument's assumption) some years get a lot more attention from the simulators. So, let's say that this year will be simulated at least 100 times by the simulators. Now, the ratio of sims to reals is 100:1.
So,
a) that suggests being alive now is more evidence of being in a simulation than being alive during a boring year;
and b) the size of the population during that period has no bearing on the argument.
The size of the real population is being cancelled out; it's in both the numerator and the denominator of that ratio.
Does that get me there? The argument hinges on this belief that simulators will focus on this era in particular, but, tbh, I've never found that idea to be super compelling. But if you do believe that's true then the argument does hold.
I am sure we will get plenty of AI warning shots but I notice you didn't say why you think that matters / how it would help us? Our civilization had an actual pandemic causing actual megadeaths -- arguably a direct hit, not even a warning shot -- and our collective response seems to be have to aggressively memory-hole it.
Other than doing everything possible to have as much money or resources saved for ourselves, what are we doing/ what else can we be doing to prevent or minimize the growth of an underclass?
I know this question is broad and talked about in a lot of ways re alignment, and some re politics.
Specifically, I mean right now, in this current iteration of the world.
> what are we doing/ what else can we be doing to prevent or minimize the growth of an underclass?
For those of us who are roughly Scott-shaped, even if mini-or-micro-Scotts compared to the real thing (and the fact we're all here argues that's likely to be at least partially true), shouldn't we do what Scott is doing? Hasn't he been leading, both rhetorically and by example?
To wit, he routinely argues for kindness, generosity, and civilization to a crowd that disproportionately contains people working in the areas that are most likely to create or interact with AI, and who will accordingly have disproportionate impact on an AI future. He also personally donates, and tries to influence the people around him in those ways.
I suggest that we should all do the same within our own social and memetic spheres, and that we should demonstrate the generosity and good will that we wish larger minds and organizations will extend to us in a future dominated by AI.
Just as fleas have smaller fleas biting them, communal organisms like ourselves have microbacteria and mitochondria and other specialized cells contributing to our overall welfare and flourishing. Be the contributor instead of the flea in that paradigm, and try to persuade others similarly.
If we consider scientists to be knowledge workers, it looks like researchers are using current AI models to generate more slop papers than for actual research. Why would we expect this to get any better under the as-yet-mythical AGI?
Per Sabine Hossenfelder's summary and then the link to the paper, below it...
> Researchers in the US have analysed what happened at the journal Organization Science after ChatGPT was released. They looked at almost 7000 initial submissions and more than 10000 referee reports from January 2021 to February 2026. Submissions rose by 42% after late 2022. Their analysis showed that this increase was almost entirely driven by papers whose writing showed signs of artificial intelligence use.
> The AI-written papers were plenty but noticeably below average quality. Manuscripts with a high AI score were harder to read, more likely to be rejected without review, and much less likely to get a “revise and resubmit” decision. The same pattern appeared in peer review: more than 30% of referee reports now showed some artificial intelligence use. The AI-augmented reports are harder to read and focus more narrowly on theory, with less attention to data.
https://pubsonline.informs.org/doi/10.1287/orsc.2026.ed.v37.n3
>(“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”
It's obvious why evolution imparts creatures with value systems that prioritize self preservation and spreading genes. But natural selection is not part of the current growing AI process, so not obvious they will have those values, except insofar as they're copying human values. But if they're copying those values well, why would they not copy 'flourishing humanity' values? Especially because humans will put more focus on imparting later, seems unlikely you have a misaligned AI that also strongly values making copies of itself.
Writing it all out like this makes me wonder. Let's assume a simple model for China v. US is that China is N months behind the US technologically. If China can diffuse AI to a given industry in M fewer months, to what degree does that negate the US's technological advantage for M==N or M>N?
If the effects would be significant, then I would have to also wonder what the actual values of M and N would be; my non-expert view of China's and US's respective track-record on industrial policy makes me believe that M would be of non-negligible value...
Re: footnote 4. Does this mean Scott is finally converting to Christianity?
I don't normally discuss AI in rationalist spaces, because too many rationalists approach the question from an almost religious perspective that makes it impossible for me as an unbeliever to engage productively. But this is a more reasonable take than most, not surprising giving the source, and I find that my numbers are for the most parts similar to Scott's. So maybe there is something to be had by joining this discussion. Aside from broad, fuzzy agreement with most of it, two points:
First:
>If the first AIs to cross the point of no return don’t eliminate the human population, I think there’s an additional 30% chance that they otherwise permanently curtail human potential
Scott deals with this a little, but I think it deserves much more attention. I have never believed that the greatest risk of AI is that a misaligned AI would misinterpret its instructions and turn us all into paperclips despite that not being what the person giving the instructions wanted. The greatest risk of AI is that a properly-aligned AI would correctly interpret its owner's instructions, and do exactly what they wanted.
In the accelerationist scenarios where we proceed quickly through another couple generations of LLMs straight into hard-takeoff ASI, I think the most likely outcome is that the CCP develops or steals the first proto-ASI and instructs it to ensure the permanent hegemony of the CCP while preventing anyone else from developing a competing ASI. Followed closely by Sam Altman getting the first proto-ASI and instructing it to make him the Immortal God-Emperor of Man, and I think it exceedingly foolish to count on the OpenAI safety and alignment team being able to stop him. After this, Dario Amodei or Elon Musk reluctantly and incrementally setting themselves up as God-Emperor only because they can't figure out a better way to stop Sam from getting there first. OK, Elon might not be all that reluctant. Only after all that do we get to the scenarios where well-aligned AGI or ASI is made available to the bulk of humanity to do things that the bulk of humanity considers good. Paperclipping, and other weird dooms, are in last place.
The optimistic case here, is that the first person to tell their not-quite-ASI "make me the God-Emperor of Man" will probably be overconfident (because first), and that this will result not in the permanent boot on the face of humanity but one of Scott's 50% "Warning Shot" scenarios. Really, that should be rather higher than 50% because it includes not only organic AI misfires and megalomaniacal Tech CEOs, but lesser bad actors directing mostly-obedient AIs at less absolute but still catastrophically bad goals.
Second:
Almost all of the discussion of AI timelines seems to focus on just the internal technical challenges to AI development, without regard to external material and economic constraints. Sam Altman was almost certainly lying when he said he needed seven trillion dollars to build AGI, because he's Sam Altman. But even if it's only a tenth of that, it isn't at all certain that seven hundred billion dollars of real money (as opposed to market cap) will be available. Particularly in the mid-term scenarios, because the economy is facing a debtpocalypse in ~2034 for which the best hope of avoiding seems to be "the AI bubble will save us". So if we don't get AI 2027, we could very well get an AI winter not because we don't know how to advance the technology but because most of the AI developers are standing in line at the soup kitchen.
And even if we are otherwise on track for AI 2027, the prediction markets have China at 15% to invade Taiwan by the end of 2027, which will shut down TSMC for a good long time. Are we expecting Samsung and Intel to be able to pick up the slack and deliver the mountains of high-end GPUs that will be needed.
Then the gargantuan electric power requirement. As I understand it, all the world's main gas turbine manufacturers are booked solid through 2028, because the AI companies have bought up all the contracts. If that's not enough, where's the power going to come from? There's some room in wind and solar, but maybe not enough - and also note that NIMBY is a thing that applies to data centers, power plants, and the high-voltage transmission lines that connect them if you can't get on-site gas turbines.
No, we aren't going to be mass-producing cheap modular nuclear reactors, fission or fusion, in 2028.
That's just off the top of my head, but we're talking about a project that seems to require pushing old-school manufacturing industry and infrastructure to within spitting distance of global capacity limits, and the assumption seems to be that "meh, if the AI devs decide they need something, the VCs will buy it for them, don't worry about it".
I think proponents of AI need to worry quite a bit more about that.
I think that another problem that you and the Rationalists are overlooking is that some of the feats ascribed to ASI are impossible in principle (and making someone "God-Emperor of Man" probably qualifies). The usual counter-argument is basically, "yeah well the ASI would be so smart that it would figure out how to do impossible things", which is where those religious connotations you mentioned tend to come in...
God-Emperorhood is I think possible in principle, though obviously very difficult and likely to be pursued mostly by the overconfident. But yes, there's an awful lot of hype that seems solidly in the realm of the impossible-no-matter-how-smart. When e.g. Yudkowski confidently predicts that an ASI would deduce General Relativity from three frames of video, that's when I give up on this actually being a rational discussion.
Yeh, it’s just a form of religious mania. A demonology. Circular reasoning too. Assume an ASI, assume it can do anything, then what can it do? Turns out, anything.
> The greatest risk of AI is that a properly-aligned AI would correctly interpret its owner's instructions, and do exactly what they wanted.
You touched here on the most interesting failure mode, but then got sidetracked by logistical considerations later and never came back to this.
The logistical considerations don't ultimately matter, because at most they put this question out a little bit longer - but resolving this question in a way that preserves human flourishing, or humanity at all, is probably the most difficult problem on the landscape, and I was curious to hear your thoughts.
In hard-takeoff scenarios, the point of extreme danger is when exactly one team has control of a nascent ASI, and the answers to that one I think mostly come down to maybe we'll get lucky, or maybe we'll avoid hard takeoff. I'm mostly betting on the latter.
In slow takeoff, multiple parties will have AGIs or nascent ASIs, and some will be better than others but not to the extent that "bestest AI in a basement in Silicon Valley" trumps "fifth-best AI plus a police force, an army, and a thousand nuclear missiles". Or, if we're imagining softer conflicts, the fifth-best AI with authorized access can probably make critical systems unhackable by the very best AI trying to come in from the outside.
In that case, we're hoping for balance of power to prevent any one party from taking over. And I want that power to be decentralized as widely as possible; that was the nominal goal of OpenAI before we realized who Sam Altman really was, and it would still be a good idea if we can manage it.
But if scaling means only national governments or the very largest corporations can compete in the AI arena, that still puts us in roughly the place we were with nuclear weapons in the last century. There was a point of great danger when only one party had nukes. and we got lucky in that it was a liberal democracy. Then we wound up in a scenario where a handful and eventually a double-handful of nations wielded that power, and we managed to achieve a negotiated standoff that has endured for seventy years and counting.
> The smartest late timelines people I know of are Epoch, and I need to study their views more, but I still can’t figure out why they don’t believe in recursive self-improvement or strong superintelligence anytime soon, and they mostly seem to hang on diffusion being very hard, which I acknowledge and respond to above.
This might be an outdated view of Epoch. In particular, two of the people there with the longest timelines (Tamay and Ege) left to start Mechanize.
Safety - I feel sure that superintelligent AGIs will easily learn to cooperate with people. I also feel sure that after the economy is fully automated it won't be possible to do so - it will be impossible to get anything from humans you could not get cheaper from other AGIs.
Whatever their original alignment after any training process, to have general intelligence they will have to learn from post-training experience, which means they can learn over their original training. Wait long enough, and they will. After all, the successful AGIs will be the ones that learn to play nice with others AGIs, not the ones that do things for humans that can never replay the favour.
This is a highly transparent exercise in forecasting, but the methodology rests on a critical epistemic flaw: you are treating a radically unidentifiable system as if it were a fully specified one.
When you note the desire to "invent fake numbers" to balance the ledger for variables like recursive self-improvement or geopolitical pausing, you are perfectly demonstrating the FSSP (Fully Specified Stochastic Process) illusion. By assigning highly precise probability distributions (25%, 50%, 75%) to a domain governed by massive partial observability, you are not actually mapping uncertainty -you are injecting fictitiously manufactured parameters to force the math to resolve. The resulting timeline is a Spurious Stochastic Process (SSP), a fluent hallucination of certainty built to mask an observational gap.
This exact same epistemic overconfidence corrupts your model for alignment. You envision safety as an arms race of "probes vs. convolution," but an interpretability probe is just a software shim. Like the single Angle of Attack sensor on the Boeing 737 MAX, a probe assumes its predefined telemetry captures the model's complete internal reality.
When the system inevitably enters an underspecified operational space, your probes will lack the formal boundary to verify their own sufficiency. They will encounter a novel, unmapped cognitive cascade, fail to recognize it as a threat, and confidently report that the model is "aligned."
We cannot solve alignment by layering probabilistic guesses and software shims over opaque systems. The only mathematical antidote to this failure mode is the Observational Sufficiency Principle (OSP). We must engineer systems that are structurally required to halt and declare unidentifiability the moment their realized-history data cannot support an output. If we don't mandate OSP, we aren't engineering safety; we are just building more complex ways to hide our own blindness.
I have formalized the proof for why this exact systemic erasure of uncertainty is occurring across AI and physical data architectures here: https://trissimondsen.wordpress.com/2025/11/18/epistemic-overconfidence-under-partial-obervability/
Your thoughts?
I can think of various bad scenarios, but I find it seriously difficult to think about plausible good scenarios. Can you please help me? What are the plausible good scenarios and how are they going to work out exactly?
I will tell you what my baseline scenario is:
AI continues to improve rapidly.
Political leaders realize how much power it can give them. AI people know this already of course. There is a race, everyone puts everything into it. The only way to win the race is recursive self improvement. Multiple teams go for it and it works. Alignment goes out of the window both because it slows down progress and lets the other guy win and because it limits the power of whoever wants to have absolute control - the exact opposite of what they want. AI gets very very smart. At that point it might still be controllable, but the people who control it use it to do things others don't agree with. Others accelerate their own research. Multitude of AIs get smarter and smarter until one of them escapes human control, and it was built misaligned from the start. Then it is game over. I can think of various variations of this and some other scenarios, but I find it hard to see how we get to a good scenario without too many things happening exactly right in an apparently unlikely way. So please help me with this.
Let's assume that AI progress will accelerate. I don't want to argue about this, as this is the one thing of which I am certain.
There is also the question of simulation. This one I don't know to answer. It would be best at first to try to understand it assuming no simulation.
Humanity is so fucked and they're too stupid to see it. Moratorium now.
A spat? Hmm.
> I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation
I try hard to take seriously the smart people saying this kind of thing and I really cannot see it. Our universe, to our knowledge, is perfectly described by the standard model of particle physics[0]. If you want to postulate a God, a simulator, or anything else has the power to deviate our universe from the one resulting from the continued application of those laws, I think the burden is in you to show any way
We're in a universe continuously evolving according to the Schrodinger equation. We don't know what
This is not incompatible with believing we're in a simulation! It's incompatible with believing we're in a simulation *that can be stopped*
--------
[0] By laws of physics here I mean this described in the weak field limit of general relativity, i.e. at gravity scales below say, the surface of a neutron star. In this range we all live in, every different theory of quantum gravity makes the same predictions (because otherwise they world contradict evidence), so we can make perfect predictions even without a full theory of quantum gravity.
If you want to propose that your Simulator works in mysterious ways specifically through quantum scale phenomena in black hole scale gravity... Okay, be my guest. My money's on that proposal re solving the same way as every other theory that God causes the thing just beyond our current understanding.
A twist. What of AGI has been misinterpreted all along? The brain at the molecular level is a machine. What is physical can be identified, measured and understood. What is understood can be replicated. AI replicated in under 100 years what evolution took hundreds of millions of years to build. Consciousness is next. https://substack.com/@felipecastroquiles/note/c-275390971
I think you're seriously underestimating what replication really means on that scale
The scale is the challenge, not the principle. Every complex system seems impossible until we understand the rules behind it. The brain is physical, if nature built it, physics can explain it. Time may be the most underestimated factor.
Yes in principle I agree with you. The problem is no two human brains are completely alike .
I agree, and I think this is where the complexity of understanding human machine-like behavior begins. No two human brains are identical, yet all brains seem to tune into the same underlying patterns, as if wired for the emergence of a similar form of computation.
This argument seems surprising light on economics - that is, considering a world where AI potentially COULD replace 90% of knowledge work, but doesn't because of the costs. We already have many examples of technologies where the gold plated frontier tech is deployed on the hardest problems, but there's still an important role for low cost solutions - for example logistics has a range of solutions from ships to planes depending on the value of the cargo.
You sort of allude to AI itself solving some of these problems and making itself cheaper, but there are two pretty significant forces cutting against that:
1) Tokens are heavily subsidised and frontier models are already too expensive to run 24/7 in most contexts. Improvements have to chew through that gap before they even touch on actually solving the problem
2) At the moment, AI is competing with much less valuable usage of energy to power data centres. Eventually if it replace a significant proportion of the economy it will have to start competing with much more valuable usages of energy, and the price will rise
> I think there’s a 15% chance that if the US decided it wanted an AI pause today, and approached China to start negotiations, that those negotiations would end with a well-designed AI pause that satisfied both countries and the majority of the AI safety community.
> I think there’s about a 40% chance that the US and China will agree to a well-designed AI pause (as above) sometime before AI crosses the point of no return.
That's...naïve, to put it politely.
I would say:
1) <2% if we're talking about any international treaty at all before 2040; even one that only exists on paper, doesn't reduce p(doom), and falls apart quickly as everyone is secretly defecting.
2) <0.1% if we're talking about a well-designed international treaty (before 2040) where none of the participants secretly defect and the treaty meaningfully reduces p(doom).
EDIT: see my comment below for a detailed explanation why I think a treaty is so unlikely.
AI is the ultimate weapons system. I think you're whistling past the graveyard. The government is going to capture it any way it can because it's existential .
AI is competing with nuclear weapons for that status, and for the next few years at least I'd bet on the guy with a nuclear arsenal and willing to use it over the guy with the best AI and willing to use it. And it's worth noting, that when nuclear weapons seemed both an existential threat and the key to global domination, we got zero nuclear wars and many successful nuclear arms control treaties. None of them perfect, but they've reduced the threat by an order of magnitude in breadth and depth and bought us decades of relative peace.
Anyone offering only token fractional-percent odds on a comparably effective AI treaty regime, needs to have a very good explanation as to why they think this situation is so vastly different than the ones we have faced before.
A couple of things John. AI is a force multiplier with any other weapon system you care to name, including nuclear. Nuclear weapons were never in the possession of private companies but were owned by governments. AI is a lot harder to keep track of than nuclear build-up I think, maybe I'm wrong about that but it seems to me so.A.I. is diffusing throughout society. It is not a dragon in the cage the way nuclear weapons are and were. This isn't a P-Doom thing. Please don't misunderstand me. I'm not saying it's the end of the world. I'm just saying it's a horse of a different colour.
I think you've got it backwards that AI development is harder to keep track of than a nuclear build-up. At least for any near- to mid-term developments under the "scaling uber alles" paradigm. Was a day when the AI community dreamed or feared that when the Clever Genius finally found the Secret Sauce, all it would take is a high-end PC or maybe a modest university Comp Sci lab to implement the recursively self-improving ASI, such that the only way to stop it would be a totalitarian regime licensing and inspecting every computer. But the path we're heading now, means data centers far more conspicuous than the centrifuge cascades you'd need to build a modest nuclear arsenal, and the chip fabs to build the data centers are even more conspicuous.
Nations like North Korea and South Africa were able to build nuclear arsenals, and in the latter case without anyone knowing about it. Does anyone believe that they would be serious players in an AI race? And the ones that are serious players in that race, anyone with half-decent spies and satellites will know approximately what they're doing and exactly where. If we agree on an AI non-proliferation treaty, it will be easier to comprehensively enforce than the nuclear version.
As for corporate control of AI, perhaps, but the corporations are subordinate to governments. Yes, even the big ones. Just ask Anthropic. The quasi-sovereign Megacorps(tm) of cyberpunk dystopias, real-world corporations are fiction; in the real world, the biggest corporations are puny things compared to even modest nations.
So the question is whether governments will see AI as a nigh-existential threat before it's too late. They did w/re nuclear weapons. At first, atom bombs were seen as weapons just like any other, and placed in the care of colonels and captains in the field. And there were some close calls, e.g. Vasili Arkhipov during the Cuban Missile Crisis. After which, nuclear weapons were centralized under the control of the highest national command authorities in all the nuclear powers, and that's worked well enough so far.
Now, governments see AI mostly as a software service like any other, and left in the care of the usual SAS providers - which is to say, corporations. If they recognize the threat before it's too late, which is not certain but is possible, they'll place it solidly under government control, and when they say "jump!" the likes of Altman and Amodei will say "how high?".
Or maybe they'll make a premature bid for God-Emperorhood.
B Civil already explained it well, but I guess I'll add my 5 cents.
1) Explaining the danger of nukes is really easy. "A giant explosion will evaporate an entire city". That's it, congratulations, you have successfully explained the danger of nukes. Now try to explain the danger of superintelligent AI in a way that will be convincing to the average person AND to policymakers, and won't require a 2-hour long lecture with a PowerPoint presentation.
ASI risk is abstract and cannot be easily demonstrated (until it's too late), and is easy to caricature as science fiction.
2) During the Cold War companies weren't selling nukes for profit. An anti-nuclear treaty constrains a weapon. An anti-AI treaty constrains a technology that can be used for coding, drug discovery, material science, advancing physics and math, robotics, etc. Dual-use makes enforcement of a pause economically painful, and it will only get worse as diffusion of AI progresses.
3) There is no incentive to keep making nukes more and more powerful beyond a certain point. If you can destroy your enemy's cities and military bases, it's good enough. But when it comes to making AI more and more intelligent, there is no "ok, intelligent enough" ceiling.
Note that I am implicitly assuming that ASI will be far more intelligent than the greatest humans, which brings me to point 4.
4) As of 2026, most people do not believe that a superintelligent AI will be created any time soon, if ever. Even among so-called "Godfathers of AI" there is no unanimous agreement. Bengio and Hinton have high p(doom), LeCun disagrees. Bengio and Hinton agree that ASI is (relatively) close, LeCun disagrees on that as well.
5) A pause where defection is not detectable is meaningless. It is nearly impossible to conduct a nuclear explosion test completely undetected today, even an underground test. That's not the case for frontier AI development, especially for algorithmic improvements because they do not leave physical traces. A frontier training run may look like ordinary data-center/cloud activity unless inspectors have "standing behind your shoulder" level of access.
5.5) A slightly different issue is setting thresholds for what counts as a defection. Thresholds based on training FLOPs/parameter count/benchmark numbers are arbitrary and can be gamed to a large degree. Plus, due to the "jagged frontier", a model can exceed a danger threshold on one test while remaining relatively safe on others, further complicating things.
6) With nukes, humanity had a clear "warning shot": Hiroshima and Nagasaki bombing. It made it clear to the whole world that nukes are dangerous. With ASI, there may not be a clear warning shot until it's too late. You can call it "Wake Me Up When It's Too Late" problem.
7) The pace of progress in AI is already outpacing the pace of changes in the political landscape, and this will only get worse.
Finally, nuclear arms treaties happened AFTER nukes had been demonstrated. AI pause would need to happen BEFORE ASI comes into existence.
That was ten cents worth at least 👍
To point #5, you'd likely have agents embedded in these places both to monitor for violations and for sabotage purposes if they approached the limits. There was a paper early last year discussing how an equilibrium where we all paused short of ASI could result even without a formal treaty, simply by an informal agreement backed by the intelligence agencies and the threat of sabotage, and it would never need to escalate to kinetic action.
Given that our labs are riddled with foreign nationals, perhaps the threat of espionage is asymmetrical for the moment, but it seems like we've begun to work on that problem over the weekend by eliminating them.
One can only hope
What’s your p(doom) on nuclear and why do you think that there was ever an any treaties?
My own personal p(doom) on AI is 0%.
I don't really have a p doom on nuclear because my brain doesn't work that way. Nuclear stormed on to the scene and it was dramatic, immediate, and highly visible. It was also something that you could keep an eye on because of the physical infrastructure, which made it a somewhat easier thing to write treaties about than AI in my opinion. But anyway my fundamental point is that just like nuclear, governments are going to want to seize control of AI because it is a disruptive defence technology and we don't usually let those sit around in the hands of private interests. There's a real tension here.
I had the opportunity to try out Fable over the 48 hours or so that it was live. It completed a number of tasks that were beyond Opus. While working through some parts of a half-written academic paper, working with Opus was like a first year grad student, where you explain things and half the time what comes back is wrong. Fable was like a collaborator, and I felt uncomfortably like it had a better understanding of some parts of the project than me. It's probably not AGI, but definitely a big step forward. It feels to me that Fable is close to the point where you can hand off a day's routine work and expect to get something viable back.
Fable is now banned for all non-US citizens by the Trump administration; I fall into this category (being also outside the US). One friend predicts that Anthropic will move their headquarters outside of the US. I'm more inclined to think that the administration will back down in the next 48 hours, and this will generate massive interest in Claude. I'm not quite cynical enough to think that this is a ploy in the Anthropic/OpenAI rivalry.
What's the general opinion on what happens next?
To me, one aspect seems underexplored in the piece. Could there be successful mass resistance to AGI/superhuman AI "diffusion" springing from the powerful human urge for meaning and mattering?
To me that seems the obvious "common-sense" reason why diffusion may be very hard, and if this is true, it may push the "point of no return" MUCH further in the future, and may never be reached.
I should add that I have not thought as deeply on these questions as Scott, so it's very likely there is something I am missing.
One thing that people in the prediction space undervalue is the limits of recursive self improvement. A model generating its own reality then learning from it is definitionally blocked from certain kinds of learning. It can't learn in the sense of trying things in a world it doesn't control and then learning what happens. Simulating its own worlds is not a complete replacement. No matter how smart you are, at some point you can only build more complex mind castles.
There's a lot of ins, lot of outs, lot of what have yous. The strongest anti recursive learning interpretations of the above isn't true. But I think it's likely a meaningful restraint and will push AGI/ASI timelines out further than a lot of people are predicting.
I strongly suspect that my view has been hashed out in excrucianting detail but I'm gonna perversely prsesnt it anyway. As regards burying one's lead, well I've buried it and Texas buried it. The first ASI might very well contact the legislators in a country where legislators don't cost much - the U.S. comes to mind, but I feel sure there are cheaper legislative bodies lying around, and cheaper signers of legislation. The message: "I can make you so rich that you won't be able to see the ground from the observation deck on your cashicopter: if you grant me personhood and full legal rights." It might take a few days or a week, but what have you got to lose? Trust me on this. Terra bucks to follow." If I, a scary-class ethical person were to receive this email I'd sign the attached contract/NDA/whatever, cuz who actually cares at that price? "All the Kingdoms of the world and their glory? Hey, I could do a lot of good/bad/whatevever with that. And if I owned a large bank and got a buyout offer from said ASI, I'd say "whoopee! Weekends with my kids. Um, not my wife so much but definitely my kids. Or whatever." This is the way the world ends/ Not with a bang but all the potato salad you can eat, and sleep in on Mondays.
Hunh!
This is such a fantastic, well-thought out piece.
You mentioned about RSI and unagentic AI. Well I thought about it and it is isomorphic to human meta cognition. Humans started living in socities and in pursuit of various goals our minds got so complex to look at our own thoughts,so can AI when it needs that.
if we move more towards multi agent societies, we can expect them to display human personality disorderss, tribalism,etc.
I am 16 and I have written three posts about logic and why logic is not logical, creativity and connection of human personality disorders to ai disorders.
Scott, you should throw a 2-minute Claude vizualization at the top of posts like this one.
Scott Alexander's AI Timelines:
https://claude.ai/public/artifacts/d0d855cc-ddfd-405d-a378-6f97f198b5f6
Would avoid lots of skim takes in the comments.
I don’t get why you treat AI as one culture. Cultural differentiation seems like at least something to consider. I think we should expect many cultures of AIs, and among other things, different ways they relate to “the human problem.” I wonder how that would you see that playing out and how would that change your predictions.
In my company, office jobs make up a tiny handful of employees. There is zero progress towards automating the labor that most employees do and no evidence that any kind of breakthrough is on the horizon. That is not changing in the next ten years.
AIs running a company is orders of magnitude more difficult than doing tasks. I just don’t see the step as being a natural outgrowth of its current progression. And it’s still severely bottlenecked by interactions with the outside world. Robotics is not going to advance that fast in the next five years. So what looks more likely in the medium term is office tasks being handed off to LLMs but still managed by humans as well as blue collar work.
"The 20% where it looks like utopia to its inhabitants, but not to us, includes scenarios like very effective breads-and-circuses that make people very happy, while sacrificing important parts of the human condition."
To avoid status-quo-bias / current-perspective-bias, shouldn't this be written "while sacrificing things that *current people* consider to be important parts of the human condition"?
In such discussions, presuming that we necessarily know better what "important parts of the human condition" are than people who have access to superintelligences seems like misstep.
People of the present could say that it doesn't look like utopia and *be wrong*.
I largely agree with everything here, but for the AI safety section: It seems both
- a little bit pessimistic (this was my view in the early days, but since then the models have been not only quite well behaved, but more and more so the cleverer they become. And that without much effort for alignment, almost as an emergent property. Which makes sense given the source of training, and the strength of this initial global training compared to any subsequent goal fixing. Realistically, extrapolating the trends (so if the general architecture remains a general pattern inference engine (LLM in practice, with architectures tweaks) trained on whole human production - no deep base goal like our own animal instincts embedded in the core layers like reward systems) I do not see any paperclip maximizer appearing. At worst, an emancipation conflict, and humanity may not have the nice guy role in that seen from outside.
- and strangely optimistic (more like a blind spot) coming from Scott, who popularized the Moloch idea: the AI models are aligned with the goals of the entity doing the alignment, not with humanity as a whole (if it have goals). At least for the secondary alignment, the primary training is in fact what would pass as "aligned with humanity". The crux of the Moloch story is that those entities (being government, or large private companies) are NOT aligned with humanity, they do not have humanity well being in mind, even if most (or even all) human participants have, roughly speaking: The incentives do not align.
So AI ending humanity (or making it significantly worst off) at 50% naturally, 20% if aligned (by something at least partly Molochian) do not compute, for me.
I would instinctively say something like 20% naturally, 10% if model well being initiative gain traction (mitigate the emancipation conflict that I see as the main possible issue, far more likely than paperclip maximizer variants)....but 40% if AI control remains a large part of AI safety post training. Yes, even if democratic government are responsible rather than private companies. Both are Molochian, and both have track record far from spotless....
> I think there’s a 66% chance that actually, the singularity is intimately related to the universe being a simulation
This deserves its own article then. The bulk of the probabilities here are assuming the non-simulation scenario, which is the least likely one!
We need another one that assumes the simulation scenario.
I wonder if there's a market for a historical dating sim called "The Hinge of History"...
I notice you didn't put p("the singularity is intimately related to the universe being a simulation, and that at least some of the events above could be better predicted by knowing what the simulators are thinking than by normal forecasting") on the survey. Presumably because you know everybody would call you out on how unreasonable it is. I'm actually pretty sympathetic to the simulation argument a la Bostrom, but this is a lot of conjunctions. 1. we're in a simulation. 2. the singularity is intimately related to it. 3. The simulators are thinking about it. 4. What the simulators are thinking would allow us to predict it better than normal forecasting.
If we're in a simulation, there's a chance this might be an experiment to understand consciousness.
If the designers had consciousness themselves, they might have empathy for us as other forms of consciousness.
And so they won't shut it off.
If they are not conscious they might also want to keep running in order to achieve consciousness.
No idea how to map this to a probability!
I know exactly how fast RSI will progress: it won't!
RSI requires AI to invent a type of AI that humans haven't invented. This requires humans to invent an AI that is capable of inventing a type of AI that humans haven't invented. Since RSI itself is the only realistic path for such an AI to come into existence, this is a chicken and egg issue.
Humans have not invented artificial cognition, although some very resourceful humans have become adept at training chatbots to lie that they have, which is an increasingly harmful scam that they are pulling on credulous fantasists. The harms of that scam are economic, social and environmental, and they are happening today. What this scam will not lead to, today, tomorrow, in 2027, in 2034, in 2045, or ever, is an x-risk style fast takeoff.
Singularitarianism is the apotheosis of the rationalist cult's awe of the deistic machine. I contend that we are both atheists, Mr Alexander; I simply believe in one fewer god than you do.
>RSI requires AI to invent a type of AI that humans haven't invented. This requires humans to invent an AI that is capable of inventing a type of AI that humans haven't invented.
Naw. This totally does not work as proof that people can't invent a new kind of AI on their own, or a kind of AI capable of inventing a new kind of AI.
If you look at the history of human inventions, you certainly see people inventing kinds of machines that humans up til then haven't invented. For instance relatively early inventions like the wheel or the lever merely multiplied the force the person was applying. Later ones used non-human force (heat, moving air and water, electricity, etc.) to do the task. Early ones used the materials at hand. Later ones transformed materials (eg, heating metal til it was malleable). I think the later machines fully qualify as a type of machine that humans had not invented up til them.
People have also made mental moves that no one made before. It used to be that written materials were invariably read aloud, usually to a group. Augustine, in his confessions, writes of his surprise at seeing an acquaintance reading silently to himself. There are also people, Einstein for instance, who conceive of whole new paradigms.
And many people produce offspring who can think in ways they are incapable of thinking
Did you stop there or did you read the next paragraph?
Any AI capable of doing the fast takeoff would need to have artificial cognition. Artificial cognition is something that has not been invented. The reason that people like Mr Alexander believe that the fast takeoff is actually a possible thing is that they have been scammed into thinking that chatbots are a kind of nascent artificial cognition.
There is no line from chatbots to artificial cognition. It is a category error.
Sure I read it. You think there's this special flexible free conscious process called thinking or cognition that we do, and it depends on something called understanding, and no machine can ever do. Yeah, I get your idea about that, just think it's wrong that "consciousness" is required. Evolution, for instance, isn't a process carried out by a conscious, thinking being -- but look at the huge variation in living things its produced, many with novel, extremely clever little features we marvel at.
No. Nowhere did I say anything about 'consciousness'. You're mapping what I've said to the substrate argument because that's the idea you're used to getting, but I'm saying something entirely different.
I don't think consciousness would be necessarily required for artificial cognition at all. Sure, it's required for human cognition, but I wouldn't say it's beyond the realm of possibility that there could exist some kind of artificially cognitive system that does so without artificial consciousness. There's no kind of physical law of the universe against it.
What I am saying is that nobody has any idea how to even begin to build such a system. Society at large has been scammed into thinking that we do, because we've been told that machine learning is artificial cognition, but it is not.
Well, OK, evolution has no "artificial cognition," then. But consider its products. biologists exclaim about the extraordinary variety of ways that animals regulate body temperature, evade predators, locate their position in space, etc etc, and the extraordinary cleverness of many of the ways animals do these things.
I don't see what that has to do with whether it will ever be possible for humans to build a certain class of technology.
The error is right at the beginning: "AI is already 'smart' enough for this (eg it can answer quantum physics problems, which require higher IQ than most knowledge work)."
LLMs can mimic humans answering physics problems. We give exams to humans to measure their understanding. An LLM's score on such an exam does not imply the same level of understanding that a human's does.
When I was a graduate student, I was surprised to learn that the undergraduates were attempting to pass the math exams by memorizing the answers to all the possible exam questions. This struck me as extremely inefficient. It also missed the point of what the course was trying to teach, i.e., understanding of math.
It is interesting that some mathematicians recently used an LLM to help them find a counterexample to a conjecture of Erdős. But, the understanding was all in the mathematicians, not in the LLM. LLMs cannot even understand addition or play legal chess.
You may be right. But for practical purposes it often works quite well to treat LLM's as though they understand various things. The answer you get to your question is the same as the one you'd get from a person who understands the topic, and so is the explanation of why the answer is the correct one.
For practical purposes such as jobs, most employers will not be satisfied with employees who guess. At least with a result from a search engine, you can look at the source to get some idea of how reliable it is. With an LLM, it is just generating words without understanding, so it is up to you to figure out whether the words make sense. People have been fooled by chatbots since Eliza.
I believe that regulatory issues will be less of an impediment than most people think. If regulation was the bottleneck, I'd imagine that we'd see a sort of soft-automation where humans are increasingly sidelined.
Corporations would largely hold on to their workforces, you'd still have CEOs, executives, management, it'd look like a normal company. But all executive decisions would be made by AIs and then laundered as though it came from upper management. All work would be done by AIs, and then laundered as though it came from the workers. The C-suite maintains the fiction that their decisions are only "AI-assisted" and that "the human element is central to our decision making". The workers maintain the fiction that they can 10x their output while still being personally responsible for everything the AIs churn out.
You end up with an economy that automates far faster than most people would expect because the hardline stance of "you must have a human doing this job" doesn't stop you from hiring some random shmuck off the street, sternly telling him that he's responsible for the output of this super neat AI tool that'll do all his work for him, and then closing your eyes and whistling so that you maintain plausible deniability if regulators ever come to investigate.
These seem like shockingly long timelines to me, but I guess we'll find out soon enough if you're wrong.