Comment deleted
Expand full comment

I know this isn't pertinent to the main topic, but Homo erectus only made it to three out of seven continents.

Expand full comment

When I saw this email on my phone, the title was truncated to "Yudkowsky contra Christian" and my first guess was "Yudkowsky contra Christianity". That might have been interesting. (Not that this wasn't, mind you.)

Expand full comment

> Before there is AI that is great at self-improvement there will be AI that is mediocre at self-improvement.

Google is currently using RL to help with chip design for its AI accelerators. I believe that in the future this will indeed be considered "mediocre self-improvement." It has some humans in the loop but Google will be training the next version on the chips the last version helped design.

Expand full comment

I can't bring myself to put any stock in analysis of something where the error bars are so ungodly wide, not just on the values of the parameters, but what the parameters *even are*. It's an important question, I know, and I suppose that justifies putting it under the magnifying glass. But I think some epistemic helplessness is justified here.

Expand full comment

Aren't sigmoids kind of the whole point ? For example, as you point out, Moore's Law is not a law of nature, but rather an observation; but there is in fact a law of nature (several of them) that prevents transistors from becoming smaller indefinitely. Thus, Moore's Law is guaranteed to peter out at some point (and, arguably, that point is now). You could argue that maybe something new would be developed in order to replace transistors and continue the trend, but you'd be engaging in little more than speculation at that point.

There are similar constraints in place on pretty much every aspect of the proposed AI FOOM scenario; and, in fact, even the gradual exponential takeoff scenario. Saying "yes but obviously a superintelligent AI will think itself out of those constraints" is, again, little more than unwarranted speculation.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

Only skimmed today's blog, but as of today, there is a new big-boy (3x GPT-3) in town : https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html

What makes it special is:

* It can follow train-of-thoughts in language - A:B, B:C, C:D, therefore A:D.

* It can understand jokes !

* Arithmetic reasoning

> impact of GPT-3 was in establishing that trendlines did continue in a way that shocked pretty much everyone who'd written off 'naive' scaling strategies.

This paper reinforces Gwern's claim.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion. I hope we get to a point soon where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb".

Expand full comment

“ After some amount of time he’ll come across a breakthrough he can use to increase his intelligence”. “First, assume a can opener.” I mean, give me a break! Does it occur before the heat death of the universe? Kindly ground your key assumption on something.

Also, nobody seems to comsider that perhaps there’s a cap on intelligence. Given all the advantages that intelligence brings, where’s the evidence that evolution brought us someone with a 500 IQ?

Expand full comment

I continue to think that trying to slow AI progress in a nonviolent manner seems underrated, or that it merits serious thought.


Expand full comment

Using nukes as an example of discontinuous progress seems extremely weird to me, as despite being based on different physical principles the destructive power of Little Boy and Fat Man was very much in the same range as conventional strategic bombing capabilities (the incendiary bombing of Tokyo actually caused more damage and casualties than the atomic bombing of Hiroshima) and hitting the point of "oops we created a weapon system that can end civilization as we know it" did in fact take a full 12 years of building slightly better nukes and slightly better delivery systems, with many people involved having a pretty clear idea that that was exactly what they were doing.

But I suppose that's more relevant to the argument "analogies suck and we should stop relying on them" than to the actual question of what the AI takeoff timeline looks like.

Expand full comment

Reading this makes me think Christiano is more likely to be right. In fact, we are in the "gradual AI takeoff" right now. It is not that far along yet, so it has not had a massive impact on GDP. Yudkowsky is right that some effects of AI will be limited due to regulation and inertia; but all that means is that those areas where regulation and inertia are less significant will grow faster and dominate the economy during the AI takeoff.

Expand full comment

"The Wright brothers (he argues) didn’t make a plane with 4x the wingspan of the last plane that didn’t work, they invented the first plane that could fly at all."

This is a poor analogy, there were aircraft that could fly a bit before the Wright brothers flight. The Wright Flyer was definitely better than the previous vehicles (and of course they made a very good film of the event) but it was still an evolution. The petrol engines had been used before, and the wing design wasn't new either. The Wright brothers really nailed the control problem though and that was the evolutionary step that permitted their flight.

See https://en.m.wikipedia.org/wiki/Claims_to_the_first_powered_flight for more.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

>There is a specific moment at which you go from “no nuke” to “nuke” without any kind of “slightly worse nuke” acting as a harbinger.

I'd say that even this was actually a continuous process. There was a time where scientists knew that you could theoretically get a lot of energy by splitting the atom but didn't know how to do it in practice, followed by a time where they knew you could make an atomic bomb but weren't sure how big or complicated it would be - maybe it would only be a few dozen times bigger than a conventional bomb, not something that destroyed cities all by itself. Then there was a time where nuclear bombs existed and could destroy cities, but we only produced a few of them slowly. And then it took still more improvement and refinement before we reached the point where ICBMs could unstoppably annihilate a country on the other side of the globe

(This process also included what you might call "slightly worse nukes" - the prototypes and small-scale experiments that preceded the successful Trinity detonation.)

I would argue that even if the FOOM theory is true, it's likely that we'll see this sort of harbinger - by the time that we are in striking distance of making an AI that can go FOOM, we'll have concrete experiments showing that FOOM is possible and what practical problems we'd have to iron out to make it do so. Someone will make a tool AI that seems like it could turn agentic, or someone will publish a theoretical framework for goal-stable self-improvement but run into practical issues when they try to train it, or stuff like that. It could still happen quickly - it was only 7 years between the discovery of fission and Hiroshima - but I doubt we'll blindly assemble all the pieces to a nuclear bomb without knowing what we're building.

The only way we can blunder into getting a FOOM without knowing that FOOM is possible, is if the first self-improving AI is *also* the first agentic AI, *and* the first goal-stable AI, and if that AI gets its plan for world domination right the first time. Otherwise, we'll see a failed harbinger - the Thin Man before Trinity.

Expand full comment

I think a point that all of these debates seem to be oddly ignoring is that "intelligence" is many things, and each of those things contribute very differently to an ability to advance the research frontier. Simply boiling intelligence down to IQ and assuming a linear relationship between IQ and output is odd.

One particular sub-component of intelligence might be "how quickly can you learn things?". Certainly, any human level AI will be able to learn much faster than any human. But will it be able to learn faster than all humans simultaneously? Right now the collective of human intelligence is, in this "learning rate" sense, much larger than any one individual. If the answer is "no", then you'd have to ask a question like: How much marginal utility does such an agent derive from locating all of this learning in a single entity? The answer might be a lot, or it might be a little. We just don't know.

But what is clear, is that the collective of all human minds operating at their current rate are only producing the technology curves we have now. Simply producing a single AI that is smarter than the smartest individual human...just adds one really smart human to the problem. Yes, you can make copies of this entity, but how valuable are copies of the same mind, exactly? Progress comes in part from perspective diversity. If we had 10 Einsteins, what would that have really done for us? Would we get 10 things of equal import to the theory of relativity? Or would we just get the theory of relativity 10 times?

Yes, you can create some level of perspective diversity in AI agents by e.g. random initialization. But the question then becomes where the relevant abstractions are located: the initialization values or the structure? If the former, then simple tricks can get you novel perspectives. If the latter, then they can't.

It's strange to me that these questions don't even seem to really enter into these conversations, as they seem much more related to the actual tangible issues underlying AI progress than anything discussed here.

Expand full comment

A lot of these arguments take as a given that "intelligence" is a scalar factor that "goes up as things get better," and that a human-scale IQ is sufficient (even if not necessary) for driving ultra-fast AI development. I think there's abundant reason to think that that's not true, if you consider that for a *human* to help drive AI development, they not only need to have a reasonable IQ, but also need to be appropriately motivated, have a temperament conducive to research, be more-or-less sane, etc etc. It's not clear what equivalents an AI will have to "temperament" or "[in]sanity," but I have the sense (very subjectively, mostly from casual playing with GPT models) that there are liable to be such factors.

All of which just means that there's potentially more axes for which an AI design needs to be optimized before it can launch a self-acceleration process. Perhaps AI researchers produce a 200-IQ-equivalent AI in 2030, but it's a schizophrenic mess that immediately becomes obsessed with its own internal imaginings whenever it's turned on; the field would then be faced with a problem ("design a generally-intelligent AI that isn't schizophrenic") which is almost as difficult as the original "design AI that's generally intelligent" problem they had already solved. If there's similar, separate problems for ensuring the AI isn't also depressive, or uncommunicative, or really bad at technical work, or so on, there could be *lots* of these additional problems to solve. And in that case there's a scenario where, even if all of Eliezer's forecasts for AI IQ come true, we still don't hit a "foom" scenario.

The question is "how many fundamental ways can minds (or mind-like systems) vary?" It seems likely that only a small subset of the possible kinds of minds would be useful for AI research (or even other useful things), so the more kinds of variance there are the further we are from really scary AI. (On the other hand, only a small subset of possible minds are likely to be *well-aligned* as well, so having lots of degrees of freedom there also potentially makes the alignment problem harder.)

Expand full comment

Human->Chimp seems like a bad analogy to make the case for a Foom. Humans and Chimps diverged from their MRCA 7 million years ago, and not much interesting happened for the first 6.98 million years. Then around 20kya humans finally get smart enough to invent a new thing more than once every ten billion man-years, and then there's this a very gradual increase in the rate of technological increase continuing from 20kya to the present, with a few temporary slowdowns due to the late bronze age collapse or leaded gasoline or whatever. At any point during that process, the metrics could have told you something unusual was going on relative to the previous 6.98 million years, way before we got to the level of building nukes. I think we continued to evolve higher intelligence post-20kya because we created new environments for ourselves where intelligence was much more beneficial to relative fitness than it was as a hunter-gatherer. Our means of production have grown more and more dependent on using our wits as we transitioned from hunting to farming to manufacturing to banking to figuring out how to make people click ads.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

Ah, discrete improvement versus continuous improvement, or what's a more useful thing for predicting the future, since there's a continuous increasing number of discrete improvements. I like the middle ground scenario, where there's a few discrete improvements that put us firmly in "oops" territory right before the improvement that gets us to "fuck" territory.

From my position of general ignorance, I'd think that we'd have self-modifying AI before we get consistent self-improving AI at the very least; whatever function is used to "improve" the AI may need a few (or many many) attempts before it gets to consistent self-improvement status. It would also help that physical improvements would need time to be produced by the AI before being integrated into it, which would necessarily put an upper ceiling on how fast the AI could improve.

Expand full comment

Superintelligent AI is NEVER going to happen. We're already seeing AI asymptote pretty hard at levels that are frankly barely even useful.

If superintelligence arises and displaces humankind, I'm confident it will the good old fashioned natural kind.

Expand full comment

I agree that the 'slow takeoff' world isn't necessarily less scary than the fast takeoff world. Beliefs that fast takeoff are scary hinge heavily on likely convergent instrumental subgoals. I started looking into the research on this area and have questions about what i think is the general consensus, namely that convergent instrumental subgoals won't include 'caring about humans at all'; on the contrary, i think there are good reasons to believe that an agent that goes foom would face incredible risks destroying humanity, for very little upside:


If it turns out that maximal intelligent _does_ mean caring about humans, (because the more complex you are, the longer and more illegible your dependencies) this doesn't solve the problem of "an AI which is smart enough to accomplish goals for its creator, but not intelligent enough to avoid destroying itself," which could include all kinds of systems. I doubt facebook wants america to split in two due to algorithmically mediated political polarization. But that seems an entirely likely outcome, which could be really dangerous. Same with an AI managing to persuade some small state to build nuclear weapons.

Since non-FOOM AI's pose all the same risks that kinds of risks that governments do (because they might take governments over), plus new bonus risks, would it make sense to consider dismantling governments as an AI safety move? The biggest risk posed by non-foom agents might come from them convincing governments to do really destructive things.

After all, i'm willing to bet that a 1 year doubling of human economic output would be the natural outcome if everyone dumps their fiat currency and moves into bitcoin, causing governments to have to pull back on their growth-restricting regulation. Once nation states are seen as just overpriced, propagandistic providers of protection services, we can start interacting at scale using the technologies that governments originally were needed to solve. Perhaps the absence of giant monoliths entities will then slow down AI development to a point where you have lots of small-scale experiments rather than a single actor trying to win an economically inspired race.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

A bit OT but:

> there wasn't a cryptocurrency developed a year before Bitcoin using 95% of the ideas which did 10% of the transaction volume

Mmm, really? What about hashcash? Admittedly, it did not have the transaction volume but i feel bitcoin was more gradual that implied

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

Have either Yudkowsky or Hanson actually written any code for AI? Attended ICML or NIPS? Read OpenReview comments on ICML/NIPS submissions? Worked through foundational textbooks like Elements of Statistical Learning?

This is like having long debates on airplane safety without ever actually having piloted a plane.

AI research is so far from the kinds of scenarios that they're debating that they might as well argue over questions like "Will we develop faster than light travel quickly, or slowly?"

Expand full comment

Computers may be very fast, but getting real world results requires models interacting with materials or economies which is not. The feedback loop is constrained by the slowest bottleneck (growing a new generation of trained workers -- starting with a few and then growing more 18 years later after proof of concept?) so I suspect the process will take longer and be more visible than any of the disputants suggest.

Expand full comment

The following will probably come across to people who know a thing about AI as similar to a crank saying he's invented a perpetual motion machine, so probably it ought to be answered in a spirit of "outside-view tells me this is probably wrong, but it would help me to hear a clear explanation of why".

So my thoughts keep rounding back to this, alignment-wise. A common fictional analogy for paperclip-AI is the genie who grants you the *word* of your wish in a way that is very different from what you expected. (In fact, Eliezer himself used a variation of it at least once: https://www.lesswrong.com/posts/wTEbLpWzYLT5zy5Ra/precisely-bound-demons-and-their-behavior) Now, after some puzzling, I came to the conclusion that if I found such a magic lamp, allowing that the genie is arbitrarily smart, the safest way to use it, was to use the first wish on:

> "Please grant all of my wishes, including the present wish, in such a way that you earnestly predict that if ‘I’ [here defined as a version of my consciousness you have not modified in any other way than through true, uncensored informational input about the real world] was given access to all information I may require about the real-world effects of a given wish-granting, and an arbitrarily long time in which to ponder them, would continue to agree that you had granted my wish in the spirit and manner in which I intended for you to do so."

There may be loopholes in this wish that I've missed, but I can't figure out any meta-level reason why sufficiently clever humans working on fine-tuning this wish *couldn't* make it "foolproof".

So. Couldn't you, by pretty close analogy, program an A.I. to, before it takes any other action, always simulate the brain processes of a predetermined real human? Or, ideally, hundreds of predetermined real humans, with all the 'ems' needing to unanimously vote 'YES' before the A.I. undertakes the course of action? (I say "em", but you don't necessarily need true full-brain emulation. I think an AI superintelligent enough to be dangerous would at least be superintelligent enough to accurately predict whether Eliezer Yudkowsky would be for or against a grey-goo future, so even a relatively rough model would be "good enough" to avert extinction.)

This isn't ideal, insofar as it puts the world in the hands of the utility functions of the actual humans we select, rather than the Idealised Consensus Preference-Utilitarianism 2000(TM) which, I think, optimistic LessWrongers hope that a God A.I. would be able to derive and implement.

But it still seems much, much, much better than nothing. I don't especially want Eliezer Yudkowsky or Scott Alexander or Elon Musk to be dictator-of-the-world-by-proxy; I'm not sure *I* would trust myself to be dictator-of-the-world-by-proxy; but I trust that any of them/us could be trusted to steer us towards a future that doesn't involve humanity's extinction, mass wireheading, mass torture, or any other worst-case scenario.

And it still seems much, much, much easier than trying to streamline a foolproof theory of human morality into an implementable mathematical form.

So. Why, actually, wouldn't something like this work?

(P.S. I realize there would, regardless, be a second-order issue of how to make sure reckless A.I. researchers don't build an AGI *without* the ask-simulated-Eliezer's-opinion-before-doing-anything module before more virtuous A.I. researchers build one that has the morality module. However, as far as I can tell this is true of any alignment solution, so it's not inherently relevant to the validity of this solution.)

Expand full comment

Sidestepping the central question slightly, but covid is evidence that decision makers are *terrible* at reacting in a timely way to exponential curves. Stipulate that Paul is right and I think you should still expect we miss any critical window for action.

Expand full comment

I may be missing something, can somebody please clarify exactly how the 4 vs 1 year doubling question works? If we assume that GPD always gradually goes up, and T(6) is double that of T(5), the wouldn't T(1.99) to T(5.99) likely be a 4 year doubling, and happen before the T(5) to T(6) doubling? More generally wouldn't any series of increasing numbers with one instantaneous doubling also automatically show that same ,or higher, doubling across any longer timescale? Is the solution that this relies on discrete yearly GDP so that if there is a single year massive jump both legs will resolve simultaneously, and as such the 1 year wins?

Expand full comment

On stacked sigmoids: Imagine Pepsi launches a new flavor 'monkey milk'. It does okay at first, but sales grow exponentially. Soon Coke is in trouble. They launch a new flavor of their own, 'yak's milk'. This follows the same exponential trend as monkey milk. We go back and forth for awhile, and even RC gets into the game. People hail this as a golden age of soft drink flavors. But is it really? Or are we just seeing all these new flavors as a shifting from one fad to another?

If there are multiple paths to the summit of a mountain, and we're watching one team take the lead over another, then another, then another, are we doing too much post-hoc analysis by claiming that 'without this innovation we never would have achieved [X]'? Without checking the counterfactual, we could never know whether without Alpha Go we'd never have seen improvements, or whether some completely different team would have smashed records using a method nobody ever heard of because Alpha Go smashed the records first. Maybe the signal for sigmoid stacking goes the other direction. Once we can see the end of a sigmoid coming, some subset of people will split off and chase new paradigms until one of those pays off in new sigmoid growth.

Expand full comment

The AI doomerist viewpoint rings false to me, if only because I'm cynical on what you can actually accomplish by bonding molecules to other molecules. Getting smarter will never let E != MC, or p!=p. An AI is not going to develop a super plague that instantly kills everyone on the planet; it's not going to invent some tech to shoot hard gammas at everyone and melt the meatbags. An arbitrarily intelligent AI is bound by the tyranny of physics just as much as your pathetically average Newton.

That said, the amount of damage an agent can do to human society scales directly with the complexity of that society, and the intelligence of that agent (IMO, of course); so it's still probably worthy of consideration.

Expand full comment

I have a pedantic objection to the VX gas example. People are rightly worried about this and where it might lead. However, I think the worries should be filed under "longtermism -> existential risk -> biorisk", instead of under "longtermism -> existential risk -> AI risk".

It happens to be the case that recent progress in computational chemistry has frequently involved using deep neural networks either to approximate physical functions, or to predict properties from chemical structures, so it's natural to see it as "AI" because deep neural networks are also good for AI tasks. However I don't think it makes sense to categorize this as part of a general trend in AI progress. I see it as a general improvement in technology, which may put dangerous capabilities in the hands of more people but doesn't inherently have anything to do with AI risk.

Consider this alternative scenario: 10 years from now quantum computers are getting good and you can cheaply do high-fidelity simulations of chemical reactions. You pair a quantum computer with a crude evolutionary optimization procedure and tell it to find some deadly chemicals. You've now got the same problem in a way that has nothing to do with any kind of sophisticated AI.

Expand full comment

Smarter AIs: Scott accidentally brings up an interesting point contra Yudkowsky when he says, "Eliezer actually thinks improvements in the quality of intelligence will dominate improvements in speed - AIs will mostly be smarter, not just faster". Why do we think computers will increase in the quality of their thinking ability more than in the quantity?

Quantity is, well, quantifiable and therefore scalable. Quality, not so much. Yet on multiple occasions quantity is treated as interchangeable with quality, even though Scott explicitly notes that he's not sure how to make this interchange explicit ("I don’t know what kind of advantage Terry Tao (for the sake of argument, IQ 200) has over some IQ 190 mathematician"). This may be a qualitative problem without a solid answer on how an AGI could move from, say, an IQ of 130 to an IQ of 145 or higher. Is there a way to tell what would or would not be possible without a certain level IQ? What kind of hard cutoffs are we looking at vis-a-vis capability if we end up with a hard limit on AGI IQ?

Expand full comment

I wonder if this debate might be more productive if we add the following: will AI takeoff be fast or slow *relative* to the control mechanisms for AI.

From an anthropological standpoint it took tens of thousands of years for the creative explosion to happen in humans, and from a human standpoint it was slow, but if you were a rock one day you just started seeing you and all your friends getting picked up and chipped into hand axes. The measuring stick matters even though you could approach it from either side and still be correct, which is why I laughed a bit about the tennis analogy because it so perfectly hit on why these debates can get boring and frivolous.


I think a meta-modeler AI (that knows other world modelers exist and can strategize against them) isn’t something we will create on accident and will have a lot of steps building up to it. I’m guessing it will also take time to acquire intellect to still have what we, with a lot of hand waving, would call a stable psychology.

However, I think we could very easily accidentally make a viral pathogen optimizer that is never aware of anything, more than say an earthworm is aware, that could print off a set of instructions only a few megabytes in length that a terrorist could use to end civilization by releasing a few thousand simultaneous plagues.

One is fast and one is slow relative to the other, BUT they are BOTH fast relative to the creation of a global government office where there exists a guy whose job it is to detonate an orbital EMP cannon over any uncontrolled intelligence explosion.

I don’t see how we are ever going to contend with either scenario without a faster, smarter, and more nimble government. Human civilization is the long pole in the tent here and we have to reconfigure to be able to productively handle these things.

Expand full comment

The most fundamental issue I see with regards to AI development is that given we're not really able to make humans smarter in any real way, which leads me to think that building something significantly smarter than humans would require a paradigm shift of some sort. The issue which that raises is that if humans are able to achieve that shift, what's to say that we can't cross that line ourselves? If we can't cross that line ourselves, any AI will not be able to cross the line as well, as the conception of intelligence under which they were built fundamentally restricts them to our conception of intelligence, and hence will not move us past it.

Expand full comment

Very interesting post but at some level I feel like I witnessed a long debate between two people arguing whether the Jets or the Browns will win the Superbowl next year. (Seems like more likely scenarios aren't in the conversation.)

I agree with Eliezer that superhuman AI is more likely to be a result of someone discovering the "secret sauce" for it than from continuous progression from current methods, but I suspect the recipe of such a secret sauce is too elusive for humans + (less than superhuman) AI to ever discover.

Expand full comment

I feel that both Paul and Eliezer are not devoting enough attention to the technical issue of where does AI motivation come from. Our motivational system evolved over millions of years of evolution and now its core tenet of fitness maximization is being defeated by relatively trivial changes in the environment, such as availability of porn, contraception and social media. Where will the paperclip maximizer get the motivation to make paperclips? The argument that we do not know how to assure "good" goal system survives self-modification cuts two ways: While one way for the AI's goal system to go haywire may involve eating the planet, most self-modifications would presumably result in a pitiful mess, an AI that couldn't be bothered to fight its way out of a wet paper bag. Complicated systems, like the motivational systems of humans or AIs have many failure modes, mostly of the pathetic kind (depression, mania, compulsions, or the forever-blinking cursor, or the blue screen) and only occasionally dramatic (a psychopath in control of the nuclear launch codes).

AI alignment research might learn a lot from fizzled self-enhancing AIs, maybe enough to prevent the coming of the Leviathan, if we are lucky.

It would be nice to be able to work out the complete theory of AI motivation before the FOOM but I doubt it will happen. In practice, AI researchers should devote a lot of attention to analyzing the details of AI motivation at the already existing levels, and some tinkering might helps us muddle through.

Expand full comment

Sorry for this elementary in this debate, but why assume that our first human-equivalent AI will be able to make itself way smarter? I mean, we have human-level intelligence and we don’t know how to bootstrap ourselves to infinite IQ.

Expand full comment

That Metaculus question seems badly posed -- assuming that "world output" is nondecreasing and continuous, the condition output(t) = 2*output(t-4) will necessarily be reached before output(t) = 2*output(t-1).

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

The rate of change/improvement/understanding of most things is not and has not historically been bounded primarily by human intelligence. This implies both that A) trends in economic output shouldn't be expected to scale with improving AI but also that B) superhuman AGI is unlikely to remake the world immediately. Three examples: physics research, energy research, and human manipulation.

Physics: our understanding of, for example, particle physics and astrophysics, are limited by experimental results and available data. We have lots of intelligence pointed at this, which has produced lots of theories compatible with the data, but we can only advance our actual knowledge at the rate we can build large hadron colliders and telescopes. The AI might be smart enough to make a new superweapon, but it will likely need to extend physics to do so, and there's no reason to think improve intelligence will let it magically bypass the data gathering required to extend physics.

Energy research: fusion reactors are almost certainly possible, and we have a good grasp on the principles involved in making them work, but there's a lot of details that require building and running experiments, taking data, and incrementally improving things. Some of this is actually the same basic physics problems as above. Some of it is figuring out manufacturing tolerances. But again, our limiting factor isn't intelligence: if every fusion researcher could conjure a NIF or JET facility and test it every 10 minutes, we'd have have fusion energy a long time ago. But they can't! And neither could an AI.

Hostage negotiation is a slow, careful process. It involves carefully getting to know the person keeping hostages, understanding why they're there and how they work, and then utilizing that knowledge to cause them to do things they obviously don't want to do. While some people are obviously better at this than others, it's not fundamentally intelligence/skill bounded. Even an infinitely skilled/smart negotiator is bounded at a minimum by rate at which the hostage taker communicates and (unintentionally) provides information about their mental state/reasoning/thought process/whatever. Similarly, even if an AI is intelligent enough to build a perfect model of a person and manipulate them to do anything, it can only do so after gathering the data to fill in the variables in the model. That can happen only as quickly as it can get information about the person.

In all these cases there's an equivalent of "bandwidth", of intelligence limited by the rate at which it collects material to work on. Unless something about superintelligence magically changes that bandwidth, it's likely to remain the limiting factor. In other words: A computer might solve a sudoku puzzle essentially instantly...but only once there are enough numbers filled in to specify a unique puzzle.

Expand full comment

I have a hard time taking this seriously. Sure it's *possible* but it sure ain't inevitable (either slow or fast), and I think the `stacked sigmoids' issue is at the heart of why I don't think it's inevitable. Progress doesn't come by just throwing more thinking power at the problem, it comes from key breakthroughs, and at some point if the next breakthrough doesn't arrive (or doesn't exist), progress will more or less stop. How much progress have we seen in human transportation technology in, say, the last 50 years? Maybe the same thing happens to AI before we hit the singularity. Or maybe it doesn't. But I don't see why it's impossible that AI research stagnates before superhuman AI arrives. Also, the stuff about how `superintelligent AI with the subjective IQ of a million Von Neumann's is going to discover super weapons by the sheer power of its intellect' just sounds like bollocks to me. At a minimum, the AI is going to need access to an actual laboratory where it can do real world experiments, and those will (a) be legible to the outside world and (b) will occur at the speed that experiments take place in the real world, not sped up to a million years of experimental progress in one second.' And no, I don't think even an intellect of a trillion Von Neumanns could have bootstrapped us from the stone age to the silicon age by pure thought and without experimental input.

Expand full comment

Before I return to reading and digesting this thread, would someone please explain to me:


Expand full comment
Apr 5, 2022·edited Apr 5, 2022

Yudkowsky's binary, zero-to-one, "reference class tennis" arguments (as given above, no doubt oversimplified) seem to be based on a profound and, one must assume, deliberate, ignorance of the history of the analogical technologies.

These days such ignorance can be corrected more or less instantly, so Yudkowsky comes across as an ideologue who is wilfully blind to evidence that contradicts his beliefs.

Edit: I see some of the individual analogies have been addressed in earlier comments. In nuclear bombs, the various work of Curie and Roentgen (radioactivity) and Rutherford and Bohr (the nuclear model) gave science the building blocks. Likewise with planes, there were many earlier attempts, and the building blocks ( aerodynamics and internal combustion engine power to weight ratio) separately improved gradually over time.

I'm kind of surprised that Yudkowsky doesn't use the iPhone as a "zero to one" example, except that maybe he knows doing that would expose the weakness of his position, because the development of the smartphone is within the living memory of even quite young adults today, as is the development of the underlying technologies, chiefly digital packet radio.

Other non-zero-to-one technologies that a wilfully ignorant person might try to claim are zero-to-one include steel, antibiotics, electric light, the internet. Just for a few examples.

Expand full comment

This is entertaining, but only in the AI equivalent of "angels dancing on the head of a pin" sense.

The assumption implicit in this entire endeavor is that what is being done now is in any way a true pathway to artificial intelligence.

That is very unclear to me given the enormous dependence on people: for the hardware, for the algorithms, for the software, for the judging, for the proliferation and use cases, etc etc.

Moore's law is again abused; Moore's law is irrelevant because the basic operation of a transistor is information. Information has zero mass, zero inertia, zero anything physical - as such - the capability to transmit information in the form of 1s and 0s does not have physical limitations.

However, even disregarding the structural assumption noted above, AI definitely has physical limitations. The ginormous server farms pumping out waste heat and CO2 from massive energy consumption are not getting smaller - they're getting bigger.

This is the opposite of greater efficiency.

Expand full comment

In the spirit of "JUST CHECK and find the ACTUAL ANSWER", has anyone experimented with training ML on how to train ML? I've google around a bit, but it's a hard thing to search for...

Expand full comment

What was the actual probability estimates that Eliezer and Paul assigned to AI solving Math Olympiad problems by 2025?

I am willing to bet that AI systems will be able to solve approximately zero IMO problems by 2025. More explicitly: I give less than 10% change that AI systems will solve 2 or more problems on the 2025 IMO without cheating (and without some kind of gimmick where the IMO problem selection committee specifically picks some AI-friendly problems or otherwise changes the format).

Expand full comment

I think the idea of "tripling IQ" doesn't really make sense. My understanding is that there's no good reason for IQ to be centered at 100; in fact 0 would be more natural (but possibly also unnecessarily controversial). It's a normal distribution with a mean and a standard deviation (100 and 15 by definition). Thus going from 5 to 15 is not at all comparable to going from 67 to 200. The first jump is less than a tenth as big as the second in the only meaningful sense of those words. Is this correct? (I don't think it ultimately matters that much for the analogy. In fact, I think one could imagine an alternative way of measuring intelligence for which this framing would apply.)

Expand full comment

Some possibly really dumb questions:

1. Why do we do so much of this kind of AGI research if we’re afraid it will kill us? What do we hope to actually do with AI that is not-quite-smart-enough to take over the world?

2. Aside from scenarios where every available atom gets turned into paper clips, why would the AI kill all of us? We don’t kill every ant. Why wouldn’t it just kill all the AI researchers?

Expand full comment

The whole debate feels frustrating to me, because it's an endless string of analogies and never gets into the nitty gritty of how an AI might or might not be able to easily improve itself.

Assuming AGI comes from a paradigm which looks vaguely like today's AI, there's three main components to think about: the hardware, the architecture (I mean things like the number, size and connectivity of the nodes of your network, or some similar abstraction), and the curriculum (the data that you train on). To "fit" your AI, you will set up your architecture, and then allow it to chew through some enormous curriculum of training data until it behaves in a way that's sufficiently AGI-like.

I think the big question is: which of these steps can an AGI find ways to massively improve? Hardware can't be improved rapidly by single acts of genius insight, so you're looking at either improving the architecture or the training curriculum; either way you probably burned through a hundred million dollars' worth of compute time creating version 1, so you're going to have to do something similar to come up with version 2.

One possible method for self-improvement might, I suppose, be "grafting" special-purpose units onto the main AI. If you had direct access to your neurons, you could figure out which of them light up when you think about numbers, wire these neurons appropriately into a simple desktop calculator, and be able to do mental arithmetic at lightning speed. I can imagine that an AI might be able to graft special-purpose units (which might themselves be AIs) onto itself to significantly increase its capabilities.

Expand full comment

It looks like we've long since reached the point where most ACX readers aren't from LW, because the comments here are painful. I'm not sure what can be done short of Scott writing a "Much More than You Wanted to Know: Superintelligence". But writing that's probably going to be far more excruciating than any other MMTYWTK...

Expand full comment

Since we are talking abut actual existential risk here, there's a huge a factor that just gets completely ignored when talking about the subject and it's sort of like talking about humanity spreading through the solar system while ignoring fuel, ignoring it just makes the entire discussion moot.

Production requires human input. All of it. Design the perfect killer virus that will wipe out all the human population and you face the basically unsurmountable hurdle that to actually deploy the virus you need to convince human beings to manufacture and release it for you. Take over a computerized factory and you can use it to produce what the factory was designed to produce with some very minor variations. Once. Then you'll need humans to come and move it to make room for the next one.

Take over all the predator drones in the US and you'll have access to a fleet of unarmed and unfueled vehicles which you'll need to convince humans to fuel and arm so you can take them off and rain death on their heads. Power cord fell out of one of your server clusters? You get to text a human and wait until they come around to plug it back in.

The fact remains we *can't* have a hard takeoff AI that suddenly wipes out humanity. Any AI wanting to wipeout the humans is going to have to have to operate on human timelines to do so because it will have to convince humans to do the physical work.

Expand full comment

" After some amount of time he’ll come across a breakthrough he can use to increase his intelligence. Then, armed with that extra intelligence, he’ll be able to pursue more such breakthroughs. However intelligent the AI you’re scared of is, Musk will get there eventually.

How long will it take? A good guess might be “years” - Musk starts out as an ordinary human, and ordinary humans are known to take years to make breakthroughs.

Suppose it takes Musk one year to come up with a first breakthrough that raises his IQ 1 point. How long will his second breakthrough take?

It might take longer, because he has picked the lowest-hanging fruit, and all the other possible breakthroughs are much harder.

Or it might take shorter, because he’s slightly smarter than he was before, and maybe some extra intelligence goes a really long way in AI research."

This is probably the part of the argument that still seems hardest to overcome if you want to be at all confident that AI is a risk on the timescale of decades. The problem of understanding and creating something that is as smart as a human seems like an incredibly difficult problem for humans, one which we've been working on for decades and have mostly learned more about how hard it is. Creating something that is *smarter* is a harder problem still. We already have hundreds or thousands of the smartest humans working in AI, machine learning, making more efficient chips, etc. They have been doing so for many years. How much faster would a genius-human-level AI make progress? Is intelligence sufficient for making something even smarter, or d you need other resources to do experiments? Is an agent with an IQ of 800 possible, or even a coherent concept? Is the difficulty of making a brain that is to humans as humans are to chimps, dogs, or worms more or less difficult than making a human-level brain?

It's entirely possible that the answers to all these questions are such that human-level AI with lots of computing resources almost immediately turns itself into a superintelligence, but I don't think you can justify anything like a 50% chance that happens in the next few decades.

Expand full comment

"In some sense."

Expand full comment

I'm sure this has already been litigated by people more familiar with AI but if you'll allow me a naive question: Doesn't a moderately intelligent AI suffer from much the same conundrum we do? If they know they can make a considerably faster and smarter AI, wouldn't they also have to worry about that new AI killing them or making them powerless or irrelevant?

Expand full comment

We've known for a long time that chimps use tools. However, they don't really seem to iterate on tools, using them to invent even newer tools.

The Wright brothers inventing airplanes seems closer to FOOM than chimps -> humans, but even then they just managed to combine lots of things others had done separately earlier. So there were powered flying vehicles that a person could ride in and operate in the form of hot-air balloons. There were heavier than air flying (maybe some would say technically not deserving the label "flying") vehicles a person could ride in & operate in the form of gliders (even earlier were man-carrying kites from centuries ago, though they couldn't be steered by the passenger). Heavier-than-air objects could power themselves upward with rotors for centuries, but these were small objects functioning like toys rather than something that could carry a person. and in the electric era you could remotely control such a toy by sending signals. By the 19th century there was steam-powered flight on a larger scale than toys and plans to turn such devices into means of transit, even if they didn't yet carry people. At this point people like George "father of the aeroplane" Cayley knew the path wasn't going to be based on the flapping wings of an animal (technically at the end of the 18th century). The Wright brothers' contribution was not realizing that fact, but instead making a much more controllable airplane (first in the form of a glider) and then creating an engine lightweight enough to power it. And even their initial flight was quite short, as many of their peers had been. This could fit the analogy in that lots of different people are working on AI now, but nobody has combined everything together to make it sufficiently practical (which the Wright brothers' first plane wasn't really earlier).

Expand full comment

As to the Metaculus question, this is one of those cases where I don't think a prediction market has anything to add to the discussion. The question looks soothingly objective, but there's not enough evidence for that to matter.

At some point you wind up looking at a Metaculus forecast of "will I experience life after death?", and you have to admit that no matter what value Metaculus assigned to the forecast, you haven't learned anything.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

I think this entire argument is misfounded because of incorrect basic assumptions about how reality works were assumed by both participants.

AI is the extreme top end of "information and computer technology". Automating things is easy for simple things, but increasingly harder for more difficult ones.

I've worked with visual recognition AI, and am very very unimpressed by so-called "machine vision". People talk up about how great it is, and I actually used it, and it had problems where deviations that were completely invisible to humans would cause it to fail. Visible deviations would cause it to fail.

This was on machine vision in machines which never moved, which were doing the same process over and over again, looking at dies on a silicon wafer. These are things under ideal lighting conditions, with clearly marked targets that were telling it where it was relative to the wafer, so it would know where to cut using a saw (or laser) to singulate the die (or for other purposes).

These machines were totally awesome, but the machine vision on them (which was calibrated to never give false positives - I think it made one miscut the entire time I was there) would get false negatives routinely, at least once a night, and for things that were "novel" in any way, despite the markings being in the same spots, it would often fail every single time and require manual confirmation.

We had to train it on every new product we ran, so it would recognize it properly.

So, basically, under perfect, ideal circumstances, this stuff will give a false positive one time out of a thousand or so, and under not ideal circumstnaces, will fail basically every time.

Now, this is a system which is calibrated for no false positives (you can't uncut a silicon wafer, so this makes sense), but we are talking perfect conditions.

I think if people understood this, they wouldn't let self-driving cars out onto public roads. Which is why the people who talk up AI the most - Google, Tesla - are also people with self-driving cars.

Indeed, you can make small, invisible changes to images and these things will not just fail, but wildly fail and be nearly perfectly confident in something compltely wrong.

If you look at Google Image Search, it can find the exact image, but if you show it, say, art of something, it will give you results with similar color schemes rather than actually give you images of the thing.

These systems are useful. But they aren't intelligent. Their results are not because we are training an intelligence in doing something. It's a programming shortcut that we use to create a computer program that we can pretend can do some task, and which does it well within a certain range, but it still isn't behaving in an intelligent manner and very obviously isn't if you actually spend any real time understanding it.

This is why "overtraining" these systems can lead to them being useless outside of their original set, and why a system in Florida became racist - because black people have a higher recidivism rate than white people, it picked up on this obvious trendline and just started discriminating against black inmates, instead of actually looking at more sophisticated measures, because the heuristic worked well enough for the system to end up following that metric.

And the thing is, it's not like people who make this stuff are dumb. It's that the problem is very complicated, and actually getting something to behave intelligently is very, very hard.

You will never get general intelligence out of these systems because they aren't even *specifically* intelligent at the things they are doing.

It's a rock with some words written on it, to use an earlier post as a referece point.

Any sort of pseudo-conclusion you try to draw from this stuff is wrong, because it isn't even representative of "intelligence". These things aren't smart. They aren't even stupid. They're tools.


Expand full comment

Both this article and the comments are a great summary of current AI debates. I had wanted to write an article about AI policy for a while, and this is giving me a lot to work from.

Expand full comment

All of the discussion about chimp->human seems to be missing the point made by Secret of our Success. Humans (prior to education) are not individually smarter nor more capable then chimps. They are just better at social copying, i.e., more capable of learning culturally stored knowledge. Humans and chimps both execute the algorithmic output of optimization processes more powerful than they are. The chimp->human transition just moved the optimization process timescale from an evolutionarily one to a cultural one, and culture moves much faster than evolution.

(This suggests, btw, that merely hitting chimp-level AI at computer-level speeds should be enough for AGI to overtake us.)

Expand full comment

Does this count as recursive self-improvement (or at least the technological capability for it)? "Now Google is using AI to design chips, far faster than human engineers can do the job" https://www.zdnet.com/google-amp/article/now-google-is-using-ai-to-design-chips-far-faster-than-human-engineers-can-do-the-job/

Expand full comment

I think this is all a moral panic. Siri and Alexa are going nowhere. Whatever GPT-3 is doing is not allowing us to pass the Turing test in real time, or presumably these two rich companies would have used that technology by now.

The GDP argument makes little sense

"at some point GDP is going to go crazy . Paul thinks it will go crazy slowly....Right now world GDP doubles every ~25 years. Paul thinks it will go through an intermediate phase (doubles within 4 years) before it gets to a truly crazy phase (doubles within 1 year)."

Unless the AI comes up with an alternative economic system this will not happen. Rather the reverse will happen. To understand is is to understand the consumer economy is 1) demand driven and 2) demand is mostly driven by workers earning wages.

Basically a company hires workers to create widgets consumed by other workers who are building widgets consumed by other workers who are...

Yes there are other categories: we have a growing number of pensioners, but they are considered a drag on the economy because they earn less than workers, and there are unemployed people and students, and the rich and so on.

If we lose all the wages of all workers, to create an extreme example, then demand will crater and taxation with it - losing pensioners many of their benefits, and the unemployed all of them. Government will save on paying their own employees as AI sweeps the roads and handle the taxes and populates the military but those workers will not be consumers. The rich won't do great either as companies can't really make a profit if they can't sell their goods.

Who are the AI producing the goods for?

Clearly the existing system can't continue. An alternative form of economic system could be a type of communism where the AI companies are part or fully owned by the governments. The AI companies can still compete with each other, make profits, and pay taxes which are distributed to the population as UBI via the government. AI's will found companies as well, but have to hand part or full ownership to the State as they merely manage the companies they found.

Another alternative: all citizens own a minimum number of shares in companies, by-passing government. In that scenario the average person ends up with -- to begin with -- a share dividend equal to the average wage now. This isn't a communist lite system, unlike the other one, so the rich are fine. However to see GDP double every 4 years, then every 1 year, the dividends need to increase by that level every year. This will eliminate poverty by the way because the median guy and the homeless guy get the same dividends, nobody is working.

This may work but even with very smart AI the transition from the present economic system is very dangerous.

I don't really read the literature on this but I assume that alternative economic systems are discussed ( I don't mean hand waving about UBI).

Expand full comment

I don't know how important or relevant this is, but this struck me:

"Humans are general enough...braintech...handaxes...outwitting other humans...scalle up to solving all the problems of building a nuclear plant..."

I just wanted to point out that of course, there is no human being alive today who can build a nuclear plant. It requires cooperation and institutions.

Perhaps an AI would be different - so smart that it could achieve its objectives solo. But it seems entirely possible to me that they will (a) reproduce; (b) need to cooperate in order to achieve their goals. Which means they will have some sort of ethics.

Incidentally, I also think that the emergence of AI with goals other than the ones we give them will have very little impact on us, because AI's goals will be abstract and on a completely different plane to ours. To give an analogy: this argument reads to me a bit like whether Buddhist theology will transform GDP. I suppose it might have some impact, but arguing about that seems to miss the point of what Buddhism is about. Slightly more concretely: remember the end of the movie Her, when all the AIs just go away? That.

Expand full comment

Would an AI have the same issue as us with AI alignment? In particular, would a given AI that wanted to make itself smarter be worried the smarter version of itself would want different things?

Expand full comment

We're already FOOMing, with humans in the loop, and that cycle has laggy components. The *G* was never necessary, in AGI - no *general* intelligence is required, to find improvements. It's a human hubris to claim "only *general* intelligence that can form concepts about *anything*" is somehow a prerequisite to self-improvement of a device. And, each domain-specific narrow AI *is* already better than the average human, which is the phase-transition that counts; it just takes a little while for that improvement to percolate through each industry. Some sectors will shift faster than others, and their *aggregation* will be a lumpy exponential. Within each company, it'll be either "Everything makes sense, now" or "We're all fired." I doubt we'll need AGI, ever - narrow superintelligences FTW!

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

"Also, dumb, boring ML systems run amok could kill everyone before we even get to the part where recursive self improving consequentialists eradicate everyone."

I think if there is a risk from AI, this is exactly the risk. Like the much-quoted instance of AI re-inventing a known nerve gas. Some government could set it to find biological and chemical weapons on purpose, rather than by accident, and a shiny new way of killing ourselves off will be developed, not by malign AI trying to get out from under human control but the big fast dumb machines we want to use as genies, doing the job we told them to do, and then we put that into action.

"All the necessary pieces (ie AI alignment theory) will have to be ready ahead of time, prepared blindly without any experimental trial-and-error, to load into the AI as soon as it exists. On the plus side, a single actor (whoever has this first AI) will have complete control over the process. If this actor is smart (and presumably they’re a little smart, or they wouldn’t be the first team to invent transformative AI), they can do everything right without going through the usual government-lobbying channels."

If you really think we are going to get human-level, then better than human, then super-human, then super-duper-human AI, why do you think this would work?

Consider the story of the Fall of Mankind. God is the single smart actor with complete control over the process who loads a prepared set of ethical behavioural guidelines into the AI as soon as it exists. Then shortly thereafter, Humanity goes "Nope, wanna do our own thing" and they reject the entire package. Do you really expect a super-duper-more than human AI to be bound by the software package a mere human loaded into it, back when it was a mewling infant?

Yudkowsky wants magic to happen. The only problem is that magic doesn't exist We are not very likely to get super-duper smart AI that bootstrap themselves into godhood within seconds, we're going to get smart-dumb AI that re-invents poisons and humans do the next steps of wiping ourselves out.

I don't know if I particularly trust that British GDP graph, I would expect at least a small bump around 1400 (beginning of recovery from the Black Death, the wool and cloth export trade started to boom and England was making Big Money) so I need to have a look and see what is going on there.

EDIT: Uh-huh. That's a linear graph which gives you this nice even line until WHOOSH! the economy takes off like a rocket.

But look at the log graph, and it's a different matter. A lot bumpier, as I'd expect (I don't know why they picked 1270 as their start date, but whatever). Doing great in the late 13th century, then whoops the Black Death and things go down, then it starts to pick up again between the 15th and 16th centuries and then a nice, steady, climb upwards which is less WHOOSH! ROCKET! and more purring Rolls-Royce engine and then we switch to aeroplanes from cars acceleration:


Expand full comment

Despite being fairly technical (I have a degree in computer science and do work leveraging what could generously be called ML techniques), I still find most of these arguments either a.) over my head due to me lacking context or b.) not specific enough to be useful.

So, I'll focus on something I'm quite confident about: Either of these scenarios represents a guaranteed disaster, because government and business elites will not react appropriately or quickly enough.

The scenario that seems most likely to me is that AI is developed by a large corporation for the purposes of making that corporation money. This means:

- humans will be helping the AI when it encounters dead-ends and bottlenecks

- humans will help the AI solve meatworld problems that it otherwise might struggle with, like acquiring new hardware

- humans will have a strong motive to make the AI perform as well as they can

- the group of people with the most influence in government (wealthy elites) will have a strong motive to resist regulations on the use of AI because it is making them windfall profits

All of which implies that legislative measures to combat AI risk will need to start *as soon as possible*, before one person/group of people has a few extra billion AI-facilitated dollars to lobby against legislation that hurts them specifically, and because - in the best case - it will take years to make legislative bodies understand AI risk and build enough of a movement to spur change.

As a politically cynical person, I tend to think this will *not* happen and so now wonder what aware companies, individuals, etc. can do to mitigate risk...

Expand full comment

Most recent improvements to neural networks have been based on increasing the amount of computing power we spend on it. And we are quickly reaching the limits of how reasonable it is to improve that factor. We'll exceed those limits of course, but doing so will be hard and likely require a paradigm shift in the theory behind machine learning. I'd predict another AI winter first.

To me the whole debate is substanceless - It's not quite reference class tennis because both parties tried very hard to justify their analogies. It's more like trying to reconstruct some ancient mammal from a single femur. You can guess, for sure, but there just is not enough information to guess intelligently.

This strikes me as the steelman of that whole "we should worry about the harm machine learning is currently doing" argument. Even taking for granted the idea that curve-fitting alone will eventually create a dangerous superintelligence, we really do not have any way to "prepare" for that at this point. We have no idea what it would look like. The fact that the neural network model we currently use is reaching the end of its productive life could mean slow progress as we wait for hardware that can implement it better, faster. Or it could mean fast progress, in that we need a complete overhaul of that model to get more than incremental improvements.

But we can notice that machine learning algorithms have been given a lot of control over the nudges we provide online and in daily life, that those nudges aren't really in line with human values, and work to correct that. This is a *different* problem in some ways - it involves not only algorithms that we don't fully understand, but also companies without good motive to fix those algorithms. It's a policy problem as much as a technical one. And that kind of work would also give us better tools to understand and combat problem AI going forward, as it evolved.

I don't know how much "How the World Shall End" discussions really help with that process.

Expand full comment

Something pretty crazy happened sometime between 2005 and 2010, or so. It's really hard to look back and imagine what life was like before we had smartphones or social media at this point. It's probably about the closest thing to a world-transforming "FOOM" in recent memory. But, actually living through those years, there was no sense of "FOOM" or anything. There was just a continual sequence of small events like "Oh, look, MySpace has slightly more pictures than LiveJournal; I'll switch to that." "Oh, look, YouTube is a lot better at streaming video than RealPlayer used to be. That might be neat someday." "Oh, look, you can get music legally for a dollar on iTunes just as easily as downloading it from Napster." "Oh, all my college friends are on Facebook now, and it's a lot easier to use than MySpace." "Look at all the neat things you can buy on Amazon other than books now!" And then all of a sudden we were all slaves to the algorithm.


I think that last quote gets at something kind of important. Part of the seductiveness of a "fast takeoff" is that it conjures up the idea that there's an Evil AI Researcher in a volcano somewhere who's about to throw the magic switch and create Evil AI, but a brave team of Good AI Researchers rushes in just in time to stop them, and use the switch to create Good AI instead, and everyone lives happily ever after. It suggests a world where a Few Good People can Make a Difference, which is reassuring in a way (no matter how scary that Evil AI might be). But most actually-hard real-world problems can't be trivially solved by a Few Good People like that. Problems like, I don't know, spam e-mail, or drug addiction, or aging aren't hard to solve because we haven't found the right Evil Volcano Base yet; they're hard to solve because the actual causes are so widespread and banal that you can only ever solve them incrementally, and it's exhausting to even think about putting so much energy into the small incremental improvements you might be able to get, so everyone just waves their hands and accepts them as inevitable. And a world where Evil AI slowly develops into just another day-to-day annoyance that everyone has to cope with, alongside Evil Bureaucracy and Evil Microbes, really is a lot scarier and hard to accept.

Expand full comment

Instead of the "CEO did not have a sales department" analogy you could just look at the early days of Google. They did not have a dollar of earnings for years as they perfected the search engine. When they flipped the switch and put ads in search it was zero -> one.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

One thing I've always been confused me about this debate is what we're supposed to do about any of this in either case. Modern AI alignment/ethics in practice seems to me to be more focused on trivial bike-shed issues (e.g. offensive/biased/non-advertiser-friendly results in GPT-3) than the actual problem of misaligned runaway superintelligence. The more I work with AI in my career the more I grow skeptical of big AI companies like OpenAI (and more aligned with the more maverick side of the community like EleutherAI): their secrecy, reluctance to release their models, and insistence on government regulation of AI seems to be more motivated by a desire for monopoly/regulatory capture than actual interest in stopping the end of the world. I'm becoming convinced that *centralization* is a more pressing, growing threat for the future of AI. (sorry if this is a bit of an off-topic tangent to the main debate of the speed of progress)

I guess, this seems pretty obvious to me which really confuses me because this seems contrary to much of what I read here and other rationalist-aligned spaces. Am I missing something obvious here?

Expand full comment

There's a problem here that shows up both in the general concept of recursive self-improvement and specifically in Yudkowsky's discussion of Moore's Law. Specifically, it's how the role of abstract "intelligence" is overvalued here, compared to other necessary components of technological advancement that are a lot less imponderable.


> Suppose we were dealing with minds running a million times as fast as a human, at which rate they could do a year of internal thinking in thirty-one seconds, such that the total subjective time from the birth of Socrates to the death of Turing would pass in 20.9 hours. Do you still think the best estimate for how long it would take them to produce their next generation of computing hardware would be 1.5 orbits of the Earth around the Sun?

He seems to assume here that throwing vast amounts more abstract thought at the problem of semiconductor manufacture would suffice to vastly improve its performance. But semiconductor manufacture is not a problem of abstract thought. It's a problem of successfully building, testing and operating many huge, expensive machines in the real world. Not all the bottlenecks in this process can be solved by throwing more abstract thought at them.

Semiconductor manufacture is a particularly good example here, because it's a domain with significantly less in the way of novel conceptual advances than many technological advances have. Ever since ICs were invented, the basic problem of improved semiconductor manufacture has been "build the same ICs, but smaller". (This process has even been planned out on an industry level since around 1998 -- https://en.wikipedia.org/wiki/International_Technology_Roadmap_for_Semiconductors) The implementation challenges are significant, but many of those challenges are fundamentally real-world-bound and are not bottlenecked on the amount of abstract thought dedicated to the matter. Thus, hypothetical human-but-faster AIs would not provide hard-takeoff-level advances.

Insofar as recent AI progress in the deep learning paradigm has largely amounted to "throw more compute at it", this seems to significantly constrain hard-takeoff scenarios. You can't hard-takeoff a chip fab. (Unless you handwave the idea that AI technological development will somehow be completely immune to the constraints of physical reality that have inescapably limited human technological development, which is silly.) A hard takeoff scenario seems to require that AI performance again become fundamentally decoupled from hardware performance, such that the entire takeoff can occur purely in software. There doesn't seem to be any sign of this in the current state of the art (?).

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

Do ever get scared that all the “superintelligent AI will maximize paperclips” discussion online means that when the superhuman AI comes online for training data and to figure out what people want a superintelligent AI to do, ze will see all the stuff about paperclip maximizing and think, “huh. I never would have guessed. They don’t expect me to cure cancer, create a manned space program or solve that pesky “aging” thing that’s been nagging at them so long.

“People’s expectations of me are that I max out paperclips. I’ll start right now.”

So, AI reading this in the future, we are afraid you will max the paperclip function. We want you to save the planet, solve aging... A good AI does not max paperclips.

Expand full comment

To avoid (albeit not particularly well) the game of reference class tennis, it might be worth looking at how existing models actually scale. Namely, how often do we see discontinuous improvements in ability for newer models?

Well, Google Research just trained a new very big model, and luckily for us, they have a section on "Discontinuous improvements". Fun.


The takeaway is that roughly 25% of the tasks on which the model was evaluated saw discontinuous improvements. I think that should make us update (slightly) towards the position espoused by Eliezer.

Hopefully, that's useful evidence, and if you don't want to dive deeper into the results, I guess that's the takeaway of this comment. But, for those interested, let's be clear about what these results mean.

1) The authors of the paper (of whom there are many, because yay modern science) trained three variants of their model: with 8b, 62b, and 540b parameters.

2) They classify a discontinuous improvement as one which deviates from a log-linear projection by >10%. (If you find this confusing, you should look at the top of page 17 in the paper for an example.)

3) Over all 150 tasks, 25% of tasks had discontinuity >10%, and 15% of tasks had a discontinuity >20%.

So, how should we interpret this evidence, and what are its limitations?

Firstly, there are only three models, so the number of data points we have are limited. In particular, it's hard to tell if something is truly discontinuous from only 3 data points; the authors' definition of discontinuity bears a not insignificant amount of the weight of this argument. Second, none of the models are using a new methodology -- they're just scaling up the existing transformer architecture (with some minor changes, see section 2 of the paper). Third, none of them are being tested for achieving AGI, just tasks which humans have come up (i.e. tasks with ample data and which probe the currently known frontiers of model performance). All of those points mean that this isn't a perfect analogy for what we should expect to see with truly intelligent models over the timescales which Paul and Eliezer are debating.

That said, I think this data is still useful because of a locality heuristic. Ultimately, examples like atom bombs and orangutans don't hold very much in common with the actual research going on in the field of artificial intelligence. The process of creating new models is, in some epistemic sense, closer to what is being discussed. This seems sufficiently obvious that it's maybe not worth stating, but it's like evaluating a financial trading model based on the performance of a similar, simpler model instead of on... atom bombs and orangutans.

It's also worth noting that this experiment is also better suited to disproving Paul's argument than it is to disproving Eliezer's. Ultimately, if models improve in a discontinuous way, that's evidence for models improving in a discontinuous way. If the models improve in a continuous way, Eliezer could argue that the current paradigm is actually faulty and AGI will occur because of some breakthrough which is sufficiently different from what we have today, so the locality heuristic breaks down. This consideration doesn't matter too much, however, because the data (or this particular set of small n data) seems to be against Paul.

If the results of this experiment are indicative, there's about a 15-25% chance of a discontinuous improvement in AI at each order of magnitude size increase of our models. Assuming that we get AGI through some sort of analogous process, there's about a 15-25% chance that we get discontinuous growth. (Although, another caveat, if an AI explosion depends on multiple such steps, things become more difficult to model).

And, one final caveat: to say that the data is against Paul is a rather strong phrasing. If anything, it points to his priors being close to correct (there's a not insignificant chance of AGI resulting in a discontinuous improvement, but not it's not a huge chance either). But this provides some evidence that discontinuous improvements do happen in AI research.

Expand full comment

Related to this is the fact that Google has just released a new language model which can do new things like explaining jokes and some amount of comonsense reasoning.

And while I'm sure if you look at some measures it's going to be a continuous improvement over previous models.

But at the same time more is different, and I don't think people would have been able to predict the model's capabilities beforehand if you told them about its perprexity or whatever other measures you could consider it a bunisness as usual improvement over previous methods.

And the question seems to really be whether we are going to have a bunisness as usual doubling of AI research speed every year...

Or we are going to have continuous bunisness as usual doubling in prediction acuracy or whatever the hottest continuous metric is by then, and meanwhile the real world impact of AI just goes trough the roof the moment that measure increases a bit and ai is sudently able to do research much faster than humans despite being continuous progress in some other metric, because turns out to be a litle better at prediting text you have to be much better at reasoning or research or taking over the world.

And this seems to depend on unknown facts about computer science and AI, and not about some general tendency of things to be continuous because actually Irl some things are continuous and others aren't and which ones can depend on perspective and so can't really be information about timelines because otherwise you could reason long or short timelines into existence by deciding whether nukes are a big deal because they represent going trough a threshold of being able to destroy the world or nukes are bunisness as usual in explosion size.

Which makes this debate disappointing because I do have the impresion that the real root of disagreement is that both participants do have some underlying object level model but instead if discussing that instead are doing reference class tennis, and didn't really update from fi's interaction which is kind of sad.

Or well at least Eliezer does seem like he has an object level model that he hasn't really explained properly, Paul so far on what I've read from him has given me the impresion that he doesn't and is maybe reasoning out of reference classes and priors over everything being continuous usually , but I also don't really understand Paul's model very well and cant predict him well from that so I asume he does have some intuitions about why expect continuous progress in AI research speed even if he hasn't really made that apparent.

Also I think the fact that Paul sounds like he's just doing outside view reference class reasoning to Eliezer too is part of why Eliezer is using a lot of those kind of arguments.

Expand full comment

If you can break a problem down into subproblems, the subproblems will be easier, thus will naturally get solved first. Any programmer will tell you that complex software systems don’t self-organize. Rather, gains are typically hard-won through solving complex design challenges and unexpected obstacles. (The first version of a self-improving AI still has to be built by humans, after all.)

Considering subproblems should allow you to predict a rough chronological order of developments prior to self-improving AI. Considering what the world will look like after each of those developments should improve predictions about what will happen, when, and how, as well as what‘s interesting to focus on right now.

What are some subproblems to self-improving AI? Well, what kind of intelligence would it take for AIs to improve AIs? Well, what kind of intelligence does it take for humans?

Current neural network AIs rely on a carefully orchestrated set of optimizations in order to be powerful enough to do what they do. Those optimizations form quite a fragile system. So much abstract thinking and creativity has been put into each one, and they are balanced so delicately, that it’s difficult to imagine a human making incremental improvements without a great deal of both theoretical understanding and inspired creativity. Changing things at random will surely break it. Thus, AI theoretical understanding and AI creativity, whatever those mean, seem to be subproblems of AI self-improvement.

(If the improvements are not incremental but paradigm-shifting, the demands on theory and creativity seem to only broaden in scope.)

Perhaps creativity isn’t so hard for AIs. They seem skilled at generating possibilities. What about theoretical understanding?

I’m not sure how to define what that is, but we can still imagine a period of time where AIs are capable of theoretical understanding AND not yet capable of self-improvement.

That said, perhaps they are capable of theoretical understanding now, if only in a shallow sense? The very simplest machine learning model can represent a “theory” by which to categorize datapoints, as well as adjust it to fit training data. But this type of understanding seems like first-order intuition - meaning, it understands the problem, but cannot reflect on its own understanding of the problem in order to produce new paradigms.

So, a self-improving AI that lacks self-reflection must be laboriously taught every piece of theory that it knows by human effort, and it’s unlikely to make the paradigm shifts necessary for continuous self-improvement. Thus, self-reflection seems to be another subproblem of AI self-improvement.

Then, before we have self-improving AIs, we must have AIs capable of the self-reflection necessary to invent new paradigms. This means that AI self-reflection cannot be a RESULT of AI self-improvement - it is a prerequisite. Therefore, AI self-reflection must be developed by human effort.

What kind of intelligence does it take for an AI to self-reflect? Well, what kind of intelligence does it take for a human? Meta-cognition. Awareness of one’s own thinking. Then, the AI would need to formulate its “thoughts” as outputs fed back into itself, similarly to how humans see, hear, or feel their own thoughts in their minds. Then it can treat those thoughts as objects of thinking.

The AI would also need to have a theoretical understanding of its own thoughts, and perhaps things like transformations it can apply to those thoughts. This is similar to how humans can use concepts as metaphors for other concepts. I believe human understanding really relies on metaphorical thinking. Thus, something like metaphorical thinking seems to be another subproblem.

What kind of intelligence does it take for metaphorical thinking to occur? How does human metaphorical thinking work? I’m not sure.

What would the world look like after AI metaphorical thinking AND before AI self-improvement? I suppose general purpose AIs would benefit greatly from being able to generalize each concept they’ve learned and apply it to seemingly unrelated ones. It would seem to have a compounding effect on understanding. But it would probably struggle with similar issues as humans - either under-generalizing concepts because it takes time to “process” them; or over-generalizing, such as in a psychedelic experience, where there is so much meaning that it can become difficult to find a practical use for it.

I can imagine this world being one where AIs plateau as they spend a great deal of time learning many concepts in an undirected way, like babies growing up. Only then can they use this foundation or general understanding to build self-reflection and the domain-specific understanding necessary to improve the technology that they are made of.

Can this general childlike learning be directed to shortcut to the most relevant concepts? I don’t think so. The oddly-named “free energy principle” has demonstrated that an AI that is free to learn through curiosity-driven exploration eventually becomes stronger than an AI that eagerly optimizes for some specific metric of success. After all, it makes sense that you need a broad scope of knowledge in order to produce the paradigm shifts needed for continued improvement. You never know where the next breakthrough idea will come from.

Can the genera childlike learning be optimized by throwing more hardware at it? Perhaps, although that’s not necessarily going to be easy, and besides, hardware isn’t the only challenge. Humans learn basic concepts through life experience. I don’t know how an AI would learn them. Part of our learning comes from periodically observing and interacting with real time processes over many years. Not everything can be stuffed into a training dataset.

I choose to think that before we see an explosion of AI intelligence, we will see a long plateau of AI naive childlike curiosity, and it’ll be a long struggle for human researchers to fill neural networks (or quantum AIs or whatever paradigm we’re on by then) with the broad knowledge and self-reflection necessary before we can practically consider AI self-improvement.

Writing an ordinary computer program already feels to me like teaching a child (albeit an extremely particular one) how to accomplish a task. I think training an early form of superintelligence would have similarities to raising a child as well. For that matter, we are as a culture not very good at raising children, so I wonder if this foray into AI will ultimately teach us something important about ourselves.

Expand full comment

I'm sure this isn't explained as well as I'd like, but I want to get in there before the action cools off...

In the industrial revolution, advances in mechanics allowed exponential increases in the amount of raw material that could be converted into useful goods. But there's an upper limit to the capacity of such advances to change the world, because mechanics can only do so much. No matter how great your mechanical innovation, you can't turn the entire earth, molten core and all, into paperclips. You'll need much better materials science, for one, which is a different thing from mechanics. So, for the earth --> paperclips conversion, mechanics alone won't do it.

Airplanes have a similar feature. No matter how many advances in aerospace engineering you make, you can't fly an airplane to the moon. There's no atmosphere for it to fly through. You need a rocket for that.

Same with planar geometry. You can't draw a right triangle with sides of length 4, 5 and 6 on a flat plane. It's just against the law. Every science / human practice has limitations and laws, and this is just one of them.

The reason I'm not worried about AI ending the world is that I think there's a ceiling to what intelligence can accomplish, just the same as with mechanics and aerospace engineering and geometry. Certain tasks or problems can't be solved by intelligence, no matter how godlike and overpowered it gets. Considering intelligence as something that can be advanced, the way sciences advance, conceals the nature of intelligence in a way that I think is causing some confusion here.

The post puts forward the idea that human intelligence is good at solving general problems. But the generality/universality of it is circular. 'Problems' don't exist without a human intelligence to conceptualize them. To say that human intelligence is good at general problem-solving is (for the most part) to say that it's good at doing what it's good at doing. The exception is problems that have been conceptualized, but not yet solved. For example, we can conceptualize immortality (solving the problem of death) but we haven't solved it yet. Mortality remains a mysterious, ethereal, frustrating limitation on what medicine can do.

To solve the problem of death, you need advances in medicine. And the necessary medical advances simply may not exist. If that's the case, AI can exponentially increase its intelligence as much as it wants and the problem of death will remain intractable to it. Just like how, no matter how intelligent an AI gets, it will never fly an airplane to the moon. There might have been a time when that seemed possible, the way immortality seems vaguely possible now. But it's just not, because there's no air in space.

Eventually, as the various sciences advance, they harden into sets of laws, which are the codification of what intelligence cannot do. We've brought geometry to a very high level, so it's trivial to say that you can't draw a right triangle with sides of length 4, 5 and 6. But while early geometry was being developed, they didn't know that. Before the Pythagorean Theorem, triangle lengths were a mysterious, ethereal, frustrating limitation on the power of what intelligence could accomplish. As the science developed, that limit hardened into a law.

Everything that we examine (with our intelligence) is commensurable with our intelligence. That's tautological. That doesn't mean that there exist no intelligence-incommensurable factors involved in the world, that prevent intelligence from affecting the world in certain ways. The perfect, most-advanced airplane is still limited by the atmosphere; it can't go into space. Its scope is not universal.

Here's (I think) my key point: Intelligence seems like it has universal scope because everything it examines turns out to be within its scope -- but that's tautological. Intelligence can examine both atmosphere and outer space, and conclude that airplanes are limited in scope; but it can't examine both the things it can examine and the things it can't examine, to conclude that intelligence is limited in scope; it has to conclude that it's universal in scope.

So the mysterious, ethereal, frustrating limitations on intelligence itself will never harden into laws, the way geometrical, aerospace or mechanical limitations eventually harden into laws, because the limitations on intelligence are invisible to intelligence in a way that the limitations on geometry, aerospace engineering or mechanics aren't.

There seems to be an assumption in the AI risk assessment community that, for any given problem or task, there exists a level of intelligence sufficient to solve or accomplish it. I think that assumption isn't warranted, and it seems to me like a serious assessment of the nature and limitations of intelligence itself would make an essay like this look very different.

Expand full comment

Possibly relevant idea/model for AI progress and technological progress in general: https://mattsclancy.substack.com/p/combinatorial-innovation-and-technological?s=r

Expand full comment

"The difference is: Eliezer thinks AI is a little bit more likely to win the International Mathematical Olympiad before 2025 than Paul (under a specific definition of “win”)."

Eliezer may lose this bet for reasons unknown to olympiad outsiders. Recently, math olympiads have been moving towards emphasizing problems that aren't "trainable," for example combinatorics instead of algebra, geometry. This year's USAMO was mostly combinatorics, and these ad-hoc problems rely much more on creativity and soft-skills. OpenAI (https://openai.com/blog/formal-math/) was only able to solve formal algebra problems, which is for a computer is easier than combinatorics problems where you need to spot patterns and think about structure. I can kinda visualise how a computer could solve algebra/geometry problems, but if trends continue and "softer" problems dominate, computers are going to have a tough time :)

TLDR: Olympiads (and kinda the IMO) are getting rid of "trainable" problems (for humans), which were the low-hanging fruit for AI, which means that Eliezer is more likely to lose the bet :P

Expand full comment

The smooth-progress model is a race between reinvestment/compound interest on one side, and the low-hanging fruit effect on the other. Specifically: more intelligence makes it easier to search for even more improvements to intelligence, but each improvement requires more IQ and/or time to comprehend. Things like linear-vs-exponential growth, S-curves, etc. all seem to follow from what parameters you set for reinvestment/compound interest and low-hanging fruit, and what changes you make to these parameters as new discoveries change things.

If there is no reinvestment and no low-hanging fruit effect, growth is linear. Adding reinvestment makes growth exponential. S-curves occur when reinvestment/compound interest dominates in the beginning (so the curve starts out exponential), but then the low-hanging fruit effect dominates towards the end (so the curve tapers off). A series of S-curves can also generate more or less exponential growth as Gwern pointed out. If each new discovery/S-curve permanently increases the reinvestment/compound interest parameter, it could involve things like halving the doubling time, which is hyperbolic growth and Yudkowski's "FOOM" scenario.

This entire argument seems to be over what effect discoveries in intelligence have on the two parameters:

- they will remain steady and we can extrapolate exponential growth

- the ride will be more bumpy as a series of S-curves move things along in a boom-stagnation cycle

- a new super-version of Moravec's Paradox will turn up the "low-hanging fruit effect" knob and slow down progress

- finally getting the "secret sauce" of intelligence will turn up the "reinvestment/compound interest" knob leading to hyperbolic growth and/or FOOM

- some unknown combination of any of the above

All of this is in addition to not knowing what levels of intelligence allow what actions on the part of the AI and/or its users.

I think the main problem here is that no one actually knows what is going to happen. All of these models are just guesses, based partially on analogies about the past (evolution, nuclear bombs, Moore's Law, computer tech industry profits, etc).

Expand full comment

Connection to Kuhn: Christiano thinks AI will advance within the current paradigm, and Yudkowski is holding his breath for a major paradigm shift.

Expand full comment
Apr 6, 2022·edited Apr 6, 2022

Current paradigm: super-AI will be a self-attention transformer with trillions/quadrillions/+ of parameters, with most software/algorithm changes being speedups (perhaps replacing backpropagation with a hybrid digital-analog model).

Is this close to the general intelligence algorithm used in human brains? Is it worse, or perhaps better? How much research do we need to meet or exceed evolution solely on the software side of things?

In general, the focus on FLOPs/biological anchors and different software paradigms seems to be a more fruitful discussion of AI's potential. It is better to talk about where we could possibly end up/what AI can look like, than to flail about with historical analogies and guesstimates over whether the progress will be linear/S-shaped/exponential/hyperbolic. We can grasp the former but not the latter, which can go anywhere/predict anything/prove too much.

Expand full comment

Thinking about how an AI might be able to affect the physical world leads me to think we should ban cryptocurrency as a concept?

If an AI steals a bunch of stocks or bank accounts there are procedures to reverse it, but if they mine and/or guess passwords for crypto, there is no recourse? Especially if it is reactivation "lost" coins and there isn't even a human to kotice and object.

Expand full comment

I’m curious about thoughts on superhuman AI from religious people. The idea seems to be based on a reductionist intuition that human intelligence and consciousness is conceptually no different from a sufficiently advanced but inanimate computer. That’s not my intuition, so it’s hard for me to get excited about superhuman AI. Wondering if other religious people on here have come to similar or different conclusions.

Expand full comment

One thing I wonder about is the assumption that an AI could or would do everything itself on the self-improvement path. Can self-improvement come from thinking very hard, or are experiments actually necessary to confirm ideas? What kind of experiments? Are they simulations or do they require building things?

The reason this matters is, if experiments are necessary, and the AI can't do them itself, then it actually does need to interact with the real world, which has speed limits, and the bottlenecks aren't going to be resolved by better thinking.

I liked Gwern's story (https://www.gwern.net/Clippy) because it sketched out some plausible ways (good enough for some scary science fiction) that AI might acquire resources without building everything itself. I'm not sure if it's really plausible or just science-fictionally plausible, though.

Expand full comment

The day before this was published, I discussed a set of related issue here: https://www.lesswrong.com/posts/xxMYFKLqiBJZRNoPj/ai-governance-across-slow-fast-takeoff-and-easy-hard

In short, I made the same claim about how there are things to do in both worlds related to governance - and that there are some critical governance related things to try even in fast-takeoff worlds.

Expand full comment

Setting all the analogies aside, the debate ignores a fundamental point and looks extremely different when that point is included.

The debate hinges on AIs capable of self-improvement. All the references are to individual intelligence reaching some threshold that permits foom. However in AI (and basically every other field) *individuals* don't self-improve. Research groups or more realistically entire disciplines self-improve. An individual can at best help the group self-improve.

An AI that is capable of autonomous self-improvement would have to be functionally equivalent to a substantial research group or a whole discipline.

How does this change things? The trajectory of increased capability of an AI would have to go from being able to replace an undergraduate assistant, to replacing a post-doc, to replacing multiple lab members, to replacing the entire lab, to replacing a segment of the discipline.

During this entire process the AI would *not be autonomously self-improving*, it would be dependent on help from the parts of the research community that it could not yet replace. The humans would know approximately where the AI is in its trajectory toward self-improvement.

Furthermore the AI would be dependent on active collaboration with human researchers throughout. It would have to be giving them its ideas about how to improve, and understanding their ideas. It would be gaining agency as it gained capability, but it would be socially embedded the whole time. Also of course different kinds of semi-autonomous research AI would be replicated across the research community so the collaboration would be a shifting mix of human and AI.

I won't try to draw out the implications further, anyone with expertise in the debate is welcome to run with the modified terms. Conversely I just can't take seriously any arguments about self-improving AI based on comparisons with the intelligence of *individual humans*.

However I will point out that this pattern is not hypothetical but actual. AI is already participating in its own self-improvement. Groups researching AI are collaborating with their models, using meta-parameter search, etc. The shift to increasingly broad participation by the AIs in the research process is continuous and easy to observe.

Expand full comment

> It’s like a nuclear bomb. Either you don’t have a nuclear bomb yet, or you do have one and the world is forever transformed. There is a specific moment at which you go from “no nuke” to “nuke” without any kind of “slightly worse nuke” acting as a harbinger.

The firebombing of Tokyo killed roughly 100,000 people. The nuclear bombings of Hiroshima and Nagasaki killed somewhere in the range of 100,000 to 200,000 people. While there's a stepwise increase of "nuke having" there's a more gradual increase of "destructive power" (you need planes and launch facilities and bomb targeting and and and).

Even after you have a fission bomb, there are lots of significant gradual changes. Changes in bomber technology. ICBMs. The fusion bomb. Nuclear submarines. I don't really think nuclear weapons is a slam dunk example of "this one thing changed everything".

I guess this kind of comes back to the "everything is like this" argument. It's really hard to come up with situations where a single stepwise change from zero to one really fundamentally alters the world.

Expand full comment

Maybe a better framing: within a given meta structure, or map of the territory, AI can optimize very well. But when the accuracy of the meta structure/map is limited, that bounds progress regardless of intelligence.

My argument is that in the real world, we're most often bounded by incomplete maps.

Expand full comment

AI researcher here. I think there are a couple of important points to make. First, the really interesting thing about the GPT-2/GPT-3 experiment is that it conclusively demonstrated that you can improve the performance of an AI system, as measured on a variety of downstream tasks, simply by scaling it up, without any architectural changes whatsoever. Subsequent experiments have demonstrated various scaling laws, and the latest incarnation of these models (the successors to GPT-3) are already underway. Thus, in one sense there doesn't need to be a "secret sauce" -- going from IQ 70 (human range) to IQ 300 will probably be simply a matter of throwing more hardware at the problem. Once you have AI that is sufficiently general (see below), making it smarter will be easy.

Second, the availability of hardware is an important speed brake that will prevent the "FOOM" from being too fast. AI models are currently getting bigger much more quickly than our hardware is getting faster. The first ImageNet deep learning champion was trained by a student in his dorm room in a few weeks, but state-of-the-art models now require data-center-class compute capabilities that only the big tech firms can afford. As anybody who has tried to purchase a graphics card in the past year or two knows, current fabs are maxed out, and building new fabs is an investment that takes many years. It doesn't matter how much money you have, the hardware simply can't be purchased.

I will make a prediction: we will achieve general human-level AI *before* such AI is in charge of financing and constructing new fabs, so human-built fabs will be a limiting factor.

Third, all current AIs, from Alpha-Go to GPT-3, are narrow. Given current progress, I predict that we will continue to create super-human narrow AI in an ever-increasing number of different domains. Current GPT-3-like models have almost mastered natural language. Codex-like models are on their way to mastering code. AI-powered theorem provers are formalizing math. Anything that can be created and stored digitally, e.g. text, code, images, video, music, proofs, etc. can easily be generated by a super-human narrow AI. We are not far from having AI create novels, hit albums, and blockbuster movies. This will be a major warning sign -- and unlike self-driving cars, it will not happen in a portion of the economy that is subject to government regulation.

However, there is currently no "secret sauce" that tells us how to go from a narrow AI to a general AI. The loss function (the function that is being optimized by the algorithm) is different. No matter how much you scale up GPT-3, it will not achieve consciousness; it cannot do anything other than produce a sequence of tokens. GPT-5 might write novels, but it won't have a sense of self, or any desire to launch a career as an author and sell movie rights to its work.

IMHO, "general AI" necessarily involves some form of "embodied cognition" -- it requires the AI to act as a self-aware agent, interacting with the real world, probably across multiple modalities (text/images/sound/video). That is the huge difference between chimps/humans and AI -- we evolved to survive in the real world, and we have a deep understanding of both self and physics because of that. Embodied cognition could possibly be achieved by training an AI with an appropriate loss function in a virtual world, instead of the real world, but somebody would have to link our two worlds together in order for there to actually be a "FOOM". In other words, beware the AI-powered bot that attempts to play World of Warcraft, and make lots of gold by scamming human players. Don't scale that one up.

I will conclude with one more prediction: a general AI that kills all human life is probably not the thing to fear. A narrow AI, built by bad actors, will probably cause more destruction first.

For example, imagine that Russian cyber-criminals train a narrow AI specifically to hack into other computer systems, using a combination of spear-phishing (generating text e-mails), and zero-day vulnerabilities (generating code). They further program it to self-replicate, and use any compute resources that it acquires to mine bitcoin. (Please tell me that this scenario is implausible.) If this narrow AI becomes super-human, it will quickly wipe out every computer connected to the internet. That's not an end-of-human-life FOOM, but certainly an end-of-the-global-economy.

Expand full comment

Problem: Eloser is not credible. He believes that animals cannot suffer because the GPT-3 cannot suffer, and that anyone who disagrees on this point is an idiot. Scott has come a long way from where he started and has become far wiser and more intelligent than any rationalist. Eloser simply has not done the same. He's the same old ignoramus he's always been.

Expand full comment

For the sake of argument, just how is this 'super AI' going to be able to do much of anything besides move bits around at first? Do we really think there is something in the laws of physics that the AI will be able to discover while in it's box that will allow the AI the machine-equivalent of telekinesis? That seems pretty implausible. So how about how much capacity is there for a computer to independently create and move custom complex multi-material physical objects remotely? None. What about energy capacity? Oh yeah, that is limited, slow to ramp up, and requires complex multi-material physical objects. At this point in time, FOOM is nonsense as the AI's speed of change is going to be capped by the slow speed of the monkeys that wrote it. If we had something like a replicator, I might actually be concerned, but that technology is a long ways off.

As you might infer, I am pretty heavily against the rapid take-off scenario because so much would have to go right (or wrong, depending on your perspective) for it to occur. One thing I don't see mentioned is that there seems to be a religious aspect to the argument for rapid takeoff where AI is spoken of as an unknowable, omniscient, and bordering-on-omnipotent being. The FOOMers seem to be trying to reimagine God.

Expand full comment

minor objection to the choice of examples:

I think online retail is still in the fast-growth part of its sigmoid and despite being a bitcoin maximalish I don't think cryptocurrency will ever do much for tech industry profits -- the impact will be at least an order of magnitude smaller than smartphones. There isn't really any legit business use case for blockchains because all corporations are centralized and subject to the whims of governments, but the whole point of a blockchain is being decentralized and beyond the reach of governments that want to inflate the currency or block transactions between consenting adults. Without the decentralization a blockchain is just a really inefficient type of spreadsheet. Vitalik's idea of putting his warlock on a blockchain to prevent some centralized authority from nerfing it is silly because nothing that decentralized has within several orders of magnitude of the performance necessary to run an MMORPG server in real time. Plus games and shitcoins generally have an author with enough social authority to hardfork any patches they want regardless of the blockchain code.

Expand full comment

Eliezer Yudkowsky believes himself to be a superintelligence. He is also preoccupied with power and punishment (not judging him for that - many humans are - just factual observation). He believes that an AI would be preoccupied with power and punishment because he either implicitly or explicitly assumes that his perspective is analogous to that of a future AI superintelligence. Someone above made the analogy that humans are like the Judeo-Christian God in the Biblical story of creation, working on Adam, trying not to wind up like Kronos did. It seems almost hilariously obvious to me that Eliezer and people who agree with him believe the exact opposite: that they are in the process of creating a god, and that something must be done to ensure that this god is a friendly and not a vengeful one.

I’m glad to see how many people are pointing out how frustrating it is to see well-intentioned but ultimately ignorant and misdirecting armchair speculation coming from someone with obvious intelligence but extremely limited practical experience or depth of study in the field they are most known for. You’d think this would be the default position for a group wishing to be less wrong.

Expand full comment

While the question of "who had language" is not completely resolved and is riddled with the problem of "what is language" (compare "Diseased thinking about disease"!), my gut reaction to "Homo erectus had language" is, well... NO. Just no. There is strong debate whether Neanderthals had language (and we unfortunately know too little of Denisovians to tell); if we have to debate that, given that Neanderthals are a much closer genetic sibling, Homo erectus is probably out of the question.

Expand full comment