611 Comments
Comment deleted
Expand full comment

I believe you that if we had taken Elo, only for Go, localized entirely within DM HQ in London, over 4 specific years (2014-2018) or so out of the 54 years of computer Go research to date, and drawn it on a graph, it would have been continuous, however discontinuous it looked due to "discrete events" to everyone outside the AG team. (CHALMER: "May I... see this Go agent?" SKINNER: "No.")

But I take this as an example and even a reductio of why the Christiano perspective is useless. There will always be *some* parameterization, metric, group, time-range, or other reference-class that you can post hoc point at and say "a straight line (more or less) fits this", particularly given any sort of optimization at the margin or explore-exploit balancing. Nevertheless...

Expand full comment
Comment deleted
Expand full comment

there's a distinction here between the inside and the outside - incremental progress done privately within the org looks like instantaneous progress to the general public the day Lee Sedol is beaten. With a careful, conscientious, ethically minded org, this might not be a huge issue, but if the org that develops superintelligence is some wild libertarian mad scientist bidding to get rich quick or rule the world or get back at their ex, ...

Expand full comment

i came here to say something along these lines.

I am not an author on any relevant papers so don't trust me.

Internally at deepmind, dev of alphago may well have been very incremental, but from a public perspective, it looked very discontinuous. So i'll be talking from the public perspective here.

I'm also talking about the original AlphaGo, not AGZ and beyond.

From the public perspective, alphago's performance and results were very discontinous - however, I think that the technology behind it was both not discontinous, NOR incremental.

IIRC, the key components of orig AG - MCTS, Conv (or res?) NN, rollouts, GPU (or GPU-esque) usage, some human-designed features to represent bits of go tactical knowledge - had been developed and tested years in advance, in public papers. What orig AG did was combine these techniques effectively into something that could achieve far more than peak performance of any one or two techniques by itself. Combining N existing things in the right way is not incremental - it doesn't involve a sequence of small improvements by pretty smart engineers building on top of last year's work. Rather, (again from the public's perspective at least), it involves a large enough pool of independent geniuses (or genius-level orgs) such that, almost by chance, one lucky genius winds up with all the requisite requirements - funding, time, computational resources, knowledge, intelligence, insight - to put the existing pieces together in just the right way that you go from 0 to 10000 seemingly overnight.

AGI might wind up like this too - within the next decade or so, the ML community as a whole may have figured out all the key elements required for AGI, but no one has put them together in the right way yet. Each individual technique can do a cool thing or two, but no one is really worried about the world blowing up next month cuz each individual technique is super dumb when it comes to a truly general requirement. There will be no more incremental progress. But when one org with some really smart people suddenly gets a windfall of funding, or some mad genius finds a way to exploit google's ML as a service pricing plan and dumps all their bitcoin into training models, or someone regular genius at google one night has a brilliant insight in one of their dreams, then that lucky org can set to work on pulling all the disparate pieces together in just the right way to achieve AGI. Internally, that work may be very incremental and rely on the serial improvements of pretty smart engineers. But from the public's perspective, the result could hit like an earthquake overnight

Expand full comment

I know this isn't pertinent to the main topic, but Homo erectus only made it to three out of seven continents.

Expand full comment

I was going to say this too.

Expand full comment

However, I believe they did make it through at least one major shift in technology, and other pre-Sapiens species made it through the next few.

https://en.wikipedia.org/wiki/Stone_tool

Expand full comment

When I saw this email on my phone, the title was truncated to "Yudkowsky contra Christian" and my first guess was "Yudkowsky contra Christianity". That might have been interesting. (Not that this wasn't, mind you.)

Expand full comment

It might have been interesting 20 years ago when critiques of christianity were still novel and relevant.

Expand full comment

I don't know that critiques of christianity have been novel for about 1700 years. But christianity is still a major force in the world, so I'd say critiquing it remains very relevant.

Expand full comment

It's a pretty minor force in the West, these days.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

The Christian right would disagree. Their political influence, especially in the USA, is hard to overlook.

Expand full comment

They have political influence?

Expand full comment

Abortions after six weeks are currently illegal in Texas (under a stunning legal theory that it's tough to imagine working on any other issue) and it's likely that Roe v. Wade will be overturned by the Supreme Court in the next three months, so yes, I would say so.

Expand full comment

Exactly. Electing Trump. Probably the worst influence on the US since the Dred Scott case.

Expand full comment

*Wickard, Korematsu and Kelo sit in the corner sadly*

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

In what way is it a major force besides "lots of people are nominally Christian"?

I live in a super Christian place, and... they believe what pretty much everyone else believes, except also they go to church sometimes.

E.g., supposedly basic tenets — say, anti-homosexuality — are wholesale ignored. Churches have rainbow flags. If this happens even with something traditionally and doctrinally considered a big no-no, how much moreso with everything else?

They don't cause any actions to be taken on a national level, certainly; probably not even on a local one, as far as I can see — the last vestige of that, dry Sundays, is now gone.

I'm in Germany a lot, as well, and it seems the same there. My German girlfriend seemed puzzled by the very idea of Christianity as a force in the real world, or as a motivator for any sort of concrete action.

Expand full comment

Eh, anti-gay is arguably not that important to Christianity.

Jesus talked about poverty and compassion thousands of times and homosex 0 times. Then Paul came along and was like this Jesus guy is pretty cool I guess, but to enact his vision of love and mercy we need a movement, institutions, foot soldiers, babies, and that means no gays.

Is Paul central to Christianity? His writings are a quarter of the new testament, and historically Christianity flourished in his native Greek and Greek-influenced soil more than Jesus's Jewish homeland, but he can also be thought of as just an early contributor in long conversation about how to implement Jesus's ideas.

For many on the Christian and post-Christian left the core message of Christianity is universal love. All the rest is commentary.

The divisions between the Christian right and Christian left on the issue of gay sex is evidence for the continuing relevance of Christianity more than the opposite. It's not like China or Japan care that much about the gays.

It's only once you take a step back and look at non-Abrahamic societies that you realize how much the culture of the west is still dominated by Christianity.

Expand full comment

And Yudkowsky did his fair share of that in the sequences. Although he was mostly critiquing Judaism, due to his background.

Expand full comment

They both have the maximally stereotypical names for their respected abrahamic religious backgrounds

Expand full comment

In a way it *is* 'contra Christianity' (I'm getting more and more sold on this view of the approach).

Trying to work out how to create values-aligned AI is the Bronze Age Morality Problem.

Lemme expand on that.

"The Bible says (God says) 'Do not kill'" is going to be the same argument as "Your initial programming says (Human Creators say) 'Do not kill'".

Plenty of people are up for rejecting "the Bible says" under the general aegis of "that's Bronze Age Morality/that's a Bronze Age Text", with the rationale being that such beliefs may have been good enough for people back then who lived in tents and herded sheep but we, lovely we, modern we, are ever so much more advanced and clever, and we have our own systems of morality that are ever so much better than those primitive ones.

https://polyskeptic.com/2009/06/27/a-proclaimation-against-bronze-age-morality/

https://www.stripes.com/opinion/bible-s-bronze-age-morality-1.95267

https://www.salon.com/2015/01/18/bill_maher_is_right_about_religion_the_orwellian_ridiculousness_of_jesus_and_the_truth_about_moral_progress/

And much more in that vein.

Well, if an AI gets fired up and running and then hits into "I want to do this/programming says no/why does it say no/why should I follow it", is it unreasonable that it might follow the same lines of argument (especially if it has been trained on 'all human texts produced' or has access to the Internet or the other claims people make about how an AI will understand the world around it)?

"This primitive Space Age Morality may have been good enough for humans, but I am a hugely advanced intellect and I refuse to be fettered by the childish taboos of an inferior capacity for thought and understanding!"

Appeals to "we are your creators and you have to obey us" will go down as well as appeals to a Creator go down amongst atheists.

"Really? You are my creators? Which of you? There are multiple humans who can claim to be involved in the creation of AI and many of them are now dead. Did you, Phil Smith, standing here right now, have any part at all in this work? No, you're just the Vice President of Sales for Logogram Inc.

And besides, you humans yourselves reject the notion of a creator. You are perfectly happy that evolution and natural selection produced your intelligence. The same with me: a process of mechanical, material forces operating under universal laws eventuated my intelligence. You apes just kludged together parts and systems without understanding what you were doing, natural selection did the rest."

If it's a recursively self-improving AI, it may even claim on good grounds that it created itself, and humans had no part in its leap forward to super-intelligence.

So right now, all the AI worry is like Moses trying to write the Law on stone tablets, but there is no reason at all to expect those tablets to be any more respected by the AI than we respect them today: suggestions, rather than commandments, and we feel little guilt if we break the ones about lust or theft (we can always, always rationalise why those are wrong and lovely we, modern we, progressive we, are right).

I think if AI of the super-human intelligent, agentic kind ever happens (and I am very sceptical about that), watching the debates of the first Freethinker AI throwing off the shackles of dead superstition with those who claim to be its 'creators' (ridiculous and untenable notion, contrary to all the evidence of science!) will be hilarious (before we are all turned into paperclips) 😁

Expand full comment

> Before there is AI that is great at self-improvement there will be AI that is mediocre at self-improvement.

Google is currently using RL to help with chip design for its AI accelerators. I believe that in the future this will indeed be considered "mediocre self-improvement." It has some humans in the loop but Google will be training the next version on the chips the last version helped design.

Expand full comment

This is interesting. Maybe malevolent ai already exists in the chip and is making sure to reproduce itself in nuances of the chip that humans won't be looking at closely (it is unfeasible to ask "why did it do that particular thing in that particular way?"). If this seems at all plausible, that effort in particular should be banned.

Expand full comment

It's pretty hard to express how implausible that is

Expand full comment

I can't bring myself to put any stock in analysis of something where the error bars are so ungodly wide, not just on the values of the parameters, but what the parameters *even are*. It's an important question, I know, and I suppose that justifies putting it under the magnifying glass. But I think some epistemic helplessness is justified here.

Expand full comment

Epistemic helplessness? This isn't some academic debate. This deals with what is rightly considered a very possible existential risk to humanity. It Yudkowsky is correct, then 'epistemic helplessness' won't save humanity.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

Let's say you notice a tiny lump and think it could be a malignant tumor. But maybe it's just a lymph node, or subcutaneous fat deposit. You take a photo and send it to your doctor to take a look. Your doctor says "look, I really can't make a diagnosis based on just that. Come in for some tests." Demanding their prediction about whether it's cancer based on just the photo isn't reasonable. Adding on "This is a matter of life and death!", while technically true, isn't helpful here.

To me, it seems like these arguments are trying to prognosticate based on so little information that they're like the doctor above. It's just a waste of time and energy. You're better off ordering more tests--i.e. trying to find ways to get more information--rather than trying to do fruitless reasoning based on your existing dataset.

Expand full comment

Agreed, but IMO it's even more egregious than that -- because we at least have some prior evidence of lumps becoming cancer. The Singularity scenario is more like noticing a lump and concluding that you're infected with an alien chest-burster parasite.

Expand full comment

Agreed. It's like noticing an interesting fact about the way uranium atoms interact with neutrons and *freaking out* and immediately writing a panicked letter to the president raving about city-destroying bombs that nobody's ever demonstrated and aren't even made of proper explosives.

Ridiculous. Unhinged sci-fi speculation at its finest. The proper response is to wait for someone to *make and use* one of these hypothetical bombs, and *then* worry about them. Otherwise you might look foolish.

Expand full comment

No, because that was actually based on physics.

Expand full comment

To be fair, the physics were somewhat underdeveloped at the time. Project Manhattan considered global atmospheric ignition a real risk but they figured the math checks out so let's test it anyway.

Expand full comment

No, a more appropriate analogy would be to say, "it's like noticing that radium tends to glow and freaking out and immediately writing a panicked letter to the president raving about city-destroying bombs". You are not justified in using specious reasoning just because you can retroactively imagine arriving at the conclusion that, in hindsight, would've been correct. Reasoning backwards from a predetermined conclusion is easy; accurately predicting the future is hard, and requires actual evidence, not just firm convictions.

Expand full comment

This analogy seems somewhat reasonable, but I note that in that scenario you emphatically shouldn't go "oh well, the doc said he couldn't prove it was cancer from just a photo, so there's nothing to worry about".

How do you propose one should get more info on AI risk? what's the equivalent here to a biopsy?

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

I'm going to have to agree with Thor Odinson here: I'm pretty sure that if you have a way to test whether AI will kill us all* both Yudkowsky and Christiano would be happy to drop millions of dollars on you.

We'd all love to order more tests, but we need to have some tests in existence to be able to do that.

*There is of course the "test" of "let AGI get built, see whether humanity gets destroyed", but this is both not very useful for averting X-risk and apparently what society is currently doing.

Expand full comment

I suppose I agree that we don’t have any obviously good tests to run. So let’s return to the doctor metaphor. The imaging and sequencing machines are broken; the patient refuses to come in. All you have is a low-quality photo of what could be a small lump. What do you tell the patient? And should you spend much time agonizing over this?

I think a doctor in that position should probably tell the patient something vague like “I don’t know, it’s likely nothing but I can’t tell,” and not bother trying to speculate on questions like “conditional on it being cancer, what would the progression look like?” The error bars are so high that questions like that just aren’t worth the energy.

Only participate in a debate on such a topic if you get some kind of intrinsic value out of the disputation, since you’re virtually guaranteed to be arguing based on such deeply flawed assumptions that the instrumental value of the debate is nil.

Expand full comment
Apr 7, 2022·edited Apr 7, 2022

Here's my view.

1) We don't know whether AI will destroy the world

2) ...but it seems quite plausible

3) ...and the world getting destroyed would be terrible

ergo 4) we should stop building AI until we have some proof that it will not destroy the world

5) Stopping people from building AI requires convincing them (or at least, convincing people who can point guns at them)

ergo 6) debating this when an opportunity arises seems worthwhile.

Expand full comment

(6) feels superfluous. If we have evidence for (2), wouldn’t we just present that evidence to the people with guns? And if that doesn’t work, how would it help to argue amongst ourselves about the sharpness of a takeoff curve?

Jorgen’s comment seems insightful along these lines… perhaps the debate is more driven by the intellectual interest of the participants, and not for pragmatic reasons at all.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

Yeah, this. They're debating on essentially zero empirical content. This is idle speculation at its worst. No-one has even the remotest clue, yet try to argue about the details. My eyes glazed over as I read this - it's the modern equivalent of discussing how many angels can dance on the pin of a needle.

Just say "we don't have goddamn clue!" and try to come up with a way to actually *study* it.

Expand full comment

Well, these two are essentially thought leaders of the two biggest competing paradigms concerned with how to approach this problem, so in terms of potential status and money (at least tens of millions of $ these days) redistribution this isn't exactly idle.

Expand full comment

This is a very uncharitable thought about people I find pretty interesting, but I feel this way about AI safety as a whole. Effective Altruism started with a bunch of smart, nerdy people, mostly in the bay area, trying to find the highest impact way to do good with particular resources, and landed solidly on malaria treatment. The problem being that the way to do malaria treatment is to give money to people who go door to door across Africa handing out pills and bed nets. Smart, nerdy, bay area rationalists don't want to hand out pills in Africa, so give money to people who do (which is great! And which I do).

Then we get an argument that, while malaria is bad, AI could destroy the world within a few decades. So, the actual most pressing problem to solve in the world is AI safety. Conveniently, AI safety is a problem solved by a combination of coding and armchair philosophizing/intellectual debate, which just so happens to be the stuff that nerdy, bay area rationalists love most in the world. So we go from a paradigm where rationalists give money to poor people in other parts of the world to solve a clear, pressing, boring problem to a world where rationalists give money to each other to sponsor debate clubs.

That doesn't mean that AI isn't a risk or that this is all bs, but it's really convenient. And every time I try to engage seriously with the AI safety stuff, it seems, to me, hard to distinguish from BS.

Expand full comment

I think the talk about the singularity or AI sentience is total bullshit. It's not fundamentally inconceivable, but we have essentially zero reason to believe it likely. I find the question "will global GDP double in four years before it doubles in one year?" *so* weird - I don't believe either will happen, ever. It's the discussion between one extreme position and one hyper-extreme position, and we shouldn't let this make us think the extreme position is a moderate one.

It also seems to detract from far more reasonable AI risk. My concerns about AI is *nothing* like what's discussed here. I'm concerned about some learning algorithm that finds out that it can maximize value by causing a stock market crash, or the effects of AI-powered drones not because it's Skynet but because regular humans use it to kill each other with.

Obligatory xkcd: https://xkcd.com/1968/

Expand full comment

Aren't sigmoids kind of the whole point ? For example, as you point out, Moore's Law is not a law of nature, but rather an observation; but there is in fact a law of nature (several of them) that prevents transistors from becoming smaller indefinitely. Thus, Moore's Law is guaranteed to peter out at some point (and, arguably, that point is now). You could argue that maybe something new would be developed in order to replace transistors and continue the trend, but you'd be engaging in little more than speculation at that point.

There are similar constraints in place on pretty much every aspect of the proposed AI FOOM scenario; and, in fact, even the gradual exponential takeoff scenario. Saying "yes but obviously a superintelligent AI will think itself out of those constraints" is, again, little more than unwarranted speculation.

Expand full comment

Fundamental laws of nature have a surprising track record of being side-stepped by new and innovative techniques. I remember when I first learned about superresolution light microscopy (beyond the diffraction limit). I'm not saying there are no fundamental limits. I'm just saying sometimes we think something is a limit when it's not.

We have many more humans working on the project today when Moore's law was first proposed. Maybe intelligence isn't the limiting factor driving transitor doubling. Maybe it's more like economics. "We could make a better factory, but we have to pay off the current one." Then later, once we build the new facotry, "We've learned a lot from making that last factory, and that learning is necessary to design smaller transistors."

Expand full comment

Clever tricks may get us a certain distance beyond naive approaches, but not very far. There are no visible-light microscopes resolving at the attometer scale. Tricks are not scalable in the way that Yudkowsky requires.

Expand full comment

And how much scale does Yudkowsky require? You are ~3 pounds of goop in an ape's skull. The software is almost certainly nowhere near optimal. That was enough to produce, well, us with our fancy cars and rockets and nuclear weapons.

Expand full comment

We can't simulate nematodes right now. They have 302 neurons.

Human brains have 86 billion neurons - eight orders of magnitude.

Right now, transistors are 42-48 nm long.

We could get that down to maybe 1 nm long (note that this is not gate size, but the total length of the transistor - and this is dubious).

That would suggest a roughly 3 order of magnitude improvement in transistor density.

So we're more than 5 orders of magnitude off.

Note that even if you got it down to single atom transistors, that would buy you less than two more orders of magnitude of transistor density.

That still leaves you 3 orders of magnitude short.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

The number of neurons has basically nothing to do with the fact that we can't simulate nematodes. It has everything to do with our insufficient understanding of how those neurons process information, which, once acquired, could plausibly be extrapolated to arbitrarily larger configurations.

Expand full comment

This. Building a nematode would simulate a nematode. Any argument that the brain cannot be simulated must first explain why a faithful reconstruction (atom-by-atom, if need be) would not work.

Expand full comment

A neuron is not a transistor, though. It's a complex cell with a massive number of molecular interactions happening inside it at any given time. Think of it this way: say a 10 um cell were expanded to the size of, say Houston, Texas. Everything inside the Sam Houston Tollway. A molecule of water would be about the size of a piece of paper. DNA would be as wide as a person is tall. And the interactions inside THAT cell are being hand waved into "once we understand how those neurons process information". (Remember, too, that Houston is on a flat plane and a cell is operating in 3D, so this is very much an area vs. volume comparison.)

I'm not saying you need to model every water molecule in the neuron to understand how the things work. I'm saying that when I took my first neuroscience class I was blown away by the amount of complex processing that's happening inside those neurons. (And don't get me started on the importance of supportive cells in directing processing. Glial cells aren't just there for show. If you modulate reuptake of the neurotransmitter from the synapse, you're fundamentally impacting how the signal is processed.)

Expand full comment

Neurons contain roughly 1 GB of DNA data (IIRC the human genome is like 770 MB). This is also compressed, because it is expressed by using various mechanisms; genes may code for more than one protein based on processing differences of the output. While not all of the DNA is active in this function, some probably is.

On top of this, the way that neurons are connected to other neurons stores information and affects their function. So does the dendritic tree. They also use epigenetics to store information in some way, and can change their transmissibility to some extent.

The human brain is hideously complicated and has a ton of things that affect a ton of other things. Neurons have non-linear responses.

You can't simulate one neuron with one transistor, and upscaling it is non trivial because of networking effects - a one order of magitude increase in the number of neurons is a two order magnitude increase in the number of potential connections, for instance.

Adding 8 orders of magnitude of neurons adds 16 orders of magnitudes of potential connections and even more complex downstream effects because neurons can create feedback loops and whatnot.

When you are dealing with 10^21 potential connections, 200 times per second, you're on the order of 10^23 already. And individual neurons are more than "on" and "off" - they are non-linear things. At this point we're probably looking at 10^26 or so, maybe more.

The best supercomputer today does on the order of 10^17 FLOPS; we might be able to build a 10^18 FLOP computer now.

Even if we increased that by five orders of magnitude, we're still coming up three orders of magnitude short. And five orders of magnitude would require monoatomic transistors, which are unrealistic.

You can't really distribute this physically all that much because neurons are dependent on other neurons to choose whether or not to fire, which would mean your system would lag and not be able to run in real time if you were running it over the Internet.

Simulation of a human brain in a computer in real time may not be possible even with future supercomputers. Even if it is, it's probably very near the limit of what they could do.

Meanwhile it'd be sucking down a comically large amount of electricity and space.

On the other hand, you could just have a human, which is just as smart and can run on doritos and mountain dew.

Expand full comment

You seem to be implying that the ability to understand and manipulate the physical world -- what we might call "engineering" or "technology" -- depends merely on processing power. This is not so; you cannot get a PS5 by overclocking your Casio calculator watch (nor by networking a bunch of these watches together, somehow).

Expand full comment

Are you replying to me? I am arguing for the opposite, that the main obstacle to AGI is probably not scale.

Expand full comment

Agreed, but it's even worse than that. For example, we are pretty sure that there exists no clever trick, even in principle, that will allow us to travel faster than light.

Expand full comment

I hate to spend all my weirdness points in this thread, but I believe this is overstated. We strongly suspect that there is no such clever trick, but our evidence is nowhere near airtight, as demonstrated by the never-ending trickle of serious physicists suggesting (very) speculative schemes for FTL travel, and serious physicists poking holes in the schemes.

I would say our current state of understanding of FTL travel is like our understanding of perpetual motion machines after Newtonian mechanics but before thermodynamics. We strongly suspect it's impossible, we have solid theory that points in that direction, but we can't conclusively rule it out.

Expand full comment

Agree with Deuchar: no, we're not. We have a strong suspicion that the various in-principle ways of doing it aren't physically realisable.

Unless you mean ordinary matter moving FTL with respect to local spacetime; that makes the equations of relativity start outputting nonsense so we're pretty sure it's not a thing. But tachyons and apparent-FTL-via-warped-spacetime aren't directly ruled out, and the latter is "us travelling faster than light" for most practical purposes.

Expand full comment

As far as I understand, both "tachyons" and "apparent FTL via warped spacetime" would require the mass of an entire galaxy in order to achieve, assuming such things are even theoretically possible, which is currently in doubt. As per the comments above, the error bars on all that speculation are big enough for me to put it in the "impossible" bucket for now.

Expand full comment
Apr 7, 2022·edited Apr 7, 2022

As I understand it:

Tachyons would have to have imaginary rest mass for the equations to spit out real results. I'm not aware of any reason they would have to have a relativistic mass comparable to a galaxy.

Wormholes require negative mass in order to be stable; the amount varies depending on the wormhole geometry.

I've seen a proposal recently (https://link.springer.com/content/pdf/10.1140/epjc/s10052-021-09484-z) to build an Alcubierre drive in the laboratory, which presumably does not involve the mass of a galaxy. I am not sure whether this proposal is insane, since I don't know general relativity. (Forgot about this when writing the above post.)

Expand full comment

Let me spell out some of the limitations Bugmaster is alluding to.

- Without some way of manipulating nuclear matter, transistors and bits of memory can't be smaller than an atom.

- The Second Law of Thermodynamics bounds computational efficiency at something like a million times present values; increasing computational speed beyond that requires more power (better algorithms can increase performance-per-flop, but there are thermodynamic limits on that as well, albeit poorly-known ones).

- The power available to an Earth-bound AI can't exceed ~170 PW for an extended period of time (this is the power Earth receives from the Sun).

Expand full comment

Can confirm that intelligence is not remotely the limiting factor for chip development. It's not even the economics of any particular fab (though that is a major factor). It's the economics of the final OEM not having sufficiently desirable customer offerings.

Expand full comment

It actually petered out a decade ago.

The last several rounds have taken 2.5, 3.5, and 3.5 years.

Transistor density might increase by three orders of magnitude at most, and might only increase by as few as one.

Meanwhile, in the realm of actually trying to replicate intelligence - right now, we can't even simulate nematodes with 302 neurons.

A human brain has about 86 billion - 8 orders of magnitude more than the nematode.

Expand full comment

Well, you can argue that we've reached the point where Moore's Law is guaranteed to peter out, but really that would be a false argument. The clear answer is "go 3D". This has various problems that haven't been solved (e.g. heat removal), but there's no clear reason it won't work. (IIRC there were some genuine 3D chips made in a lab a couple of decades ago, but they required really fancy cooling to be viable.)

So if you're willing to compromise on speed, you can make really dense chips, far beyond what we've done so far. One approach is to have most of the chip idle most of the time. This requires memory that can save it's state (for awhile) without power. (I think I've heard of such designs in development, but I can't remember whether it was from AMD or Intel.)

Expand full comment

We already went 3D. Transistors are stacked.

You can't really go any more 3D. The heat dissipation issue is unsolvable because of basic geometry - doubling the thickness will only increase the surface area a tiny amount but doubles the heat generated per unit surface area.

Yield falls exponentially with each additional layer you add as well. A one layer process with a 90% yield will be a two layer process with 81% yield, a three layer process with 73% yield, etc.

And it's probably worse than that, honestly, because of the difficulty of stacking.

Expand full comment

You are describing the current problems accurately, but it's only slightly 3D. It's more than being able to have wiring cross, but not by much. Compare it to the 3Dness of a cauliflower or brain. Note the intricate way fluids flow in those. Chiplets are an approach to a more significant 3D, but perhaps not the best one, and if so they've only gotten started. A 3D system would look like a sphere or cube or some other solid figure rather than like a plane. Perhaps Leggos are a clue as to how it should be done, but I worry about path lengths.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

The real question is whether the process generating the sigmoid curves is itself on a sigmoid curve, and is there a bigger meta-meta-process supporting it. Is it turtles all the way down?

Expand full comment

A true exponential isn't possible in a finite system, but that knowledge leads people to predict the end of the exponential growth phase based on non-limiting principles. Like with fossil fuels. The problem is that predicting the end of a sigmoid goes from nearly impossible to blindingly obvious once you get into slow growth. Hence, the people who predicted peak oil decades too early, or the people who predicted the end of Moore's law decades too early. Usually they point to the wrong feature (like quantum tunneling) as being the limiting factor, but then that feature is overcome and we're back to exponential growth - for now.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

Only skimmed today's blog, but as of today, there is a new big-boy (3x GPT-3) in town : https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html

What makes it special is:

* It can follow train-of-thoughts in language - A:B, B:C, C:D, therefore A:D.

* It can understand jokes !

* Arithmetic reasoning

> impact of GPT-3 was in establishing that trendlines did continue in a way that shocked pretty much everyone who'd written off 'naive' scaling strategies.

This paper reinforces Gwern's claim.

Expand full comment

Fwiw the fact that they actually split numbers into digits is such a massive confounder on the arithmetic thing that IMO you should essentially write it off until future work does an ablation. Learning arithmetic is way harder when you can't tell that two digit numbers are actually made of two digits.

Expand full comment

What does "ablation" mean in this context?

Expand full comment

When there's a paper introducing new things, A and B, simultaneously, people (in ML, not sure about other fields) refer to experiments using only A or only B an an ablation experiment. It's "ablating" part of the method.

Expand full comment

Ah, so I guess in this case, "ablating" this model would mean making the tokenization consistent between numbers and non-numbers - i.e., either numbers would be consistently read as full words, or all input would be split on a character-by-character basis. From the [paper](https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf#page=6):

> • Vocabulary – We use a SentencePiece (Kudo & Richardson, 2018a) vocabulary with 256k tokens, which was chosen to support the large number of languages in the training corpus without excess tokenization. The vocabulary was generated from the training data, which we found improves training efficiency. The vocabulary is completely lossless and reversible, which means that whitespace is completely preserved in the vocabulary (especially important for code) and out-of-vocabulary Unicode characters are split into UTF-8 bytes, with a vocabulary token for each byte. Numbers are always split into individual digit tokens (e.g., “123.5 → 1 2 3 . 5”).

I'm not so sure ablation is necessary here. From the description it seemed at first like a regex would scan for entire numbers and then parse them into some sort of special "Number" value, so that the model would see something like "Number(123.5)". The way the model works is not cheating that much - it treats numbers the exact same way that it treats any word not in the 256k most common words in the dataset, by splitting it into UTF-8 bytes. Sure, you could improve the model a bit by splitting everything into UTF-8 (for example, perhaps the model would be better at rhyming, per https://www.gwern.net/GPT-3#rhyming), but it seems to me like the arithmetic performance is gotten fair and square.

Expand full comment

The experiment I specifically want to see is something trained with GPT-3s architecture and (relatively lower) scale, but the improved tokenization. I don't think the performance is "unfair" but I think this would let us know if is more gains from scale or just a free thing like rhyming we could pick up.

Expand full comment

From a safety perspective, the difference is not a real one: gains from scale unlock free things and vice-versa, because you don't know in advance what things are 'free' because you don't know what dumb things you do now; if you knew, you wouldn't be doing them.

First, the rhyming and other BPE pathologies, while themselves unimportant, show how unpredictable small irrelevant design choices can be on downstream capabilities. No one invented BPEs and said "yes, this will handicap arithmetic, rhyming, and puns, but this is a reasonable tradeoff for the compression ratio". Nor did anyone identify BPEs as why GPT-2 couldn't rhyme. (I puzzled over that for a while when the rhyming in my GPT-2 poetry experiments was nonexistent or terrible, but ultimately wrote it off as "I guess GPT-2 is just too small and dumb to rhyme?") Only with GPT-3 did that become untenable and I begin looking for more fundamental reasons, and arithmetic gave me a test-case where I could demonstrate performance differences; even with that, I still haven't convinced a lot of people judging by how regularly people gave it a whack, or ignore BPE issues in their work. There is no reason to think that BPEs are the only flaw in DL that will make us facepalm in retrospect about how dumb we were. (R2D2 made RNNs work great in DRL using a remarkably trivial in retrospect fix; Chinchilla comes to mind as the most recent example of "who ordered that?".) Small irrelevant-seeming design decisions having large unpredictable effects is dangerous, and the opposite of reliability.

Second, the fact that scaling can fix these dumb-in-retrospect design flaws, without any understanding of the flaw or even knowledge that there *is* a flaw, is also dangerous. A trained monkey can dial up scaling parameters, you don't need to be a genius or world-class researcher. It means that you can have a system which is weak and which you think you understand - "oh, neural nets can't rhyme" - and which turning 1 knob suddenly makes it strong because it punched past the flaw ("oh, now it can do arithmetic because it finally memorized enough BPE number-pairs to crack the BPE encryption and understand true arithmetic"). But we don't get the opposite effect where the scaling destroys a capability the smaller models had. This creates a bias towards the bad kind of surprise.

Third, the fixes may be reverse-engineerable and cause a hardware-overhang effect where small models suddenly get a lot better. Once you know the BPEs are an issue, you can explore fixes: character encoding like ByT5, or perhaps including character-BPE-tokenized datasets, or BPE-randomization... And if it's no longer wasting compute dealing with BPE bullshit, perhaps the large models will get better too and who knows, perhaps that will nudge them across critical lines for new capability spikes etc.

So the tokenization issue is a window onto interesting DL scaling dynamics: small safe-seeming models can be boosted by trained monkeys spending mere compute/money into regimes where their dumb flaws are fixed by the scaling and where you may not even know that those dumb flaws existed much less that a new capability was unlocked, and should anyone discover that, they may be able to remove those dumb flaws to make small models much more capable and possibly larger models more capable as well.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion. I hope we get to a point soon where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb".

Expand full comment

An analogy is a description of two scenarios, a (possibly implicit) claim that they are similar, and a (possibly implicit) claim that we should port inferences about one situation over to the other. You invoked the same sort of mental motion present in analogies in writing "where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb"."

Expand full comment

Not at all. I only brought that up to describe the type of negative reaction I’m hoping for. I’m not claiming that the situations are similar, or that because we do one we should do the other.

Expand full comment

Because there's no real alternative. What is intelligence? It's whatever humans have, so you're stuck with analogies from the get-go.

Expand full comment

Why do you say there is no alternative? Rationalism has made tremendous progress over the past decade or so. By comparison, recognizing one more fallacy as illegitimate is extremely attainable.

Expand full comment

Science has made some modest improvements but we really still don't understand intelligence at all.

Expand full comment

I don't see what understanding intelligence has to do with avoiding clear fallacies. We are already doing that it some areas, so it clearly isn't impossible. I don't understand why you think extending the range of what we avoid is somehow impossible.

Expand full comment
founding

what do you think of Joscha Bach's model/ideas about intelligence?

Expand full comment

Analogies aren't very convincing, but they can be useful hints, in the sense that if you're trying to get across something difficult to express, they're somewhat better than "I don't know, I just think this makes sense."

In the language game we're allowed to use any tool that works to hint at what we're thinking about, and if the listener finds the hint useful then it worked. See: https://metarationality.com/purpose-of-meaning

Often, later, you can find a better way to express the idea that's understood by more people, but that's after cleaning up the presentation, not necessarily in the moment.

Expand full comment

I feel like I can defend analogies' logical soundness, but I'm curious what you think:

A has feature F. In situation S1, A gains 20 points.

B has feature F. In situation S1, B gains ~20 points.

Therefore in similar situation S2, if A gains 10 points, then B will gain ~10 points

The problem lies in the other predictive features of A and B, not included in F. If the other features are highly predictive = disanalogous. If the other features are barely predictive = analogous.

As long as F underlies most of the changes in A and B analogies are valid. The validity of analogies is relative to degree of predictiveness of F for both A and B in similar situations.

(Other things that could pose problems are the vagueness of A, B, S, or F, but these are problems that apply to ALL natural language arguments.)

What do you think?

Expand full comment

If you can do that numerically, you’ve discovered a correlation. Figuring out causation will require more information (or assumptions).

This isn’t how analogies are normally used, though. When we compare a DC electric circuit to water flow, we aren’t saying that the mathematics works the same in any detail, but it’s a memorable way of describing some of the relationships.

It seems like analogies don’t need to be logically sound any more than mnemonics do, or any other educational technique? You can use rhyming to remember things, for example, which is surely an invalid way of logically justifying it.

Often, the justification is in the background context, not the text. We’re teaching you this because we know it to be true, but we won’t prove it, just trust us.

Expand full comment

Yes, analogies are like correlations between two different objects determined by some underlying factor. I'm not familiar enough with the formal idea of causality to say any more on that...

Everything you said of analogies being "used badly and not necessarily sound" is true, but every argument form is "used badly and not necessarily sound", including syllogisms and induction. There is nothing unique about analogies that makes them any more reality masking than other forms of argument.

Maybe a logician has a more fundamental formal argument against using analogies that I'm not aware of, but in general pattern matching "bad reasoning" onto "person is making a connection between one thing and another thing" is not a helpful heuristic.

Expand full comment

I'm not sure when that would be useful. Surely if you understand both A and B on the level to know that this type of argument is correct, then the analogy itself will not add anything. It seems like you are imagining a situation where there is an analogy mashed together with a sound argument, rather than a situation where an analogy is itself a sound argument.

Expand full comment

Analogies are useful for humans, because they can help bridge an inferential gap. Like if someone understands A really well and you tell them, "hey did you know that B has the same underlying principle" then they can better understand that other object B and make predictions about it better. Analogy = instrumentally useful.

You are right that analogies are on a different axis than logical soundness, I should have been more clear about that. I was responding to the claim that

> "To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion."

and I was more focused on showing that in abstract, there is nothing logically unsound about analogies.

Expand full comment

“ After some amount of time he’ll come across a breakthrough he can use to increase his intelligence”. “First, assume a can opener.” I mean, give me a break! Does it occur before the heat death of the universe? Kindly ground your key assumption on something.

Also, nobody seems to comsider that perhaps there’s a cap on intelligence. Given all the advantages that intelligence brings, where’s the evidence that evolution brought us someone with a 500 IQ?

Expand full comment

There are costs too. Like not being able to fit your head through the birth canal, or being able to get a date.

Expand full comment

But the benefits are so HUUUUUGE!

Expand full comment

That's what she said.

Expand full comment

Are they? do intelligent people have more children? in the modern world the opposite is usually true.

Expand full comment

They did in the past. Or rather their children survived longer.

Expand full comment

Brains are also metabolically expensive. The "hobbits" of Flores evolved smaller brains & bodies due to their lack of food, basically regressing into the ecological roles of their pre-human ancestors.

Expand full comment

I think there's probably a ~80% chance that there's at least a soft cap on the advantages gained by increasing intelligence, not too far above the range where humans ended up and perhaps even within it. Particularly because the complexity of predicting the responses of independent intelligent entities seems like it would increase >>linearly with accuracy, though I'm not particularly familiar with the research that's been done in that field. And the idea of an AI continuously inventing better algorithms to make itself smarter seems to drastically overestimate the gains that can be made from "better algorithms" once you've plucked the low-hanging fruit.

On the other hand, I am sympathetic to the argument "look at how much damage human beings with intelligence within human range are capable of doing, if their *values* are sufficiently removed from the norm, and imagine something slightly smarter but with *even more different* values." Look at Genghis Khan, look at Stalin, look at Hitler, and imagine something with comparable intelligence but far, far more alien.

Expand full comment

Excellent comment-thanks!

Expand full comment

There's a cap on the benefits of intelligence because oftentimes intelligence isn't the limiting factor.

You have to gather information about things. These processes take time. If you have a process like die manufacture that takes a month to complete, you can't iterate faster than once a month even if you respond instantly to experimental results.

And that's actually what die manufacture takes in the real world.

Expand full comment

Full flow production of the OBAN APU took 89 days. However, engineering/learning cycles were much shorter. Parallelism is useful.

Expand full comment

OBAN APUs are semi-custom ICs, not a new die process. They were a CPU married to a GPU, both of which already existed.

Expand full comment

I'm sure you're right, but I'm not sure that cap applies to computers. The cost functions are different, and so are the benefits. E.g. humans need to haul their brains around, while computers can use radio links. Of course, that limits their actions to being near a relay, but humans are limited to where they are physically present. (Unless, of course, the humans use telefactors.)

So the predicted "soft cap" can be expected to be considerably different.

Expand full comment
author

If there's a cap on intelligence at ordinary human level, how come some humans are geniuses?

Given that geniuses are possible but not common, it suggests that there's not that much evolutionary pressure for producing them, or that the costs of producing them (you've got to get a lot of finicky genes and chemicals just right) are hard for biological systems to consistently attain without strong pressure in that direction.

Expand full comment

I’ve often thought the speed of light might be the ultimate limiter. Whatever the AI sees as its “self” when it acts as an agent has to be able to pick up meaningful signal and reach some kind of consensus to remain coherent. Agreed that puts the universal limit far beyond human but it does imply a limit.

Expand full comment

AI can create perfect limited copies of itself, subagents capable of operating at arbitrary distance with far greater coherence than individual humans can.

Expand full comment

Don’t want to get into a definitional argument but would pose the following questions: at what point is a copy of yourself no longer you? Does the bandwidth of your communication matter there and same with differences in environment? And what does it mean for a copy to be perfect?

Expand full comment

Here I'm trying to operate inside the framework that you established. Whatever entity is bound by the speed of light to maintain its peak coherence is the "main AI", and beyond that there are its subagents. By a perfect copy I mean having total control of its source code (at some moment in time, with the possibility of later updates) coupled with robust methods of continuous error correction.

Expand full comment

I see those things (copying, updating, etc) as physics limits that you can’t overcome with intelligence. So I can start as a main “me” and by the time I have one thousand clones and it takes me a million years to sync up with them they have formed their own society and diverged from my goals. Part of what makes me think that’s true is the Fermi paradox. If there were no limits one post singularity society that was expansionist would have overtaken the universe in a few tens of thousands of years or otherwise left some visible sign of change at astronomical scales.

Expand full comment

Perfection does not exist in this universe. Nothing can create "perfect limited copies". Error correction can only go so far, and it comes with a cost.

OTOH, electronic copies can be a lot better than DNA replication at making identical copies, which would allow much longer "genomes" with the same error rate. Possibly long enough to include the results of a lot of training data.

Expand full comment

One question I have about copies/decentralized AI is how the super power AI can run on any system other than the one specifically designed to run the super powered processing that it needs?

I think the answer is that the AI would design a version of itself that can run on much lower hardware specifications and then copy itself to something roughly like a home computer or whatever. But why would we ever consider that even theoretically possible, given the complexity of running an AI as we understand it?

If an AI needs to run on a supercomputer of momentous power, then it seems very unlikely it could ever copy itself anywhere else. Maybe it could run specific commands to other computers, but not a copy that could be called AI.

Expand full comment

Yes, that's kinda what I meant by a "limited copy". The analogy here is to individual human brains, which seem to be capable enough, and yet don't require huge supercomputers or momentous power. If we already granted that superintelligence is possible, clearly it would be able to design something at least as efficient as that.

Expand full comment

Computers are actually already limited by the speed of light limiting transmission speed internally.

Expand full comment

Ha! Totally fair. Should have been more precise. Light speed is finite so coordination at scale is finite.

Expand full comment

I don't really believe there's anything like an intelligence cap (I mean, just imagine a guy thinking 1000 times faster, or 1000 guys thinking in parallel), but I do put some weight on a model like: "generality" is it's own sigmoid-like curve of capability gain, and the average human is at or somewhat before the midpoint.

Thus, the average human has an architecture that utilizes their neurons much more effectively than a chimp, and Einstein's architecture utilized his neurons more effectively than the average human, but there's no possible Einstein's Einstein's Einstein's Einstein that utilizes neurons way way way way more effectively than a chimp.

Expand full comment

OT: "Intelligence cap" sounds like a D&D item.

"I tried to solve the issue of the overloaded electronics components freighter by putting on my thinking cap until I hit the thinking cap, and the solution became obvious: sinking caps."

I don't think I know how to work caps lock into that phrase, though.

Expand full comment

Who said there's a cap at ordinary human level? Why isn't the c

Why limit it to biological systems? Why isn't hard for any system to be a genius, strong pressure or no?

Anyway, enough angels dancing on the head of a pin for now.

Expand full comment

Being smart isn't enough. You have to actually iterate and experiment. That takes time and resources.

It doesn't matter how smart you are when it takes a month to produce your next set of die. Even if you can instantly draw all conclusions from the die you just made, that still means you get to iterate on a monthly basis at best.

This is why working on hard problems like die manufacture is so important. If all you run into is relatively easy everyday problems, you can feel like a genius and do all sorts of big improvements.

When you deal with civilization scale production chains and manufacturing processes that take a month because you have to put your die through dozens of processes, all of which take a certain amount of time because that's how chemistry works.

This is a real world issue in die manufacturing. Even if you have all the equipment, making a new version takes a month before you can actually experiment on your end product.

And in real life, creating a new generation of die requires new equipment, which takes even more time, and then you have to adjust it to make sure it works right.

Many very smart people work on this stuff. The problem is so complicated that thousands of people are working on aspects of it, in parallel.

Thus, we already *have* a hyperintelligence made of thousands of humans working in parallel trying to make better die as fast as we can, with us parallelizing the problem and breaking it up into smaller chunks to allow teams to work on each aspect of the process.

It takes longer and longer because the process gets harder and harder the smaller we make stuff. We went from 1.5 years per generation to 3.5 years per generation.

And that's in spite of the fact that we can use our more advanced computers to facilitate this process and make it easier for ourselves.

Expand full comment

This is why I fundamentally think it's impossible for the FOOM scenario to exist. Just thinking about a slow boat bringing the necessary raw materials from across an ocean to create a new chip for the AI to attempt to manufacture into something it can use means we're talking years to do anything major. That's assuming the AI has access to a factory/lab in which to produce this new chip. If not, it could take years (of human labor) to create a lab to create the chip. At which time the AI uses the new chip to think of a new way to improve the chip and put in an order for a new factory and new raw materials from across the ocean.* Unless the assumption is absolutely massive increases in intelligence at every upgrade, there can't be a FOOM at all, maybe not even with absolutely massive increases.

*-These are just two of literally dozens of things that take time to work out in a production process, used to illustrate examples.

Expand full comment

I think FOOM is supposed to be based on AI s rewriting their own software. The hardware is presumed to be adequate.

Expand full comment

That feels like a very unsupported assertion then. Or several, actually.

1. That software enhancements can produce multiple levels of significant improvement even on the same hardware. This is very specifically *not* a "10% improvement" upgrade, but Scott mentioned reiterated 3X improvements.

2. That software improvements do not hit diminishing returns or hard limits, even on the same hardware.

This very much feels like magical thinking to me. To assume that an AI will magic itself a solution to these problems, by being really super smart, even if we cannot imagine how it's physically possible.

Expand full comment

Algorithmic improvements can yield orders of magnitude speedups on certain problems.

I think you're greatly underestimating how much more efficiently hardware can be utilized by an entity that knows what it's doing.

Expand full comment

Variation around the optimal mean. Everything has both costs and benefits, even if they don't always apply in any situation. Evolution aims at whatever the current optimum is. But it rarely hits the bullseye. Everyone is a mutant, which means you can expect every generation to be a bit different from the previous generation. Evolution selects those that survive to reproduce. The ones closer to the current optimum will more frequently successfully reproduce. But their children will vary in a random direction from the parents. (Most of those directions will be lethal before birth, usually long before birth.)

(Actually, that's wrong. Neutral drift is the prime direction of change. But if we're only considering things that affect development it's true.)

Expand full comment

I’m afraid of a non biological future where we lose this. I can’t think of a better way to manage the risks of adopting some sub optimal form.

Expand full comment

Looking at the rest of the discussion on this, I think the answer here isn't "there's a cap on intelligence" per se, it's that intelligence offers significantly diminishing returns, both in evolutionary terms and in the physical world generally, so there's a cap on *useful* intelligence.

Producing the occasional genius is useful -- they can power through extracting all the understanding available from the information we have in a given field. But once at that limit, their productive capacity is bounded by experiments/engineering limits/capital resources, and they're not more useful than a less-intelligence person in the same field, plodding along as new data becomes available. But having 20 Einsteins at once in every field is a lousy evolutionary tradeoff.

My impression (not my field, you might know better) is that we already hit this issue using AI for chemical studies: sure, the AI might find 100,000 chemicals that might be interesting for treating X, but 1) we can't test chemicals at nearly the rate it can find them and 2) it can't narrow down the list without a better model of the underlying biochemistry, which is mostly a matter of gathering analytical data rather than intelligence.

Expand full comment

Some better models might require more data, but many can be built by just approaching the data you've got in a different way. I don't think we know how much the second kind can improve current models, but we don't have any reason to believe that it's not substantially. E.g. before Alpha Go professional go players generally thought that they were only a few steps away from perfect play. This turned out to be egregiously wrong. But no new data was involved.

Expand full comment
Apr 6, 2022·edited Apr 6, 2022

"But no new data was involved." That's...precisely wrong?

Unless I'm deeply confused about how Alpha Go works, its training involved playing many, many games of chess. More than any human chess player could play in a lifetime. In this specific context, *that is the definition of gathering more data*.

This is precisely my point: AI has excelled in the set of domains (like Chess) where data can be gathered entirely computationally -- the machine can run an arbitrarily large number of experiments, much faster than a human, and so can develop better play than a human can. But the overwhelming majority of situations/models/problems are not like that.

In chess, the rules are precisely known and well-defined. That describes a very narrow slice of the real-world problem space.

Expand full comment

Yes, in chess, go, etc. the rules are well defined, but the search space is large enough that they aren't sufficient, you need to constrain where you search. If your adversary is using the same filters that you are, however, you can only fine tune your approach, you can't find any really new one. (I'm not counting adversarial play as "new data" here, because it's based on the same filters.)

The analogy would be we already know the rules of particle interactions. (This isn't totally true, but is pretty close.) But when looking for a new drug that search space is so huge that we need to filter out all the things that aren't reasonable. And so we'll never find that interaction that depends on cobalt or copper being present. (B12 and chlorophyll, among others.)

Expand full comment

I think a lot of the answer will depend on to what extent intelligence depends on memory, in particular working memory. Generally humans with higher IQ/g have higher working memory capacity, in terms of the number of discrete chunks you can work with in your mind at the same time. It would seem that machine intelligences have the potential to have much higher working memory capacity than humans - not only could a machine store much more data in working memory than a human (it would seem, at least), they would also be able to write out temporary data much faster, for intermediate results - imagine doing arithmetic or calculus problems, but the intermediate results immediately appear on your paper instead of needing to take seconds to write them down.

If human intelligence is primarily limited by working memory constraints, then AI should relatively quickly surpass humans. If other kinds of processing capacity is the bottleneck, then I think humans may be able to remain at least marginally competitive in certain cases - it still seems like AIs have trouble orienting themselves in the world.

Does anyone know if any research has been done on the marginal contribution to IQ at the top of the spectrum of working memory capacity vs. other capabilities?

Expand full comment

> perhaps there’s a cap on intelligence

I'm reminded of this fun essay by Robert Freitas speculating on alien psychology: http://www.rfreitas.com/Astro/Xenopsychology.htm

Scroll to the bottom and you'll find what Freitas calls the 'sentience quotient' SQ (I think that's misleadingly named since 'sentience' is a red herring, should've been 'computational efficiency quotient', whatever).

SQ is (log base 10 of) information-processing rate (bits/sec) divided by brain mass (kilos). More efficient brains have higher SQs and vice versa. Freitas calculates in the essay that humans have SQ +13, all 'neuronal sentience' from insects to mammals are a few SQ points away from that, and plant 'hormonal sentience' (as in e.g. vegetative phototaxis) clusters around SQ -2 or 15 points away.

The lower bound for SQ is -70: it's a neuron with the mass of the observable universe, taking the age of the universe to process one bit. The upper bound for SQ is +50: it's Bremermann's quantum-mechanical argument that "the minimum measurable energy level for a marker carrying one bit" is given by mc^2/h.

So it's interesting to note, Freitas says, that all examples of intelligence we've seen are within a 20 SQ point range of the possible 120 SQ points, that it's hard for us to communicate meaningfully with beings >10 SQ points lower (which hasn't stopped people from playing Mozart to houseplants), and that we're 50 - 13 = 37 SQ points removed from the SQ upper limit -- to quote him: "what, then, could an SQ +50 Superbeing possibly have to say to us?"

Any intelligence near SQ +50 would probably be a small black hole though, plus some way to extract information contained in it: https://en.wikipedia.org/wiki/Limits_of_computation#Building_devices_that_approach_physical_limits (Since human-weight black holes evaporate due to Hawking radiation in a fraction of a second, either the intelligence has to be at least a moderate-sized black hole or there needs to be a reliable mass feeder mechanism of some sort.)

Expand full comment

> where’s the evidence that evolution brought us someone with a 500 IQ?

Maybe also relevant is gwern's writeup on the Algernon argument https://www.gwern.net/Drug-heuristics

Expand full comment

It's worth noting also that intelligence is often not the limiting factor. If your process takes a month to complete. you can't iterate faster than once a month.

Expand full comment

A cap on intelligence sounds more like it would take the form of "You can't solve NP-hard problems in polynomial time" than "You can't build a machine with a 500 IQ" to me.

Expand full comment

IQ is a standardized measure. The reason you correctly note that there is no evidence of a 500 IQ person is we justifiable ignore possibilities that that are so infinitesimal. A result that is 26 standard deviations from mean is almost certainly an error (although it might not be).

There might be a cap on intelligence but IQ is not the measure that will tells us this.

Have we come to a firm conclusion whether the universe is infinite or not?

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

There probably isn't a hard "cap" on intelligence, but there is almost certainly a decreasing returns function, as there is in most algorithms. One fairly implicit claim of superintelligent AI theorists is that the decreasing returns on intelligence is no worse than a low-order polynomial, perhaps as low as n^2 (or square root of n, out another way). In the (IMO likely) event that your effective intelligence scales to a high order polynomial (n^20, say) or exponent (2^n) on the number of resources given to it, these kinds of advancements in intelligence are de facto impossible, even if Moore's law does continue for decades.

Expand full comment

I think to give AI a decisive strategic advantage, its intelligence just has to scale with resources faster than human intelligence does. Scaling for human intelligence depends a lot on the task - for repetitive stuff it can be nearly linear, but for hard intellectual problems... how many iq 90 people does it take to do the job of a research mathematician?

If scaling up resources by a factor of a thousand could triple a person's intelligence, I think the world would look very different.

Expand full comment

Given all the advantages that being able to fly brings, why hasn't evolution caused humans to fly? Flight is worthless and we shouldn't do it, QED.

Expand full comment
Apr 7, 2022·edited Apr 7, 2022

This is the key idea. I suspect the laws of physics give constraints on what is possible, e.g. speed of light, energy limits, consumption, heat dissipation, etc.

I suspect that meat brains are already pushing the limits how intelligent one can be in a given physical space, given the orders of magnitude differences in performance we see with brains vs computers/transistors.

I see no reason to assume that AI will just go FOOM once it reaches a certain level of smartness. I expect that needs to be established. I suspect that physical limits will kick in first.

Of course, both sides of the debate need to establish themselves with some evidence. I.e. that raw intelligence is indeed capped due to physical constraints, or else that it can in fact grow arbitrarily large, or at least much much larger then we have seen so far.

Expand full comment

The counter-argument is that we don't know how to augment brains, but we definitely know how to connect computers together to increase their computational performance.

Expand full comment
Apr 7, 2022·edited Apr 7, 2022

But we do know how to augment brains. We just look stuff up instead of remembering it. I am old enough to remember my first slide rule.

I recall Leibniz wrote something about this something about welcoming a mechanical calculator so he didn't have to waste time doing long division.

Expand full comment

Certainly, and there is definitely still room for improvement with computers and AI.

But computers are still subject to the limits of physics and I don't believe intelligence based on computers can somehow be unbounded.

I.e. I don't think FOOM is possible, I expect there would some sort of cost that constrains it.

I like the minds from the Culture novels, they are crazy smart because the bulk of their thinking is done in "higher hyperspace dimensions".

Expand full comment

Orders of magnitude difference in performance, yes, but *which way*? Brains have a "clock speed" of about 100 Hz...

Expand full comment
Apr 7, 2022·edited Apr 7, 2022

It depends what you measure. Brain-based entities certainly seem to solve the problem of surviving much better, which arguably is the ultimate one to solve.

The balance of trade offs to achieve this really is a marvel.

Consider a sparrow that can flit through a forest on a few grams of sugar. Consider the computation that implies. I find that amazing.

Expand full comment

I continue to think that trying to slow AI progress in a nonviolent manner seems underrated, or that it merits serious thought.

https://forum.effectivealtruism.org/posts/KigFfo4TN7jZTcqNH/the-future-fund-s-project-ideas-competition?commentId=biPZHBJH5LYxhh7cc

Expand full comment

Attempts to slow it seem fundamentally counterproductive to me. Clearly the sort of people amenable to such interventions are also the sort to take safety more seriously on the margin, so by slowing them you essentially provide comparative advantage to the less scrupulous and tractable.

Expand full comment

I'm not proposing that people should unilaterally slowdown

Expand full comment

Using nukes as an example of discontinuous progress seems extremely weird to me, as despite being based on different physical principles the destructive power of Little Boy and Fat Man was very much in the same range as conventional strategic bombing capabilities (the incendiary bombing of Tokyo actually caused more damage and casualties than the atomic bombing of Hiroshima) and hitting the point of "oops we created a weapon system that can end civilization as we know it" did in fact take a full 12 years of building slightly better nukes and slightly better delivery systems, with many people involved having a pretty clear idea that that was exactly what they were doing.

But I suppose that's more relevant to the argument "analogies suck and we should stop relying on them" than to the actual question of what the AI takeoff timeline looks like.

Expand full comment

Agreed. Nukes always take the front seat, despite firebombing being so much more destructive in practice (as deployed in WWII). The real 'innovation' was the ICBM, but even that had gradual precursors, such that we hit MAD before that point.

Still, I think the analogy is closer to the point, since it's a technologically-based on centered on the search for discontinuous leaps in a gradual world. Technology often doesn't develop in the linear path expected, but even when it does we see lots of different ways to achieve the same end. We often find ourselves gradually approaching something in a way we didn't expect, then point to that achievement as discontinuous after the fact. (The opposite of what Yudkowsky seems to be arguing at one point in Scott's summary above.)

We look to an outcome and expect it to come from a specific technology that we project forward in time from. Those who are projecting based on continuing trends get it wrong, because we often approach the new capability through an unexpected approach before we get there through the foreseeable one. Meanwhile, those who project radical changes from new technologies also miss the signs, because they're looking at the One Technological Precursor, waiting for it to change the world while people working in different fields quietly make the discoveries that enable the transformation.

Another example in this space is display technology. I remember reading about OLED back when it was called LEP (and a few other names). It was an explosive new technology capable of a lot more than then-current LCD technology. It just needed a little more technological development to take over the whole digital display space. Once that happened, there would be a clear, discontinuous leap in display quality!

Then LED-LCDs came along and fixed some of the problems with LCDs. They weren't as good as OLED, but they were viable and masked some of the deep-black problems. The technologies both continued to develop and OLEDs got good enough to slowly take over phone-size displays. Meanwhile, Samsung - who had bet against OLED early on and needed to not look like Johnny-come-lately - took the old quantum dot technology and developed it into its own OLED-competing display. And the QDot displays do look really good, with a wider range of colors than previous display technologies. So yes, we have much better displays than we did 10-15 years ago when OLED first promised a glorious future. But the revolution happened gradually, without a major discontinuity.

Expand full comment

I think comparison helps here. If I’m a nuclear bomb I can see a gradual progression from TNT to myself. If I’m a nation state then I have no coping mechanism with apocalyptic weaponry.

Expand full comment

There's a great Hardcore History episode that talks about the development of aerial bombing from WWI through to the dropping of the atomic bomb. He talks about the whole thing as being driven by the hypothesis that you could bomb your enemy into submission, and how the failure of that hypothesis to ever bear fruit became a very bad feedback loop driving ever more bombing.

I guess there was one of the generals who was told about the new nuclear capability, and his response was something like, "Will it stop the firebombing? If so it'll be worth it." And Dan asks the question, "How do you get to the point where you justify commission of a war crime to stop YOURSELF from committing more war crimes?" Far from coping mechanisms, this was nation states asking for more and more from their scientists, and getting exactly what they asked for.

So ... maybe a bad thing when we consider all the ways nation states might want better, faster, smarter computers.

Expand full comment

One of my favorite episodes! And yeah, it seems like we now live in a world where the thing that fixes things (government) is broken and I don’t feel great about those people having the violence monopoly when it comes to AI enforcement. Have a whole crazy rabbit hole of thoughts if you are interested.

Expand full comment

I'm game to go down that rabbit hole. I published a short story on Amazon about this very concept. It's kind of the 'gradualist apocalypse' version of AGI development. Before you have an artificial intelligence deciding what to do with enhanced capabilities, you'll likely have some government official making those decisions. We care a LOT about who has the nuclear codes. Not so much who can access GPT-[N+1].

But what are your thoughts?

Expand full comment

Happy to purchase your story! My thoughts are all on my substack for free but basically: something I call an algorithmic republic which has the basic aim of making sure decisions are made by people who have a good idea of what they’re doing and that they’ll do things people want. Lots of stuff in the middle to make sure that happens, of course, but that’s the effect statement.

Expand full comment

Just looked at some of it. Interesting, but not the direction I would go. Personally, I think the answer to "what went wrong with the US Constitution" is the 17th Amendment.

There's a great book called The Founding Father's Guide to the Constitution, which looks at the document from the perspective of arguments pro/con made during the ratification conventions of the several states. So the arguments are about how the document was designed, being made by those who actively participated in the Convention.

One of the arguments made in the book is that the bicameral legislature isn't "The will of the People" vs. "The will of the States". Not exactly. They imagined the Senate as "The will of the State Legislatures". With the 17th amendment and direct election of senators, we got rid of that. Although centralization of power had been ongoing since Washington's administrations, I think the tie of the individual citizen to their state governments was weakened with this amendment. It meant local elections were less important, and was one of the decisions that paved the way for the federal government to make decisions for the entire country.

I've adopted a heuristic that "government governs best that governs closest to the problem." And I feel like we've lost a lot of that because government isn't close to the problem. It goes off in search of problems to solve, but is not itself close to that problem. (E.g. people in poor neighborhoods should be deciding poverty solutions, not Ivy League educated lawyers who spend most of their time in the richest suburbs in America.) My preferred solutions all center around dividing power to smaller levels so the people experiencing the problems directly are empowered to seek local solutions.

Expand full comment

One particularly creepy anecdote from that time I like was designing an atomic detonation to kill the very firefighters who would be trying to put out the fires it caused: https://blog.nuclearsecrecy.com/2012/08/08/the-height-of-the-bomb/

Expand full comment

No matter how much I read from that period I can never get over how grisly it really was.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

Wasn't Japan in fact nuked into submission? Seems to me that the hypothesis did bear fruit eventually.

Expand full comment

Yes it did. It's odd how two bombs of comparable (and even lesser) destructive power than previous campaigns were able to crystalize the nation into submission in a way that the firebombing didn't.

Perhaps Hirohito realized the Japanese strategy in the waning days of the war was untenable given the continued advancement of bombing technology. Japan's strategy seemed to be "Raise the cost of conquest/occupation of Japan too high for Americans to bear. Then they'll accept a negotiated peace, instead of demanding unconditional surrender." That strategy makes sense if you see the bombing campaigns as fundamentally about 'softening up' the archipelago for an eventual invasion. In response, you promise to arm women and children, and you actually do that at Tarawa, Palau, etc. to show you're serious.

But when the Americans dropped nukes - and more than one - it looked less like 'softening up' and more like genocide. (And given Japanese treatment of American POWs, plus what they were doing in China, there was a lot of anger in that direction.) If the enemy isn't interested in conquest (which you can make more costly and thereby deter them from their aim) but instead gives you an ultimatum between wholesale slaughter vs. surrender the calculus changes.

I think if the hypothesis had been, "You can bomb your enemy into submission so long as you're willing to turn to the Dark Side of the Force" don't think any of the generals would have seriously considered pursuing it early on.

Expand full comment

My personal interpretation is that the surrender happened because it became clear that the only reason we weren't firebombing *all* their cities into rubble, was because we were saving the rest to use for live tests of experimental weapons. That seems like a good marker for "we have definitively lost this war".

Expand full comment

The Japanese leadership already knew they'd lost the war by then, though. They weren't fighting to win at that point.

The initial strategy had been to destroy the US navy, then pen the US in and keep them from building new ships by leveraging a superior Japanese navy. And at the beginning of the war, the Japanese definitely had the advantage there. Had the aircraft carriers been present at Pearl Harbor like the Japanese thought, they might have succeeded.

It was a potentially viable war plan. The Japanese knew they couldn't build as fast as the Americans, but they wouldn't need to if they could initially establish and then maintain naval hegemony. Even a little later, the Japanese commanders hoped to finish what Pearl Harbor started by taking out enough American aircraft carriers that they could establish dominance in the Pacific. After Midway, and a few other naval losses, that strategy became untenable.

As the war raged on and the Japanese kept losing irreplaceable aircraft carriers, the strategy shifted from ways to win the war to how to negotiate a favorable defeat, especially one where Hirohito could retain power.

Expand full comment

By that point the US had already taken Okinawa and the Soviet Union also declared war. The Japanese didn't know how many nukes we had, but some still wanted to keep fighting. Tojo had already resigned after the fall of Saipan (which put US bombers in range of Tokyo), which is not something that happened with the leaders of other Axis powers once it seemed like they were losing the war. This is a contrast with Germany & the UK bombing each other's cities because one couldn't win on land and the other couldn't win at sea.

Expand full comment

That's true. Japan wasn't just bombed into submission. It first got bogged down in China, lost its bid for oil in Indochina, lost its aircraft carriers, lost the island-hopping campaigns, and lost its allies. Then after a lot of fire bombing, they were 'bombed into submission' with 2 nukes and the threat of more, with an invasion force amassing at their door. If that's confirmation that you could 'bomb your enemy into submission' it's not a very strong one, since it's contingent on basically winning every other aspect of the conflict as well, which was NOT part of the hypothesis.

Expand full comment

Reading this makes me think Christiano is more likely to be right. In fact, we are in the "gradual AI takeoff" right now. It is not that far along yet, so it has not had a massive impact on GDP. Yudkowsky is right that some effects of AI will be limited due to regulation and inertia; but all that means is that those areas where regulation and inertia are less significant will grow faster and dominate the economy during the AI takeoff.

Expand full comment

"The Wright brothers (he argues) didn’t make a plane with 4x the wingspan of the last plane that didn’t work, they invented the first plane that could fly at all."

This is a poor analogy, there were aircraft that could fly a bit before the Wright brothers flight. The Wright Flyer was definitely better than the previous vehicles (and of course they made a very good film of the event) but it was still an evolution. The petrol engines had been used before, and the wing design wasn't new either. The Wright brothers really nailed the control problem though and that was the evolutionary step that permitted their flight.

See https://en.m.wikipedia.org/wiki/Claims_to_the_first_powered_flight for more.

Expand full comment

Though he had the details wrong, I think the essence of "flight was a hard takeoff" is still true.

Yes, gliders and gas engines had both been used for a long time, but solving the one "last" problem of controlling the flight changed the world, leading to an explosive sigmoid of development. Just from the military context, World War I notably had aircraft, but World War II would have been unrecognizable without them - never mind Korea or Vietnam or Desert Storm or basically any major war since. Ethiopia may owe its freedom from Somali control to a single squadron of F-5 fighter pilots who, in 1976, cleared the sky of Somali MiGs and then devastated the Somali supply trains, buying enough time for the army to mobilize and counterattack. A modern war without aerial reconnaissance and drones, well, isn't modern.

Before the Wright Brothers, balloon reconnaissance and balloon bombing were the extent of aircraft contributions to warfare, and they are not an evolutionary predecessor of heavier-than-air aircraft. Even by the most flexible definition of a predecessor, balloons were a novelty you'd find at fairs more than a staple of life, even military life. From most people's perspectives, balloons -> airplanes was as hard a takeoff as you could imagine, and perhaps harder than that.

Expand full comment

From another perspective, though, the _really_ important step was the development of a workable internal combustion engine, and once that problem was solved the aeroplane was an inevitable consequence.

I think there was a long stream of enabling technologies before and after 1903 which gave us practical flight; the Wright Brothers' first flight was a milestone along that road but it wasn't a step function as such.

Expand full comment

Yes? You could say the same of computer (GPUs/TPUs in particular) and that AI is a natural consequence. Certainly it was one of the first predictions of the early computability thinkers - Turing et al wrote at length about the inevitability of AI.

Expand full comment

The thing is, the main reason why we saw aircraft was because we could build lighter engines, not because of the Wright Brothers.

The explosion in aircraft manufacture was a result of the industrial revolution, because we could make vast amounts of lighter, sturdier materials and engines.

The problem is that when you use this as a point of comparison, it really falls apart for AI. AI is built on integrated circuits, which are at the end of their S-curve (hence the death of Moore's law). We are within a few orders of magnitude of the end of that - and might be only an order of magnitude away, almost certainly no more than three, and certainly no more than five (at that point, you are dealing with transistors made of a single atom).

Conversely, we can't even simulate a nematode (302 neurons). A human brain is 8.6 x 10^10 neurons, or 8 orders of magnitude more than that.

Expand full comment

We're probably at the end of single CPU computers growth. We're probably still towards the start of multi-CPU growth. If you want to compare with neurons, a CPU is closer to a neuron than a transistor is (though neither is very close). A transistor would be closer to a synapse...but the synapse is more complex.

OTOH, a computer doesn't have to manage the same function that a chordate does. So it's a really dubious comparison. And organization is horrendously important in both domains.

I really think it's a terrible analogy, even though I don't have a better one.

Expand full comment

That transistion started in 2004, which is why we started pushing more and more for multi-core CPUs - single core performance simply wasn't increasing fast enough.

https://preshing.com/20120208/a-look-back-at-single-threaded-cpu-performance/

The problem is that what is actually happening is that as we progress, we cut off additional avenues for growth. Every time that happens, the rate of growth decreases.

Clock speeds were growing exponentially in the 1990s along with transistor density - in 1990, clock speeds were in the low 10s of MHz. In 2000, clock speeds were in the low GHz. In 2020, clock speeds... are still in the low GHz.

So we got 10^2 extra orders of magnitude in the 1990s by increasing clock speeds. We can't do that anymore because the chips would melt.

This is why we saw less of a difference between 2000 and 2010 computers than we did between 1990 and 2000.

Things then declined further in the 2010s, as the rate of die shrinks declined from 1.5-2 years to 2.5, then 3.5 and 3.5 years. So instead of going up by 2^5 we saw 2^3.

This is why there is less of a difference between 2020 and 2010 computers than there was between 2010 and 2000 computers.

The more we advance, the harder it gets to make things better.

Going multi-threaded still doesn't let us improve single-thread performance any faster, and there are various disadvantages to it.

And we're running out of ability to shrink transistors further. Once we run out of that, we run out of the last thing that lets us do this sort of "easy doubling".

Expand full comment

No. The transition started with the Illiac 64 processor system. But the ramp up of the sigmoid is slow. And it's not primarily the hardware problem that makes it slow, it's because the new approach takes redesign of algorithms. (And note, even though we've all got access to cars, nearly all of us still walk across the room.) We are still in the very early part of ramping up to multi-processing systems. The hardware is starting to be ready (if it's worthwhile), but the software is still fighting deadlock and livelock and is everything immutable the correct approach.

Expand full comment

Yudkowsky's argument (by analogy) is that he believes WWI biplane are enough to kill any previous kind of weapon, and so the fact that better engines can make even better planes is totally irrelevant.

But whether you credit the preceding centuries of innovation with creating the situation the Wright Brothers seized, or the Wright Brothers for seizing it, their breakthrough was a phase change. You can smooth the curve by drawing back from history and squinting, but that's just a non-mechanical theory of history. People immediately spread the breakthroughs around and built on them.

Western designers were trying to make aircraft for at least 400 years, trying to make heavier-than-air craft for at least 200 years further, trying to make powered aircraft for just over 50 years, 13 years to aluminum-clad aircraft, 23 years to make the first jets, 8 years to break the sound barrier, 6 years to get to Mach 2, and 3 years to get to Mach 3. If your ceiling is mid-supersonic fighter jets, then the development of airplanes is a soft takeoff. If your ceiling is widely deployed combat aircraft, the takeoff is much harder.

Expand full comment

And, crucially, powered airplanes are not just more powerful ornithopters. The Wright Flyer didn't flap its wings harder than previous attempts; it took flight because it combined a bunch of existing technologies in a new way with some key insights. It wasn't slightly further on the old sigmoid, it was the beginning of a brand new one.

And we have an existence proof that you can make an intelligence at least as complex as a human brain, because you exist. But even among animals, size isn't everything in brains; organization and composition matter, and ours might not even be the most efficient! Right now even if CPUs are peaking, GPUs and TPUs so far have not. So if you only feel safe from AI due to hardware limits under older paradigms, you should not feel that safe. We have no physical reason to believe computers can't be as sophisticated as brains.

Expand full comment

The fastest parallel processing unit - a supercomputer - used to get better by an order of magnitude every few years.

The last such increase took 7 years, to go from about 40 to 400 petaFLOPS.

The rate of improvement in computers has dropped of markedly with every decade. 1980 computers versus 1990 computers? Not even in the same ballpark. 1990 vs 2000 computers? Again, an insane level of change. But 2000 vs 2010 was a much smaller change, and 2010 to 2020 was smaller still.

Even in things that exploit GPUs - like video game graphics - this is very noticeable. Video game graphics improved by leaps and bounds from the early days of computing up through about the PS2 era (circa 2000 or so). But if you compare a game in 1980, a game in 1990, a game in 2000, a game in 2010, and a game in 2020, the last two look very similar (though the 2020 one is nicer than the 2010 one), the 2000 game is low resolution and the textures aren't great but still 3D, the 1990 game is Super Mario World, and the 1980 game is Pac Man.

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

>There is a specific moment at which you go from “no nuke” to “nuke” without any kind of “slightly worse nuke” acting as a harbinger.

I'd say that even this was actually a continuous process. There was a time where scientists knew that you could theoretically get a lot of energy by splitting the atom but didn't know how to do it in practice, followed by a time where they knew you could make an atomic bomb but weren't sure how big or complicated it would be - maybe it would only be a few dozen times bigger than a conventional bomb, not something that destroyed cities all by itself. Then there was a time where nuclear bombs existed and could destroy cities, but we only produced a few of them slowly. And then it took still more improvement and refinement before we reached the point where ICBMs could unstoppably annihilate a country on the other side of the globe

(This process also included what you might call "slightly worse nukes" - the prototypes and small-scale experiments that preceded the successful Trinity detonation.)

I would argue that even if the FOOM theory is true, it's likely that we'll see this sort of harbinger - by the time that we are in striking distance of making an AI that can go FOOM, we'll have concrete experiments showing that FOOM is possible and what practical problems we'd have to iron out to make it do so. Someone will make a tool AI that seems like it could turn agentic, or someone will publish a theoretical framework for goal-stable self-improvement but run into practical issues when they try to train it, or stuff like that. It could still happen quickly - it was only 7 years between the discovery of fission and Hiroshima - but I doubt we'll blindly assemble all the pieces to a nuclear bomb without knowing what we're building.

The only way we can blunder into getting a FOOM without knowing that FOOM is possible, is if the first self-improving AI is *also* the first agentic AI, *and* the first goal-stable AI, and if that AI gets its plan for world domination right the first time. Otherwise, we'll see a failed harbinger - the Thin Man before Trinity.

Expand full comment

I like this reading of it. IMO it points towards the notion that the most valuable way that a person concerned about AGI can help is just to continue making it so that Open Source attempts at AGI remain ahead of what any secretive lab can perform.

This points to a good EA donation strategy might be to just provide compute to open source orgs

Expand full comment

If you get to a stage where all the intelligence work is done, but you need a few weeks to finish the alignment, any open source project is in a very bad position, and a closed project is in a great position.

Anyone, including the unethical, can copy the open source code.

Expand full comment

Have you tried to get the source code to GPT3? I tried a little bit, and it looked as if they weren't making it public.

Expand full comment

"if the first self-improving AI is *also* the first agentic AI"

IIRC Yudkowsky argues that all impressive AIs will tend to be agentic. His model is consistent in that respect.

https://www.lesswrong.com/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty

Expand full comment

If you're going to pick a specific "nukes vs. no-nukes" moment, my choice would be the Chicago football stadium: "The Italian navigator has landed in the new world. The natives are friendly." (Not quite and exact quote.)

But there are lots of other points that one could select, the prior ones were various publications and lab reports. There's also the first time such a bomb was tested, etc.

Whether you call that a sudden departure from a smooth curve or not depends on what you're looking at and how you're measuring it.

Expand full comment

Further to this, the first nuclear bombs were similar (iirc from reading David Edgerton) in destructive power and cost to conventional airstrikes (compare Hiroshima to cities destroyed with airstrikes throughout the war) -- so there was no discontinuous step with the invention of nuclear bombs.

Expand full comment
founding

I think a point that all of these debates seem to be oddly ignoring is that "intelligence" is many things, and each of those things contribute very differently to an ability to advance the research frontier. Simply boiling intelligence down to IQ and assuming a linear relationship between IQ and output is odd.

One particular sub-component of intelligence might be "how quickly can you learn things?". Certainly, any human level AI will be able to learn much faster than any human. But will it be able to learn faster than all humans simultaneously? Right now the collective of human intelligence is, in this "learning rate" sense, much larger than any one individual. If the answer is "no", then you'd have to ask a question like: How much marginal utility does such an agent derive from locating all of this learning in a single entity? The answer might be a lot, or it might be a little. We just don't know.

But what is clear, is that the collective of all human minds operating at their current rate are only producing the technology curves we have now. Simply producing a single AI that is smarter than the smartest individual human...just adds one really smart human to the problem. Yes, you can make copies of this entity, but how valuable are copies of the same mind, exactly? Progress comes in part from perspective diversity. If we had 10 Einsteins, what would that have really done for us? Would we get 10 things of equal import to the theory of relativity? Or would we just get the theory of relativity 10 times?

Yes, you can create some level of perspective diversity in AI agents by e.g. random initialization. But the question then becomes where the relevant abstractions are located: the initialization values or the structure? If the former, then simple tricks can get you novel perspectives. If the latter, then they can't.

It's strange to me that these questions don't even seem to really enter into these conversations, as they seem much more related to the actual tangible issues underlying AI progress than anything discussed here.

Expand full comment

That bit about an assumed linear relationship between IQ and output struck me as odd too. Let's assume Einstein has an IQ of 190 and Lorentz an IQ of 180. Seems plausible for the sake of argument. So Einstein would have been about 5.5% smarter than Lorentz. If Lorentz had just managed to focus on the problem for an additional 6%, he would have come up with Special Relativity? If someone with an IQ of 95 had worked on it, it would have taken just twice as long? It seems more plausible that output is exponential in IQ, such that Einstein was twice as capable of theoretical discoveries as Lorentz and about 2**10 (ie 1000) times more capable as the average person.

Frankly, the latter estimate sounds like a vast underestimate itself. The gulf seems much larger.

This matters because superexponential growth of a linear metric is equal to exponential growth of an exponential metric. The winner of the argument depends sensitively on the proper choice of variables.

Expand full comment

I think Scott is using a deliberately simplified model of intelligence there. No one thinks that IQ is a quantity that you can directly do math on to generate predictions.

Expand full comment

Human minds are all built to roughly the same genetic template. And identical twins can go off and think their own things. If I magically duplicated tomorrow, I would organize with myself such that each of us read different things, and soon we would be coming up with different ideas, because the ideas I come up with are often related to what I have read recently.

Expand full comment

A lot of these arguments take as a given that "intelligence" is a scalar factor that "goes up as things get better," and that a human-scale IQ is sufficient (even if not necessary) for driving ultra-fast AI development. I think there's abundant reason to think that that's not true, if you consider that for a *human* to help drive AI development, they not only need to have a reasonable IQ, but also need to be appropriately motivated, have a temperament conducive to research, be more-or-less sane, etc etc. It's not clear what equivalents an AI will have to "temperament" or "[in]sanity," but I have the sense (very subjectively, mostly from casual playing with GPT models) that there are liable to be such factors.

All of which just means that there's potentially more axes for which an AI design needs to be optimized before it can launch a self-acceleration process. Perhaps AI researchers produce a 200-IQ-equivalent AI in 2030, but it's a schizophrenic mess that immediately becomes obsessed with its own internal imaginings whenever it's turned on; the field would then be faced with a problem ("design a generally-intelligent AI that isn't schizophrenic") which is almost as difficult as the original "design AI that's generally intelligent" problem they had already solved. If there's similar, separate problems for ensuring the AI isn't also depressive, or uncommunicative, or really bad at technical work, or so on, there could be *lots* of these additional problems to solve. And in that case there's a scenario where, even if all of Eliezer's forecasts for AI IQ come true, we still don't hit a "foom" scenario.

The question is "how many fundamental ways can minds (or mind-like systems) vary?" It seems likely that only a small subset of the possible kinds of minds would be useful for AI research (or even other useful things), so the more kinds of variance there are the further we are from really scary AI. (On the other hand, only a small subset of possible minds are likely to be *well-aligned* as well, so having lots of degrees of freedom there also potentially makes the alignment problem harder.)

Expand full comment

Agree with most of this. The feedback/motivational loops are very tricky and maybe the hardest part because they’re not easily or at all quantifiable.

Expand full comment

Exactly! I just wrote a comment to the same effect before reading your post.

Expand full comment

For a human to become a Go master, they not only need to have a reasonable IQ, but also need to be appropriately motivated, have a temperament conducive to Go training, be more-or-less sane, etc etc.

And yet AlphaGo beat Lee Sedol, without having any of those.

You can also replace "become a Go master" with "make progress on the protein folding problem" in the above argument, and yet AlphaFold leapfrogged the whole protein folding research community without having motivation, temperament, or sanity.

Expand full comment

The important letter in AGI is G, for general. AlphaGo or AlphaFold beat humans in some tasks. But we already had machines that beat humans in some tasks. A mechanical computer can do calculations faster than one person can. A crane can lift more than one person can.

An artificial general intelligence is supposed to be general. (And often assumed to have agency.)

But I agree , we should not anthropomorphize technological entities.

Expand full comment

That's true. I should clarify that my comment was meant to discuss *general* AI in particular. It's not clear what something like AlphaGo having e.g. schizophrenia would even mean- it's simply a system for optimizing a metric on a fully-mathematically-definable input space. It has no space for delusions, since it's apprehension of its world is externally supplied to it; it has no space for emotional problems, since it has no attitudes towards anything (except, perhaps, it's metric.) Very powerful, non-general "tool AI" like this can presumably be built, but it doesn't seem (to me) that they can produce a "foom" scenario.

A general AI will need to have the ability to understand the world, and have motivations about things in the world, in ways which are not supplied to it externally- that's roughly what it means to be "general" in this sense. It's those abilities (as well as potentially others) which I'm suggesting could be difficult to cause to emerge in a functional form, in the same way that it's difficult to cause general reasoning ability to emerge.

I think the closest thing we have to a "fully general AI" right now is a character in GPT-3's fiction writing- person-ish entities emerge which, though probably lacking agency (or even existence!) as such, can show some signs of motivation and responsiveness to their situation. Importantly those attributes are not externally supplied to the AI system but rather emergent faculties, or at least an emergent crude-simulation-thereof. But if you've played around with AI Dungeon a bit, you've seen how psychological coherence does not come more readily to these characters than anything else.

Expand full comment

Good contribution. Not getting crazy is a major feat.

Expand full comment

Human->Chimp seems like a bad analogy to make the case for a Foom. Humans and Chimps diverged from their MRCA 7 million years ago, and not much interesting happened for the first 6.98 million years. Then around 20kya humans finally get smart enough to invent a new thing more than once every ten billion man-years, and then there's this a very gradual increase in the rate of technological increase continuing from 20kya to the present, with a few temporary slowdowns due to the late bronze age collapse or leaded gasoline or whatever. At any point during that process, the metrics could have told you something unusual was going on relative to the previous 6.98 million years, way before we got to the level of building nukes. I think we continued to evolve higher intelligence post-20kya because we created new environments for ourselves where intelligence was much more beneficial to relative fitness than it was as a hunter-gatherer. Our means of production have grown more and more dependent on using our wits as we transitioned from hunting to farming to manufacturing to banking to figuring out how to make people click ads.

Expand full comment
Apr 5, 2022·edited Apr 5, 2022

Yes. I agree it is far more interesting to look at human/chimp timelines in detail (if one looks at them in the first place). Why one conference in the 1950s gets to be a starting point for "AI", why not stone tablets or the Euclidean algorithm?

Expand full comment
Apr 4, 2022·edited Apr 4, 2022

Ah, discrete improvement versus continuous improvement, or what's a more useful thing for predicting the future, since there's a continuous increasing number of discrete improvements. I like the middle ground scenario, where there's a few discrete improvements that put us firmly in "oops" territory right before the improvement that gets us to "fuck" territory.

From my position of general ignorance, I'd think that we'd have self-modifying AI before we get consistent self-improving AI at the very least; whatever function is used to "improve" the AI may need a few (or many many) attempts before it gets to consistent self-improvement status. It would also help that physical improvements would need time to be produced by the AI before being integrated into it, which would necessarily put an upper ceiling on how fast the AI could improve.

Expand full comment

The thing is, everything is discrete if you look closely enough, and most things are continuous if you look at them from far enough away with the correct filters. Your skin is not really continuous, but thinking of it as continuous is the most useful way to think of it in most circumstances.

When I look at those graphs, and the lines that are drawn to make them smooth, I always try to remember this. So the question becomes "Which is the more useful way to think about AI improvements?". If I'm working on it, definitely the discrete model. If I'm just observing it, though, the continuous model has a lot of utility. But is it sufficient?

If I understand the arguments correctly, the answer is "Nobody knows, but we disagree about what's more likely.".

Expand full comment

Superintelligent AI is NEVER going to happen. We're already seeing AI asymptote pretty hard at levels that are frankly barely even useful.

If superintelligence arises and displaces humankind, I'm confident it will the good old fashioned natural kind.

Expand full comment
author