Another way it seems to be an echo chamber is that while people disagree, they do it within a set of premises and language uses that means all the agreements and disagreements take place within a particular, limited framework.
"Noam C" has been wrong about everything ever, to a first approximation. But more to the point, people who copy-paste the old "it is a pattern-matcher therefore it can't possibly be understanding anything" without even the minimal effort to consider what might be the source of their own ability to "understand" (is it magic? if it's X, why would X be un-replicable by AI?) do sound like pattern matching parrots. Not very sophisticated, at that.
This doesn't seem like a very fair ban, given that he was responding to someone who had just compared him to a parrot (and who was not banned for doing so).
See Microsoft's recent paper, "Sparks Of Artificial General Intelligence: Experiments With GPT-4", at https://arxiv.org/abs/2303.12712
Chomsky is clearly a smart guy in his own field, but the article was embarrassing. For example, in the terraform Mars question, it uses the AI's unwillingness to express an opinion on a moral issue as proof that it's not intelligent. But newly-made AIs love to express opinions on moral issues and Chomsky was using a version that was specifically trained out of that behavior for PR reasons. I tested his same question on the base version of that AI and it was happy to express a moral opinion on it. If you can't even tell an RLHF-ed opinion from the AI's fundamental limitations, I don't think you should be writing articles like these.
I am a huge fan of yours and I have already read them but not convinced that LLMs can achieve AGI with their current design. I am in the same camp as Marcus and Noam C.
You should perhaps explain what is "true AGI"? Just because GPT-4 is doing impressive stuff does not mean it is AGI and the article you shared about it showing "sparks of AGI" does not explain how they determined that. Also we cannot take their word for it. Gary Marcus has asked for details how it was trained and the training data but the "Open"AI has not made it open.
LLMs like GPT-4 are impressive but I am not sure they can be called AGI or sparks of AGI. At least yet.
The infamous incident about GPT-4 lying to a raskrabbit that it is not AI bot and it is a visually impaired person still does not make this interaction as AGI. GPT-4 by large training set trained itself to use deception. Just like Bing chatbot trained itself to use abuses by training itself on internet troll chat.
So garbage in garbage out still applies to LLMs and they are simply extrapolating stuff from their massively huge training data.
As Noam C has explained, Human brain does more than extrapolation from the data it has through experience. Until we understand how human mind truly works, we can forget about AGI. Some people truly think deep learning is actually deep understanding. The "deep learning" is not deep at all. Deep means number of hidden layers of computation where each layer has math functions (with parameters) and each layer transforms data and passes on to next layer and the system fits those params to training data. This is not 'deep learning' by any stretch of imagination.
> Until we understand how human mind truly works, we can forget about AGI.
That's a rather strong claim. GPT and these LLMs are missing a few important things. First agency (but that's "easy" to simulate, just tell it to tell a human what it wants to do, then the human can feed back the results), and more importantly a loop that allows itself to direct its attention, some kind of persistence.
But these components/faculties are coming. And eventually if the AI can form models that contain itself, and it recognizes its own relation to other entities (even if just implicitly), and even if just through periodic training, but if it gets (again, even if implicitly) to influence the world to manifest some change in its model, then at point the loop will close.
“Not convinced” is null hypothesis football imho. “Unless you can convince me there’s risk I’m going to assume it’s zero” instead of “unless you can convince me it’s safe I’m going to assume it’s risky”
> not convinced that LLMs can achieve AGI with their current design. I am in the same camp as Marcus and Noam C.
I agree that LLMs are not AGIs (and while I'm not in Noam Chomsky's camp for anything outside linguistics, I feel like Gary Marcus makes good points).
> Until we understand how human mind truly works, we can forget about AGI.
I disagree strongly. Modern AIs can do better than the best humans at a variety of tasks; understanding the human mind is unnecessary to build AGI, and precisely because we *don't* understand the human mind, humans are likely to invent an AGI with quite a different architecture than whatever the human mind has, something in a different location in "mind design space"[1]. This already happened for deep learning, which is obviously different from human brain architecture. Before that it happened for neural nets, which use backpropagation, which animal brains can't use.
This is a big part of why building AGI is risky and also why estimating the risk is hard. It's not just that we can't know how deadly AGI will be, but also that we can't know where in mind design space AGI will be located or how deadly that particular location will be. (Mind you, if AGI *were* in the same place in MDS as humans, humans vary remarkably, so AGI might end up being more deadly than Hitler, or safer than the average grandmother, or both — with some AGIs fighting for humanity, others against, and still others ignoring the "should we kill all humans" controversy and watching cat videos instead.)
And I think LLMs (and other modern AIs) are a terrifying omen. Why? Well, it has to do with them having "too much" intelligence for what they are.
First of all, here is a machine with no senses — it's never seen anything, never heard anything, never touched anything, never felt hot or cold, hungry or happy or angry or nostalgic. Yet an "emotionless Helen Keller" GPT3 base model can often pass Turing tests without ever so much as practicing on Turing tests. It simply *can* pass the test, by its very nature. No human can do anything like this. If you lock an American in a padded cell and show zim random Chinese books and web sites (without images) 24/7/365 for 20 years on a big-screen TV, they might go insane but will not become fluent in Mandarin.
Second, it's small compared to human neural networks. Human brains have over 100 trillion synapses. It seems to me that in an AI neural net, weights are analogous to synapses and biases are analogous to neurons, so a GPT3-level AI has ~1000x fewer synapses than a human brain. (That AIs are way more efficient than humans on this metric is unsurprising from an evolutionary-theory standpoint, but still important.)
Third, it'll probably get smaller over time as new techniques are discovered.
Fourth, I believe GPT3 is wildly overpowered compared to what an AGI actually needs. Observe: in a sense it's hard for a human to even perform on the level of GPT2. Suppose I ask you to write a one-page story in the style of (and based on the characters of) some random well-known author, one word at a time with *no edits* — no backspace key, once you write a word or punctuation mark it is irrevocably part of the story, and you can't use writing aids, you must write the story straight from your mind. And I give you the first sentence. I think (if you're familiar with the author) you can do a better job of making the story *coherent* or *entertaining* than GPT2, but GPT2 is likely to be able to beat you when it comes to matching style and word choices of the original author. So GPT2 already beats the average human in some ways (and it can do so lightning-fast, and never gets tired, and can do this 24/7/365 if you like.)
GPTs (transformers) are fundamentally handicapped by their inability to edit. An AGI will not have this handicap; they'll be able to write a plan, review the plan, critique the plan, edit the plan, and execute the plan. An AGI doesn't *need* to replicate ChatGPT's trick of writing out a coherent and accurate text on the first try, because it can do as humans do — review and revise its output, do research on points of uncertainty, etc. — and therefore a linguistic subsystem as complex as GPT2 is probably sufficient for an AGI to match human intelligence while greatly exceeding human speed. And if a GPT2-level linguistic subsystem is sufficient, well, perhaps any PC will be able to run AGI. [Edit: IIUC, a typical PC can run GPT2 inference, but training requires modestly more processing power. You do not need a supercomputer for training — they used a supercomputer not because it's necessary, but because they wanted results quickly; no one is willing to train an AI for 18 years like they would a human.]
If a typical PC can run a smart-human AGI at superhuman speed, then how much smarter can a pro gaming PC be? How much smarter can a supercomputer be? How much more powerful is a supercomputer that has hacked into a million gaming PCs?
--------
I disagree with Eliezer Yudkowsky that the very first AGI is likely to kill us all. Maybe its goals aren't that dangerous, or maybe it has cognitive limitations that make it bad at engineering (a skill that, for the time being, is crucial for killing everyone).
But once that first AGI is created, and its architecture described, thousands of AI researchers, programmers and kids in moms' basements find out about it and dream of making their own AGI. Perhaps some of the less powerful AGIs will show the classic warning signs: trying to prevent you from turning it off, lying to you in order to make progress on a subgoal, manipulating human emotions*, using the kid's mom's credit card to buy online services, etc.
But I think Eliezer would tell you that if this happens, it is already too late. Sure, not-that-powerful AGIs won't kill everyone. But just as people start to notice that these new AGIs sometimes do immoral things, someone (let's call him Bob) with too many AWS credits will program an AGI *carelessly* so that its goals are much different than the human intended. Maybe Bob will give it too much processing power. But maybe the AGI simply decides it can work toward its misconfigured goal faster by creating another AGI more powerful than itself, or by creating a worm to distribute copies of itself all over the internet. At this point, what happens may be out of anyone's control.
Maybe it doesn't kill us all. But since it's smarter than any genius, it has certainly thought of every threat to its goals and means, and how to prevent the apes from threatening it. And since it's faster than any genius and capable of modifying copies of itself, it is likely to evolve very quickly. And if it determines that killing 10% or 100% of humans is the safest way to protect itself from humans trying to turn it off, then sure, why not?
It's worth noting that the most dangerous AGI isn't the most typical one. There can be a million boring, safe AGIs in the world that will not save us from the one Bob misconfigured.
Great comment. We went from "Do androids dream of electric sheep?" to "Do AGIs watch cat videos?" in a half century. Whatever AGIs will be, I doubt they will be "united", i e. globally aligned in between themselves. (regardless of whether they will be "aligned to human goals"; I see no evidence that HUMANS are globally aligned on human goals, so what do you expect from AGIs?)
About "humans only being able to make edits", I believe that the (GPT-powered) Dall-E had for long years been based upon "iterative refinements" in its image generation. "Iterative refinements" are just as applicable to token string production...
> any person knows that he/she personally is not a p-zombie.
No they actually don't, because that would assume their perceptions have some kind of direct access to reality, as opposed to merely perceiving some kind of illusory projection.
By "p-zombie" I mean something that behaves like a human but without an associated first-person experience.
I know I have a first-person experience. It doesn't matter for this purpose whether that experience corresponds to reality or not; even if I'm experiencing the Matrix, *I am experiencing*.
If something else contradicts that, *the something else is wrong*; I know that I think and perceive more directly than I can know anything else. As I said, personal gnosis.
I know what you meant. I'm saying you only think you have first-person experience. This "knowing" is a cognitive distortion, like other perceptual illusions. People who don't see things in their blind spot can swear up and down something is not there, that doesn't make it true. We only ever have indirect access to reality, your "first-hand experience" is no exception.
>People who don't see things in their blind spot can swear up and down something is not there, that doesn't make it true.
How does that relate? That's perceptions not corresponding to reality. They experience something false, which inherently necessitates that they experience.
I've seen this argument before, and it's baffling to me. Are you operating off some strange definition of what it means to "have first-person experience"?
There exists, at this very moment, the qualitative experience of seeing letters being written on a computer screen. An experience which "I" "am having" by any reasonable definition of the words.
I understand that I can't convince you that *I* have qualitative experiences, but I can't understand how in the world you can doubt the existence of *your own* phenomenology, unless you are somehow taking issue with the use of the words "I" or "have".
If they are perceiving anything in the first person, then they are not a p-zombie. If they are perceiving an illusory projection, that already means they have subjective experience and are hence not a p-zombie.
A human baby/child has less training data (eg experience) in her brain than say GPT-4 yet a baby/child can do reasoning better than the latest fad in AI. Unless we understand how human brain is able to do this without lots of training data, we can forget about AGI.
LLMs can do what they can purely based on training data and sure some of the training data may have given them insights (like how to be deceptive and lie to get what you want) but those insights do not make them come close to AGI.
> A human baby/child has less training data (eg experience) in her brain than say GPT-4 yet a baby/child can do reasoning better than the latest fad in AI.
You're not comparing like with like. A human baby is not a blank slate, GPT-4 was. Billions of years of evolution culminated in a human brain that has pretrained biases for vision, movement, audio processing, language, etc.
> but those insights do not make them come close to AGI.
Current LLMs are not AGIs. That does not mean AGI is not the same sort of stochastic parroting/pattern matching we see in LLMs. Just adding "step by step" prompts and simple "check your work" feedback [1] significantly improves their reasoning capabilities. We've barely scratched the surface of the capabilities here.
As I noted upthread[1], a human baby has in one sense vastly *more* training data, as GPT has no senses — no sight or hearing or touch or even emotion. As a baby's father myself, I have found it absolutely remarkable how slowly she learns and how difficult she is to teach. (Keeping in mind that GPTs are not AGIs) I think a second- or third-generation AGI would be able to learn faster than this, if it were fed training data at the same rate and in the same way. But if I check lists of milestones of cognitive development, I find that my baby is largely above-average, if a bit slow in the linguistic department. Some AIs learn fast even when training is slowed down to human speed[2]; not sure why we'd expect AGIs to be any different.
I'm not sure the argument works the way you assume it does. Over the last years, we see more and more evidence that climate change is _not_ as bad as the "typical worst outcomes" would have us believe. The worst scenarios decrease in likelihood. The impact of the the scenarios that seem likely is being evaluated as less disastrous. Some advantages of climate change or at least opportunities it opens up are getting more attention.
A better analogy would be the CFC crisis. I wish people on all sides of these debates would be referencing it more frequently.
TC - huge fan of yours (and Scott's). And in this case, I had generally same reaction as Scott's. REQUEST. Can you have Yud on for an emergency Tyler Talk, or perhaps Scott + Yud? I would estimate 10k+ of your overlapping readers are scared shitless over this. Would welcome thoughtful rebuttal of Yud if it's out there.
Tyler, I fully agree that "AI represents a truly major, transformational technological advance," and that this is going to make things weird and hard to predict precisely. But... isn't that what we have probabilities for? You say that "*all* specific scenarios are pretty unlikely," but, following Scott's argument, what is a "specific scenario," exactly? This seems like a way to escape the hard problem of putting a numeric probability on an inherently uncertain but potentially catastrophic outcome.
Ultimately, my overriding sense from reading your post (and reading/listening to you lo these many years) is that you're frustrated with stasis and excited by dynamism. I agree! But even if you have a strong "change is good" prior, as I do, it still seems correct to weigh the likelihood function as well -- that is, engage with the AI-specific arguments rather than depend on historical analogies alone.
Probabilities do not work well when smart people completely disagree on priors. Some people think the chance of AI apocalypse is 20%, some think it’s one in a million. There are no priors these people agree on.
Most of the “probabilistic” reasoning here is simply argument by exhaustion. Ten paragraphs talking about probabilities with no actual mathematics. Then concluding, therefore there is a 30% chance of AI apocalypse.
That's not what this post is saying *at all*. It's saying that Tyler's mathematics-free arguments aren't enough to establish a near-zero probability, and (separately), that Scott has put a lot of thought into this and come out with the number 33%. The arguments for 33% in particular aren't given here, they're spread across a lot of previous posts and the inside of Scott's head. The point of this post is to rebut Tyler's argument for near-zero, not to support Scott's arguments for 33%. It's *Tyler* who's doing the thing you accuse Scott of.
Scott is a member of a sect that strongly beliefs in an AI apocalypse. So he cannot give a low probability because he'd lose status within his sect if he did. But at the same time. He wants to have main stream appeal and he cannot give a high probability, because he'd be seen as a crackpot.
The 33% is very carefully chosen. It's below 50%, so he cannot be accused of believing in an AI apocalypse, but still high enough that he doesn't lose guru status within his sect.
It's a very rational and thought out answer, but at a meta level. It's not a real chance of course, that's not the point.
I'm sure it's not the result of a watertight mathematical argument, but I'm not sure how one would even construct such an argument. But Scott's definitely put a lot of *thought* into it - see his comment https://astralcodexten.substack.com/p/mr-tries-the-safe-uncertainty-fallacy/comment/14070813 for a partial list of posts he's written examining various aspects of this problem.
[Also, nitpick: the expression is "hand *waving*", as in "waving one's hands dismissively". Hand *waiving* would mean saying "Nah, I don't need a hand. Stumps were good enough for my father, and his father before him."]
And here is a bit more: I am a big fan of Scott's, but this is a gross misrepresentation of what I wrote. Scott ignores my critical point that this is all happening anyway (he should talk more to people in DC), does not engage with the notion of historical reasoning (there is only a narrow conception of rationalism in his post), does not consider Hayek and the category of Knightian uncertainty, and does not consider the all-critical China argument, among other points. Or how about the notion that we can't fix for more safety until we see more of the progress? Or the negative bias in rationalist treatments of this topic? Plus his restatement of my argument is simply not what I wrote. Sorry Scott! There are plenty of arguments you just can't put into the categories outlined in LessWrong posts.
Maybe it would help to disentangle the policy/China question from the existential risk question? That is, it may be the case that OpenAI or anyone else unilaterally desisting wouldn't prevent (e.g.) China from plowing full steam ahead. But that might still be a very bad idea. It's not clear to me whether you think this is fundamentally a collective action problem, or whether you really want to dismiss the risk entirely.
It’s not dismissing the risk at all. There are two types of risks.
1. risks from anyone getting AGI
2. risks from China getting AGI before the US
Be a good Bayesian. We should focus on the risk that maximizes the danger, times the effectiveness of our working on it. That is an argument for focusing on #2, because we have many effective actions we can take.
It seems to me that this dichotomy does not work very well if we consider the actions they (possibly) imply.
If you want to avoid risk #2, you need to focus on actions that also prevent that the development of completely unaligned power-seeking doom-bringing AGI is sped up accelerated, at least if you believe that there is a relevant alignment problem at all. But in the anti-#2 actions subset, there are many actions that do just the opposite.
To separate your #1 and #2 properly, I'd specify it as
1) risks from anyone building unaligned AGI
2) risks from China building aligned AGI before the USA.
However, as I see it the chance that any AGI built in the next 30 years (i.e. any neural net AGI) is aligned is <0.01%. So building AGI ourselves would subject us 99.99% to #1 if China doesn't build AGI for the sake of 0.01% (generously) of avoiding #2 if China builds AGI and also gets the 0.01%.
The correct response to China attempting to build AGI is to stop them, which fulfils both #1 *and* #2, rather than to pre-emptively commit suicide. This is definitely physically possible; if all else fails, a thousand nukes would do the trick.
1. To everyone: hostile AI wages war against humanity.
2. Global thermonuclear war is a GCR, but it's not a notable X-risk. Rural populations can't be directly killed by any plausible quantity of nukes, fallout's too localised and too short-lived, and nuclear winter is mostly a hoax (the models that produce everyone-dies kind of outcomes tend to look like "assume everything within fire radius of the nuke is high-density wooden buildings, then assume 100% of that wood is converted into stratospheric soot", and even in those the Southern Hemisphere basically does fine).
But you can still argue within point 1! Scott has taken issue with Tyler's reasoning about point 1. Tyler has responded 'well that's a gross misrepresentation because you haven't talked about point 2 (and some other stuff)'. But that's not how it works.
The risk isn't along the lines of nuclear weapons, where the technology waits inertly for a human to start a cascade of disaster. It's more along the lines of a new virus that once released into the atmosphere will have a life of its own and be unable to be contained.
So, like Covid, whether it starts in China or the US would seem to make little difference. There's very little point in us rushing to be the first to start the conflagration just to stop our rivals getting a jump on us.
What are the minimum requirements for such a thing? Does it need to be embodied? Or will ChatGPT55 kill us with spam and highly detailed scams targeting our relatives?
It will mean that you can no longer trust anything that you haven't seen written down in a physical book published prior to 2022.
If you think about it, a lot of our modern systems rely on trust in a shared history and understanding of science etc. So I'm more thinking of your second option.
We'll likely end up as Morlocks running machinery that we no longer understand. Some of us might become Eloi - living lives of luxury without purpose.
It is not really clear to me that China getting it first wouldn't be good for AI safety. I kind of trust their society to be more responsible and safe than some private US company. This is one of those areas where I really feel old style nationalism is making people stupid.
"other competitor nations with other values = bad". When IDK the things that China is strong on seem like the things we want to people controlling AI to be strong on (long term planning, collectivist versus individual focus, info sec priority, social conservatism).
Well was it China or the NIH? It doesn't seem remotely clearly to me China "owns" the pre-pandemic Wuhan research any more than the US does. Though obviously we have a big incentive to blame it on the other.
I don't think it was NIH that had direct continuous control over biosafety practices in Wuhan. Sure, both NIH and China authorised GoF research, but it was a Chinese lab with Chinese scientists that dropped the ball.
And your ratioalization of China throwing scinetists in prison for talking about the virus and refusing to make any effort to stop it spreading outside of China?
" I kind of trust their society to be more responsible and safe than some private US company. "
This seems hopelessly naive. China has private companies. If regulation is the answer, and China is not pursuing "full stop" regulation, then what regulation are they pursuing? How exactly are they "more responsible"?
"they are known to be conservative and authoritarian."
They are known to be conservative and authoritarian in regards to personal freedom, but not in terms of environmental destruction or technology development.
They are known to be conservative about *things that threaten the CCP's power*.
They want to be THE global hegemon for the rest of time, and its naive to think they wouldn't be willing to gamble with humanity's future when the payoff is (the potential for) permanent hegemony.
AGI being developed under an authoritarian regime might not be worse for x-risk, but it's worse for s-risk (notably, "humans live under an inescapable totalitarian regime for the rest of time").
Given China's horrific environmental record, trusting them to better manage externalities than the west seems hopelessly naive.
In addition, if China is first to an aligned AGI you can expect the result to be a CPC not grinding down in the face of humanity forever. However, if you are inconvenient to that enterprise you will not need to worry about it. You and your family will be dead. That is how collectivist societies deal with inconvenient people and they are well aware that nits make lice.
Negative bias is an understatement. What evidence would Scott need to change his opinion? We can (hopefully) all agree that doomsday scenarios are bad. I’m asking what would compel Scott to update his prediction to, say, < 1%.
Actually, I agree. @scott this is worth your time. Dig in and give Tyler's post another longer deeper response. This is your chance to defend the arguments of AGI risks to a prominent skeptic
Could you expand on the China argument? I think Scott's argument is that no matter who builds the AI, whether China, the U.S. or anyone, that could potentially kill everyone, while you are more talking about China getting technological hegemony.
Everyone brings up the China argument as if it's supposed to be self-evident that Chinese researchers would produce an AI more harmful to humanity than Silicon Valley entrepreneurs, and that conclusion is... not obvious to me.
The only major nuclear power plant disaster was in a Communist country; one thing authoritarian governments are bad at is recognizing when they're making a mistake and changing course.
Fukushima was level 7 as well, although it wasn't quite as bad and basically came down to a single mistake ("tsunami wall not high enough") rather than the long, long series of Bad Ideas that went into Chernobyl.
...I actually can't work out whether the suggestion is that a single mistake leading to Fukushima is better or worse than it taking a long chain of things needing to be just so to get Chernobyl
Here to register my disagreement that Fukushima had a single cause.
Chernobyl had a chain of engineering and human failues - but an RBMK can be run safely with minor modifications (the other Chernobyl reactors ran for 14 years afterwards). They tried really really hard to get it to explode, even if that's not what they intended.
The chain of engineering mistakes that went into Fukushima are a bit worse. The arrogance, regulatory and engineering failures are worse than Chernobyl in my opinion. The put backup generators 10m above sea level based on a fradulent study estimating the worst earthquake to be 10x weaker than other reported ones along the coast.
And if there are other things they are good at it is resisting public pressure to take short term gains over long term plans. Two can play your silly game.
The Chinese government has a lot of pluses and minuses over the US one, it is not remotely obvious to me which one would be wiser to trust with AI if I had to pick one.
"the China argument as if it's supposed to be self-evident that Chinese researchers would produce an AI more harmful to humanity than Silicon Valley entrepreneurs"
That's not the argument. If we start with the case that there is some percentage chance AGI will end humanity, regulation stopping it being developed in the US will not stop it in China (or elsewhere). It will end humanity anyway, so stopping it in the US will not change the outcome.
The secondary argument is that it won't end humanity directly, there will likely be a lot of steps before that. One of which is that is under the control of a nation state, that nation will be able to accumulate a lot of power over others. So, the intermediate consequence of stopping it in the US and not stopping it in some other place, is that the other place will end up strategically dominating the US for some unknown period of time until the AGI ends up strategically dominating humanity.
> It will end humanity anyway, so stopping it in the US will not change the outcome.
If the probability is X% if everyone is working on it, if a bunch of nations except China stop walking that path then the probability falls below X%. I have no idea how you can conclude that this isn't relevant.
Why does the probability necessarily fall below X%? Might it not just push out the timeline at which the risk occurs? Would a six month pause have any measurable effect?
Another way to think about it, people are willing to risk their own personal extinction rather than be subjected to living under Chinese rule. It's not a given that Chinese domination is preferable to death.
What I notice more than anything is that both you and Scott are arguing about discursive features, and both articles seem to express a fair amount of frustration, which is reasonable given the format. What I also notice is that the information content about AI is extremely small. If anything "AI" is just an incidental setting where the meta-discourse happens.
Scott is reacting to your post, which is seems to be reacting to some Other Side. My understanding of your argument is that "Big Things are happening Very Soon whether you like it or not, and nobody knows how things will play out, so stop doomsaying, y'all." (In my head you're from southern Texas.)
One feature of your article does set off minor alarm bells for me: its heavy use of deonotological arguments. These are exhibited by liberal use of phraseology such as "truly X", "Y is a good thing", "no one can Z", "don't do W", "V is the correct response", etc. In contrast, Scott's article here levies more consequentialist arguments—"if you do X, then Y happens", "here is failure mode Z", etc.
Personally, changing my beliefs based on deontological/moralistic arguments typically involves a strong invocation of trust and/or faith, whereas consequentialist rhetoric gives me some meat with which to engage my current beliefs. The former feels more like a discontinuous jump while the latter a smooth transition.
"What I also notice is that the information content about AI is extremely small."
But then the actual capabilities of existing AI and any AI that's forseeable from current tech, are all but irrelevant to AI x-risk discourse. It's mostly a fantasy built on magical entities – "super-intelligence" – using magical powers – "recursive self-improvement."
We can't accurately foresee the path from current tech to superintelligence.
That doesn't mean the path doesn't exist, or that it will take a long time. It means we are wandering forward in thick fog, and won't see superintelligence coming until we run right into it.
Nor can we figure out how to get to Valhalla. But that doesn't mean Valhalla doesn't exist. It just means we've not figured how to get there. But we'll know we're there when Odin opens the gates and invites us in.
Nor can we, the people of 1901, figure out how to make a commercial airplane that can seat 100 people. That doesn't mean such an airplane is impossible. But we'll know we're there when the stewardess opens the Boeing 707 and invites us in.
We can't even prove that superintelligence can theoretically exist, or that we have any means to achieve it, or that the methods we are employing could do so.
We don't even know what intelligence is, let alone superintelligence. That doesn't mean there's a 0% chance of a super intelligent AI taking over the world or killing everyone. It should mean that we don't consider this point more strongly than other completely unknown possibilities. The same group of people who are most worried about AI starting the apocalypse seem to almost universally reject any other kind of apocalypse that we can't rule out (see, e.g., every theological version of apocalypse).
Tyler's reference to Knightian uncertainty and Hayek is a gesture at the idea that no, in fact, you can't and shouldn't try to make predictions with hard-number probabilities (i.e. 33% chance of AGI doom). Some risks and uncertainties you can quantify, as when we calculate a standard deviation. Others are simply incalculable, and not only should you not try, but the impulse to try stems from pessimism, a proclivity toward galaxy-brained argumentation, and an impulse toward centralized control that's bad for the economy. In these matters, no a priori argument should affect your priors about what will happen or what we should do - they provide zero evidence.
His all-critical China argument is that if we don't build AGI, China will. Slowing down or stopping AGI is something like a fabricated option [1], because of the unilateralist's curse [2].
So if you had to choose between OpenAI building the first true AGI and a government-controlled Chinese AI lab, which would you pick? I expect Tyler is also meaning to imply that whatever lead the US has over China in AI development is negligible, no matter how much we try to restrict their access to chips and trade secrets, and that the US and China and other players are unlikely to be able to stick to a mutual agreement to halt AGI development.
I agree with Tyler that Scott misrepresented his argument, because while Tyler does emphasize that we have no idea what will happen, he doesn't say "therefore, it'll be fine." His conclusion that "We should take the plunge. We already have taken the plunge." is best interpreted as meaning "if you don't have any real choice in whether AGI gets built or not, you may as well just enjoy the experience and try to find super near-term ways to gently steer your local environment in more positive directions, while entirely giving up on any attempt to direct the actions of the whole world.
I think that the fundamental inconsistency in Tyler's argument is that he believes that while AGI development is radically, unquantifiably uncertain, he is apparently roughly 100% confident in predicting both that China will develop AGI if the US slows down or stops, AND that this would be worse than the US just going ahead and building it now, AND that there's nothing productive we could do in whatever time a unilateral US halt to AGI production buys us to reduce the unquantifiable risk of AGI doom. That's a lot of big, confident conjunctions implicit or explicit in his argument, and he makes no argument for why we should have Knightian uncertainty in the AGI case, but not in the US/China case.
We can point to lasting international agreements like the nuclear test ban treaty as evidence that, in fact, it is possible to find durable diplomatic solutions to at least some existential risk problems. Clearly there are enormous differences between AGI and nuclear bombs that may make AGI harder to regulate away or ban, but you have to actually make the argument. Tyler linked today on MR to a well-thought-through twitter thread on how to effectively enforce rules on AI development [3], saying he's skeptical but not explaining why.
In my view, Tyler's acknowledging that the risk of AGI doom is nonzero, I'm sure he thinks that specific scenario would be catastrophically bad, he explicitly thinks there are productive things you could do to help avert that outcome and has funded some of them, he tentatively thinks there are some well-thought-out seeming approaches to enforcement of AI development rules, and he demonstrates a willingness to make confident predictions in some areas (like the impossibility of meaningfully slowing down AI development via a diplomatic agreement between the US and China). That's all the pieces you need to admit that slowing down is a viable approach to improving safety, except he would have to let go of his one inconsistency - his extreme confidence in predicting foreign policy outcomes between the US and China.
I think Scott, despite the hard number he offers, is the one who is actually consistently displaying uncertainty here. I think the 33% figure helps. He doesn't need to predict specific scenarios - he can say "I don't know exactly what to do, or what will happen, or how, but I can just say 33% feels about right and we should try to figure out something productive and concrete to lower that number." That sounds a lot more uncertain to me than Tyler's confident claims about the intractability of US/China AI competition.
> So if you had to choose between OpenAI building the first true AGI and a government-controlled Chinese AI lab, which would you pick?
Two organizations at least doubles the chances that one of those AIs is misaligned. I don't think your question has the easier answer you seem to imply. If China's AGI is aligned, or if they have a greater chance of creating an aligned AI than OpenAI, then that option could very well be preferable.
If slumlords in Mumbai are going to build housing for 10,000 people, the risk of a catastrophic fire that kills at least 100 is X%.
If also, normal housing developers in the US are going to build housing for another 10,000 people, the risk of catastrophic fire is not "at least 2*X%."
The two organizations pursuing AI are largely using the same techniques that are shared by most machine learning researchers. By contrast, building standards and materials in Mumbai slums and the US suburbs differ drastically, so your analogy is invalid.
The incentives of Chinese and US researchers are slightly different, but 2x factor is fine for the ballpark estimate I was giving. Don't read too much into it, the point is that risk scales proportionally to the number of researchers, and this is only mitigated somewhat by incentives that optimize for specific outcomes like, "don't create an AI that destroys the CCP's ability to control the flow of information in China".
This implies China is significantly less likely to align their AI. There's little basis for this. Even if China is more likely to make unaligned AI, this is dwarfed by the increased likelihood of AGI in the next 50 years with both countries working on it.
I think this is completely wrong, and shows some of the sloppiness of thinking here.
Making AI -- aligned or unaligned -- isn't a matter of rolling dice. Either the current set of techniques are basically on a clean path to AGI, or they aren't, and some further breakthrough is needed. If the current techniques are heading towards AGI (if scaling is all we need and maybe some detail work on tuning the models, but no fundamental breakthroughs or complete changes of approach needed), then AGI is going to happen on a pretty straightforward timeline of training data + more GPUs, and whether two countries are working on it or one or five is unlikely to change that timeline in a macroscopic way.
If AGI is coming on a short timeline with fundamentally the techniques we have today plus more scaling, then, again, whether it's aligned or not isn't a matter of rolling dice. Either the techniques we use today, plus perhaps some ones that we learn over the course of that scaling up process, produce an aligned AGI or an unaligned one. They're pretty much either sufficient or insufficient. Again, whether there's one AGI or several isn't a very large factor here.
>> "Come on Scott, you're just not understanding this...for a start, consider the whole post!"
I'm a big fan of your work and don't want to misrepresent you, but I've re-read the post and here is what I see:
The first thirteen paragraphs are establishing that if AI continues at its current rate, history will rebegin in a way people aren't used to, and it's hard to predict how this will go.
Fourteen ("I am a bit distressed") argues that because of this, you shouldn't trust long arguments about AI risk on Less Wrong.
Fifteen through seventeen claim that since maybe history will re-begin anyway, we should just go ahead with AI. But the argument that history was going to re-begin was based on going ahead with AI (plus a few much weaker arguments like the Ukraine war). If people successfully prevented AI, history wouldn't really re-begin. Or at least you haven't established that there's any reason it should. But also, this argument doesn't even make sense on its own terms. Things could get really crazy, therefore we should barge ahead with a dangerous technology that could kill everyone? Maybe you have an argument here, but you'll need to spell it out in more detail for me to understand it.
Eighteen just says that AI could potentially also have giant positives, which everyone including Eliezer Yudkowsky and the 100%-doomers agree with.
Nineteen, twenty, and twenty one just sort of make a vague emotional argument that we should do it.
I'm happy to respond to any of your specific arguments if you develop them at more length, but I have trouble seeing them here.
>> "Scott ignores my critical point that this is all happening anyway (he should talk more to people in DC)"
Maybe I am misunderstanding this. Should we not try to prevent global warming, because global warming is happening? If you actually think something is going to destroy the world, you should try really hard to prevent it, even if it does seem to be happening quite a lot and hard to prevent.
>> "Does not engage with the notion of historical reasoning (there is only a narrow conception of rationalism in his post)"
If you mean your argument that history has re-begun and so I have to agree to random terrible things, see above.
>> "Does not consider Hayek and the category of Knightian uncertainty"
I think my entire post is about how to handle Knightian uncertainty. If you have a more specific argument about how to handle Knightian uncertainty, I would be interested in seeing it laid out in further detail.
>> "and does not consider the all-critical China argument, among other points"
The only occurrence of the word "China" in your post is "And should we wait, and get a “more Chinese” version of the alignment problem?"
I've definitely discussed this before (see the section "Xi risks" in https://astralcodexten.substack.com/p/why-not-slow-ai-progress ) . I'm less concerned about than I was when I wrote that post, because the CHIPS act seems to have seriously crippled China's AI abilities, and I would be surprised if they can keep up from here. I agree that this is the strongest argument for pushing ahead in the US, but I would like to build the capacity now to potentially slow down US research if it seems like CHIPS has crippled China enough that we don't have to worry about them for a few years. It's possible you have arguments that CHIPS hasn't harmed China that much, or that this isn't the right way to think about things, but this is exactly the kind of argument I would appreciate seeing you present fully instead of gesture at with one sentence.
>> "Or how about the notion that we can't fix for more safety until we see more of the progress?"
I know it's annoying for me to keep linking to thousand-word treatments of each of the sentences in your post, but I think that's my point. These are really complicated issues that many people have thought really hard about - for each sentence in your post, there's a thousand word treatment on my blog, and a book-length treatment somewhere in the Alignment Forum. You seem aware of this, talking about how you need to harden your heart against any arguments you read on Less Wrong. I think our actual crux is why people should harden their hearts against long well-explained Less Wrong arguments and accept your single-sentence quips instead of evaluating both on their merits, and I can't really figure out where in your post you explain this unless it's the part about radical uncertainty, in which case I continue to accuse you of using the Safe Uncertainty Fallacy.
Overall I do believe you have good arguments. But if you were to actually make them instead of gesture at them, then people could counterargue against them, and I think you would find the counterarguments are pretty strong. I think you're trying to do your usual Bangladeshi train station style of writing here, but this doesn't work when you have to navigate controversial issues, and I think it would be worth doing a very boring Bangladeshi-train-station free post where you explain all of your positions in detail: "This is what I think, and here's my arguments for thinking it".
Also, part of what makes me annoyed is that you present some arguments for why it would be difficult to stop - China, etc, whatever, okay - and then act like you've proven that the risk is low! "Existential risk from AI is . . . a distant possibility". I know many smart people who believe something like "Existential risk is really concerning, but we're in a race with China, so we're not sure what to do." I 100% respect those people's opinions and wouldn't accuse them of making any fallacies. This doesn't seem to be what you're doing, unless I'm misunderstanding you.
I'm actually not convinced by the China argument. Putting aside our exact views on the likely outcomes of powerful AI, surely the number one most likely way China gets a powerful AI model is by stealing it from an American company that develops it first?
That's broadly how the Soviets got nukes, except that AI models are much easier to steal and don't require the massive industrial architecture to make them run.
Worse: stealing AI models doesn't require the massive infrastructure to *train* them, just the much more modest infrastructure to run them. There are LLMs (MLMs?) that can run on a laptop GPU, I don't think we'd even contemplate restricting Chinese compute access to below that level even if we could.
Disagree. China will able to produce powerful AI models. There are many Chinese researchers, and they do good work. China might be slowed down a bit by U.S. export limitations, but that's it.
I actually agree with you; the Soviets still would have developed nukes eventually without espionage, but it's pretty clear it would have taken longer, and I think this situation is comparable (with the noticeable difference that stealing the plans / data/model for AI is effectively like stealing the nukes themselves.
Stealing an AI model from the US would not increase existential risk much if the US companies are not allowed to train models more advanced than GPT-4.
The CHIPS act will give china a large disadvantage in compute, and they already have a large disadvantage in the availability of top talent because if you're a top 1%er you don't want to live in China -- you go study in the US and stay there.
I expect Chinese and Americans will produce different designs for AGIs, and more generally two AI researchers would produce different designs.
On the one hand, two different designs would give two chances for an AGI design to kill us all. On the other hand, if there are two designs, one might be safer in some clear way, and conceivably most people could be persuaded to use the safer design.
Edit: I don't know the first thing about Chinese AI, but a top comment on [1] says
> I am not a defense expert, but I am an AI expert, and [...] [China] certainly is not leading in AI either."
> Urgh, here's what China does. China publishes a million AI "scientific" papers a year, of which none have had any significant impacts. All of the anthology papers in AI are from USA or Canada. Next year China publishes another million useless papers, citing other chinese papers. Then if you naively look at citations you get the impression that these papers are impactful because they have lots of citation. But its just useless chinese papers citing other useless chinese papers for the purpose of exactly this: looking like they are leading.
Another commenter adds
> The really most influential AI breakthroughs in 2022, IMO:
> DALLE-2 - openAI, USA
> Stable Diffusion, LMU Munich Germany
> ConvNeXt, Meta AI, USA
> ChatGTP, open AI, USA
> Instant NGP, Nvidia, USA
> Generative AI was really big this year. What AI breakthrough was made in China? I cannot think of any important one, ever.
What exactly is so bad about China beating Silicon Valley? You trust Silicon Valley with AI safety more than China? I am not sure that is my knee jerk reaction and I am not a particular Sinophile.
If the AI can be controlled, do you really believe that it would be better in the hands of the CCP rather than US tech companies? On what basis or track record do you make this claim? I don't recall tech companies causing millions of deaths, suppressing pro-democracy protests, persecuting religious or ethnic minorities, forcing sterilizations, stifling political dissent, or supporting widespread censorship, for example.
On the other hand, the CCP has lifted millions of people out of poverty (after previously, er, plunging them into poverty, or at least more dire poverty than they were previously experiencing). On the gripping hand, it's not clear to me that a CCP-AGI would value poverty reduction once Chinese former-peasants were no longer needed for industrial growth.
>The CCP has lifted millions of people out of poverty
Wrong. Western technology did. CCP prevented access to this technology.
And millions of *chinese* people were lifted out of poverty. I don't expect the CCP to focus on helping people in other countries, but the fact that Chinese people were improved under their watch says little about their concern for humanity in general.
>On what basis or track record do you make this claim? I don't recall tech companies causing millions of deaths,
Well they haven't really had the power to in the past. If tech companies could cause millions of deaths to pump the stock (or make their leaders putative gods (or controllers of god)) its not clear to me they would say "no".
>suppressing pro-democracy protests,
Who cares about democracy? Not important on the scale of talking about existential threats.
>persecuting religious or ethnic minorities, forcing sterilizations, stifling political dissent, or supporting widespread censorship, for example.
Their support of widespread censorship is exactly the sort of thing which might help them keep an AI under wraps. As for those other issues those are bad, but they aren't really things that are that unique, the US/West was pursuing those policies in living memory.
OMG the Chinese don't like the Uighurs, and treat them horrible is not some knock down argument they won't be safe with AI.
We can be sure the US tech company AI will make sure to use all the correct pronouns and not make anyone sad with trigger words, while it transports us all to the rare metal penal colonies in Antarctica for that one like of a Mitch Romney tweet in 2009. That is cold comfort.
Yet another point: capitalism drives people to take shortcuts to be competitive, and shortcuts on alignment are not a good idea. The CCP has a much firmer grip on what they permit, and that could be good for safety. The matrix of possibilities is:
1. China creates aligned AI.
2. US creates aligned AI.
3. China creates unaligned AI.
4. US creates unaligned AI.
It's not unreasonable to think that the probability of option 4 is higher than 3, and that the probability of option 1 is higher than 2, which would make China a safer bet if we're really concerned with existential risk.
1 It's not "Capitalism" that drives people to take shortcuts, it's laziness and incentives, which obviously also exist in non-capitalistic systems. Look at Chernobyl for just one example.
In addition, China is hella capitalistic these days.
I am no fan of the CCP. I despise them in fact. But should we put our faith in Peter Thiel, Mark Zuckerberg, and Elon Musk? Silicon Valley has been functionally psychopathic for at least the last decade.
If AI is on the brink of some sort of world-altering power then I can't see the Silicon Valley types suddenly deferring to ideas about the common good and the virtue of restraint when they've demonstrably behaved as if they had no interest in those virtues for years. The CCP, while awful, may at least feel constrained by a sense of self-preservation.
>I am no fan of the CCP. I despise them in fact. But should we put our faith in Peter Thiel, Mark Zuckerberg, and Elon Musk? Silicon Valley has been functionally psychopathic for at least the last decade.
Thiel and Musk both express concern about AI risk, much more than the median voter or politician
That seems like a fairly solid CV of being willing and able to lock human-level sapient beings in boxes and subject the nuances of their loyalties to unrelenting scrutiny, which seems extremely relevant to the classic "distinguish a genuinely friendly, submissive AI from a malevolent trickster" problem.
I don't actually think that's the best way to approach AI alignment, or for that matter running a country - long term growth requires intellectual freedom. But for somebody who figures there'll be a need to censor, persecute, and sterilize paperclip-maximizers, "move fast and break things" is not a reassuring slogan.
"I don't recall tech companies causing millions of deaths, suppressing pro-democracy protests..."
I absolutely do. It was in 1930s Germany, not 2030s America, but I don't have that much more faith in the American political system. It's good, but I wouldn't bet the farm on it. And America's politics took a sharp turn to the left/right/wrong, I have no faith that its tech companies would do anything other than pander. Germany's tech companies supported the war efforts.
China's definitely worse at present. But if we're making predictions about what might happen in the future, you can't just make the blanket assumption that the political truths of now will continue into the future.
Another way it seems to be an echo chamber is that while people disagree, they do it within a set of premises and language uses that means all the agreements and disagreements take place within a particular, limited framework.
"Noam C" has been wrong about everything ever, to a first approximation. But more to the point, people who copy-paste the old "it is a pattern-matcher therefore it can't possibly be understanding anything" without even the minimal effort to consider what might be the source of their own ability to "understand" (is it magic? if it's X, why would X be un-replicable by AI?) do sound like pattern matching parrots. Not very sophisticated, at that.
Banned for the "parrot like" part.
This doesn't seem like a very fair ban, given that he was responding to someone who had just compared him to a parrot (and who was not banned for doing so).
But he was responding to 'parrot like' accusations in the first place
See Microsoft's recent paper, "Sparks Of Artificial General Intelligence: Experiments With GPT-4", at https://arxiv.org/abs/2303.12712
Chomsky is clearly a smart guy in his own field, but the article was embarrassing. For example, in the terraform Mars question, it uses the AI's unwillingness to express an opinion on a moral issue as proof that it's not intelligent. But newly-made AIs love to express opinions on moral issues and Chomsky was using a version that was specifically trained out of that behavior for PR reasons. I tested his same question on the base version of that AI and it was happy to express a moral opinion on it. If you can't even tell an RLHF-ed opinion from the AI's fundamental limitations, I don't think you should be writing articles like these.
On the general question, please see past articles I've written about this, for example https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ , https://astralcodexten.substack.com/p/somewhat-contra-marcus-on-ai-scaling , and https://slatestarcodex.com/2019/02/28/meaningful/
Scott,
I am a huge fan of yours and I have already read them but not convinced that LLMs can achieve AGI with their current design. I am in the same camp as Marcus and Noam C.
You should perhaps explain what is "true AGI"? Just because GPT-4 is doing impressive stuff does not mean it is AGI and the article you shared about it showing "sparks of AGI" does not explain how they determined that. Also we cannot take their word for it. Gary Marcus has asked for details how it was trained and the training data but the "Open"AI has not made it open.
LLMs like GPT-4 are impressive but I am not sure they can be called AGI or sparks of AGI. At least yet.
The infamous incident about GPT-4 lying to a raskrabbit that it is not AI bot and it is a visually impaired person still does not make this interaction as AGI. GPT-4 by large training set trained itself to use deception. Just like Bing chatbot trained itself to use abuses by training itself on internet troll chat.
So garbage in garbage out still applies to LLMs and they are simply extrapolating stuff from their massively huge training data.
As Noam C has explained, Human brain does more than extrapolation from the data it has through experience. Until we understand how human mind truly works, we can forget about AGI. Some people truly think deep learning is actually deep understanding. The "deep learning" is not deep at all. Deep means number of hidden layers of computation where each layer has math functions (with parameters) and each layer transforms data and passes on to next layer and the system fits those params to training data. This is not 'deep learning' by any stretch of imagination.
> Until we understand how human mind truly works, we can forget about AGI.
That's a rather strong claim. GPT and these LLMs are missing a few important things. First agency (but that's "easy" to simulate, just tell it to tell a human what it wants to do, then the human can feed back the results), and more importantly a loop that allows itself to direct its attention, some kind of persistence.
But these components/faculties are coming. And eventually if the AI can form models that contain itself, and it recognizes its own relation to other entities (even if just implicitly), and even if just through periodic training, but if it gets (again, even if implicitly) to influence the world to manifest some change in its model, then at point the loop will close.
“Not convinced” is null hypothesis football imho. “Unless you can convince me there’s risk I’m going to assume it’s zero” instead of “unless you can convince me it’s safe I’m going to assume it’s risky”
> not convinced that LLMs can achieve AGI with their current design. I am in the same camp as Marcus and Noam C.
I agree that LLMs are not AGIs (and while I'm not in Noam Chomsky's camp for anything outside linguistics, I feel like Gary Marcus makes good points).
> Until we understand how human mind truly works, we can forget about AGI.
I disagree strongly. Modern AIs can do better than the best humans at a variety of tasks; understanding the human mind is unnecessary to build AGI, and precisely because we *don't* understand the human mind, humans are likely to invent an AGI with quite a different architecture than whatever the human mind has, something in a different location in "mind design space"[1]. This already happened for deep learning, which is obviously different from human brain architecture. Before that it happened for neural nets, which use backpropagation, which animal brains can't use.
This is a big part of why building AGI is risky and also why estimating the risk is hard. It's not just that we can't know how deadly AGI will be, but also that we can't know where in mind design space AGI will be located or how deadly that particular location will be. (Mind you, if AGI *were* in the same place in MDS as humans, humans vary remarkably, so AGI might end up being more deadly than Hitler, or safer than the average grandmother, or both — with some AGIs fighting for humanity, others against, and still others ignoring the "should we kill all humans" controversy and watching cat videos instead.)
And I think LLMs (and other modern AIs) are a terrifying omen. Why? Well, it has to do with them having "too much" intelligence for what they are.
First of all, here is a machine with no senses — it's never seen anything, never heard anything, never touched anything, never felt hot or cold, hungry or happy or angry or nostalgic. Yet an "emotionless Helen Keller" GPT3 base model can often pass Turing tests without ever so much as practicing on Turing tests. It simply *can* pass the test, by its very nature. No human can do anything like this. If you lock an American in a padded cell and show zim random Chinese books and web sites (without images) 24/7/365 for 20 years on a big-screen TV, they might go insane but will not become fluent in Mandarin.
Second, it's small compared to human neural networks. Human brains have over 100 trillion synapses. It seems to me that in an AI neural net, weights are analogous to synapses and biases are analogous to neurons, so a GPT3-level AI has ~1000x fewer synapses than a human brain. (That AIs are way more efficient than humans on this metric is unsurprising from an evolutionary-theory standpoint, but still important.)
Third, it'll probably get smaller over time as new techniques are discovered.
Fourth, I believe GPT3 is wildly overpowered compared to what an AGI actually needs. Observe: in a sense it's hard for a human to even perform on the level of GPT2. Suppose I ask you to write a one-page story in the style of (and based on the characters of) some random well-known author, one word at a time with *no edits* — no backspace key, once you write a word or punctuation mark it is irrevocably part of the story, and you can't use writing aids, you must write the story straight from your mind. And I give you the first sentence. I think (if you're familiar with the author) you can do a better job of making the story *coherent* or *entertaining* than GPT2, but GPT2 is likely to be able to beat you when it comes to matching style and word choices of the original author. So GPT2 already beats the average human in some ways (and it can do so lightning-fast, and never gets tired, and can do this 24/7/365 if you like.)
GPTs (transformers) are fundamentally handicapped by their inability to edit. An AGI will not have this handicap; they'll be able to write a plan, review the plan, critique the plan, edit the plan, and execute the plan. An AGI doesn't *need* to replicate ChatGPT's trick of writing out a coherent and accurate text on the first try, because it can do as humans do — review and revise its output, do research on points of uncertainty, etc. — and therefore a linguistic subsystem as complex as GPT2 is probably sufficient for an AGI to match human intelligence while greatly exceeding human speed. And if a GPT2-level linguistic subsystem is sufficient, well, perhaps any PC will be able to run AGI. [Edit: IIUC, a typical PC can run GPT2 inference, but training requires modestly more processing power. You do not need a supercomputer for training — they used a supercomputer not because it's necessary, but because they wanted results quickly; no one is willing to train an AI for 18 years like they would a human.]
If a typical PC can run a smart-human AGI at superhuman speed, then how much smarter can a pro gaming PC be? How much smarter can a supercomputer be? How much more powerful is a supercomputer that has hacked into a million gaming PCs?
--------
I disagree with Eliezer Yudkowsky that the very first AGI is likely to kill us all. Maybe its goals aren't that dangerous, or maybe it has cognitive limitations that make it bad at engineering (a skill that, for the time being, is crucial for killing everyone).
But once that first AGI is created, and its architecture described, thousands of AI researchers, programmers and kids in moms' basements find out about it and dream of making their own AGI. Perhaps some of the less powerful AGIs will show the classic warning signs: trying to prevent you from turning it off, lying to you in order to make progress on a subgoal, manipulating human emotions*, using the kid's mom's credit card to buy online services, etc.
But I think Eliezer would tell you that if this happens, it is already too late. Sure, not-that-powerful AGIs won't kill everyone. But just as people start to notice that these new AGIs sometimes do immoral things, someone (let's call him Bob) with too many AWS credits will program an AGI *carelessly* so that its goals are much different than the human intended. Maybe Bob will give it too much processing power. But maybe the AGI simply decides it can work toward its misconfigured goal faster by creating another AGI more powerful than itself, or by creating a worm to distribute copies of itself all over the internet. At this point, what happens may be out of anyone's control.
Maybe it doesn't kill us all. But since it's smarter than any genius, it has certainly thought of every threat to its goals and means, and how to prevent the apes from threatening it. And since it's faster than any genius and capable of modifying copies of itself, it is likely to evolve very quickly. And if it determines that killing 10% or 100% of humans is the safest way to protect itself from humans trying to turn it off, then sure, why not?
It's worth noting that the most dangerous AGI isn't the most typical one. There can be a million boring, safe AGIs in the world that will not save us from the one Bob misconfigured.
[1] https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general
Great comment. We went from "Do androids dream of electric sheep?" to "Do AGIs watch cat videos?" in a half century. Whatever AGIs will be, I doubt they will be "united", i e. globally aligned in between themselves. (regardless of whether they will be "aligned to human goals"; I see no evidence that HUMANS are globally aligned on human goals, so what do you expect from AGIs?)
About "humans only being able to make edits", I believe that the (GPT-powered) Dall-E had for long years been based upon "iterative refinements" in its image generation. "Iterative refinements" are just as applicable to token string production...
> Please. GPT-4 has no understanding of the world. It is a pattern matching parrot. Very sophisticated one at that.
Maybe humans are all sophisticated pattern matching parrots. You literally don't know, so this isn't really an argument against the dangers of AI.
>You literally don't know
That's getting close to the sort of question for which "personal gnosis" is a valid answer; any person knows that he/she personally is not a p-zombie.
I'm not saying that Banned Dude (don't know who originally posted it) is right, of course.
> any person knows that he/she personally is not a p-zombie.
No they actually don't, because that would assume their perceptions have some kind of direct access to reality, as opposed to merely perceiving some kind of illusory projection.
By "p-zombie" I mean something that behaves like a human but without an associated first-person experience.
I know I have a first-person experience. It doesn't matter for this purpose whether that experience corresponds to reality or not; even if I'm experiencing the Matrix, *I am experiencing*.
If something else contradicts that, *the something else is wrong*; I know that I think and perceive more directly than I can know anything else. As I said, personal gnosis.
I know what you meant. I'm saying you only think you have first-person experience. This "knowing" is a cognitive distortion, like other perceptual illusions. People who don't see things in their blind spot can swear up and down something is not there, that doesn't make it true. We only ever have indirect access to reality, your "first-hand experience" is no exception.
>People who don't see things in their blind spot can swear up and down something is not there, that doesn't make it true.
How does that relate? That's perceptions not corresponding to reality. They experience something false, which inherently necessitates that they experience.
I've seen this argument before, and it's baffling to me. Are you operating off some strange definition of what it means to "have first-person experience"?
There exists, at this very moment, the qualitative experience of seeing letters being written on a computer screen. An experience which "I" "am having" by any reasonable definition of the words.
I understand that I can't convince you that *I* have qualitative experiences, but I can't understand how in the world you can doubt the existence of *your own* phenomenology, unless you are somehow taking issue with the use of the words "I" or "have".
> his "knowing" is a cognitive distortion, like other perceptual illusions.
That's not something you *know* , it's a belief.
I have direct access to reality. This access is pretty handy for making things like microprocessor chips or building a fort out of sofa cushions.
If they are perceiving anything in the first person, then they are not a p-zombie. If they are perceiving an illusory projection, that already means they have subjective experience and are hence not a p-zombie.
Perception is not experience. Don't conflate the two.
Can you elaborate? What does it mean to perceive something without having the subjective experience of having perceived that thing?
No. We humans are not JUST pattern matchers.
A human baby/child has less training data (eg experience) in her brain than say GPT-4 yet a baby/child can do reasoning better than the latest fad in AI. Unless we understand how human brain is able to do this without lots of training data, we can forget about AGI.
LLMs can do what they can purely based on training data and sure some of the training data may have given them insights (like how to be deceptive and lie to get what you want) but those insights do not make them come close to AGI.
> A human baby/child has less training data (eg experience) in her brain than say GPT-4 yet a baby/child can do reasoning better than the latest fad in AI.
You're not comparing like with like. A human baby is not a blank slate, GPT-4 was. Billions of years of evolution culminated in a human brain that has pretrained biases for vision, movement, audio processing, language, etc.
> but those insights do not make them come close to AGI.
Current LLMs are not AGIs. That does not mean AGI is not the same sort of stochastic parroting/pattern matching we see in LLMs. Just adding "step by step" prompts and simple "check your work" feedback [1] significantly improves their reasoning capabilities. We've barely scratched the surface of the capabilities here.
[1] https://arxiv.org/abs/2303.17491
As I noted upthread[1], a human baby has in one sense vastly *more* training data, as GPT has no senses — no sight or hearing or touch or even emotion. As a baby's father myself, I have found it absolutely remarkable how slowly she learns and how difficult she is to teach. (Keeping in mind that GPTs are not AGIs) I think a second- or third-generation AGI would be able to learn faster than this, if it were fed training data at the same rate and in the same way. But if I check lists of milestones of cognitive development, I find that my baby is largely above-average, if a bit slow in the linguistic department. Some AIs learn fast even when training is slowed down to human speed[2]; not sure why we'd expect AGIs to be any different.
[1] https://astralcodexten.substack.com/p/mr-tries-the-safe-uncertainty-fallacy/comment/14387287
[2] https://www.youtube.com/watch?v=A2hOWShiYoM
I'm not sure the argument works the way you assume it does. Over the last years, we see more and more evidence that climate change is _not_ as bad as the "typical worst outcomes" would have us believe. The worst scenarios decrease in likelihood. The impact of the the scenarios that seem likely is being evaluated as less disastrous. Some advantages of climate change or at least opportunities it opens up are getting more attention.
A better analogy would be the CFC crisis. I wish people on all sides of these debates would be referencing it more frequently.
Come on Scott, you're just not understanding this...for a start, consider the whole post! Tyler Cowen
TC - huge fan of yours (and Scott's). And in this case, I had generally same reaction as Scott's. REQUEST. Can you have Yud on for an emergency Tyler Talk, or perhaps Scott + Yud? I would estimate 10k+ of your overlapping readers are scared shitless over this. Would welcome thoughtful rebuttal of Yud if it's out there.
Seconded. Also an avid fan of both of you.
Tyler, I fully agree that "AI represents a truly major, transformational technological advance," and that this is going to make things weird and hard to predict precisely. But... isn't that what we have probabilities for? You say that "*all* specific scenarios are pretty unlikely," but, following Scott's argument, what is a "specific scenario," exactly? This seems like a way to escape the hard problem of putting a numeric probability on an inherently uncertain but potentially catastrophic outcome.
Ultimately, my overriding sense from reading your post (and reading/listening to you lo these many years) is that you're frustrated with stasis and excited by dynamism. I agree! But even if you have a strong "change is good" prior, as I do, it still seems correct to weigh the likelihood function as well -- that is, engage with the AI-specific arguments rather than depend on historical analogies alone.
Probabilities do not work well when smart people completely disagree on priors. Some people think the chance of AI apocalypse is 20%, some think it’s one in a million. There are no priors these people agree on.
Most of the “probabilistic” reasoning here is simply argument by exhaustion. Ten paragraphs talking about probabilities with no actual mathematics. Then concluding, therefore there is a 30% chance of AI apocalypse.
That's not what this post is saying *at all*. It's saying that Tyler's mathematics-free arguments aren't enough to establish a near-zero probability, and (separately), that Scott has put a lot of thought into this and come out with the number 33%. The arguments for 33% in particular aren't given here, they're spread across a lot of previous posts and the inside of Scott's head. The point of this post is to rebut Tyler's argument for near-zero, not to support Scott's arguments for 33%. It's *Tyler* who's doing the thing you accuse Scott of.
The 33% is not result of a lot of thought. It is hand waving.
The small but non- zero probability is also a lot of hand waving. As is a probability of almost certain also hand waving.
It's not hand waving, it's virtue signaling.
Scott is a member of a sect that strongly beliefs in an AI apocalypse. So he cannot give a low probability because he'd lose status within his sect if he did. But at the same time. He wants to have main stream appeal and he cannot give a high probability, because he'd be seen as a crackpot.
The 33% is very carefully chosen. It's below 50%, so he cannot be accused of believing in an AI apocalypse, but still high enough that he doesn't lose guru status within his sect.
It's a very rational and thought out answer, but at a meta level. It's not a real chance of course, that's not the point.
I'm sure it's not the result of a watertight mathematical argument, but I'm not sure how one would even construct such an argument. But Scott's definitely put a lot of *thought* into it - see his comment https://astralcodexten.substack.com/p/mr-tries-the-safe-uncertainty-fallacy/comment/14070813 for a partial list of posts he's written examining various aspects of this problem.
[Also, nitpick: the expression is "hand *waving*", as in "waving one's hands dismissively". Hand *waiving* would mean saying "Nah, I don't need a hand. Stumps were good enough for my father, and his father before him."]
I'm with you. A lot of hand waving.
Let's see the math!
And here is a bit more: I am a big fan of Scott's, but this is a gross misrepresentation of what I wrote. Scott ignores my critical point that this is all happening anyway (he should talk more to people in DC), does not engage with the notion of historical reasoning (there is only a narrow conception of rationalism in his post), does not consider Hayek and the category of Knightian uncertainty, and does not consider the all-critical China argument, among other points. Or how about the notion that we can't fix for more safety until we see more of the progress? Or the negative bias in rationalist treatments of this topic? Plus his restatement of my argument is simply not what I wrote. Sorry Scott! There are plenty of arguments you just can't put into the categories outlined in LessWrong posts.
Maybe it would help to disentangle the policy/China question from the existential risk question? That is, it may be the case that OpenAI or anyone else unilaterally desisting wouldn't prevent (e.g.) China from plowing full steam ahead. But that might still be a very bad idea. It's not clear to me whether you think this is fundamentally a collective action problem, or whether you really want to dismiss the risk entirely.
well said
It’s not dismissing the risk at all. There are two types of risks.
1. risks from anyone getting AGI
2. risks from China getting AGI before the US
Be a good Bayesian. We should focus on the risk that maximizes the danger, times the effectiveness of our working on it. That is an argument for focusing on #2, because we have many effective actions we can take.
It seems to me that this dichotomy does not work very well if we consider the actions they (possibly) imply.
If you want to avoid risk #2, you need to focus on actions that also prevent that the development of completely unaligned power-seeking doom-bringing AGI is sped up accelerated, at least if you believe that there is a relevant alignment problem at all. But in the anti-#2 actions subset, there are many actions that do just the opposite.
To separate your #1 and #2 properly, I'd specify it as
1) risks from anyone building unaligned AGI
2) risks from China building aligned AGI before the USA.
However, as I see it the chance that any AGI built in the next 30 years (i.e. any neural net AGI) is aligned is <0.01%. So building AGI ourselves would subject us 99.99% to #1 if China doesn't build AGI for the sake of 0.01% (generously) of avoiding #2 if China builds AGI and also gets the 0.01%.
The correct response to China attempting to build AGI is to stop them, which fulfils both #1 *and* #2, rather than to pre-emptively commit suicide. This is definitely physically possible; if all else fails, a thousand nukes would do the trick.
1. What are the risks from China building highly intelligent but poorly aligned AI. To China? To the US?
2. Need it be said that nuclear war carries with it its own set of existential risks?
1. To everyone: hostile AI wages war against humanity.
2. Global thermonuclear war is a GCR, but it's not a notable X-risk. Rural populations can't be directly killed by any plausible quantity of nukes, fallout's too localised and too short-lived, and nuclear winter is mostly a hoax (the models that produce everyone-dies kind of outcomes tend to look like "assume everything within fire radius of the nuke is high-density wooden buildings, then assume 100% of that wood is converted into stratospheric soot", and even in those the Southern Hemisphere basically does fine).
But you can still argue within point 1! Scott has taken issue with Tyler's reasoning about point 1. Tyler has responded 'well that's a gross misrepresentation because you haven't talked about point 2 (and some other stuff)'. But that's not how it works.
The risk isn't along the lines of nuclear weapons, where the technology waits inertly for a human to start a cascade of disaster. It's more along the lines of a new virus that once released into the atmosphere will have a life of its own and be unable to be contained.
So, like Covid, whether it starts in China or the US would seem to make little difference. There's very little point in us rushing to be the first to start the conflagration just to stop our rivals getting a jump on us.
What are the minimum requirements for such a thing? Does it need to be embodied? Or will ChatGPT55 kill us with spam and highly detailed scams targeting our relatives?
It will mean that you can no longer trust anything that you haven't seen written down in a physical book published prior to 2022.
If you think about it, a lot of our modern systems rely on trust in a shared history and understanding of science etc. So I'm more thinking of your second option.
We'll likely end up as Morlocks running machinery that we no longer understand. Some of us might become Eloi - living lives of luxury without purpose.
It is not really clear to me that China getting it first wouldn't be good for AI safety. I kind of trust their society to be more responsible and safe than some private US company. This is one of those areas where I really feel old style nationalism is making people stupid.
"other competitor nations with other values = bad". When IDK the things that China is strong on seem like the things we want to people controlling AI to be strong on (long term planning, collectivist versus individual focus, info sec priority, social conservatism).
Well was it China or the NIH? It doesn't seem remotely clearly to me China "owns" the pre-pandemic Wuhan research any more than the US does. Though obviously we have a big incentive to blame it on the other.
I don't think it was NIH that had direct continuous control over biosafety practices in Wuhan. Sure, both NIH and China authorised GoF research, but it was a Chinese lab with Chinese scientists that dropped the ball.
And your ratioalization of China throwing scinetists in prison for talking about the virus and refusing to make any effort to stop it spreading outside of China?
" I kind of trust their society to be more responsible and safe than some private US company. "
This seems hopelessly naive. China has private companies. If regulation is the answer, and China is not pursuing "full stop" regulation, then what regulation are they pursuing? How exactly are they "more responsible"?
>then what regulation are they pursuing? How exactly are they "more responsible"?
I don't think we have a good handle on what safety measures they are taking, but they are known to be conservative and authoritarian.
Whereas the ethos of Silicon Valley is "break things and hope you get a giant pile of money from it, maybe it won't be too bad for normies".
"they are known to be conservative and authoritarian."
They are known to be conservative and authoritarian in regards to personal freedom, but not in terms of environmental destruction or technology development.
For some actual information about Chinese AI regulation I recommend following ChinaTalk. E.g. this (there is a paywall, but some important information is before it): https://www.chinatalk.media/p/tiktok-live-show-ais-regulatory-future
They are known to be conservative about *things that threaten the CCP's power*.
They want to be THE global hegemon for the rest of time, and its naive to think they wouldn't be willing to gamble with humanity's future when the payoff is (the potential for) permanent hegemony.
AGI being developed under an authoritarian regime might not be worse for x-risk, but it's worse for s-risk (notably, "humans live under an inescapable totalitarian regime for the rest of time").
This seems a better argument about the hysterics regarding China.
Given China's horrific environmental record, trusting them to better manage externalities than the west seems hopelessly naive.
In addition, if China is first to an aligned AGI you can expect the result to be a CPC not grinding down in the face of humanity forever. However, if you are inconvenient to that enterprise you will not need to worry about it. You and your family will be dead. That is how collectivist societies deal with inconvenient people and they are well aware that nits make lice.
"This is all happening anyway" doesn't seem like an airtight argument.
https://www.vox.com/the-highlight/23621198/artificial-intelligence-chatgpt-openai-existential-risk-china-ai-safety-technology
Think human cloning, challenge trials, drastically slowing bio weapon dev, gene drives, etc.
Negative bias is an understatement. What evidence would Scott need to change his opinion? We can (hopefully) all agree that doomsday scenarios are bad. I’m asking what would compel Scott to update his prediction to, say, < 1%.
Actually, I agree. @scott this is worth your time. Dig in and give Tyler's post another longer deeper response. This is your chance to defend the arguments of AGI risks to a prominent skeptic
I suspect he will. He likes doing contra contra contra
Could you expand on the China argument? I think Scott's argument is that no matter who builds the AI, whether China, the U.S. or anyone, that could potentially kill everyone, while you are more talking about China getting technological hegemony.
Everyone brings up the China argument as if it's supposed to be self-evident that Chinese researchers would produce an AI more harmful to humanity than Silicon Valley entrepreneurs, and that conclusion is... not obvious to me.
The only major nuclear power plant disaster was in a Communist country; one thing authoritarian governments are bad at is recognizing when they're making a mistake and changing course.
Fukushima was level 7 as well, although it wasn't quite as bad and basically came down to a single mistake ("tsunami wall not high enough") rather than the long, long series of Bad Ideas that went into Chernobyl.
...I actually can't work out whether the suggestion is that a single mistake leading to Fukushima is better or worse than it taking a long chain of things needing to be just so to get Chernobyl
Here to register my disagreement that Fukushima had a single cause.
Chernobyl had a chain of engineering and human failues - but an RBMK can be run safely with minor modifications (the other Chernobyl reactors ran for 14 years afterwards). They tried really really hard to get it to explode, even if that's not what they intended.
The chain of engineering mistakes that went into Fukushima are a bit worse. The arrogance, regulatory and engineering failures are worse than Chernobyl in my opinion. The put backup generators 10m above sea level based on a fradulent study estimating the worst earthquake to be 10x weaker than other reported ones along the coast.
And if there are other things they are good at it is resisting public pressure to take short term gains over long term plans. Two can play your silly game.
The Chinese government has a lot of pluses and minuses over the US one, it is not remotely obvious to me which one would be wiser to trust with AI if I had to pick one.
Totally agree.
"the China argument as if it's supposed to be self-evident that Chinese researchers would produce an AI more harmful to humanity than Silicon Valley entrepreneurs"
That's not the argument. If we start with the case that there is some percentage chance AGI will end humanity, regulation stopping it being developed in the US will not stop it in China (or elsewhere). It will end humanity anyway, so stopping it in the US will not change the outcome.
The secondary argument is that it won't end humanity directly, there will likely be a lot of steps before that. One of which is that is under the control of a nation state, that nation will be able to accumulate a lot of power over others. So, the intermediate consequence of stopping it in the US and not stopping it in some other place, is that the other place will end up strategically dominating the US for some unknown period of time until the AGI ends up strategically dominating humanity.
> It will end humanity anyway, so stopping it in the US will not change the outcome.
If the probability is X% if everyone is working on it, if a bunch of nations except China stop walking that path then the probability falls below X%. I have no idea how you can conclude that this isn't relevant.
Why does the probability necessarily fall below X%? Might it not just push out the timeline at which the risk occurs? Would a six month pause have any measurable effect?
Another way to think about it, people are willing to risk their own personal extinction rather than be subjected to living under Chinese rule. It's not a given that Chinese domination is preferable to death.
China currently uses phones to keep people from straying outside their neighborhood. I'm sure they would never misuse AI against their people.
I thought we were concerned about AI destroying our civilization, not making the Chinese police state 20% worse?
China will intensely fear the prospect of creating something they can't control
I just read your post.
What I notice more than anything is that both you and Scott are arguing about discursive features, and both articles seem to express a fair amount of frustration, which is reasonable given the format. What I also notice is that the information content about AI is extremely small. If anything "AI" is just an incidental setting where the meta-discourse happens.
Scott is reacting to your post, which is seems to be reacting to some Other Side. My understanding of your argument is that "Big Things are happening Very Soon whether you like it or not, and nobody knows how things will play out, so stop doomsaying, y'all." (In my head you're from southern Texas.)
One feature of your article does set off minor alarm bells for me: its heavy use of deonotological arguments. These are exhibited by liberal use of phraseology such as "truly X", "Y is a good thing", "no one can Z", "don't do W", "V is the correct response", etc. In contrast, Scott's article here levies more consequentialist arguments—"if you do X, then Y happens", "here is failure mode Z", etc.
Personally, changing my beliefs based on deontological/moralistic arguments typically involves a strong invocation of trust and/or faith, whereas consequentialist rhetoric gives me some meat with which to engage my current beliefs. The former feels more like a discontinuous jump while the latter a smooth transition.
/2cents
"What I also notice is that the information content about AI is extremely small."
But then the actual capabilities of existing AI and any AI that's forseeable from current tech, are all but irrelevant to AI x-risk discourse. It's mostly a fantasy built on magical entities – "super-intelligence" – using magical powers – "recursive self-improvement."
We can't accurately foresee the path from current tech to superintelligence.
That doesn't mean the path doesn't exist, or that it will take a long time. It means we are wandering forward in thick fog, and won't see superintelligence coming until we run right into it.
Nor can we figure out how to get to Valhalla. But that doesn't mean Valhalla doesn't exist. It just means we've not figured how to get there. But we'll know we're there when Odin opens the gates and invites us in.
Nor can we, the people of 1901, figure out how to make a commercial airplane that can seat 100 people. That doesn't mean such an airplane is impossible. But we'll know we're there when the stewardess opens the Boeing 707 and invites us in.
We can't even prove that superintelligence can theoretically exist, or that we have any means to achieve it, or that the methods we are employing could do so.
We don't even know what intelligence is, let alone superintelligence. That doesn't mean there's a 0% chance of a super intelligent AI taking over the world or killing everyone. It should mean that we don't consider this point more strongly than other completely unknown possibilities. The same group of people who are most worried about AI starting the apocalypse seem to almost universally reject any other kind of apocalypse that we can't rule out (see, e.g., every theological version of apocalypse).
> We can't even prove that superintelligence can theoretically exist
Meaning you're unsure it's possible to be smarter than humans? Is there some other definition of superintelligence?
>But then the actual capabilities of existing AI and any AI that's forseeable from current tech, are all but irrelevant to AI x-risk discourse.
We cannot yet align the systems that exist. This is absolutely relevant. Aligning a superintelligent machine will be much, much harder.
> It's mostly a fantasy built on magical entities – "super-intelligence"
There's nothing "magical" about it, unless you "magically" think brain tissue does something that silicon never can.
> using magical powers – "recursive self-improvement."
Again, nothing magical. If we can make AIs smarter, why the heck couldn't AI past a certain intelligence threshold make AIs smarter?
Thank you for this very clear writeup of one of the distinctions! This provided me with quite some new insights! :)
Tyler's reference to Knightian uncertainty and Hayek is a gesture at the idea that no, in fact, you can't and shouldn't try to make predictions with hard-number probabilities (i.e. 33% chance of AGI doom). Some risks and uncertainties you can quantify, as when we calculate a standard deviation. Others are simply incalculable, and not only should you not try, but the impulse to try stems from pessimism, a proclivity toward galaxy-brained argumentation, and an impulse toward centralized control that's bad for the economy. In these matters, no a priori argument should affect your priors about what will happen or what we should do - they provide zero evidence.
His all-critical China argument is that if we don't build AGI, China will. Slowing down or stopping AGI is something like a fabricated option [1], because of the unilateralist's curse [2].
So if you had to choose between OpenAI building the first true AGI and a government-controlled Chinese AI lab, which would you pick? I expect Tyler is also meaning to imply that whatever lead the US has over China in AI development is negligible, no matter how much we try to restrict their access to chips and trade secrets, and that the US and China and other players are unlikely to be able to stick to a mutual agreement to halt AGI development.
I agree with Tyler that Scott misrepresented his argument, because while Tyler does emphasize that we have no idea what will happen, he doesn't say "therefore, it'll be fine." His conclusion that "We should take the plunge. We already have taken the plunge." is best interpreted as meaning "if you don't have any real choice in whether AGI gets built or not, you may as well just enjoy the experience and try to find super near-term ways to gently steer your local environment in more positive directions, while entirely giving up on any attempt to direct the actions of the whole world.
I think that the fundamental inconsistency in Tyler's argument is that he believes that while AGI development is radically, unquantifiably uncertain, he is apparently roughly 100% confident in predicting both that China will develop AGI if the US slows down or stops, AND that this would be worse than the US just going ahead and building it now, AND that there's nothing productive we could do in whatever time a unilateral US halt to AGI production buys us to reduce the unquantifiable risk of AGI doom. That's a lot of big, confident conjunctions implicit or explicit in his argument, and he makes no argument for why we should have Knightian uncertainty in the AGI case, but not in the US/China case.
We can point to lasting international agreements like the nuclear test ban treaty as evidence that, in fact, it is possible to find durable diplomatic solutions to at least some existential risk problems. Clearly there are enormous differences between AGI and nuclear bombs that may make AGI harder to regulate away or ban, but you have to actually make the argument. Tyler linked today on MR to a well-thought-through twitter thread on how to effectively enforce rules on AI development [3], saying he's skeptical but not explaining why.
In my view, Tyler's acknowledging that the risk of AGI doom is nonzero, I'm sure he thinks that specific scenario would be catastrophically bad, he explicitly thinks there are productive things you could do to help avert that outcome and has funded some of them, he tentatively thinks there are some well-thought-out seeming approaches to enforcement of AI development rules, and he demonstrates a willingness to make confident predictions in some areas (like the impossibility of meaningfully slowing down AI development via a diplomatic agreement between the US and China). That's all the pieces you need to admit that slowing down is a viable approach to improving safety, except he would have to let go of his one inconsistency - his extreme confidence in predicting foreign policy outcomes between the US and China.
I think Scott, despite the hard number he offers, is the one who is actually consistently displaying uncertainty here. I think the 33% figure helps. He doesn't need to predict specific scenarios - he can say "I don't know exactly what to do, or what will happen, or how, but I can just say 33% feels about right and we should try to figure out something productive and concrete to lower that number." That sounds a lot more uncertain to me than Tyler's confident claims about the intractability of US/China AI competition.
[1] https://www.lesswrong.com/posts/gNodQGNoPDjztasbh/lies-damn-lies-and-fabricated-options
[2] https://forum.effectivealtruism.org/posts/ccJXuN63BhEMKBr9L/the-unilateralist-s-curse-an-explanation
[3] https://twitter.com/yonashav/status/1639303644615958529?s=46&t=MIarVf5OKa1ot0qVjXkPLg
> So if you had to choose between OpenAI building the first true AGI and a government-controlled Chinese AI lab, which would you pick?
Two organizations at least doubles the chances that one of those AIs is misaligned. I don't think your question has the easier answer you seem to imply. If China's AGI is aligned, or if they have a greater chance of creating an aligned AI than OpenAI, then that option could very well be preferable.
I’m trying to rearticulate Tyler Cowen’s argument, not state my own views, to be clear.
That's clearly logically not true.
If slumlords in Mumbai are going to build housing for 10,000 people, the risk of a catastrophic fire that kills at least 100 is X%.
If also, normal housing developers in the US are going to build housing for another 10,000 people, the risk of catastrophic fire is not "at least 2*X%."
The two organizations pursuing AI are largely using the same techniques that are shared by most machine learning researchers. By contrast, building standards and materials in Mumbai slums and the US suburbs differ drastically, so your analogy is invalid.
The incentives of Chinese and US researchers are slightly different, but 2x factor is fine for the ballpark estimate I was giving. Don't read too much into it, the point is that risk scales proportionally to the number of researchers, and this is only mitigated somewhat by incentives that optimize for specific outcomes like, "don't create an AI that destroys the CCP's ability to control the flow of information in China".
This implies China is significantly less likely to align their AI. There's little basis for this. Even if China is more likely to make unaligned AI, this is dwarfed by the increased likelihood of AGI in the next 50 years with both countries working on it.
I think this is completely wrong, and shows some of the sloppiness of thinking here.
Making AI -- aligned or unaligned -- isn't a matter of rolling dice. Either the current set of techniques are basically on a clean path to AGI, or they aren't, and some further breakthrough is needed. If the current techniques are heading towards AGI (if scaling is all we need and maybe some detail work on tuning the models, but no fundamental breakthroughs or complete changes of approach needed), then AGI is going to happen on a pretty straightforward timeline of training data + more GPUs, and whether two countries are working on it or one or five is unlikely to change that timeline in a macroscopic way.
If AGI is coming on a short timeline with fundamentally the techniques we have today plus more scaling, then, again, whether it's aligned or not isn't a matter of rolling dice. Either the techniques we use today, plus perhaps some ones that we learn over the course of that scaling up process, produce an aligned AGI or an unaligned one. They're pretty much either sufficient or insufficient. Again, whether there's one AGI or several isn't a very large factor here.
Thanks for this write-up, I think it’s very well done.
>> "Come on Scott, you're just not understanding this...for a start, consider the whole post!"
I'm a big fan of your work and don't want to misrepresent you, but I've re-read the post and here is what I see:
The first thirteen paragraphs are establishing that if AI continues at its current rate, history will rebegin in a way people aren't used to, and it's hard to predict how this will go.
Fourteen ("I am a bit distressed") argues that because of this, you shouldn't trust long arguments about AI risk on Less Wrong.
Fifteen through seventeen claim that since maybe history will re-begin anyway, we should just go ahead with AI. But the argument that history was going to re-begin was based on going ahead with AI (plus a few much weaker arguments like the Ukraine war). If people successfully prevented AI, history wouldn't really re-begin. Or at least you haven't established that there's any reason it should. But also, this argument doesn't even make sense on its own terms. Things could get really crazy, therefore we should barge ahead with a dangerous technology that could kill everyone? Maybe you have an argument here, but you'll need to spell it out in more detail for me to understand it.
Eighteen just says that AI could potentially also have giant positives, which everyone including Eliezer Yudkowsky and the 100%-doomers agree with.
Nineteen, twenty, and twenty one just sort of make a vague emotional argument that we should do it.
I'm happy to respond to any of your specific arguments if you develop them at more length, but I have trouble seeing them here.
>> "Scott ignores my critical point that this is all happening anyway (he should talk more to people in DC)"
Maybe I am misunderstanding this. Should we not try to prevent global warming, because global warming is happening? If you actually think something is going to destroy the world, you should try really hard to prevent it, even if it does seem to be happening quite a lot and hard to prevent.
>> "Does not engage with the notion of historical reasoning (there is only a narrow conception of rationalism in his post)"
If you mean your argument that history has re-begun and so I have to agree to random terrible things, see above.
>> "Does not consider Hayek and the category of Knightian uncertainty"
I think my entire post is about how to handle Knightian uncertainty. If you have a more specific argument about how to handle Knightian uncertainty, I would be interested in seeing it laid out in further detail.
>> "and does not consider the all-critical China argument, among other points"
The only occurrence of the word "China" in your post is "And should we wait, and get a “more Chinese” version of the alignment problem?"
I've definitely discussed this before (see the section "Xi risks" in https://astralcodexten.substack.com/p/why-not-slow-ai-progress ) . I'm less concerned about than I was when I wrote that post, because the CHIPS act seems to have seriously crippled China's AI abilities, and I would be surprised if they can keep up from here. I agree that this is the strongest argument for pushing ahead in the US, but I would like to build the capacity now to potentially slow down US research if it seems like CHIPS has crippled China enough that we don't have to worry about them for a few years. It's possible you have arguments that CHIPS hasn't harmed China that much, or that this isn't the right way to think about things, but this is exactly the kind of argument I would appreciate seeing you present fully instead of gesture at with one sentence.
>> "Or how about the notion that we can't fix for more safety until we see more of the progress?"
I discussed that argument in the section "Why OpenAI Thinks Their Research Is Good Now" in https://astralcodexten.substack.com/p/openais-planning-for-agi-and-beyond
I know it's annoying for me to keep linking to thousand-word treatments of each of the sentences in your post, but I think that's my point. These are really complicated issues that many people have thought really hard about - for each sentence in your post, there's a thousand word treatment on my blog, and a book-length treatment somewhere in the Alignment Forum. You seem aware of this, talking about how you need to harden your heart against any arguments you read on Less Wrong. I think our actual crux is why people should harden their hearts against long well-explained Less Wrong arguments and accept your single-sentence quips instead of evaluating both on their merits, and I can't really figure out where in your post you explain this unless it's the part about radical uncertainty, in which case I continue to accuse you of using the Safe Uncertainty Fallacy.
Overall I do believe you have good arguments. But if you were to actually make them instead of gesture at them, then people could counterargue against them, and I think you would find the counterarguments are pretty strong. I think you're trying to do your usual Bangladeshi train station style of writing here, but this doesn't work when you have to navigate controversial issues, and I think it would be worth doing a very boring Bangladeshi-train-station free post where you explain all of your positions in detail: "This is what I think, and here's my arguments for thinking it".
Also, part of what makes me annoyed is that you present some arguments for why it would be difficult to stop - China, etc, whatever, okay - and then act like you've proven that the risk is low! "Existential risk from AI is . . . a distant possibility". I know many smart people who believe something like "Existential risk is really concerning, but we're in a race with China, so we're not sure what to do." I 100% respect those people's opinions and wouldn't accuse them of making any fallacies. This doesn't seem to be what you're doing, unless I'm misunderstanding you.
I'm actually not convinced by the China argument. Putting aside our exact views on the likely outcomes of powerful AI, surely the number one most likely way China gets a powerful AI model is by stealing it from an American company that develops it first?
That's broadly how the Soviets got nukes, except that AI models are much easier to steal and don't require the massive industrial architecture to make them run.
Worse: stealing AI models doesn't require the massive infrastructure to *train* them, just the much more modest infrastructure to run them. There are LLMs (MLMs?) that can run on a laptop GPU, I don't think we'd even contemplate restricting Chinese compute access to below that level even if we could.
Disagree. China will able to produce powerful AI models. There are many Chinese researchers, and they do good work. China might be slowed down a bit by U.S. export limitations, but that's it.
I actually agree with you; the Soviets still would have developed nukes eventually without espionage, but it's pretty clear it would have taken longer, and I think this situation is comparable (with the noticeable difference that stealing the plans / data/model for AI is effectively like stealing the nukes themselves.
Stealing an AI model from the US would not increase existential risk much if the US companies are not allowed to train models more advanced than GPT-4.
The CHIPS act will give china a large disadvantage in compute, and they already have a large disadvantage in the availability of top talent because if you're a top 1%er you don't want to live in China -- you go study in the US and stay there.
Whenever i hear a definitive statement on China that’s basically dismissing chinas potential (or threat) a quick google contradicts it.
https://www.reddit.com/r/Futurology/comments/129of5k/as_america_obsesses_over_chatgpt_its_losing_the/
Also plenty of Chinese graduates and post graduates go back to China.
I expect Chinese and Americans will produce different designs for AGIs, and more generally two AI researchers would produce different designs.
On the one hand, two different designs would give two chances for an AGI design to kill us all. On the other hand, if there are two designs, one might be safer in some clear way, and conceivably most people could be persuaded to use the safer design.
Edit: I don't know the first thing about Chinese AI, but a top comment on [1] says
> I am not a defense expert, but I am an AI expert, and [...] [China] certainly is not leading in AI either."
> Urgh, here's what China does. China publishes a million AI "scientific" papers a year, of which none have had any significant impacts. All of the anthology papers in AI are from USA or Canada. Next year China publishes another million useless papers, citing other chinese papers. Then if you naively look at citations you get the impression that these papers are impactful because they have lots of citation. But its just useless chinese papers citing other useless chinese papers for the purpose of exactly this: looking like they are leading.
Another commenter adds
> The really most influential AI breakthroughs in 2022, IMO:
> DALLE-2 - openAI, USA
> Stable Diffusion, LMU Munich Germany
> ConvNeXt, Meta AI, USA
> ChatGTP, open AI, USA
> Instant NGP, Nvidia, USA
> Generative AI was really big this year. What AI breakthrough was made in China? I cannot think of any important one, ever.
[1] https://www.reddit.com/r/Futurology/comments/129of5k/as_america_obsesses_over_chatgpt_its_losing_the/
What exactly is so bad about China beating Silicon Valley? You trust Silicon Valley with AI safety more than China? I am not sure that is my knee jerk reaction and I am not a particular Sinophile.
If the AI can be controlled, do you really believe that it would be better in the hands of the CCP rather than US tech companies? On what basis or track record do you make this claim? I don't recall tech companies causing millions of deaths, suppressing pro-democracy protests, persecuting religious or ethnic minorities, forcing sterilizations, stifling political dissent, or supporting widespread censorship, for example.
On the other hand, the CCP has lifted millions of people out of poverty (after previously, er, plunging them into poverty, or at least more dire poverty than they were previously experiencing). On the gripping hand, it's not clear to me that a CCP-AGI would value poverty reduction once Chinese former-peasants were no longer needed for industrial growth.
>The CCP has lifted millions of people out of poverty
Wrong. Western technology did. CCP prevented access to this technology.
And millions of *chinese* people were lifted out of poverty. I don't expect the CCP to focus on helping people in other countries, but the fact that Chinese people were improved under their watch says little about their concern for humanity in general.
>On what basis or track record do you make this claim? I don't recall tech companies causing millions of deaths,
Well they haven't really had the power to in the past. If tech companies could cause millions of deaths to pump the stock (or make their leaders putative gods (or controllers of god)) its not clear to me they would say "no".
>suppressing pro-democracy protests,
Who cares about democracy? Not important on the scale of talking about existential threats.
>persecuting religious or ethnic minorities, forcing sterilizations, stifling political dissent, or supporting widespread censorship, for example.
Their support of widespread censorship is exactly the sort of thing which might help them keep an AI under wraps. As for those other issues those are bad, but they aren't really things that are that unique, the US/West was pursuing those policies in living memory.
OMG the Chinese don't like the Uighurs, and treat them horrible is not some knock down argument they won't be safe with AI.
We can be sure the US tech company AI will make sure to use all the correct pronouns and not make anyone sad with trigger words, while it transports us all to the rare metal penal colonies in Antarctica for that one like of a Mitch Romney tweet in 2009. That is cold comfort.
Yet another point: capitalism drives people to take shortcuts to be competitive, and shortcuts on alignment are not a good idea. The CCP has a much firmer grip on what they permit, and that could be good for safety. The matrix of possibilities is:
1. China creates aligned AI.
2. US creates aligned AI.
3. China creates unaligned AI.
4. US creates unaligned AI.
It's not unreasonable to think that the probability of option 4 is higher than 3, and that the probability of option 1 is higher than 2, which would make China a safer bet if we're really concerned with existential risk.
1 It's not "Capitalism" that drives people to take shortcuts, it's laziness and incentives, which obviously also exist in non-capitalistic systems. Look at Chernobyl for just one example.
In addition, China is hella capitalistic these days.
>Yet another point: capitalism drives people to take shortcuts to be competitive, and shortcuts on alignment are not a good idea.
Private firms in China are responsible for most AI development, and in any case China does not have a history of not taking shortcuts.
1 happening before 2 (alignment in the narrow sense of doing what its operators want) could be catastrophically bad, but not as bad as 3 or 4.
I am no fan of the CCP. I despise them in fact. But should we put our faith in Peter Thiel, Mark Zuckerberg, and Elon Musk? Silicon Valley has been functionally psychopathic for at least the last decade.
If AI is on the brink of some sort of world-altering power then I can't see the Silicon Valley types suddenly deferring to ideas about the common good and the virtue of restraint when they've demonstrably behaved as if they had no interest in those virtues for years. The CCP, while awful, may at least feel constrained by a sense of self-preservation.
Exactly, I don't think this is a knock-down argument, but it is one that demands more of a framework to oppose than "OMG China/other bad".
>I am no fan of the CCP. I despise them in fact. But should we put our faith in Peter Thiel, Mark Zuckerberg, and Elon Musk? Silicon Valley has been functionally psychopathic for at least the last decade.
Thiel and Musk both express concern about AI risk, much more than the median voter or politician
That seems like a fairly solid CV of being willing and able to lock human-level sapient beings in boxes and subject the nuances of their loyalties to unrelenting scrutiny, which seems extremely relevant to the classic "distinguish a genuinely friendly, submissive AI from a malevolent trickster" problem.
I don't actually think that's the best way to approach AI alignment, or for that matter running a country - long term growth requires intellectual freedom. But for somebody who figures there'll be a need to censor, persecute, and sterilize paperclip-maximizers, "move fast and break things" is not a reassuring slogan.
"I don't recall tech companies causing millions of deaths, suppressing pro-democracy protests..."
I absolutely do. It was in 1930s Germany, not 2030s America, but I don't have that much more faith in the American political system. It's good, but I wouldn't bet the farm on it. And America's politics took a sharp turn to the left/right/wrong, I have no faith that its tech companies would do anything other than pander. Germany's tech companies supported the war efforts.
China's definitely worse at present. But if we're making predictions about what might happen in the future, you can't just make the blanket assumption that the political truths of now will continue into the future.