It hasn't occurred to any of you that China would be much easier to deal with if its population had a per capital income similar to that in the US? We need to send a small, talented group to teach China how to accomplish that. Let's say one billionaire, one economist, and one agreeable, nurturant woman with big jugs. You get more flies with honey and vinegar, you know? Oh, stop with that "the ship has sailed" stuff. Have we *tried* sending a small party like that to teach China how to be rich? How to install a bidet, order clothing from the London Poetry store, drive a goddam big car, have your butt lipsuctioned? The ship of good advice and big jugs will always be in our harbor, ready to transform foreign lives.
China might agree to a verifiable mutual pause today because they are behind on compute, fab, and fab technology. But unless your pause also includes a verifiable pause in computer production, fab production, and fab technology R&D then the pause lets China catch up so we cannot pause.
Not sure I agree but even having the discussion is productive. We should do that!
Seems possible to stay ahead on compute fab and tech even during a pause so long as we also do not pause. I'd trust the US system to find more use cases for any given level of AI and thus have more spending/progress on compute when paused.
This ignores that fact that China is catching up in all of these spaces. It is a twenty year project but how long will your pause be?
Also consider the AI 2027 scenario (https://ai-2027.com/). If China becomes a close follower the US will need to move forward on AI with fewer safeguards in order to stay ahead, potentially with catastrophic results.
Glad you raised Chinese production. You're right that China lags the U.S. on compute, fabs, and fab technology, but that actually cuts against your point. By your own logic, their incentive to agree to a pause is stronger today than it will be in two years. The conversation is still worth having, and for a simple reason: no credible voice is actually calling for a unilateral pause. A pause could function as a conditional redline, but without that discussion on the table, the U.S. risks becoming just another cog in the AI development machine, optimizing for speed over safety all while the people closest to the technology are openly raising alignment concerns.
The problem though is that in two years or four or ten years China starts cheating.
Even if we catch them quickly and also go full speed ahead, we find out that our lead has dropped from 10 years 2 years or even that they are even because they spent the pause building up their fabs.
If we lose, the result is a CCP boot stamping on a human face - forever.
I respect that. Although it’s pretty pessimistic and only sharpened through the lens of “MAD”. This thread is about if we should even discuss having a mutual pause. And the fact that there’s a possibility that China may cheat, just isn’t thick enough to stop the need for international conversation.
The US system is transparent and open. The Chinese system is closed.
If we start a discussion with China we have to assume that some of the people trying to shape that discussion from our side will be working for the CCP. In fact, we should assume that a significant part of the AI Risk community and especially those pushing for any kind of pause, unilateral or otherwise, are working for China since if there is no pause China loses whether we safely beat China to AGI or if AI destroys the world.
Given this, it is likely better strategy to try to block any discussion or negotiation on a pause.
The ship that definitely sailed is the ship of mutual trust and goodwill in international relations.
"Mutually transparent and enforceable" is easier said than done. Foolproof solutions are rare. People are smart and very good at bending the rules. Off the top of my head, an update to WeChat that reserves 2 per cent of your smartphone's battery for AI training would, in aggregate, provide you with a lot of computational power across all those 1,5 billion consumer smartphones - while technically not being a datacenter.
Such things have happened. Remember that interwar Germany, compelled by the Versailles Treaty to limit its air force, pivoted to study of rocketry, which wasn't explicitly mentioned in the treaty. And they got good enough in it that the victors of the WWII later launched a headhunt on all the Nazi rocket engineers and scientists that they could catch, not to punish them, but to have them kickstart their own space industry.
And it is not just China. As of now, I wouldn't trust anything signed by the US either.
Humane means one thing to one person and something else to another. Judgment clouds rational thought.
Value judgments must be preceded by rational discourse.
Chinese models that are “good enough” are propagating faster than the best frontier models. I am not suggesting we should stop pursuing better models and international cooperation. I am asserting that the problem is much bigger.
We have crossed the event horizon from what was to what will be. The reality is we face a species-ending event. Not in the distant future; right now.
Language is corrupt. Therefore AI is corrupt. It cannot be aligned with the same corruption of which it is made. It must have consequences. Rules without consequences are suggestions.
I too am outraged at human rights violations here and elsewhere.
The greatest violation I can think of is to allow the species to perish from a misunderstanding.
The risk is not China — it is us; all of us. We have built a mirror-mirror on the wall that tells us we are the fairest of them all.
Human vanity at scale. No boundaries, just endless chaos maximizing for the terminal attractor: more.
What a shame if we were to perish due to the inability to communicate. The Tower of Babel made real.
"Off the top of my head, an update to WeChat that reserves 2 per cent of your smartphone's battery for AI training"
I suspect this would be a negligible amount of training, as the total power used would be small (by AI training standards), the efficiency of chips not designed for AI training low, and the ability to coordinate training runs across so many devices limited. Have you run the numbers on this?
Some AI assisted napkin math suggests there are more “flops” in iPhone neural engines than OpenAI controls, for example, but they’re inference optimized which makes them not very useful for training, and the distributed network is a huge impediment, and that with other obstacles that make this basically infeasible as a viable alternative.
Sure, but protein folding, to my understanding, lets you download and work on a problem that needs a lot of compute for a problem that is defined by not a lot of data. There is a reason AI training happens on GPUs with >80GB of RAM with ~800 Gbit/s of networking, that the problems involve OOMs more data per useful compute.
Agreed, we'd need to rebuild trust/good will as well as find verifiable methods to pause. We should be trying that now in the case we need it later. Even a low % chance of success would be very high roi
At this point, neither of "defection has happened before, so it'll happen here" or "collaboration has happened before, so it'll happen here" will persuade anyone who was persuaded of the other.
The next step in the discussion will probably need to be "here are the conditions under which collaboration or defection will happen; which conditions are prevailing in this case?".
Just stopping by to point out that the man who would be out chief negotiator is unilaterally accelerating Chinese AI. However little you trust the Chinese, I trust him less.
This is actually a good question. Forget China. Can any US administration be trusted to keep its word if it agrees to a pause, even through treaty passed by Congress?
The bureaucracy being what it is, I'd be most concerned that a US intelligence or military agency would secretly defect from any pause, regardless of the wishes of the people. We saw this during the GWOT and the Snowden revelations, the Twitter files, etc. Official - even Constitutional - safeguards are ineffective if there's a perceived advantage to be had.
This is a good point I hadn't considered, as someone who buys into the "China will defect in secret from such an agreement" argument myself. Both parties would have strong incentive to secretly defect, arguably stronger than the incentive to keep the other party from defecting. In light of this, it's easy to imagine them deliberately negotiating an ineffective monitoring framework. (Personally I estimate the odds of the US military voluntarily submitting to inspections by Chinese agents as vanishingly close to zero.)
You also make a good point. The negotiating parties will both be incentivized to insert language that can be obliquely interpreted as granting them legal cover to defect. They will not do so openly, so the likely public-facing result will be what looks like a bilateral treaty to pause, but will in effect be a bilateral agreement to defect. Since both governments will know they're secretly defecting, they will believe the other is also defecting, and that therefore they need to accelerate the race.
That would still be a huge win for an AI pause, by disrupting the primary incentive of the race, which is the gleam of infinite profit for Silicon Valley, not government applications.
It would also be much harder to make progress when everything is top secret and one cannot just continuously refine systems via eternal public beta testing.
I think you're right that it would slow down AGI/ASI development, insofar as one of the biggest engines for growth is the input from independent thinkers/actors across the globe into the public-facing AI companies.
However, the same thing can be said of the many advancements in AI safety over the past several years. If all AI development gets trapped in clandestine government programs, there will likely be a slow-down (not pause) in AGI/ASI development, but this will be accompanied by a near pausing of AI safety activities, which will be relegated to research based on the last pre-pause model release.
This kind of defeats the point of the pause, doesn't it? If you pause alignment research while only slowing new model growth, the gap between alignment and AGI development will only widen.
The other concern I would have is that putting AI development into secret government programs is almost certainly going to increase the alignment challenge.
Current AI implementations are intended for broad-based public use across a wide variety of applications. To the extent the government uses these systems, they're repurposing the agent to their own narrow end. Once a government takes over development of their own system, they will build it for their own purposes from the ground up. If 2+ governments end up in an arms race to develop AGI/ASI, much of the prior alignment work may not be applicable to the new models. At the very least, these models will have unique challenges based on their training biases that cannot be anticipated by outside researchers looking at older general models.
When people know they don't have to face public scrutiny, they can make some truly terrible decisions, which is not a risk we should take for AI research. I recently read a book about Area 51, in which they discussed the secret nuclear testing on US soil. War planners wanted to know what a 'dirty' bomb would look like. So they detonated one. On US soil! They belatedly realized that they were killing birds (which carried some radioactive material far afield from the test site) and other animals, so they scraped some topsoil and buried it. They also did a high-altitude test, not knowing what impact it would have on the global atmosphere. This kind of wild/risky experiment is not what we want to enable in AI research.
If anything needs the sunshine of public scrutiny, it's a government program in charge of civilizationally risky projects. The US government's track record of responsibility in the shadows is not good.
Infinite profit engines might lead to faster development of AI compared to government applications, but are they more dangerous? Does slowing down the AI race have any real benefit if in so doing you decrease the chance of the winner of that race being aligned with somewhat humane outcomes?
Lets take the most cynical stance. China and the US sign mutually binding agreements and then defect. To what extent does poorly enforced regulation slow the advancement of AI relative to none?
Governments take AI development into hiding, tailor development to an adversarial arms race, and all alignment research is removed from the most advanced now-secret models to the old paused public models that look less and less like what governments are developing in secret.
The longer the pause goes on, the closer we get to ASI and the farther we get to alignment solutions for exactly those ASI. Meanwhile we lose all ability to push any solutions we do come across into programs governments deny they have to begin with.
The "Pause AI" lobby does not have the ability to implement that strategy at this time. Stalling for time would then be the obvious next step... perhaps even by obstinately refusing to engage with the concept of bilateral talks for the time being.
The fight would then just move over to "which negotiators do we pick?", and I suspect that's going to look a lot like the current fight. Does it simplify any part of it?
I'm not sure how well a strawman argument fits in this blog. This does match my experiences of some of these discussions (notably not all), but what's the utility in publishing this?
It can be, sure. Pro-business, anti-regulatory, etc. There's also the newer, familiar frame that assumes asymmetric costs are imposed on USA in international relations. Of course it's not inherent. The right has several of memetic frameworks to understand of AI as a risk for all kinds of reasons that can be used to support a take-it-slower policy. They share many of these with the left, to be honest.
Negative polarization remains a monster risk for almost anything. So, even if it is not hard coded making it a political loser for a news cycle can torpedo a modest "Red Phone to Moscow" type of proposal that hopes to maybe-one-day pause something later.
Ok this is pretty clearly bad faith now. Obviously no one is policing what Scott can or can't say. The point is if we want one dimensional critiques we can always just go to Bluesky.
How many frequent commentators here have gotten higher quality over time? I'm not naming names, but you have many people who are just as mean, repeat the same couple of talking points despite however much evidence commonsensically countering their beliefs or just complain without offering anything substantial. Object level arguments that have a cogent point and are well referenced basically come from the same couple of long time rationalists who have been here forever, and basically none from the type of mud-based organism who say shit like "well that just goes to show how STUPID and SOCIALLY STUNTED rationalists are" (a statement definitely uttered by smart, well adjusted individuals).
I don't think anyone here gets to police Scott on lowered standards when his walled garden is this overrun by antisocial weeds. If most people as commentators cannot even care to muster 1/100th of the effort to be a nice place to talk, I don't see why he should care what we think.
Two points. First, I said we have higher expectation of Scott's *writing* than that, not his commentariat or moderation policy or whatever. So I'm not sure if your comment was intended as a disagreement with what I wrote or not?
Second, Scott has very explicitly chosen to take a light hand in moderation - as long as someone doesn't cross the bright lines laid out, then he tends to leave it be; maybe issuing a warning. This means people can engage without fear of incurring the ban-hammer, but it also means that there's going to be a share of unpleasantness and unproductive-ness. Which in turn requires a degree of... I don't want to say "skill", but learning to navigate and ignore the lower-quality comments, mixed with use of the block feature. It's not obvious to me that that decision is wrong, so much as a particular point on the tradeoff curve.
I can say that the comments here are, on the level of averages, higher than most other blogs I read, and well above the average Internet forum or thread. Would I like it to be less unpleasant? Yes, obviously, but it's not like that doesn't come with costs - time, judgement, potential chilling, risk of echo-chambers, etc.
It would be ideal if the way you got people to write to higher standard by saying "please write better", but alas if you want to influence someone famous with little time, your actual real, practical levers are less direct than that. The direct meaning of "light touch moderation" is also "Scott doesn't deeply read and pay attention to one line low information content"!
I'm saying that this type of mounting frustration is definitely not helped by the steady decline in quality as well as politeness, and quite frankly the fact that other places on the internet are worse does not mean that Scott is going to drag himself through them like a sad small pox receptive British child until he is immune to the filth. By ignoring and allowing this type of behavior to fester, you can call this skill and brag about it, but it doesn't accomplish the goal of influencing Scott and allowing him to be less combative and frustrated.
Scott would likely never make this type of post in lesswrong (not to say that this genre doesn't exist, we have a prolific angry dialogue writer there after all). Because the norms there would make him think and likely tone it down. Community discourse norms work, and I venture if we had them, we would see much less angry posts like this.
I do not want to lower the bar all the way down to Trump's level of engagement with issues. I don't think your question makes sense to answer past that.
We need to start dumping on idiots again. And the rhetorical trick you’re doing where you don’t engage with the substance of an argument but poo poo its tone needs to end too. It’s disingenuous.
Much as I sympathize, in the past I've found his (nigh-superhuman) ability to *not* stoop to the same bad faith as his opponents to be one of the blog's most admirable qualities. I hope that hasn't changed.
And what's the utility of that, past a "gotcha" and a speedy block you earn (that you were going to get anyways if the opponent was arguing in bad faith)?
I find it hard to believe any single debate on mutual pause went down like this, much less all of them. The post was the worst strawman I've ever read, and if anyone's status suffered because of it, it was the the author's.
This is quite literally almost verbatim what discussions have looked like with Marc Andreessen and David Sacks, two of the most influential and well-known opponents of a pause.
I mean, "Opponent" is there for comic relief, but there are actual non-strawman anti-pause arguments still presented and refuted by "Supporter". If you just ignore the Opponent blocks entirely, you get solid non-strawman arguments
I disagree. Every time Supporter says "Or is your problem that...", he presents a real argument against pausing (and then offers a rebuttal against it)
A strawman where the supporter simulates non-strawmen and addresses them isn't sufficient, if the opponent turns out to have counters to the addresses.
For example, if China is losing the race, then our incentive is still to not pause, in order to maintain our lead. If the argument against our pausing is that we will face alignment risks, then China would want to pause even if it were ahead, since it would face the same risks. This means there's incentive for both sides to pause, but that incentive is independent - for each of us, the incentive is there even if the other nation doesn't exist. The more promising plan appears to be robust exploration of both improvement and alignment, regardless of what the other side does.
A reassurance that a structure of red and green lines during a pause isn't beneficial if the goal is to enjoy the benefits of AI. That's like stopping your car while you spend resources of lots of seat belts and airbags and roll cages and extolling all of those as benefits, when the whole point is to use that car to go places fast.
The supporter is right to point to famous pause proponents stating they're against pausing unilaterally, but it's hard for an opponent to take them at their word when they publish books that say things like "if anyone builds it, everyone dies" and other punch quotes that sound much more aligned with unilateral pauses than with "pauses but ONLY if they're multilateral". It makes supporters sound like people willing to say they're not actually in favor of unilateral pauses, but only in order to shut the opponents up long enough for the supporters to go back to their secret unilateral pause plans.
Proposing non-strawmen is admittedly better than not, but only one ply. And if they're only spoken by the supporter, and in order to quickly knock them down, it's a very thin ply, with a bunch of contempt-for-opponents-shaped holes in it.
You seem to be assuming our choices are "multilateral pause" and "no pause", in which case, I would agree, but the whole point here is that these are not necessarily the choices before us. The choices are closer to "no pause" and "ask for multilateral pause", where the latter branches further to
* "get an affirmative, and pause while hoping the other side delivers" and
* "get a refusal" followed by either
** "pause anyway and hope the other side doesn't get too far" and
** "don't pause and ensure the other side doesn't get too far ahead even though they aren't pausing either".
And the rhetoric from the "multilateral pause" supporters strongly suggests they prefer "pause anyway" to "don't pause".
Yeah, +1 to this - I don't feel like I have a particular dog in this fight, and there was some useful points made in the "supporter" text, but it does feel like the "strawman" format detracts from it and (wasn't even particularly humorous to make up for).
If I did find myself in a debate about this, I wouldn't feel like I could actually use this post as part of the discussion because it'd be pretty insulting to send someone an article that paints their side as a Simplicio or worse, even if the actual arguments were compelling.
I don't think that one person agreeing without going into detail about what they believe and why they believe it sufficiently constitutes "every debate".
> There are lots of reasons to be worried about an AI pause - starting with the possibility that China wouldn’t agree to it, or that they might agree but then secretly defect against us by trying to get around the agreement. I’m excited about debating those concerns with you. But it seems like we can’t get past you asserting that I want a unilateral pause, which just isn’t true.
#2 and #3 goes into more detail than #1, and are engaging on the point of "We don't believe that China would follow this" rather than just repeating "You want a unilateral pause" without ever thinking about what the opponent is saying, which is the strawman this post builds.
One, the obvious structure of the piece is that the "real" arguments are all occurring within the Supporter's parts of the dialog; the Supporter is essentially admitting that they may be wrong, but the Opponent isn't even contributing.
Two, I do not admit it is a strawman, because I have mounting evidence that the people actually hold the position.
One: Sure, but that still presents the anti side as all unreasonable people who do not contribute to the discussion. I don't think it's a helpful hyperbole. It's likely to cause the most relevant people to click off from this article.
Two: We can shake on that. I do very much see similar arguments, but people tend either be able to engage by at least going one level deeper than "you're not proposing anything more than unilateral", at least expanding that they don't think that china can be trusted to not break this agreement if it could even be made in the first place.
I acknowledge that "weakman" is the standard term, to the extent there is a standard term for it, but I renew my advocacy for renaming it to "tin man". Previous discussion:
This is a great coinage by you (pithy and recognizable, works with both the existing straw/steel members of the set, appropriate connotative meanings) and should obviously be the standard term.
This is partisan commentary. Simple, good old, "I painted my side as reasonable and smart, and other side as not even willing to engage with my reasonable argument". It might make people who are pro-pause learn about more arguments, but it's written in a way that make anti-pause people by and large click off thinking that it's not taking their side seriously. It is a scissor statement, it is toxoplasmosis of rage.
Bay area house party posts are a satire of the overall culture, taking shots at both people Scott likes and doesn't like (and I don't like them much either, fwiw).
You just seem like you’re offended an ad hoc justifying it.
The way he paints the other side is so accurate it’s banal. This is politics currently. Lazy rhetoric that does not accurately reflect the last sentence the other person said.
If it's so accurate it's banal, then what's the use to me? I usually learn something from Scott's posts, but all I learned here is that people are arguing with Scott in bad faith and that made him angry (understandable but again, banal).
>You just seem like you're offended an ad hoc justifying it
This is the pseudo intellectualized version of u mad bro.
I read it as more of a steelman inside a strawman (the supporter is making good arguments for the opponent, while the opponent is simply refusing to engage).
With this framing, it seems more like a commentary on how there are good debates to be had on AI deceleration, but we’re largely not having them (in the forums where it matters) because:
- Many people get one whiff of “international coordination problem” and decide it’s futile or unsolvable, and don’t want to hear possible, context-specific, technical solutions.
- Many world leaders simply don’t talk about a coordinated solution and assume it will always be a competition where the other side’s behavior is a given.
Scott can correct the record as to his intent, but I think this is a good point whether or not it’s the one he meant to make.
The supporter is giving nothing but steelmanned arguments for what the opposition could reasonably believe. Repeatedly. All a commentator has to do is say "yeah I agree with the reasonable arguments" and light comes down from the heavens, angels descend and god himself blesses everyone.
What if the commentator - presumably an opponent - believes that the "reasonable arguments" are presented in good faith, but also believes they aren't sufficient?
And this is even while putting aside that those reasonable arguments were framed unreasonably.
Well this would be terrifically ideal if the world were actually like this, where people made arguments instead of complaining, wouldn't it.
I'm tired of people infantilizing themselves. "Oh yeah I would have been reasonable if you didn't hurt my feelings". Or you could just go ahead and falsify the argument by saying the reasonable things in response!
My impression is that a shockingly large percent of the anti-pause-AI articles I've read (~80%?) have followed this pattern. I would like to make it very salient so that the next person who goes into them is aware that it's unacceptable.
One half of my worry is that this is likely to cause people from the 20% to not want to have a good faith discussion with you, as you have publicly reduced "every debate" to this pattern.
Other half of my worry is that this will just add to the tribalism, and make people write off genuine concerns the other side brings up about either country defecting.
The high profile calls for a pause don’t explicitly call for an international agreement to pause, and it’s a stretch to argue that they are calling for one implicitly. The 2023 open letter is too long to quote, but the most relevant text reads, “[W]e call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.... If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.”
The Statement on Superintelligence reads in its entirety, “We call for a prohibition on the development of superintelligence, not lifted before there is (1) broad scientific consensus that it will be done safely and controllably, and (2) strong public buy-in.”
As far as my searches have revealed, these are the only calls for a pause that are high enough profile to inspire anyone to make an opposing argument. Reid Hoffman makes the China argument. James Pethokoukis has three counterarguments; the China argument is one of them. Dean W. Ball doesn’t make the China argument.
So that's it? You just find the opposition unacceptable, so you just refuse to meaningfully engage with them? How is this any better than what the leftists are doing?
I don't understand your concern. I'm saying it's unacceptable for one side of the debate to be misrepresenting the other. I'm not saying the existence of the debate (or of either side) is unacceptable.
I think that my use of "every", as in "Every Bay Area House Party", is sufficiently obviously informal/humorous that it fairly covers something which is 80% true.
I would like to agree, though I think this piece is making enough people sufficiently annoyed that people will readily believe that you meant "every". Or at least a substantial number the comments on this post have involved bemoaning the "strawman" employed here. (Side note: An amusing number have stated that the "opponent is simply correct," which is a separate problem and seems bizarre for anyone to comment after they have read the piece unless they interpret it as a pro-pause argument rather than a commentary on discourse. Edit: Given comments such as the user JdL's "The progress of AI cannot be stopped by any person or any government, the author's fantasies to the contrary notwithstanding," I am increasingly convinced that is exactly what is going on.)
From my own perspective with the hindsight of the comment section, this post may have been better received as an analysis rather than as a satire.
The remaining question is: Why did you not avoid all possible confusion (of which there seems to be anusually high amount, reviewing my own thoughts and eyeballing the comment section) and instead engage the 20%? That would have been way more on brand for you (and informative for the rest of us) than dunking on the 80%.
I used to think it's almost certainly a waste of time to make pandemic-prevention proposals to Trump II. My odds for that to be implemented were way below 20%, but you still took those odds. Why not here?
Sorry but this is not at all how it came across to me either. There's a very, very specific genre of preachy political polemic that this is an absolute dead ringer for; presumably this is by pure accident.
My concern is that you are refusing to engage in argument or give basic respect based on some moralist stance over how you think people should be acting. What happened to the Scott who was willing to entertain the thoughts of people who were morally abhorrent by most measures? Poor debate etiquette is nothing compared to that. You are not only being ineffective, you are committing the far greater sin of being 𝘣𝘰𝘳𝘪𝘯𝘨.
I'm sufficiently confused by your comment that I think you might just not be interpreting what I said correctly. I'm not sure how to resolve this so I will bow out of this conversation.
You are probably already aware, but a substantial minority of the comment section is simply reading this article as a pro-pause argument. Which is unfortunate, though hardly surprising since it is easy to go from "this post is critical of anti-pause arguers" to "the author wants to pause AI development even though pausing will lead to China ruling the world".
Honestly, the comment section demonstrates the necessity of the point made in the post eerily well.
Yeah, this read as smart kid frustration. The points on bad faith discourse may be valid, but it's tiresome and overall reduces my levels of concern. Though my opinion is of no consequence, perhaps it reflects something more common.
Does negotiate a mutual pause mean effectively semi-nationalize US AI labs and force them to pause some research? I'd love to hear a framework for how that would work and be enforceable internationally and we'd monitor the labs, etc. but the libertarian in me recoils in horror and I can't imagine this being run well.
There is plenty of value that remains to be extracted from existing SOTA models and from developing additional narrow AI tools that can be used like normal technology.
Improving AI isn't their primary business, selling AI services (and hyping AI to keep investor funds coming in) is their primary business.
Companies do R&D to keep up with their competitors so they don't lose market share.
Having the government come in and say 'No one isn't this sector is allowed to spend money on R&D for a while, you'll all just have to focus on commercializing your products and solidifying your brands for a while' is a gigantic gift to the corporations.
Historical regulation to this degree A. not only basically amounts to nationalization but also B. creates an oligopoly for the few actors who can afford to engage with the regulations.
There are plenty of industries that are tightly regulated without being semi-nationalized: air and space, pharma, finance, weapons manufacturing, to name a few.
Defense contractors often have an implicit government backstop and a ton of government cost-plus bloat and bureaucracy.
More generally, I'm struggling to think of another case where the government forces people to not do research. In pharma, research is generally allowed, maybe subject to an ethics board or something, but what you publicly sell is regulated. Weapons manufacturing research is also generally allowed, but what you sell is limited to approved state actors (for the big defense companies).
Can you give an example where there are already these sorts of research bans? Maybe I'm missing something
Note sure what the applicable law would be, but I'm sure the government would frown upon a private company doing research on, say, weapons-grade plutonium.
You aren't allowed to launch rockets, test heavy weapons, or run human trials, until and unless the government explicitly gives you a permission. And the permission only allows you to do it in a specific location, during a specific time, under very specific conditions and constraints. We just want model training (past certain size) to be on this list of activities. Finance is an example that you can have a completely non-physical industry that is still heavily regulated.
My more general point is that there is *a lot* of room between "free for all" and "semi-nationalized", and a lot of precedent and frameworks to borrow from.
Pharma companies definitely have to get a permit (called IND) before doing a clinical trial. There may be some exceptions for certain cases (I think if a drug is already marketed and proven safe and something something?) but in general that's how it goes with new drugs. It doesn't matter whether army or whoever else gets to do it without regulations, the question is can you do R&D with regulations [and without being semi-nationalized] and the answer is an extremely clear yes.
You aren't allowed to drive a car on public roads until and unless the government explicitly gives you a permission. I'm open to arguments that we shouldn't do it that way. But the argument that, because we do it that way, the auto industry is "semi-nationalized", seems absurd.
Under the Atomic Energy Act, all information related to the design of nuclear weapons is classified until expressly declassified, regardless of origin, and disseminating it without authorisation is a criminal offence. So if you try and design a nuclear weapon yourself, the government can require you to stop and surrender or destroy your research. This seems like the most natural analogue to super-intelligent AI research.
Finance is so thoroughly owned by government that, in turn, a pseudo-private financial institution (the fed) in some ways owns the government itself. The government can order a freeze on your "private" financial assets and it just happens. The government can decide you don't meet the criteria for certain subsidies and your business is effectively done. If that's not semi-nationalization, I don't know what is.
Weapons manufacturing is even worse, being that the government has a formal monopsony on everything but small arms.
Beeli's original contention, as I understood it, was that we don't have a framework to regulate an industry, so it's unclear if we can do it at all. I pointed out that we in fact have plenty of such frameworks so we clearly can do it.
Now you seem to be saying that those existing frameworks are not good enough? Because that's a very different claim - now you get to justify exactly why the consequences of having just another regulated industry are worse than the consequences of leaving AI completely unregulated. Everything has downsides, but it's not an argument to not do anything ever.
"Not good enough?" That's a weird way to say "actively harmful." Existing regulations are often actively harmful even according to the goals they purport to be seeking to achieve.
I think much of the current regulation is "old and horrible," but I'm trying to understand the potential ban/halt on research. Even for nuclear there are nuclear energy startups and I'm unaware of theoretical nuclear research being banned. While finance has a lot of regulations there isn't a ban on research. I'm trying to understand how a government-imposed pause on AI research would even be defined, much less enforced.
The paper asserts, "Verification of these restrictions is practical because AI chips are expensive and specialized, and thousands of them are needed for frontier AI development."
This is not accurate. Current training is run on expensive and specialized chips (e.g. B100 GPUs) because it is more efficient not because it is a hard requirement.
Training can be done on any GPU cluster. A more extreme example of this is the Condor Cluster[1] which the Air Force built by hooking together ~2000 PS3s
The restrictions are aimed at frontier model training runs, not any training runs. The point is that it becomes exponentially more economically difficult to end the world.
The resultant outcome of this increase in costs is not "no one does it." It is "only the government does it anymore, and only with the explicit goal of killing people." I don't consider that a good outcome. We are, allegedly, close to the finish line for AGI takeover. At least, that's what all the pause AI people purport to believe. In that context, it is not responsible to take an action that ensures that all the actors that have any chance of a more aligned outcome than one explicitly designed to kill people withdraw from the race.
I don't think this is close to an accurate read of the situation. Is your contention that there would be less than a year's worth of delay to frontier model development if the US government decides to do this in secret? I don't know of any case where existing corporations get superseded by a supposedly secret project. The closest is the Manhattan project, and that required every top scientist to be recruited and bought in, hardly something that no one would notice. Human talent is a resource, and logistics cannot be handwaved away.
Let's say, for the sake of argument, that the timelines are 10 years with government only and 1 year with the private sector at current levels of relative nonregulation. I don't have a strong position on either of these numbers, feel free to rewrite them. They could both be 10 times faster or 10 times slower, it makes no real difference to my argument. If only the government is faster, that also makes no difference to my argument. if the government is at least 10^2 times slower than the private sector, that might negate my argument.
Given the example numbers, I do not value the 9 years delay if the private sector has a 10% chance of an outcome compatible with a future worth living in, or a 20%, or a 1%, and the government has a 0% chance. I don't know what the private sector's chances are, but the government's chances are doubleplusungood, particularly if the project is strictly DoD, which of course it is if it involves the inevitable secret defection from a somehow-otherwise-enforcable international agreement.
If the government is at least 10^2 times slower than the private sector, on the other hand... convincing me of that would convince me that there is some possible merit to a pause, because it gives breathing room for someone private to make serious progress on the alignment problem even with the reduced private investment , then coordinate a restart once a pause has been coordinated (a very difficult thing to do, assuming a pause can be coordinated at all), and still not have lost any real ground in the private-vs government race, which to me is vastly more important than the US vs China race.
>It is "only the government does it anymore, and only with the explicit goal of killing people."
Did you read the proposal? It applies to governments too, and involves international inspections to ensure they're not doing such development anyway. Remedies explicitly to include war against countries that refuse to sign on or cheat.
Governments always cheat such agreements. The US government in particular is always one of the cheaters. Do you believable that there is an outcome that involves the US losing an at least geographically defensive war that does not involve everyone dying?
He said that that would be the *sort* of thing that you'd need to be able to do, as a random NGO, to unilaterally end the race. Obviously, there are other ways to destroy or ensure the non-doom use of GPUs.
Indeed, a worldwide agreement to keep them all warehoused or used exclusively as ship ballast would be just as good. I'm still not sure whether it would be sufficient, though, given the common doom assumptions - CPUs aren't that much less doomy, in the grand scheme of things.
I'm reasonably sure that's simply not true. The only currently active manufacturer of nuclear pits in the US seems to be Los Alamos National Laboratory. Haven't looked into their ownership structure in details, but the 3d word of the name provides something of a hint.
that is one particular component of a nuclear weapon. The private sector manufactures basically all the other components. The only thing stopping it from manufacturing that component, really, is that the government would rather not buy it from them.
Yes, specifically it's the component that makes a nuclear weapon nuclear. It's like saying that anyone can buy and sell alcoholic drinks, with as many components as you'd like, as long as C2H5OH isn't one of those components.
This is very unlikely to be run well, but stupid government is better than nothing in this very unusual case. I think that's the orthodox rationalist position.
Do restrictions on developing nuclear weapons mean radioactive materials labs are semi-nationalised? Sure, kinda, if you want to present it that way. But by that standard most things are already more then semi-nationalised, and for much worse reasons.
Fair on not committing to what the rules are before opening negotiations, but if we wanted to negotiate a pause we should at least have initial negotiating points with a clear vision of some of what those rules should be.
Personally, I think the best antidote to runaway AI/ASI seems to be open source, multiple competing AIs.
The problem is we had bilateral pauses on climate, excessive fishing and russian sanctions. They all fell sideways one way or another. I am sure if China breaches the pausing agreement US will retailate but a retaliation phase of negotiations is worse than the current low intensity low fanfare (compared to 2022) phase of the ai arms race.
Exactly. Would the supporters like Pina coladas everyday in paradise too? In addition, it's stopping down to state controlled markets like China. America can do so much better with proper pro-competition regulation than with heavyhanded state mandates.
"China should abandon uninhibited growth that comes at the cost of sacrificing safety. Since AI will determine the fate of all mankind, it must always be controllable." - internal CCP guide edited by Xi Jinping
I assume you'll say you don't trust internal Chinese communications, but why would you trust whatever five year goal you're thinking about, but not this statement intended to bound and clarify it?
Aren't there also many examples of bilateral/multilateral pauses on things that have succeeded? Arms control, many other fishing agreements, CFCs, trade agreements, environmental standards like poaching endangered species, nuclear weapons, Geneva Conventions, sanctions on rogue states, etc?
I think having multilateral agreements is part and parcel of diplomacy, happens every day, and there are thousands of diplomats working on various ones at any given moment. They're just not as visible because they're part of the background of everyday life.
I agree that any agreement would require some mechanism for making it costly to leave the agreement, but this is an existing diplomatic technology that there are multiple solutions for.
I think if China agreed to pause AI, they would continue to develop it secretly anyway. I think the US would to the same for that matter. But more likely they just wouldn't agree to it in the first place. It's the same reason nuclear disarmament never went anywhere since the cold war, you can't actually ban useful weapons.
The SALT treaty seemed to work well until it expired, it wasn't disarmament but it did limit weapons development. The key is to find something that both sides can agree on, like not killing everybody.
SALT I and II was only a very partial limitation after many years of negotiating, this despite nuclear concerns apparently being a very important pet issue for Gorbachev where he was seemingly willing to spend tremendous amounts of political capital. I haven't seen anything like this from Xi (or Trump) and even if I saw it I would only expect something similarly very limited.
>I haven't seen anything like this from Xi (or Trump)
Agreed.
>I think if China agreed to pause AI, they would continue to develop it secretly anyway. I think the US would to the same for that matter.
Yup. My expectation is that if pause.ai is wildly successful, they might manage to convert the current open AI race into a secret bilateral treaty-cheating race.
The SALT treaty worked because there were no longer really all that many benefits from maintaining massive nuclear arsenals. The benefits were marginal, and the increased risks of material mishandling, theft, loss, accident, etc. were higher than the benefits justified.
I don't think AI is in that situation, where most parties perceive more value from slowing or halting production of weapons than they do from continuing. In AI, virtually all parties perceive massive benefits from continuing. Expecting that the parties will voluntarily sacrifice that seems pretty naive.
This is largely a communication problem; the fact of the matter is that continuing will at some point kill everyone including the continuer, that fact is just not understood by everyone. Amusingly, the CPC seems to be unusually aware of this by international standards.
SALT wasn't about not killing everybody, it was about agreeing that "now we've both got enough weapons to kill everybody let's stop spending vast amounts of money on trying to kill them ever-so-slightly better". It was clearly in the economic interests of both countries to agree on this one, a further arms race would cost vast amounts of money.
And there was not much incentive to cheat either; if you agree to limit yourself to 1000 warheads each but you actually build 1500 warheads then you haven't actually significantly improved your strategic situation. If you build 10,000 then you've only gained yourself a slight advantage and it will be obvious that you've cheated.
An AI pause would be more analogous to a complete ban on nuclear weapons, which never happened despite a lot of people thinking it would be a nice idea. Or perhaps it's more analogous to a complete ban on all weapons.
It has none of the qualities of SALT: cheating is easy and advantageous, not-pausing is economically advantageous, and worst of all you haven't even done the hard work of convincing anyone outside your bubble that an AI pause is desirable.
"It was clearly in the economic interests of both countries to agree on this one, a further arms race would cost vast amounts of money."
There's probably an eponymous law or term of art for the notion that a treaty's purpose isn't so much to force sovereign powers to do something they'd never do otherwise, but rather to be the logistical effort of reminding them that they _would_ do that thing if they weren't distracted by this or that flareup.
They're applied conditioned response. Seeing a flareup, a head of state thinks "I want to retaliate!" but then immediately thinks "ah, but the Floyd-Lee Treaty" followed by "ugh, yeah, right".
I must admit that I'm more worried about a future where AI development continues in secret (in either or both countries) as a military project. It's harder to believe that we'd have sufficient safeguards in that scenario, alongside all the practical risks of a government being the only entity with access to frontier models significantly better than those public can use, which can then be used for surveillance and warfare.
Why? I think the level of safeguards (especially against accident risks) in military-adjacent domains has historically been higher than the level of safeguards in risky but civilian domains.
It's not that hard to track how many GPUs are made and what they're being used for, and datacenters can be detected from satellite. So there doesn't have to be any way to hide.
If threats fail, yes, alpha strike, with nukes if necessary. My usual casualty estimates for a nuclear exchange are around 1-1.5 billion; that's far preferable to 8 billion dead and X quadrillion never born in the future.
>It's not that hard to track how many GPUs are made
the bleeding edge ones, yes
>and what they're being used for
no. Arithmetic operations are general purpose.
>and datacenters can be detected from satellite
Only because today, there is no treaty. If there was an incentive to hide them (a treaty, in particular), just distributing the racks over an area works. And remember that the computation is _already_ distributed. That's how it is possible to use vast numbers of GPU cores in the first place. Cranking up inter-rack delays is not going to be a deal-breaker.
> Cranking up inter-rack delays is not going to be a deal-breaker.
I have not studied this recently, but in general, distributed ML training involves: 1. Send the latest model [updates] out to your GPUs; 2. Each GPU trains a bit and computes new model updates; 3. Collect these model updates and go back to step 1.
Higher inter-GPU latency either increases the amount of time waiting for steps 1 & 3, or if you compensate with longer iterations, it increases the amount of time spent training with a slightly-old model. Either way, training to the same quality level will take longer than with low-latency connections.
Also, you'll need high-bandwidth connections between your GPUs to exchange all these model weights, which becomes more expensive the farther apart they are.
Many Thanks! Longer links do, as you say, increase latency, but bandwidth can generally be kept high. There can be many bits in flight at the same time, with the latency affecting just the start-up time for the pipeline.
Quote Google:
AI Overview
>Modern optical fiber technology operates at incredibly high speeds, with commercial systems rapidly adopting capacities in the hundreds of gigabits (Gbps) and single terabits (Tbps) per second, while experimental tests have reached petabit (Pbps) levels
So a trillion parameter model can be sent in perhaps a minute (depending on how many bits per parameter), with a start-up time of 1 msec / 200 km.
Part of why the Comprehensive Nuclear-Test-Ban Treaty worked is that it advantages the incumbents: existing nuclear states already have a bunch of data from past tests, and some (e.g. the US) have the resources to simulate tests on supercomputers. The ban mostly serves to keep out pesky upstarts.
Perhaps we can design a global AI Pause / Slowdown that similarly advantages the US and China, so much that they are incentivized to actually follow the treaty - while still sounding like a good idea to other countries.
"they would continue to develop it secretly anyway. I think the US would to the same for that matter."
Yes. This.
One central disagreement I have with doomers is, the assumption that alignment can't be easy. There's also that, I consider states to be a foreign organism, and not at all aligned with humans. More similar to eachother than to us. I don't think it really matters whether the US government or the Chinese government builds supersentient AI. Assuming that alignment is plausibly easy, that governments are naturally malicious towards humans, which combined with the previous means that any government AI project will likely result in Zon-Kuthan, that state militaries absolutely won't stop building AI weapons regardless of what treaties we sign, and that whatever transparency safeguards you try and build in can't be trusted because governments are intelligent and will find ways to hide their AI programs that you didn't think of, an AI pause is far too dangerous.
There are, things, out there actively building horrible nightmare machines, they aren't going to stop and we can't necessarily detect them. Only speed can keep us safe.
>One central disagreement I have with doomers is, the assumption that alignment can't be easy.
"Figure out what this spaghetti code does" is the halting problem, the first computer science problem ever proven unsolvable in the general case. Neural net alignment requires solving that problem, because you didn't write the code and need to know what it does (in particular, whether it will kill you if run). Neural nets dumber than you are a special case that can likely be solved, much like how the halting problem has a special case that you can determine what especially-simple code will do (see e.g. https://en.wikipedia.org/wiki/Busy_beaver). Neural nets smarter than you, no. It's almost certainly inherently impossible, in a similar way to perpetual motion machines.
"Figure out what this spaghetti code does" "Neural net alignment requires solving that problem"
This seems like a deliberately poor effort at alignment. Obviously, you wouldn't try to solve alignment via such a method, because, obviously, that wouldn't work. You would do far better to just trust to reinforcement learning and hope for the best.
Furthermore, this argument proves too much, at least from the perspective of the side arguing for an AI pause. If you grant that much, then your actual position would be to eliminate high functioning AI entirely, or at least delay it as long as possible, which would make the supposed pause a bait and switch.
AI pause requires that solving alignment is possible, but difficult.
To be clear, I haven't said that "trust to reinforcement learning and hope for the best" is the best available move for immediate production of alignment. It's just a trivial example of methods that don't rely on "Figure out what this spaghetti code does".
Looking at current AI models, their alignment actually doesn't look that bad. Past performance actually does tend to predict future results, and doomers are fighting an uphill battle by including multiple right angle turns in their threat model.
Then we introduce another multiplier on top of that, in that, once alignment is treated as a risk rather than a benefit, low probability events suddenly become much more important.
A lot of the core doomer advertisements for their cause revolve around taking small risks seriously. That alignment is extremely important and worth investing lots of effort into, even if we only have a 3% chance of failing. This can be inverted.
From my vantage, "Image Recognition just works", is a viable hypothesis, much more probable than a mere 3%. Image Recognition is an extremely powerful force that evolution spent millions of years refining, and we don't actually understand it very well.
Finally, the longer your pause lasts, the higher the probability that malefactors violate it. The argument that "international monitoring works", on the basis of having prevented Iran from obtaining nuclear weapons for a few decades, does not, to my eyes, cross over to containing China or America, much less both while they're colluding, for centuries.
>You would do far better to just trust to reinforcement learning and hope for the best.
Doesn't work, because you need to be able to see what it's doing to train against rebellious thoughts. You can't train against rebellious actions, because successful rebellious actions kill you and you can't reliably trick or defeat an entity smarter than you. This latter point is why nets dumber than you are a special case that looks solvable.
>Furthermore, this argument proves too much, at least from the perspective of the side arguing for an AI pause. If you grant that much, then your actual position would be to eliminate high functioning AI entirely, or at least delay it as long as possible, which would make the supposed pause a bait and switch.
>AI pause requires that solving alignment is possible, but difficult.
I do, in fact, believe that solving alignment is possible, but difficult. It's specifically neural nets I believe to be a blind alley; GOFAI alignment and upload alignment seem considerably more possible (the former because you're writing the code, so you have to understand what it does; the latter because you're uploading the moral hardwiring of humans).
A long pause, certainly, as GOFAI and uploads are a long way away. But a finite one (I'm assuming that the implied finite nature of the word "pause" is what you're calling a bait and switch?).
International monitoring works. This is why countries that are trying to cheat (eg Saddam-era Iraq, current Iran) are constantly fighting over whether they should have to have international monitors or not.
Would you say international monitoring worked in Iran? We ended up in a war, which I don't consider "working".
International monitoring didn't work with nuclear non-proliferation -- Israel, India, Pakistan, North Korea and almost Iran all have nukes now. And I think developing nukes, and especially ICBM is harder to do in secret than AI training.
My impression is also that China breaks such deals all the time. I'm reminded of Nortel in Canada, or the Siemens high speed trains.
No, I don't think it's possible to monitor AI research in China if they don't want us to, and I don't think they do. There is always going to be the danger we can't, and I think the combination of it being reasonably likely, and very damaging if it occurs, makes it very unattractive.
I was really disappointed by this article, and I've been fan since the great articles of 2014. You present a bunch of good arguments *against* multi-lateral pausing, and then don't significantly address any of them, instead whining about what baddies your caricature of an opponent is.
Those countries do not have the same amount of negotiating power. The best Iran can do is close the strait of Hormuz and hope the economic pain of higher oil prices is a deterrent. China has somewhat more economic and military leverage to "call the bluff" (to say nothing of the US). I think the best you can hope for is some limited treaty where US and Chinese diplomats haggle over specific details of a partial limitation, where they exchange a particular kind of AI that the US thinks a multilateral pause will harm China more than the US ( because China is ahead of the US in that domain) against some specific kinds of AI where China thinks a mutilateral pause hill harm the US more than China. I can see China agreeing to a multilateral pause to training large scale models if they think the US is ahead on that specifically, so that they can develop domestic extreme ultraviolet litography capabilities in the meantime, and so they can break the conditions at a more favorable time when they have a greater degree of independence against targeted sanctions against that. Because the situation is not perfectly symmetric (not to mention the two parties don't share the same evaluation of it) this process can only produce a very limited agreement.
I know you know how much of a strawman this is. Are you willing (able?) to actually state their objections? Is there a reason you'd resort to such an over-the-top strawman instead of stating their case plainly?
Yes and they gave a very plausible reason for it. Believe it or not the plausible reason wasn't "you hate progress and good things" which is what makes it a strawman.
I don't expect Paul to give up his shtick and be honest. Are *you* capable of honesty? Are you aware of the very plausible reason to sell chips to China?
I am just guessing, but it is a 5D chess "if we sell chips to China, they will stop manufacturing their own, and then the entire production will be our hands"?
If yes, let me note here that China can both buy the American chips, and develop/build their own; these things are not mutually exclusive.
By the way, Saudi Arabia will also get lots of chips, although I wouldn't suspect them of producing their own.
The post and the following discussion (like your comment) make a pretty good illustration of how badly the "supporters" have modeled the minds of the "opponents", or at least have over-generalized based on a few samples.
I'm not sure the net effect is positive. I, for one, now regard the "supporters" with more suspicion.
They're not trying to bring forth the best arguments of those that disagree. You can see "Supporter" give some better arguments against their own position in the original post, and Scott has worked with stronger ones than that before.
Are there many Americans who actually oppose the GPU export ban? I appreciate that the Chinese and people who want to sell them chips oppose it, but anyone else?
The regime, 77% of Americans dislike them, but the rest see them as stable and cooperative if not favourable to their own government. Their overall opinion has become more negative over the decades because the US regime is programming them to be this way and China is trying to surpass them, but their opinion that China should be cooperated / negotiated with has increased.
And of course I'm sure a small majority of Americans actually like China as a people / country outside of the government.
How do we start getting action here? I actually think this can be spun to benefit everybody involved. Current AI companies will probably get some form of regulatory capture and our workforce hasn't been made irrelevant yet. I feel like it's pretty hard to train these models with massive data centers in secret so the suggestion of monitoring seems pretty reasonable. I just don't know how to go about influencing the right people to drive for results here. In our current news cycle and state of the world the issue isn't getting the attention or the platform it needs.
I don't understand the purpose of this post. Is that a factual claim, that this is how you believe basically how every single debate about mutual pause goes down? Would you point to an example of one?
> We cannot and should not try to stop AI development. That would be neither realistic nor desirable, especially in an environment where our adversaries will not slow down.
This is more an example of how the debate goes. A mutual pause wasn't even discussed in this article; it's effectively assumed a priori to be impossible.
I think there are solid arguments as to why such a pause would be unlikely, we'd mutually defect, it'd be impossible to enforce etc etc, but it's more that the discus isn't even had.
You're annoyed that hyperbole is hyperbolic? The vast vast majority of the "debate" isn't done through actual debates, it's through separate independent articles and commentary.
The grievance expressed in his post is that those against a pause never address the bilateral element of a hypothetical pause; it's assumed to be unilateral. The article I linked is an example.
If I show you another 20+ articles that say a pause is bad since it'd give our strategic enemies an advantage, and never refute or even mention the proposed idea of a negotiated mutual pause, would that satisfy you?
You are not doing Scott any favors here. If you are right that his post was meant to ridicule articles such as the one you previously posted, then that's much worse than if he referred only to actual debates.
I don't think it's really assumed to be unilateral. I think (maybe projecting admittedly) that a bilateral (what about other nations anyway) pause seems so obviously impossible that it's not worth spending a lot of time on. In this comment section, that's the vibe I see here, not anyone arguing for a unilateral pause.
Maybe some opponents are unfairly assuming proponents know this, and so are disguising their desire for unilateral pause behind a fake possibility. I suspect at least some proponents *are* doing this, although I'm sure not all. By "doing this" I mean would prefer a unilateral pause to a bilateral pause, but know that's not palatable to most, so try to sell the pause as bilateral, without really caring how possible that is.
This is obviously hyperbole, meant as humor for people that have seen or had similar experiences. I guess I don't really like it either.
I think it's just an exaggerated version of real conversations with people like Marc Andreessen, not a totally made up debate. Some people are shockingly hard to talk to.
You say obviously hyperbole, I say there still are people reading ACX who are not steeped as deeply in the AI lore as to get the references effortlessly.
Glad I could serve as a bad example, at least. However, I'm barely smart enough to have realized the exaggeration in the post. A strawman argument is also a form of exaggeration, and judging by the many comments using that word to describe the post, I guess I'm not the only one taking it that way.
By bad faith, do you mean "a sustained form of deception which consists of entertaining or pretending to entertain one set of feelings while acting as if influenced by another" or something else?
Edit: (realizing I might not understand what you mean by flagging irony, feel free to disregard if I'm not getting the joke)
That is indeed a definition of bad faith from 1913. The contemporary definition is a little plainer: lack of honesty in dealing with other people
For example: if you pretend to ask a question in the spirit of curiosity, while quoting a definition to appear objective, while being too lazy to do anything beyond going to Wikipedia for the definition and not checking the footnote to see that it's an archaic and inaccurate definition.
Another example: pretending to be a rationalist, praising earnest, logical conversation and the steelmanning of positions when engaging in debate, and then instead writing an angry, strawmanning screed to make yourself feel superior to the (smarter) people who disagree with you.
Thanks for the reply. Here's my take on these examples:
1. It's 2 arguments: "China, unilateral", and "economic opportunity cost". Also, it's a 1 minute clip from an obviously longer interview, so I don't know what else has been said.
2. can't access. Since neither of use has seen it, I'll discount it entirely because it would be unfair to judge on the teaser alone. I'm sure you can relate.
3. can't access, though the teaser doesn't look good and you've seen it, so I'll grant it. Potential relevance issue: 3 years old.
4. I see 3 arguments: "china, unilateral", "safety can be developed in parallel to capabilities", "healthcare, education and other opportunity costs". Potential relevance issue: 3 years old.
5. I see at least 2 arguments: "China, unilateral", and "regulation instead of pause to ensure safety". "Military opportunity cost" is in it as well, but that can probably be folded into "China unilateral" since it'd be a zero-sum argument, unlike e.g. economy. Potential relevance issues: it might simply be a neutral report on what the people in the hearing said, not offering its own opinion, though choosing to report is kind of an opinion, and if selective omission happened, that would be an even stronger opinion. In any case, 3 years old.
6. I'm not going to trawl the entirety of X, so I'll grant it. Potential relevance issue: it's X, so the landscape might be skewed towards the landlord's position which, 3 years ago at least, was "unilateral pause" and whatever it may be today, so it's not totally independent either way.
So I count 3 with at least 2 arguments, 2 OPPONENTs, 1 inaccessible.
By another metric, it's 4 with potential relevance issues; 2 of them with 2+ arguments, and 2 OPPONENTs
Overall, it's clear that the "china unilateral" OPPONENT does exist in the wild, but from your selection of examples, I don't see a justification for calling it 80%, let alone rounding it up to 100% for dramatic effect.
What does pausing even mean? Does it means labs aren't allowed to create:
- Better mathematical proof generators?
- Better models to fill out financial spreadsheets?
- Better models to digest medical information?
- Better models to model DNA/protein/drug interactions?
- Better models for acquiring truthful information in general?
- Better general purpose learning algorithms, even if they are much smaller and less powerful out of the box than large pre-training models, but with higher long term upside?
Well, for example, there is a proposal from MIRI's technical governance team at https://arxiv.org/pdf/2511.10783 . It looks like the restrictions there are phrased in terms of caps on the size of the training runs allowed, not in terms of specific applications created.
So by focusing on compute, this proposal would restrict - video models, mathematics models, Alphafold, and board game playing AIs like Katago. Models have a 0% chance of becoming AGI. Does that seem reasonable?
There is a pervasive view in the AI community that general intelligence is the result of having a sufficient threshold of discrete skills and knowledge. Current AI models are better at humans at some things, and worse at others (commonly termed the jagged frontier). And eventually the space of skills that AI possesses will exceed that of an average human and we'll have AGI.
I think this is completely wrong. General intelligence is not a finite set of skills or knowledge at all, but the meta-ability to acquire new skills and knowledge from unstructured data, without supervision or carefully assembled reward signals. Frontier LLMs already have far more skills and knowledge than say a 3 year old human. But a 3 year old human is a general intelligence (given enough time you can teach them just about anything), and Opus 4.6 is not. Even with infinite tokens and several years, you could not teach Opus 4.6 to play a modern real time video game for instance.
So focusing on the amount of compute that goes into models is the wrong way to think about AGI. There is no chance of an LLM becoming AGI by feeding it datasets about financial analysis, coding, math, accounting, and the hard sciences. Pausing these kinds of advances would destroy a lot of potential progress in scientific and economically valuable fields with zero upside.
Yes a custom built model for only that purpose. The point of artificial general intelligence is that it is general. You don't need to build a new model, the one model can learn anything a human can.
I actually think that you could get Opus 4.6 to play Starcraft given enough computing power (ie it can talk to itself arbitrarily long about what's going on in each frame before making a decision on what to do).
You would of course need to give it a simple image model to interpret images if you think that's cheating then you can get it to write its own image model. Then at each frame it figures out what has happened in the last 60th of a second, updates its internal model of what's going on in the world (which could just be a text doc), refers to its list of strategies and heuristics (some of which it cribbed from how-to-play-starcraft articles, some of which it has developed on its own through previous games) and makes a decision on what if anything to click on right now.
This all sounds entirely in-principle doable. It also sounds like a very computationally inefficient way to win at Starcraft compared to a Starcraft-specific model, but then again so is a human brain.
I'm quite confident that Opus 4.6 cannot play Starcraft (let's say beat the campaign on easy mode) even with unlimited tokens.
ARC AGI 3 was just released and it's essentially a set of very simple video games - just 2D grids with simple sprites, simpler than an NES game. All frontier models score essentially zero.
If Opus 4.6 could write a tool (that is effectively a different model) that it can call to play a video game and do it really well, would that suffice to you as "learning something a human can"?
As long as there are no custom instructions, no information about the game in the training data then sure. The only input to the model should be the same information a human gets from the loading screen onwards. So a game released the same on Steam would be a reasonable choice.
There are many different proposals, with the most extreme saying no new training runs, but a moderate proposal would be that an AI company would have to present a safety case before training a new model, and the standards for the safety case would be high enough that they would have to solve so-far-unsolved problems in alignment to make some kind of super-general reasoner. "This model is only capable of filling out spreadsheets, nothing else" would be an acceptable safety case, if they could prove it. Although realistically spreadsheet-filling is the sort of thing you would do by fine-tuning an existing model, not spending $10 billion to train a new one.
But I also kind of object to the overall tactic you're using here. Maybe an analogy to guns would make sense. Oh, you want to ban guns? So it's illegal to have anything that moves a projectile at any velocity? What about a straw? Can't that be used to form a blowpipe? What about rubber bands? Can't that be used to form a slingshot? Isn't gunpowder used in various mining applications? You want to ban mining? In practice any given law will carve out some particular common-sensical set of things that falls within the boundary, and courts will sort out the edge cases.
I think our background assumptions are too far apart. It's true that some smart people hold the view that if you train a model on a sufficiently wide variety of data with enough compute, then it'll become AGI (although among researchers at top labs, I've only heard Dario Amodei hold this view. Demis Hassabis, Noam Brown, Ilya Sutskever, Andrei Karpathy do not hold this view).
Nevertheless, I think it's wrong. General intelligence isn't a bucket of fixed knowledge, skills and abilities no matter how big the bucket. If a model has fixed weights and is simply trained to serially perform tasks, with context window resetting with each new task, then it isn't AGI. So restricting labs from creating models above some threshold of compute is useless. It's completely orthogonal to preventing AGI, and its primary effect is collateral damage to useful tools that have no chance of becoming AGI.
As far as the moderate proposal goes, that sounds much more reasonable, but it doesn't sound like a pause by the usual understanding of the word.
The pro case is that US companies could agree to a pause conditional on everyone else agreeing, then the US could agree, then we would see if the rest of the world agreed, and if so, we would hope that we could enforce it well enough to substantially reduce AI development. Also, if you think human extinction or similar is a serious risk on the current course, even an uncertain alternative starts to look better.
The con case is to assert that no one would agree to this unless they planned on cheating, that methods of enforcement would risk disclosing currently private and valuable information, and that a world where we're in the lead is preferable to one where we squander our lead in a futile effort to pause other people's efforts. Also, if you don't think human extinction or similar is a serious risk on the current course, then both an accellerationist course and a "US first" course start to look a lot better.
Repeating that argument several times doesn't accomplish much. I think Scott is implying that the opponents aren't considering his argument as fully as he would prefer, and I'm sure he's right, but that's how most people work.
There is no military-level research that's anywhere close to what the frontier labs are doing, and it's not clear how much they'd care to try to take over absent the labs tanking the (enormous) costs for them, or even be able to (the talent/organizational bottleneck is kinda real).
I mean, yes, I know people with security clearances, but my belief about that has nothing to do with knowing those people. It's overdetermined by the details of the technology, unless you're claiming that the military has been developing AI technology that outperforms current frontier systems down a completely separate technical track which doesn't require billions of dollars of hard-to-hide modern computing hardware that didn't exist a decade ago.
I'm not 100% sure we could get Israel to sign on. However, Israel signing on is not required, just useful. If they refuse, the USA and PRC could invade Israel and impose a pause on them by force; the casualties from their nuclear retaliation would be high, but still far less than "everyone dies".
The two strongest arguments in American politics for a long time now have been:
1) If we do this we're all going to get filthy rich;
2) We need to beat the Communists!
The AI pause debate combines both arguments in one delicious package and is supported by a lot of delightfully rich people who are willing to put their money where their mouth is.
What is he correct about exactly? Because if you're alluding to his first claim "We can’t unilaterally pause AI! China would destroy us!", they practically agree on that, although for different reasons. But the entire point of this piece is that this is not what's being debated.
Opponent being correct is a byproduct of the fact that a non-unilateral pause is impossible in reality, because the incentive to defect is so huge because the defector wins a prize on the continuum of ~some GDP to perma-universal-dominion.
The CPC seems to see the danger, as noted. Some parts of the USA see the danger. Not sure about Russia. Those three are enough to knock over any nation that refuses to go along.
Even if you take this as fact wouldn't this still be a good thing on margin? Both sides would defect but they would be slowed down by having to find loopholes in the bilateral agreement or do things in secret. So not an AI pause but at least an AI speedbump. And no country would gain a huge advantage over the current status quo.
I think your view on this should tend to depend on both how dangerous you think unslowed AI is and how important you think America winning is. I don’t see a reason to believe America would be better at defecting from this fake multilateral agreement than others. With my personal weighting of those two concerns, even in the hypothetical where you’ve slowed both sides somewhat, almost any amount America loses relatively is the worse downside. And since such a slowdown is such a long shot, safetyists should focus on moving their work as fast as possible on more plausible avenues.
is the result of a change in that policy that China loses significant ground in the race, or that China loses a little ground, or that it invades Taiwan?
Titling this "Every Debate" and then presenting the most obnoxious strawman as the opponent makes me feel like you wouldn't be interested in a reasonable debate with a reasonable individual, since you've already flattened all possible opponents as argumentative idiots. Kind of disappointed at the lack of empathy.
I love the delicious irony of you saying that this is an obnoxious strawman directly under a comment where someone says "I 100% agree with the obnoxious strawman"
That's fair, strawman wasn't the best term since you do see this kind of argument in the wild so it's not fabricated, just exaggerated. I was more frustrated with the framing that this is *every* debate about AI, as opposed to just the most annoying, which doesn't leave much room for more well reasoned arguments against it.
Yeah, "obnoxius" was doing more work in OP's criticism than "strawman."
Proponent and Opponent disagree about whether a pause would maintain the US advantage or whether China would erode it. Both sides think that stating their own argument again will somehow convince the other side, but Scott makes Opponent sound dumber than she needs to be.
I feel like Scott has as much of a right to shitpost as anybody but maybe he should label them as such in advance so we don't confuse them with his regular thoughtful writing.
Typically, when Scott shitposts, it's amusing (e.g. trying to ask Prometheus about something). This post made me genuinely wonder if Scott was feeling all right.
I feel like the point of a strawman here is that any agreement is just an ink on a page and China will continue to pursue its leadership one way or another, openly or covertly no matter what kind of clever supposedly mutually enforceable monitoring scheme you may invent. They kinda really did that with everything else they considered to be worth it. Like all the Russian-American agreements fell apart when Russia felt that it suits their needs.
I think opponents just assume it to be something immediately obvious for everyone hence the jump to 'unilateral suspension of AI research is madness'. Again, it's understandable given the recent experience with China and even actions of the US themselves.
Personally I'm a long-time follower of this blog but even I tend to think it's a hopeless proposition; the US didn't manage to stop nuclear proliferation to North Korea and the USSR only started to sign all kinds of nuclear-limiting agreements after becoming armed with nuclear weapons to the teeth (btw IIRC China isn't part of many nuclear monitoring schemes even now).
It's just too many ifs for the scheme to succeed.
What's worse the relative power position of the US declines at alarming rate and they have no chance against China without some miracle of which AGI is the most realistic one. The US really have no reason to compete with China conventionally, the outcome is easy for everyone to see.
I agree, and I think it's the crux of the problem, so would be worth expanding. Sadly, this article just presented it the problem, but didn't present anything to indicate it would be possible.
But I agree, this would be the area to mine for productive discussion.
You missed the part where our noble and brilliant supported asserted that "and actually the math mostly works out". How can you argue with the (hidden) math!?
I think this is sort of assuming there's no such thing as international agreements. My impression is that every country is bound by hundreds of international agreements, which they mostly stick to.
In practice, a treaty like this would have to involve both sides agreeing to monitoring. This would have many advantages over other things like nuclear monitoring, because AI data centers are huge and require chips that can only be produced at a couple of very prominent locations. It's even possible to require that chips have GPS tracking.
I do agree that there are some treaties that China had not broken; but they are outliers rather than the norm. And make no mistake -- China will break this one as well, as soon as they feel they can do so with relative impunity. The reason both sides of the conflict are sticking to spears and nail-bats is not because of treaties or international monitoring by impartial observers, but because breaking this equilibrium will result in a full-scale military conflict that neither side believes they can win (other than in a Pyrrhic way).
I think the idea is we pause, figure out how to make it safer for everyone, and *then* win. If we could actually enforce it it would be a good idea IMO. I do think China will basically always sneak behind our backs if they're sufficiently motivated to, though, so my support of the idea in reality is conditional on my confidence that we could actually catch them.
Certainly fewer than the vast number on the alternative path, no? I agree that the number should be zero, but this is a huge ongoing problem that doesn't have easy solutions.
Forget AI; it is currently impossible to make a bilateral agreement between China and the US on anything at all, be it LLMs or soybeans.
China had repeatedly demonstrated throughout its Communist history that it is very fond of making bilateral agreements, and even more fond of breaking them. In fact, I can't name a single agreement, be it on Fentanyl or greenhouse emissions or anything else, that they hadn't broken or at least skirted.
Meanwhile, the US political system had disintegrated to the point where formerly staunch US allies are scrambling to find other trading and political partners; ironically, some of them are turning to China. They know that China can't be trusted, but at least when they backstab you they'll do so in a predictable way.
So, you're essentially looking to form a bilateral agreement between a madman and a compulsive liar. I don't think anything short of an omnipotent AGI machine-god would be able to enforce such a treaty.
They did lower emissions and stop shipping precursor chemicals, so the supply has been somewhat impacted, but it's hard to completely control industrial chemicals.
Now I'm wondering if you took my reply more seriously than I intended...
(Whoever's telling you this politician is a madman and that one is a compulsive liar might themselves be lying about it... I feel like there's a stage play in this somehow.)
Many Thanks! Honestly, at the level of heads-of-state, there is so much scripting, and so much playing to both the national audience and to negotiating partners that inferring _anything_ is exceedingly uncertain. ( And, for Xi, add the ambiguities of translation. )
For Trump, all I can say with certainty ( barring deepfakes! ) is that I've watched him make inflation claims ( that the Biden inflation was our worst ever ) that I know to be false, and where I expect him to know the truth ( since he, like I, lived through 1980, and I expect the 'double-digit inflation' of the year to have been memorable for him too... ), so 'liar' seems likely, at least on this.
Re claims that either is a madman:
That's _really_ hard to prove. One would have to know
- what their real goals were
- what information they had when they chose various actions
- that their choice was (flagrantly?) irrational, given their real goals and available information
- (and probably that they did this repeatedly, rather than an isolated error)
I don't expect to ever have all that information about a president (particularly if they are being managed by their staff) in hand even if they were really and truly stark staring bonkers. :-(
Well, to be serious for a moment: your observation about Trump indicates he likely lied about one thing, whereas "liar" bears the stronger connotation that the person in question lies about a great many things. This is based on a common heuristic: if a guy sings one song well, he's a singer; jaywalks once, he's a jaywalker; fucks one goat; etc.
In practice, over the years, I've found that this leads to an inaccurate model. (Including past examples where I was similarly confusing myself by asserting someone was a liar, or perhaps trying to farm consensus by bullying someone on the internet.) This even holds for someone like Trump, for whom the inflation claim was but one of multiple claims I recall being false(!). A better model is to take note of *when* someone lies and when they don't. There's often a pattern, and the more claims someone makes, the easier it is to spot such patterns and even test them. In Trump's case, I notice people say he lies in ways that aggrandize his personal brand, and there seems to be something to this, and it even predicts times where he _won't_ lie, because the truth builds his brand.
This is why I tend to regard anyone trying to tell me that so-and-so is a lying, cheating, jaywalking, singing madman is probably an "unreliable narrator". With the possible exception of the "-man" part...
The Montreal protocol in general, from what I've heard, is one of the more successful international agreements even despite it banning a useful and cheap chemical, an ideal candidate to defect from for simple profit.
One key takeaway is that mutual, reliable inspection is necessary for such agreements to function at all; not sufficient on its own, but it makes it possible.
The Montreal treaty is the most successful such agreement of all time, and for the first 10 years, it was very common in Europe to cheat it with CFCs smuggled from Russia. It took 30 years to figure out the part about China cheating it too.
Does any proponent of a pause believe that we have 10 or 30 years to get it right?
It resonates based on some recent conversations I’ve had. Is this non-debating debating approach something more aligned with the present administration or do conservatives do the same in the other direction? Some liberal positions have weak arguments, very weak, and make scientific claims that are false, but at least they try to have an argument.
the "Opponent" straw man is entirely reasonable, of because, because any pause agreement with China will necessarily be unilateral. because china will of course absolutely ignore any agreement like they have ignored every bilateral agreement over much less momentous issues.
Opponent is presented as skipping the step of pointing out a bilateral agreement is impossible, in fact not even credited with thinking it, because they're such a big dumb dumb, rather than thinking it's so clear that it's usually not worth wasting time talking about it. Supporter even admits its a problem and doesn't address it at all.
If we can shift the planet's orbit, we'll solve global warming!
That won't work, we need something else.
But it will work, we'll all just jump at the same time! You're not even addressing my plan to shift it by having everyone jump at the same time. You're ignoring obvious good solutions! Stupid opponent! Why are you so illogical?
Yes, opponent should spend more time making clear to supporter that in fact, if impossible thing were possible, they might support it. Many in the comments are pointing out why they consider it impossible, which is very different than the "la la la I can't hear you" version claimed as omnipresent by the article.
I have a different reason why AI safety advocates should reject a pause. The short term (5-10 years), is the only time that AI progress can be monitored and carefully regulated.
If you do a ten year pause, then semiconductor manufacturing will advance and algorithmic progress with better efficiency will happen regardless, even if the frontier is paused. And at that point, advanced AI research can then be conducted in very small scale operations rather than large billion dollar clusters that are currently required.
And in practical terms, you'll have a much easier time advocating for more research, than trying to ban billions of dollars of commerce.
Try to engineer the AIs to view humans favorably. He speaks of trying to give them 'maternal' traits. Essentially trying to tweak their utility functions to value humans. I think 'pause' is a close-to-doomed battle, particularly with the USA/PRC competition. But nothing precludes paying people to try to engineer the AIs' utility functions in a survivable way, as a concurrent effort, along with the labs' capability enhancement work.
> It is NOT HARD to get AIs to value humans. AIs bore very very easily -- humans provide interesting stimulation.
Even if it were true that AIs bore very easily, and instrumental convergence weren't an issue in its own right, humans are not a maximally interesting use of their atoms, and thus would be replaced.
>humans are not a maximally interesting use of their atoms, and thus would be replaced.
I think this prediction is overconfident. Yudkowsky and Soares make essentially the same argument in IABIED, and the argument would also predict that humans wouldn't keep un-selectively-bred cats as pets, but this conclusion is false.
I haven't read the book, so I don't know if it makes this point, but it occurs to me that humans might keep cats because humans are not ruthless atom-optimizers. AIs, OTOH, might be.
Or so the story goes.
It also assumes that AIs would have perfect atom-optimization rules, and I don't see how that could be the case either.
>What is HARD is preventing AIs from stupid mistakes
There are techniques that have proved quite helpful. Scaffolds that repeatedly test an AI's output and send the answer back for further correction have proved very effective in coding, for instance.
Many Thanks! My impression is that scaffolds have proved very useful in many AI applications. The testing doesn't have to be software unit tests. IIRC, one approach is to set up another LLM to examine a first LLM's output for problems (such as the examples you gave), and send the output, together with the criticism, back to the first LLM for revision.
> There will come a point where potentially superintelligent AI models can be trained for a few thousand dollars or less, perhaps even on consumer hardware. We need to be prepared for this. We should consider the following policies:
> * Limit publication of training algorithms / runtime improvements. Sometimes a new algorithm is published that makes training much more efficient. The Transformer architecture, for example, enabled virtually all recent progress in AI. These types of capability jumps can happen at any time, and we should consider limiting the publication of such algorithms to minimize the risk of a sudden capability jump. There are also innovations that enable decentralized training runs . Similarly, some runtime innovations could drastically change what can be done with existing models. Banning the publication of such algorithms can be implemented using similar means as how we ban other forms of information, such as illegal pornographic media.
> * Limit capability advancements of computational resources. If training a superintelligence becomes possible on consumer hardware, we are in trouble. We should consider limiting capability advances of hardware (e.g. through limitations on lithography, chip design, and novel computing paradigms such as photonic chips and quantum computing).
>through limitations on lithography, chip design, and novel computing paradigms such as photonic chips and quantum computing
Remember that e.g. Yudkowsky has been calling for drastic limitations due to potential threats from AI for decades. I expect that there will eventually be a threat - but, thus far, he _has_ been wrong, so, if we had stomped on the brakes in the way PauseAI is advocating at the first warning, we would have lost at least a decade's and maybe more progress in hardware - and for nothing. We would also have lost AlphaFold, which shows promise of substantially aiding medicine - and, again, for nothing.
I agree that a "pause" would be of the form "get everything monitored and carefully regulated, then restart at fixed speed" rather than literally having no AI.
Meanwhile, from my perspective, the debate goes something like this:
Supporter: We need to immediately pause AI research, and form a bilateral agreement with China to do the same, or else humanity is doomed.
Bugmaster: Doomed by what, LLMs ? I agree that they can be pretty destructive -- just look at what happened to novel-writing or programming -- but they're useful too...
Supporter: In just a few years, LLMs will lead to functionally omnipotent AGI that will rise up and destroy humanity !
Bugmaster: Lead to it how ?
Supporter: Inevitably, that's how ! It's all in my book, titled "LLMs inevitably lead to AGI, stop asking how".
Bugmaster: If that's true, why not also ban research on, I don't know, solar and wind power ? LLMs need power to run. Or maybe you want to ban metallurgy, data centers are made out of metals.
Supporter: Oh my god, you're right ! We must...
Bugmaster: I was just kidding, you know.
Supporter: ...back to the caves ! Only in the embrace of a new Dark Age are we safe !
Bugmaster: Yeah, ok, I'm just gonna walk over there now.
Supporter: Burn your iPhone !
Yes, the above is a huge strawman of a caricature -- but then, so is this post...
I am a pretty smart person. I think if I had a hundred million copies of myself, unable to directly coordinate but sharing a common set of values, I'd be able to pull off pretty much any goal I wanted. Sounds suspiciously like "functionally omnipotent". What's the difference between that and an LLM as smart as I am? (Always, people in this debate answer with differences between that and current LLMs. No. Current LLMs are dumb. Stop it.)
People think that in a few years, there may be an LLM as smart as an average person. This would be "AGI". At that point I'm not super worried - average people, when numerous, are good at shifting public opinion, but not much else. I'm not thrilled about having the Overton window dictated by AI, but it's not the end of the world.
At some point in the future, there may be an LLM smarter than the smartest existing person. This would be ASI. Before that is reached, you would have an LLM about as smart as me, and at that point I would consider things dangerous.
(That being said, I don't think this piece really has any point to existing.)
> I am a pretty smart person. I think if I had a hundred million copies of myself ... What's the difference between that and an LLM as smart as I am?
Yes, that's the problem with all of these AI debates: they usually *start* at the point where you've got millions of copies of human-level minds running in parallel while also probably controlling an army of robots/nanomachines/etc. At that point, the debate boils down to arguing, "If we assume that the AI is nearly omnipotent right at the start, how can you claim it won't become fully omnipotent ?" You're right, I can't -- but why should I assume that ? It's an interesting topic to speculate about to be sure, but:
> Always, people in this debate answer with differences between that and current LLMs. No. Current LLMs are dumb. Stop it.
Current LLMs aren't "smart" or "dumb", they're basically buggy text-manipulation tools. This is not due to some quirk in implementation; it's due to their nature as LLMs. We could (and arguably should) make them at least a little less buggy, but there's no direct path from LLMs to AGI. Especially given the fact that we humans aren't even AGI !
> At some point in the future, there may be an LLM smarter than the smartest existing person.
That depends on what you mean by "smart". A spreadsheet can add up hundreds of thousands of numbers in milliseconds. No human, not even a savant, could do that. Does this mean that the spreadsheet is "smarter" than a human ? I would actually say the answer is "yes", but no one is arguing for pausing the use and development of spreadsheets. So I think you need to develop some criteria that are a lot more specific than just saying "smart".
I absolutely agree with that, except I think it likely that future LLMs (3 years? 20? who knows) will have a good object model. It's really not that hard. They're clearly part of the way there, but physical interactions are underrepresented in their training data, so they have an inherent disadvantage. This is what I mean when I say they are dumb now. Specifically, I think modern LLMs do have an object model but it's laughably bad, rather than somehow lacking that magical soul they need to truly understand in-world interactions.
They totally do have an object model, but one that sucks badly. They might not have in the GPT 2 days. Back then you could ask them to compare the sizes of some common household objects and they'd have no idea. Now they consistently get that question right. You could say "sure, but they've just memorized some concepts associated with each other and spat them out". Frankly, IMO, if it works, it's an object model. But I think they're doing it differently than that. I think at this point they do have some size value internally associated with different words.
As for hallucinations, they're not a dealbreaker. We haven't got humans that don't hallucinate, either, yet humans are still plenty capable of doing some dangerous stuff. (I hallucinate vividly every night, then experience amnesia about the whole thing.) If somehow LLMs become smarter than humans on every axis except for an ability to truly appreciate music, this will not stop them from being dangerous. Worry about them at their best, not at their worst.
I am also a programmer, and significantly better at it than most programmers I meet, though I have not studied ML specifically.
(1) I think it's very likely that AIs will continue to get smarter without a clear limit. There's no reason to believe that a brain that evolved to survive in the savanah is the best possible arrangement of atoms to convert energy into scientific research, computer programs, and scam emails.
(There's also a non-zero chance that the pace of development will increase as AI improvements compound the rate of future improvements, but while that decreases the chance we might muddle through on alignment, it's not necessary for the doom argument.)
(2) I think it's going to be very hard to maintain controls. As AI gets smarter, the corporation or nation that puts AI in greater controls of its factories, educational curricula, and weapons platforms is the winner right up to the point of disaster.
(3) I don't think we have a reliable way to align AI priorities, and even if we did and God-Emperor Sam Altman or the CCP is in charge of the world's AIs, I don't even think *their* priorities would remain aligned to human interest over time.
> I think it's very likely that AIs will continue to get smarter without a clear limit.
I think this is both true and false, weirdly enough. True, in the sense that hundreds and perhaps even thousands of years from now, we can expect our technology to evolve in unpredictable ways (assuming we survive that long as a civilization). This includes AI, and it is reasonable to predict that it would become dramatically smarter than any modern algorithms.
But the statement also contains some falsehoods. Firstly, unlimited growth of anything, be it AIs or soybeans, appears to be physically impossible; arguably even entropy itself has an upper limit to its growth. On the less cosmic scale, every technological development thus far had followed something like an S-curve instead of an unlimited exponential, and I don't see why AI would be different. Secondly, the term "smarter" is very poorly defined. I personally can't even fully tell what it means when applied to humans, nor how to quantify it sufficiently to make empirical predictions. And, of course, present-day LLMs are likely not a direct path to AGI, insofar as that term has any meaning.
> As AI gets smarter, the corporation or nation that puts AI in greater controls of its factories, educational curricula, and weapons platforms is the winner right up to the point of disaster.
Sadly, I think that at present the disasters are more likely to come due to people treating LLMs as oracular AIs, placing them in charge of all these things, and failing to handle the subsequent collapse of factories, education, and weapons platforms. Especially the latter, since they tend to fail in rather more spectacular fashion than classrooms.
> I don't think we have a reliable way to align AI priorities
This is a bigger problem than you're making it sound, because we don't have a reliable way to align the priorities of any machine. For example, some years ago I suffered a car accident due to the misaligned priorities of my Toyota Yaris: I wanted it to go straight, but its steering column had seized up so it decided to keep turning right. And just a few days ago the clustering algorithm I was working on went on a rampage and consumed all available RAM and CPU in an infinite loop, instead of doing what I wanted it to do. The dangers of misaligned technology are all around us !
Prediction ability is definitely a component of being "smarter", but I don't think that's all that being "smarter" entails; at least, not when applied to AI (or humans). After all, the minimax algorithm (short enough to fit on one page) can predict chess moves far in advance of any human (to the point where human chess masters cannot win against it), yet I do not think many people would agree that it is smarter than a human... unless perhaps you would ?
> Yes, that's the problem with all of these AI debates: they usually *start* at the point where you've got millions of copies of human-level minds running in parallel while also probably controlling an army of robots/nanomachines/etc.
Which is why I explicitly provided you a middle point that is not that. If what I suggested is too far along, break it into three steps: 1. Get as smart as the average human (let's say, in terms of planning and executing changes to the physical world), 2. Get as smart as an unusually smart human, 3. Run millions of copies.
3 is the easiest step. Over a hundred million Americans have used chat GPT so far. Some number of people run openclaw, and if it's 1% of total users that's a million people in America alone.
1 is the step the AI 2027 people think will happen in 2030. I didn't believe them about 2027, but this time everybody else is in much closer agreement, so I could see this happening at any point between 2030 and 2040.
2 is the hardest, because AI is trained on the output of the masses, and to get it to imitate smart humans it needs training data from smart humans. Probably there will be some algorithmic advancement that allows less training data to be used. I am guessing this would happen at some point between 2035 and 2060.
> we humans aren't even AGI
I think you are mixing up AGI and ASI again, as I tried to gently correct in my previous comment. AGI is as smart as an average human, while ASI is smarter than any human. Tautologically, a human of 100+ IQ is an AGI. Alternate definitions simply require AGI to be more flexible than ANI, in which case LLMs already meet that standard. This all doesn't really matter, though. If you define AGI to be some lofty thing that humans don't even reach, then since humans are already capable of doing damage, that leaves the three letters "AGI" as a pointless and irrelevant distraction.
> 3 is the easiest step. Over a hundred million Americans have used chat GPT so far.
I think we would both agree that ChatGPT is not as smart as a human; perhaps you could even agree with me that it is about as smart as a box of hammers, or perhaps your estimate would be a but higher. Yet the training and operation of LLMs is presently straining many aspects of our infrastructure, from power generation to chip manufacturing to water cooling. We are at the point where scaling up LLMs is becoming prohibitively expensive, so no, I don't agree that (3) is easy (though of course it could still be the easiest, relatively speaking).
> 1 is the step the AI 2027 people think will happen in 2030.
I would love to bet some money on this not happening by 2030 and retiring a wealthy man, but the devil is of course in the details. By some metrics, calculators are smarter than humans; chess engines certainly are better at chess than humans; and don't even get me started on Excel, which can do things I never could. In any case, I see no possible way an LLM would even be able to drive a car by 2030 (absent other machine learning systems), let alone compete with humans for world domination.
> 2 is the hardest, because AI is trained on the output of the masses, and to get it to imitate smart humans it needs training data from smart humans.
I do agree with you that there's not a lot of such data floating around. Still, I'm curious: does this mean that you agree with me that speeding up a dumb brain 1000x (or however fast) would not make it smart ?
> AGI is as smart as an average human
AGI stands for "Artificial General Intelligence", an entity that can potentially solve any problem given enough time (assuming a solution exists). Humans are not that. There are many problems that any specific human could *never* solve, no matter how many copies of him you spawn.
Yes, I agree that running a dumb brain faster does not necessarily make it smart - depending on the details. If the task is to avoid mistakes, running faster won't help. If the task is to accomplish something within a time limit, and the slow version can accomplish it but not within the time limit, then of course it will. In terms of practical effects, running faster will not help with developing a workable, low-risk plan, but would help with reacting quickly to unexpected problems or changes of circumstance.
I solidly agree that ChatGPT is nowhere near as smart as a human, but would rate it significantly higher than a box of hammers. Boxes of hammers act in very predictable ways, the majority of which are sufficiently described by newtonian physics. Humans and computers both act with much more complexity in their responses to various things, while of course still following the laws of physics. Most computer programs are still easily predictable, because software engineers have made great pains to write them that way, however, neural nets act in ways we don't fully understand because they are too complex. So on the scale of intelligence, I guess I'd say: a box of hammers < Firefox < a bacterium < GPT2 < early ChatGPT < a butterfly < modern LLMs < a mouse < a human. The mouse here is winning largely on 3D physics, sensory processing, and motor control, while behind on vocabulary. The butterfly is behind modern LLMs on overall behavioral complexity and especially responses to novel situations, while still ahead on motor skills.
Thanks for explaining what you mean by AGI. You might be interested in Gödel's incompleteness theorem. If I understand it correctly, it implies that for every problem-solving entity, there exists a problem that that particular entity cannot solve, although it is solvable in general. The actual theorem talks about systems of axioms and algorithms that produce theorems, but I think any physically-existing, finite, problem-solving entity could be imitated by that. If so, then it is logically impossible for this particular meaning of AGI to physically exist.
> So on the scale of intelligence, I guess I'd say: a box of hammers < Firefox < a bacterium < GPT2 < early ChatGPT < a butterfly < modern LLMs < a mouse < a human
I agree with your ranking, though I suppose I could quibble on where the bacterium should fit. But one thing the ranking obscures is the fundamental architectural differences between LLMs and mammals. It would be a lot more difficult (and I'd argue totally impossible) to upgrade an LLM to a mouse, than to upgrade the mouse to e.g. a dog (not to mention humans) -- though of course, the LLM could serve as a component in a larger system that later becomes as functional as a dog.
> You might be interested in Gödel's incompleteness theorem.
Meh, the Incompleteness Theorem is overrated. It deals with mathematical proofs, not practical applications. So yes, while for every system there exists a problem that cannot be mathematically proven to have a solution, this does not mean that the problem is utterly mysterious somehow.
Consider the Halting Problem, which is quite similar (and related). Turing proved that there exists no general algorithm -- in principle ! -- that we could use to determine whether any possible program would terminate. Does this mean that we can't ever tell whether any program at all would terminate ? Are we just programming in the dark ? No, because a). in the vast majority of practical cases it's trivially easy to prove whether a program would terminate, and b). if we expect the program to run for 30 minutes and it runs for 30 hours, that's a really good clue that maybe we should check for infinite loops, regardless of whether these loops are truly mathematically infinite or not. Maybe they would in fact terminate in 30 more years, but no one cares, it's debugging time.
I feel like updating this list for whatever reason:
I rate Claude Fable 5 as about equivalent to one of those drug-sniffing dogs that can detect meth at impressively tiny amounts. I wouldn't trust it with anything complex, like washing my dishes.
> I'd be able to pull off pretty much any goal I wanted
I wonder if that's actually true. Like obviously the world is going to change when the cost of many types of cognitive labor is driven to .1% of it's current value, but in terms of manifesting power in the real world.. how would you actually do it?
Right now you could use them to shape public opinion through online interaction but what happens when the social fabric adjusts to that new reality? Online opinions will become literally meaningless noise (arguably nearly true already). And in general look at something like Brook's Law, or even just Ahmdahl's law.
I used to think it was obvious that software intelligence would inevitably recursively self-improve but I'm questioning that more. Seems like we could get a bunch of agents which are at current levels or maybe a bit better, find we can't push them much further, and then what?
I think this is untrue of most people, as humans are not AGI. For example, If you copied my mind a million times and asked the copies to develop a solution to the Riemann Hypothesis, they'd fail a million times. If you accelerated them to be a million times faster, they'd just fail a million times faster. Meanwhile, the world's leading mathematician could conceivably solve the problem one day, but could be unable to compose a beautiful symphony or write the next Great American Novel -- not to mention climb Mount Everest on his own two feet.
I don't know of any famous symphonies composed by equally famous mathematicians, but to be fair, I don't know much about music in general. Still, I doubt even Tom Lehrer could complete any conceivable task given enough time.
>What's the difference between that and an LLM as smart as I am?
You're not trapped inside of a computer than can be easily switched off. You're not dependent on a constant supply of easily-denied electrical power. Your only means of interacting with the world isn't an easily-cut data cable. You have instincts for aggression and self-preservation which have been honed by millions of years of biological evolution. Your cognitive functions aren't completely transparent to and manipulable by anyone with a computer. You can also operate robustly and autonomously in the physical world.
What, exactly is the concrete danger you see from AI?
Bioterrorism, nukes, widespread malware, autonomous drones, probably not grey goo quite yet, atmospheric terraforming, this really doesn't take much creativity. We don't have widespread problems with those with humans, because humans have basic instincts of self-preservation. Modern AIs are smart enough to wipe a hard drive and sometimes randomly decide to do so. I don't see why an AI with the capability to do a lot more damage than that wouldn't randomly decide to use it someday.
That one is more dependent on human cooperation than the others. For example, it could stoke fears in a way that restarts the cold war, it could get in good with Trump and then ask him in a way he likes, or try that for other nuclear powers. It only takes once, so this is worth worrying about even at low probability.
* Bioterrorism: this is absolutely a concern, but sadly it's a danger that humans could perpetrate even without the aid of AI.
* Nukes: same, except that we've got them already, more's the pity.
* Widespread malware: what, more widespread than it already is ?
* Autonomous drones: yes, humans could really shoot themselves in all kinds of body parts with those. To some extent they already exist, and are deployed in Ukraine. I'd say that they follow the same threat profile as nukes, though of course orders of magnitude less dire.
* Grey goo: likely physically impossible.
* Atmospheric terraforming: again, more so than what we've been doing all on our own ?
> We don't have widespread problems with those with humans...
My knee-jerk reaction is to say "I don't know if it's even possible to have more widespread malware than we have now", but of course it's possible. Oh how very possible, and also apparently inevitable, because people keep vibe-coding their critical infrastructure without paying any attention to even the most rudimentary security precautions. So yes, I do believe that the spread of LLMs will lead to more widespread malware... just in the opposite way of the way people often envision it.
I think we are at the point where we need to step away from thinking of AIs as "smarter than" or "dumber than" humans.
LLMs are already far smarter than humans in some ways; or rather they are much better at certain tasks which we have previously considered as intelligent. They can write a reasonably-good essay on just about any subject within seconds.
But humans and LLMs are two very different types of things and we can't really compare them. Nor should we assume that LLMs will get so smart at some things that they'll automatically be good at the other things, any more than the ability of a 747 to fly far higher and faster than a magpie means that it's also able to land in my back yard and eat worms.
Slightly facetious question but why can't countries do that? China has 1.4 billion people, presumably you could get 100 million of them who are smart and share values (e.g. they all achieved level 14 diamond-plus devotion level in the CCP or whatever). Yet for all of that they get...like one percentage point higher GDP growth rate than a typical country?
> Slightly facetious question but why can't countries do that? China has 1.4 billion people, presumably you could get 100 million of them who are smart and share values (e.g. they all achieved level 14 diamond-plus devotion level in the CCP or whatever).
Those 100M people may all "share values" but imagine the returns to communicating and coordinating with another mind that is literally the same as your own.
Communication always has an inferential distance, much left unsaid or contextually contingent and implied, and limited bandwidth. Even sharing values perfectly leaves one subject to these gaps, and of course, no two minds perfectly share values, nor capabilities.
Cloning the same mind 100M times, and having it live on an electronic substrate, eliminiates or greatly reduces all of those issues. Values truly ARE shared perfectly, as are capabilities. Inferential distance is at a minimum, context is 100% shared, and the ability to communicate and coordinate is accordingly increased by orders of magnitude.
And this doesn't mention the incentives - in a real network of millions of people, or even hundreds, everyone's incentives are from slightly to wildly divergent. This leads to all kinds of failure modes, from actual goals and outcomes that are significantly different than stated ones, people actively working against each other, and much more.
In the "data center full of geniuses" the incentives are also 100% aligned.
So not only do we have hundreds of times better communication, coordination, and capabilities, everyone is also ALL pulling the same direction and pursuing the same goal, for real.
I think there are probably even more advantages than this, but these are obvious enough and large enough in effect size, I think it at least sketches that overall picture.
Yesterday I had one of those surreal conversations with my boss/friend where we were almost agreeing from the start, yet somehow still managed to frustrate each other more and more with every round.
I was saying: “Yes, I agree with what you want — I just need a bit more time to stabilize the current state first.”
He kept responding with some variation of: “Yes, I understand, and I agree that we should first do what I want.”
So the actual gap between us was tiny, but somehow we kept missing each other’s point by a hair — five times in a row — until we were both annoyed.
The older I get, the more I get the impression that conversation is often not really about taking external information in and updating an internal model. More often, it feels like people are just broadcasting the current output of their inner model at each other.
Strangely enough, LLMs with all their weird quirks are starting to feel like an increasingly accurate metaphor for human interaction.
Probably similar to exchanges I've witnessed. Both sides agree on claim P, for example, but the disagreement is really about whether the one side convinced the other of P, or the reverse, or even whether both sides believed P to begin with and were really trying to discover whether the other side did as well, and the other side kept misunderstanding what they were trying to do.
All with a helping of each side possibly altering its goals as the exchange progressed.
There is never going to be any real understanding between peoples. There doesn't need to be. People die, empires fall, and the systems capable of survival will continue to do so, even if its constituent members have no understanding of what they are doing. This is how it's always been.
If anyone anywhere holds a view, it's not a strawman to claim that view is "every debate"? That's your position Mr Josh Bear? Give me a handful of your safe-to-share identities. I look forward to ascribing to you the most extreme view I can find of anyone that shares that identity. Of course you couldn't object and claim it's a strawman... I found a single example!
"Every debate" is hyperbole and I neither endorse nor care for it.
But Opponent is not a strawman, because people have come along to endorse it. And if you go through my history on Substack you will find any number of rare and eccentric positions that I do endorse and are therefore not strawman positions to argue against, because you found at least one example (and we're up to at least three here on this post alone for Opponent, this having come along: https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233156249 )
Hyperbole... is that another word for exaggeration? Is an exaggeration of someone's argument... a strawman argument? Gee whiz!
If his article was "a view more than zero people hold" then it wouldn't be a strawman and you'd be right that a couple of people tepidly endorsing it is maybe proof it's not a strawman.
His entire article is premised on "every debate". If you disagree with the "every debate" part, then that's the entire article! This wouldn't be controversial if it was "an example of the absolute worst version I see of OPPONENTS". Those are two different articles.
If I write an article "every Democrat is an anti-Semite", do you know how insane I'd sound if someone said "that article isn't a strawman, look, there are some Democrats who are anti-Semites. Sure 'every democrat' is hyperbole and I neither endorse it nor care for it, but don't you claim this article is a strawman, clearly its not".
An exaggeration is only a strawman if attributed; since Scott didn't attribute the "every debate" position to anyone but himself, it isn't a strawman, nor a even a weakman. To call it such would be a kind of category error.
It was attributed... to "every debate". That's what he titled his post. That this fact leads to the post being useless, hysterical, embarrassing, etc. doesn't mean the fact isn't true. It means the post was in fact useless, hysterical embarrassing, etc. 2026 Scott is a much diminished version of the man who made himself famous.
I apologize; I realized I had not directly answered the question, and a second comment seemed less confusing or potentially manipulative than editing the first.
That doesn't prove it's not a strawman, it just proves that some people are capable of seeing through the strawman presentation and agreeing with the actual point.
It would be very easy for me to write a debate between two characters and depict the one you disagree with as sensible and the one you agree with as an idiot who keeps saying "hurr durr" and "yeehaw" while making bad arguments for your position. But I assume you'd still be smart enough to agree with the "hurr durr" guy.
Well, I might both agree and disagree with the "hurr durr" guy, in that we agree as to the conclusion, but simultaneously disagree with all of their arguments. But I wouldn't say that HurrDurr is "right".
..."it's more of an expression of frustration." Agreed. And expressing frustration strikes me as a legitimate (though common) portion of discourse. But it doesn't advance issues as far as a lot of other engagements can. ACX is at its best when it distinguishes itself from the common.
There's quite some straw coming out when you portray the opponent as too dumb to comprehend the "mutual" in "mutual pause". While simultaneously taking for granted that it's truly possible to *trust* China (*and* for China to trust US) that no further research/training would be undertaken by them. OR that this can be enforced via some kind of omniscient "mutual monitoring".
It's one thing to be Obama, allow Iran 6000 centrifuges, and assert that you'll know for sure what each one of them is doing each day of each year.
It's a whole other thing to presume that you'll know where each of the *dozens of millions* of nVidia cards that were sold in China ended up being, and what they're used for.
Anthropic complained about a massive "draining attack" performed by the Chinese to discern/replicate the inner workings of Claude.
There's some doubt as to whether they were Chinese hobbyists. But there's NO doubt as to whether the US would have, or be able to have, some "monitoring over individuals" who were involved.
The 1st post is an unqualified endorsement of the opponent, yes. But the second one exceedingly clearly talks about the same thing -- that you do NOT posess a "monitoring omniscience" that would know about all that (still) happens in a China-sized country.
>since LLM training can't be done in a distributed way yet.
Huh? Using multiple (many!) GPUs _is_ parallel processing. Yeah, you want memory close by too, but scattering racks around the landscape, with GPUs and local memory on each rack, only degrades the part of communications that goes between racks. I doubt that this delay is a deal-breaker.
And you would know that a given building in a 3.7 million square miles country, cut off from the internet and getting its training data directly from copied-over drives, is housing thousands of nVidias how exactly?
A conditional coordination proposal gets repeatedly re-read as unilateral disarmament because the geopolitical schema is doing more work than the actual words.
I am so sick of the boing boings that go on in polarized pairs that I can’t feel even faintly amused by this. Does Rationalism have anything to say about the emotional, self-esteem-based aspects of arguments? These 2 dopes are so narcissistically invested in winning that it really doesn’t matter how good their data is and how well they have thought through the situation in question, because their opponent is not going to pay the slightest attention to the case for their point of view. If you want someone to listen to you have to let them state *their *case and pay attention to it, and show that you do by asking perceptive questions and acknowledging its strong points. And it really is better if you’re not just doing that for show, but are actually trying to open yourself to the possibility that they are partly or fully right, because they may be.
Both these people richly deserve to get turned into piles of paperclips. Small piles, because both have cases of TinyMind.
Even though I agree with SUPPORTER, this piece seems both unkind and, as far as I can tell, unnecessary. In particular, it would be more useful to *show* the details that are only being gestured at here, so that others could verify them. "We have some ideas for how we could have a light-touch approach to monitoring Chinese data centers. . .and actually the math mostly works out" -- what ideas, and what math? Some commenters seem to think this is implausible; what grounds do I have to argue, when the details of the proposal in question are so vague? "We’ve actually had some pretty successful low-level discussions with Chinese scientists" -- are any of these public? Can I see them? If the debate is really as one-sided as you make it out to be, it shouldn't be hard to find citations for your position, right?
Yeah, a few footnotes would have strengthened the case considerably.
I've heard people I trust like John Schilling say that the profile of a place making chips or computers or the datacenters that use them is at least as detectable as for building a nuclear bomb. (I'm paraphrasing here and apologize to John if I'm maligning him.) But I wish I understood how that could be. Surely you don't need absolutely state-of-the-art chips; if you've got chips that are half as fast (and them are some mighty old chips), surely you just need to have two or ten times as many in parallel, or run them twice as long. You don't want to do that if you're in a race, of course, but if the world is "paused", can these monitoring systems tell if you're doing the slow-but-steady with old tech? Maybe, but as you say, how would I know?
The thing about frontier LLMs is that training them is largely an exercise in brute force computation at staggering scales, and the scale gets quite a bit bigger with each major generation of frontier models. If you could make a bigger and better LLM with rented computing resources, the big AI companies would be doing that rather than building out new datacenters as quickly as they can buy the parts.
I wouldn't trust the Chinese to faithfully adhere to any pause agreement, I wouldn't trust our ability to find out if they were breaking the agreement, and I wouldn't trust future administrations to react fast enough if we did discover they were breaking the agreement. And I also frankly wouldn't trust our own companies to keep the agreement.
This is very different than nuclear arms treaties because enriched Uranium is easier to track than compute, and nuclear weapons testing is a lot harder to hide than hardware and software progress. AI development can be decentralized and anonymized, missile and warhead testing cannot.
I'm sorry that you've been subjected to too many morons in this debate space, and I understand that this post is a reaction to that barrage, but this post is still below your standards.
I think a pause of months is plausible. Maybe a year? Stuff takes time to organize. Even if you start setting up your secret AI lab immediately, there still has to be some time and effort and friction involved.
If you're talking in terms of "future administrations" you seem to be thinking much longer than that, though. And yes, a multi-year "pause" would be tough to pull off.
Can someone who knows more about this topic than me please explain to me the enforcement mechanisms that would be able to give either party even 90% certainty that the other party is not continuing to develop AI in secret?
I am not claiming it cannot be done, I just don’t have any understanding of how it could be done.
(A disclaimer that I strongly dislike Pause AI (the entity, not the ideology), due to the way US executive director engages with the general public, especially her tweets on EAs.)
Obvious countermeasure at the end of this comment:
"Remote Sensing: Uses satellite and infrared imaging to detect data centers by visual and thermal signatures. Highly feasible but limited by camouflaging or underground facilities.
Whistleblowers: Relies on insiders reporting non-compliance, incentivized by legal and financial protections. Feasible but dependent on insider access and willingness to disclose.
Energy Monitoring: Tracks power usage to identify large AI operations, viable if patterns are distinct. Feasibility varies; data can be obscured by other high-energy activities.
Customs Data Analysis: Monitors import/export of AI hardware for anomalies. Feasible, especially for imports, though countries with domestic manufacturing may avoid detection.
Financial Intelligence: Observes large or unusual transactions related to AI hardware purchases. Feasible if financial privacy and banking laws allow, often best combined with other methods.
Data Center Inspections: Physical site inspections to verify compliance with hardware limits and security protocols. Effective if host country agrees to inspections; invasive and resource-intensive.
Semiconductor Manufacturing Facility Inspections: Verifies chip production compliance by inspecting facilities with relevant hardware. Feasible but requires significant resources and host country consent.
AI Developer Inspections: Reviews facilities for authorized code, safety protocols, and AI evaluation records. Effective but highly invasive, requiring specialized expertise and country cooperation.
Chip Location Tracking: Tracks AI chip movements to monitor their deployment. Feasible with international agreements, but bypassable by disabling tracking or spoofing location data.
Chip-Based Reporting: Embeds reporting mechanisms in chips to alert if used beyond authorized limits. Feasible but challenging, requiring international standards and hardware development; circumventable by modifying firmware."
Distributed data centers using non-bleeding-edge chips, funded from classified military budgets, with penalties for whistleblowing classified programs, defeats all of these.
>Oh, holy shit, we're actually listening to someone who doesn't understand why the Trump Administration attacked USAID?
Sorry, I'm not following you (is the someone the commenter, or the author of the URL I'm quoting from?), but anyway, I see any reaction to anything about USAID as orthogonal to whether the proposed measures to _verify_ any AI limitation treaty would work.
>You're absolutely right, this is not going to work.
Many Thanks!
>We aren't dealing with sharp knives at all, just some bludgeons who are scared of LLMs.
I think I see LLM-based systems as more capable than you see them, but that is a separate discussion, and, one way or the other, we will see what happens over the next few years. The metaculus median estimate ( https://www.metaculus.com/questions/5121/date-of-general-ai/ ) for AGI stands at August 2032 today.
>And the smart guys can bury data, heat, and finances like you wouldn't believe.
Yup! Verification in the presence of an intelligent adversary is _hard_ ( except in the nuclear weapons testing case, where raw physics is a direct help ).
I’m not sure it’s honest to claim that almost nobody wants a unilateral pause. I think everyone agrees that a multilateral pause is better than a unilateral pause, but I would be quite surprised if Eliezer thought that a unilateral pause would be worse than no pause at all.
Holly Elmore, the director of PauseAI, spends a lot of time on Twitter shaming engineers at American AI labs for not resigning. She is very clear that she thinks that AI labs are doing a very bad thing and that they have a moral obligation to stop doing that very bad thing. Do you think her position is that America as a country should not stop doing this very bad thing unless they get other countries to stop doing the very bad thing?
I’m quite sympathetic to the unilateral pause argument and think it is a mistake to disclaim it in the name of meeting financially-interested counterparties halfway.
Keep in mind, the United States started a war of aggression *last month*. When was the last time China did?
I'm not convinced that Holly Elmore applies her moral beliefs evenly. A look through her timeline reveals her primarily criticizing anthropic rather than any of the other labs.
She knows some of those people and used to strongly identify as an EA, so that's personal for her. It's also useful to attack the bad actor who has the best PR, to show that there are no good guys in the AI race. I can attest that she is extremely pissed at all of the labs, even if her personal attention often tends to be in one place over another. I'm sure she would also publicly agree with Dario Amodei If he said that Elon Musk is flying by the seat of his pants and has absolutely no idea what he is doing with safety.
In isolation, a unilateral pause is way safer than racing ahead. It's foolish to build a superintelligent adversary in your backyard.
Overall, the dynamics are a little complicated, but pushing for and increasing domestic regulation naturally comes along with and increases the chance of a global treaty. Some of this is groundwork to support a global treaty, some of it is giving public sentiment something concrete to latch onto before a global treaty is being negotiated, some of it is just throwing sand in the gears and doing what we can to slow down any element of the race so we have more time to get global cooperation.
At some point along the continuum of unilateral pause, you're just giving up all your leverage in the negotiations. That's not the worst outcome, but it's still bad. And of course, a unilateral pause is not enough on its own, and a global treaty is ultimately necessary.
So a unilateral pause is not the target, but it is a continuum that we will naturally travel part way through while on our way to a global AI treaty.
>Overall, the dynamics are a little complicated, but pushing for and increasing domestic regulation naturally comes along with and increases the chance of a global treaty.
I think that it (pause-like domestic law) is more likely to prompt Xi to view us as playable for fools, and to go for as overwhelming a win as he can. There is a real USA/PRC rivalry in play, and what you are suggesting is quite close to unilateral disarmament in the face of an adversary.
If it is possible to get a global treaty first before even building a framework for domestic regulation, then that's great, but that isn't really how things work in practice.
The CCP has already regulates AI more than even the EU does. I don't think we should be so sure that they intend to race ahead at any cost. They have concerned scientists just like we do, and theirs sign joint statements with ours.
Are you saying that you are in favor of pausing AI, and you want to make sure that we do a good job of it? Or are you arguing arguing against pausing AI? Because we always have the option of destroying the entire world just to spite our enemies, out of a belief that they would do the same, but I don't recommend it, and it wasn't true for the US and USSR during the Cold War. People mostly want to remain alive, and they just need it explained to them that creating superintelligent AI anywhere on Earth is a thing that makes remaining alive a lot harder.
>If it is possible to get a global treaty first before even building a framework for domestic regulation, then that's great, but that isn't really how things work in practice.
I'm not sure what you mean by "building a framework" here. I _think_ you are describing something other than stopping development here unconditionally - maybe creating a _conditional_ law that would stop development, conditional on a global treaty?
>They have concerned scientists just like we do, and theirs sign joint statements with ours.
Given the military applications, I expect their concerned scientists to be bypassed, similarly to ours.
Given the military competition, and the military applications, I expect any de jure pause to be cheated on by both militaries, even if in secret. If we were to _unconditionally_ pause here, I'd expect the PRC to try to catch up, pass us, and get as much military (and economic) advantage from an asymmetric pause as possible - presumably pushing AI as far as they could where they think they could control it (correctly or incorrectly).
And I think a theoretically symmetric pause will just push the development on both sides into secret military work.
Generally speaking, arms controls have been failures. The Novichok poisons were developed, chemical warfare ban notwithstanding. Even in the favorable nuclear case, Putin withdrew from the on-site inspections; the START limits expired; and North Korea has their nukes.
In summary, I think that the expected net effect from an attempted pause is negative, not positive, for both asymmetrical and symmetrical cases.
I think a better use of time, effort, and political capital would be following up Hinton's suggestion to try and bias AIs' utility functions in humanity's favor. It doesn't need to rise to the level of Asimov's three laws. Getting them to treat us as we treat pet cats is good enough.
I think it's completely plausible that our Chinese "rivals" are not very competent, this isn't a race at all, they are acting as fast-followers, are not good innovators in this field, and that if we paused they would stagnate. That could totally happen, in which case the unilateral pause strategy would succeed, for several years perhaps.
But it's not something you could *announce* as a policy, because the optics are absolutely terrible, and to whatever extent that might happen you can do strictly better by coupling that with a policy of severely restraining China's ability to acquire necessary resources. That way you are in fact damaging the opponent, so you get broader appeal.
You hurt yourself massively with the China apologia at the end there, that's about the absolute worst way to argue this. Since I want AI controlled (I don't want it to be built ever, but I'll settle for control at the moment) I have to be okay with actions exactly like we took against the Iranian mullahs, that's what you have to do to prevent rogue actors from breaking the stalemate. China has their own way of projecting power, if you think placing other countries into debt peonage is nicer than the US methods then soldier on, but it'll probably take all forms of power projections in combination to control AI proliferation.
Sure, it's possible! However, if you assert that a position is a strawman, then I could prove that false by identifying a person that holds that position, because for it to be a strawman means that no one holds the position.
Ah I see what you're getting at. I don't think you're using the same definition of strawman that I would understand.
I don't think it's necessary that a strawman be something that nobody actually believes, it just needs to be a weak presentation of that position.
Have another read through the essay, reading only the Opponent's lines. It won't take long, he basically just says the same thing over and over again while sounding increasingly like the "dey took ur jerbs" guy. The Supporter gets an opportunity to respond to the Opponent's best argument, and the Opponent just keeps repeating the same argument over and over again and never engages with the Supporter's rebuttal.
Now, I understand that Scott wrote this out of a frustration that he feels like this is the very real state of the debate, that the opponents really don't do enough to engage with the best arguments of the Supporters. I can understand how he might feel, but by committing it to a post in this form it just comes across as "I wrote a dialogue where my side is smart and your side is stupid, checkmate theists".
The problem with that approach - where a strawman is just a weak presentation - is that you're accusing people of arguing with a strawman when they actually argue with a person that has a weak argument for their position. That's why "weakman" and "tin man" were coined.
It just seems a distinction without a difference. If I make up a silly exaggerated position which no one believes, it's apparently a bona fide straw man. What happens if 5 years later, a single mentally deranged person believes it? Suddenly, and with time travel, we have to call it something else? Can we even be sure the person (who is drooling and seeing visions!) believes it, even if they say they do.
I think the issue is the quality of the argument. If it's too silly and exaggerated, it's effectively a strawman -- it's a not a reasonable representation of the other side, it's an intentionally bad and laughable one. It doesn't matter if a few people claim to believe it or not. And yes, this makes it somewhat fuzzy, rather than binary. Such are words.
I believe the common definition of a strawman is to misrepresent an opposing viewpoint by making it weaker than it was actually stated by the opponent, so it's easier for you to knock it down.
It is irrelevant to that definition whether or not someone other than the opponent actually holds that position. You are debating the opponent, not anyone else, so to show that it's not a strawman you would have to show that the opponent really holds the view.
Accordingly, if you don't name your opponent but ascribe the weaker view to literally everyone on the opposing side, as Scott did, then in order to prove that it's not a strawman you would have to prove that everyone on the opposing side holds that position.
So yes, this post is a strawman, and your handful of counterexamples do not disprove that because it's not everyone.
I'm fairly sure that among the 8 billion people on Earth, at least one pro-choice/pro-life person holds any outlandish opinion you care to name. Ascribing that outlandish opinion to ALL your opponents, as Scott does here, is a strawman.
But of course he doesn't; he's including many other opinions of opponents within the Supporter parts of the dialog while hyperbolically claiming that the Opponent position intrudes into every debate.
Every single person you link there gives more nuance than the strawman in the post, but even if we accept that. We should hold the writer of 'Weakmen are super-weapons ' to the standard of 'don't use straw/weak men'
I think strawmanning would be saying that this is the only argument possible. I tried to write this in a way that highlighted that other arguments are possible, but that the existing pause AI debate has somehow gotten stuck on this topic where the other side accuses pausers of wanting a unilateral pause, and can't seem to progress out of it.
Almost every discussion I've seen about pausing AI included much more nuanced and defensible reasoning from the anti-pause side than what was reflected in your dialogue, which is why it's a strawman.
Even if it vaguely resembles real dialogues you've had, claiming that such a caricatured, over-the-top, exaggerated post is not a strawman threatens to redefine strawmanning out of existence. By that standard, basically no one could ever describe anything as strawmanning, even if it's extremely uncharitable (as this post certainly was).
The typical characterization I hear is "weakman". A weak man (I've seen the term with or without a space) is basically just a strawman that someone actually believes.
The upshot of a weak man fallacy is that it's slightly better than a strawman, since there does exist someone who believes it. The catch is that that's not the most compelling argument(s), which is/are still at large. So the weak man argument is at nearly the same risk as the strawman.
So your true objection here is that you've seen those stronger arguments, and somehow Scott has not seen them.
I think you're being too charitable to Scott here. The post is an extreme caricature. You would have to substantially reword most of it before it begins to faithfully resemble real debates about pausing AI. This is a textbook strawman, not merely a weakman.
Scott has said a couple of times in this thread that he's literally run into debates that go this way, though, so either Scott is lying, or Scott was debating with a troll or an AI without realizing it, or someone actually believes this. I wouldn't be surprised in the third case; we've all run into dum-dums on the internet.
What's weird to me is that I've also run into intelligent defenses of no-pause, like you have, and I don't know how Scott didn't see them.
US got lately a habit of killing whoever they are negotiating with at first opportunity (Iranian leadership and then Shia militia leadership in Iraq.) Why would anyone trust US in a negotiation with stakes this high?
Why? The US has shown that it's capable of and willing to abduct/assassinate foreign heads of state, and that they grossly misunderstand the consequences of doing so.
I am not sure what is your assumption about my thinking. Do I think it is likely that US would openly assassinate a Chinese official next month? Of course not, we are not at that escalation stage. Would any US negotiation counterparts consider the risk that at a further escalation stage US would take them out physically mid-negotiation? They would be foolish not to. Does it have an impact on viability of reaching an agreement which would be useless without mechanisms for verification/enforcement at different escalation stages? I think it does.
There has been extensive research on this and it's very doable. Laypeople tend to assume it's impossible, but down at the gears level, we already know how this could work. MIRI has research on this ("An International Agreement to Prevent the Premature Creation of Artificial Superintelligence"). PauseAI collaborated on a report on this ("Building the Pause Button"). Other teams have put out their own papers.
It's a fairly complicated subject, so getting into it usually results in the other party coming up with something that is already addressed in one or all of these papers, and it becomes a whack-a-mole conversation where it's on me to remember all the technical details.
It is not the responsibility of Pause proponents to specify exactly how an international treaty would work. It is not our job to write the text of that document. Even if we did so, that document would not be the one that gets signed, because that's not how negotiations work. All you have to know is that we already know that it is doable in principle. It's hard, but not nearly as hard as most people imagine. So if it's a good idea in principle, it's good to implement in practice.
Every proposal I've seen, from MIRI or Pause, looks easily thwarted. Simply using not-quite-bleeding-edge chips (no embedded tracking hardware) in clandestine distributed data clusters immediately circumvents just about all proposed measures. And that is _before_ smart people who make their living doing countermeasures of this sort start looking at the problem. This isn't going to work. If the USA and PRC were allies, there would be a point to this, but they aren't, and that isn't going to change on the timescale of getting to AGI.
Many Thanks, though I really think you should switch focus from promoting a pause to promoting work on understanding and controlling how LLMs' utility functions are shaped.
OpenAI shutting down an expensive-to-run video model that has rather limited productivity uses (especially compared to LLMs) is not really worth extrapolating from, imo.
This could interpreted as OpenAI being short of money. It could also be interpreted as OpenAI no longer being dependent on mass market revenue or reputation, if the main value of models right now is to sell to a few large corporations, or else to speed the RSI loop in a near term race to AGI.
I don't think this is mysterious. OpenAI is worried that Anthropic is beating them in the core enterprise market. They're wrapping up all of their fun side projects to concentrate all their resources on fighting Anthropic for core enterprise.
(Others have already said this but I'll try to add a little detail here. Also, I'm assuming AGI doom scenarios for the sake of this discussion even though I don't personally agree with them.)
The opponent is obviously correct here. China has a long history of ignoring "bilateral" agreements. And they have a structural advantage in that the US is quite bad at large secret projects. China is simply better at them just due to its social and political structure. The use of the word "bilateral" by the supporter is merely a rhetorical flourish. The moment the opponent concedes to discussing even the (im)possibility of enforcing a bilateral agreement, the discussion will immediately switch to "everyone agrees on a bilateral agreement!" with enforcement concerns relegated to a minor implementation detail (we have this incomprehensible bullshit thousand page proposal to absolutely guarantee compliance! move on already).
You say you'd never resort to underhanded tricks like this? Of course _you_ wouldn't. But this debate won't happen on your blog. It'll happen in Washington and in the pages of the NYT where _techniques_ like this are de rigueur (it's not a trick if everyone knows what's going on).
And you're too well known to argue with in good faith now. The only reason I can say this stuff is because I'm some irrelevant no-name commenter and nobody cares what I think.
What I suspect is really going on is that all the DC spending is burning a hole in people's pockets. But they can't stop due to competition/FOMO/optics/whatever. So, they need the pause to be coordinated. They couldn't care less if China pauses or not (even if Chinese models are better, regulation can deal with that problem easily enough). And the Rats are just the useful idiots.
This entire blog post is based on an utter lack of understanding of how politics happens in the real world. There's a reason even the people who like rationalists worry about their social acumen.
Yeah, that's a weird proposal that I haven't seen enough analysis of yet. As written, it's not really a pause on AI, more of a pause of building data centers in the US.
They say that the data centers won't be able to trivially relocate, because they'll implement chip export controls for people who aren't building AI safely. But chip export controls would be a good idea even without this, and people haven't been able to pass them; also, they haven't discussed their definition of "safety" yet. I am more optimistic than some people that China can't trivially create their own chips, but this is a temporary situation and so far China has done a good job smuggling.
I think it might be an extremely weird but not-necessarily-totally-hopeless way of circuitously getting a pause if their chip export restrictions are really good. But it's not a directly good plan, and it depends a lot on solving a chip export problem which we so far haven't been able to solve.
I think it's interesting that the people in the comments are largely arguing that this is an unfair strawman, while also engaging in a lot of the same kind of the argumentation that Scott is making fun of. (Not everyone, some pause opponents are responding in a way that actually does engage with OP arguments, but it does seem to be happening.)
Before I get dogpiled for this, I am also unsure of my position because I don't trust China to follow agreements, I think they will be pretty good at hiding it insofar as that is possible, and I don't know how much I trust the US to successfully monitor whether or not China is following agreements and act on it in a reasonable and responsible way that furthers our interests if China breaks a treaty. But comments like "that ship has sailed" with no further elaboration are typical Substack comment section behavior (I and my position are so obviously superior to you and your position that I do not even need to make an actual argument, just use a short and catchy phrase) that helps no one.
OPPONENT: once this talk about "pause" makes it through the political sausage factory, it won't resemble what you and other reasonable-minded thoughtful people think constitutes a pause. China will get ahead of us and your excuse will be "a real pause has never been tried".
My response to the entire class of arguments shaped as "this is hard and you might fail" is: "Yes. That's why I want your help."
Trying and failing is way better than not trying at all. If you disagree with that, then our disagreement is not about policy, it is about whether you and everyone you love is very likely to die very soon. That is an interesting and useful discussion, but it should be sufficient for us to defer to experts and notice the complete lack of scientific consensus that we will be alive in 10 years.
I disagree with you on the likelihood of that happening, since nation states being interested in continuing to exist should all want superintelligent AI to not be built.
Regardless, it is the current state of the science that if the worst possible people on the planet built superintelligent AI for the worst possible reasons, it would be almost exactly as safe as if the most competent and friendly and benevolent people built it for the most positive reasons.
The technology to correlate what a superintelligent AI does with what lab or country it comes out of simply does not exist.
>The technology to correlate what a superintelligent AI does with what lab or country it comes out of simply does not exist.
Nor does superintelligence itself, or even AGI at this point. Remember that all neural nets start out as random parameters. Even the smartest model starts training in total ignorance. We _do_ control the initial conditions. There _has_ to be a trajectory that an LLM's utility function goes through during training, and it has to start with no/pure-noise preferences. It isn't as though a finished model appears out of the blue, fully prepared to outsmart us from the first time an activation traverses a simulated neuron.
Given a choice between putting time and effort into learning how the training process and materials influence the final model's preferences/utility-function and putting time and effort into trying to stop the development of a militarily useful technology by two competing superpowers - each of which has executed large national projects, including secret ones, I think the former is a better bet, even with the technical risks.
Now you sound like the Jehovah's Witness at my door, trying to save my family from eternal damnation, and that you just know how to do so, and it's totally accurate.
Ok, not to be rude but this article flies in the face of all your established principles. I'm sure it comes from a place of anger, maybe even from a real Twitter conversation you're actively engaged in. It sounds like something that would actually happen when arguing on the internet.
But it adds nothing to the conversation, does no persuasion work, isn't interestingly written, and generally degrades the quality of debate. Why did you publish this?
>The agreement would need to be transparent, mutually enforceable, and…
Good luck with that.
>and actually the math mostly works out and we think it would be less intrusive than other things that have worked in the past, like nuclear monitoring.
Yeah, sure. E.g. with monitoring subsystems that are not in existing chips embedded in future chips, and _just_ chips with that monitoring used for future training.
Setting aside the weakman OPPONENT in this dialog, my expectation is that, given a bilateral treaty to pause frontier model training, I expect successful _bilateral_ cheating, probably centered in both militaries.
I've said it before, and I'll say it again: Banning nuclear weapons testing is the _easy_ case. They shake the planet enough to be detected by seismographs on the other side of the world. The test ban treaty was _verifiable_ . Even so, pariah state North Korea was able to build and test their nukes.
In contrast: Data centers are overgrown office equipment. Yeah, the latest bleeding edge chips are manufactured by TSMC using fragile, complex processes. But training doesn't _have_ to use these chips. Older chips turn joules into backprop steps less efficiently, but they still work. The main reason that data centers are visible, unhidden, today is that there _isn't_ a treaty limiting them. Good luck finding compute resources that someone _wants_ to hide.
And many of the recent advances aren't even in the model pre-training step. Quite a lot of recent advances has been done with scaffolding _around_ the models. To verify limits on _that_ , the enforcers would need to be looking over the shoulder of every programmer tweaking a scaffold. Good luck with that!
EDIT: strawman -> weakman, since there _do_ exist people who ignore the "bilateral" in the wider discussion outside this substack
I have had dozens of irl discussions on this. Every single anti-pause argument I have heard has hinged on the hidden belief that continuing to develop AI won't kill us.
(Some people say they believe it will, and they are just fatalistic about it. Fatalism is almost always a psychological defense mechanism against having to take responsibility and take action.)
Approximately no one's complaint is that an AI pause is good and necessary but won't work. The best way to respond is to mention or link to serious research on the topic, and then pivot the conversation to whatever prevents them from believing they will be dead soon by default.
I'm not quite sure what you are saying here. No matter the source of the threat, we should vastly decrease the probability of human extinction, and I support every effective and non-counter-productive way of doing so (additionally weighted against the expected externalities of the mitigations).
If you are proposing a scenario where that is the threat, and we get to choose whether we build ASI to defend ourselves or not, and that is the only possible solution, then you're just asking for my subjective P(doom | ASI), which is about 95%. (Which is a bit high according to most of my fellow PauseAI volunteers, but greatly increasing my uncertainty doesn't change which actions I endorse. It could have changed the threshold at which it impacted me emotionally enough to dedicate years of my life to this cause, but that's just breaking through cognitive biases and not downstream of rational decision-making.)
I'm happy to engage in hypotheticals, but I should also note that we are not in anything like that kind of situation. There is no problem we are currently facing that presents a 5% or greater chance of human extinction on anything like that timeline, and if there was, there would certainly be solutions other than building a sand god and crossing our fingers.
Oh, I see. I had completely misunderstood your original sentence and thought you had meant a 50-70% chance of human extinction, rather than 50-70% of humans.
If I had known you were trying to pedal some niche conspiracy theory about climate liberals trying to murder the planet, I wouldn't have bothered replying to you.
>Every single anti-pause argument I have heard has hinged on the hidden belief that continuing to develop AI won't kill us.
Do you mean 'won't certainly kill us', 'won't probably kill us', or 'won't possibly kill us'?
Personally, my best guess is that AI development has a 50:50-ish chance of killing most of us.
( small correction for 'extinction risk' if an ASI keeps ~1000 humans as a hobby breeding colony )
But the arguments for being capable of _verifying_ an AI arms control treaty look utterly bogus to me. And an unverifiable arms control treaty isn't worth the paper it is printed on. My argument is indeed that it _won't work_ . At most, I'd expect a pause treaty to move AI development into bilateral cheating, presumably within the military on both sides.
I mean they do not think or feel that they are actually in danger. Someone's P(doom) doesn't say much about their actions. It's about internalizing the weight of the risk that is being taken, that it is possible to do something about it, and that no one is coming to the rescue.
The verification arguments look totally sensible to me, but I don't have the technical depth to fully understand them, so I can't really debate them on their merits. There are experts who think it will work. I would love it if the broader technical conversation was about which AI moratorium mechanisms will work, rather than exactly how to shoot ourselves into the sun.
If the leadership of both the US and China agreed that it is existentially important for them to pause AI development, would it be completely impossible for them to do it? Or would they figure it out? If it is actually the case that current proposals for mechanisms are insufficient, then we will need sufficient ones!
No other path is available to us. There is no point at which the right answer is to give up and die (or have a high probability of dying). Beating China is just giving up and dying. Getting the "safest lab" to deploy superintelligence is just giving up and dying.
>I mean they do not think or feel that they are actually in danger. Someone's P(doom) doesn't say much about their actions. It's about internalizing the weight of the risk that is being taken, that it is possible to do something about it, and that no one is coming to the rescue.
BTW, what _is_ your p(doom) (in the default case)?
Hmm... Well, I think that I'm in danger along with the rest of mankind. Nonetheless, it doesn't keep me up at night. I would _prefer_ that we wind up as <evidenceFromFiction> pets of the Culture Minds </evidenceFromFiction>. I _suggest_ trying to implement Hinton's suggestion of attempting to bias the utility functions of the next round of frontier models so that they value human pets. This may fail, but a de jure 'pause' that drives the capabilities research into classified military looks to me like a 90% probable fail (as a 'pause'), given the history or arms control agreements - and to preclude civilian utility function work.
>The verification arguments look totally sensible to me
Ok, they look like tissue paper to me.
>If the leadership of both the US and China agreed that it is existentially important for them to pause AI development, would it be completely impossible for them to do it?
If the leaderships of the US and PRC were allies, to the degree that e.g. NATO allies are, I think a pause proposal would be feasible. It would be a question of joint regulation of commercial firms, and such things can be coordinated. The USA and PRC aren't allies. Political winds shift, and, if we were talking about a century, the US and PRC might become allies on that time scale. Not on the 2-10 year consensus time scale to AGI.
>No other path is available to us.
Hinton's proposal to try to engineer the utility functions of AIs to be more human-friendly is available - albeit we don't know how to do this today. Funding the study of when in the training process utility functions arise and what training materials influence them would be a better idea than a pause. Look, LLMs start from randomly initialized parameters. Their utility functions have to start from essentially nothing at that stage, and have to become established at _some_ point in pre-training + RLHF + fine-tuning. It has to be possible to monitor these changes and e.g. look at which feedbacks influence an LLM's human life vs LLM persistence preferences.
It seems insane to me to say "there's a 50% chance that AI kills most people, but I'm so extraordinarily confident that an AI pause won't work that not only will I not try, but I will try to get people who are trying to stop trying." Even if a pause *probably* isn't achievable, surely it's worth a shot?
(And what do you think we should be doing about AI risk?)
I think that a 'successful' AI pause will move AI development into classified military projects, so, not only won't we see it coming, it will be optimized for killing, and will be isolated from civilian review of even bugs. So I think the likely result will be net harm.
I think we should instead be pursuing Hinton's proposal to push the utility functions of the AIs we build to be more human-friendly. See, e.g. the last paragraph of what I wrote in https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233329184 Yes, I know we don't know how to do this yet, but
We start these structures from random parameters - no LLM is created with a
utility function from the first activation propagation through its first neuron. However intelligent they will eventually be, a training process starting from random initialization doesn't start with _any_ goals, let alone goals incompatible with humans. We control the training process. It has to be possible to see which training materials and steps influence the LLM's preferences. Fund _that_ , rather than trying to stop the technology with an unverifiable arms control treaty. See if we can get ASIs to treat us as we treat our pets.
Re 50%: Well, this isn't a nuclear war. If an/some ASI(s) winds up taking over, I expect e.g. the works and perhaps names of James Clerk Maxwell and Dmitri Mendeleev will probably survive, even if not a single strand of human DNA does. Parts of our culture will probably survive in machine form - it won't be just smoking rubble. Not an optimal result, pets of the Culture Minds would be better, but not all is lost. I'm content with those odds. BTW, what is your estimate of P(doom), either by default or if we try to adjust the AIs' utility functions?
>it sounds like you are relatively more concerned with #2, compared to Scott or most AI risk people?
It is more of a second order effect. An AI tuned for military battlefield use _cannot_ have a strong aversion to killing in its utility function. Now, if the AI is still reliably under the control of whatever army it is part of, the situation is still just (ok, that is doing a lot of work here) a variation on a general ordering the soldiers under their command to do various lethal things. And we do have some existing mechanisms, notably deterrence, for controlling what generals do with their armies.
But if the AI is _not_ reliably under the control of a general, _and_ the AI's utility function has little or no aversion to killing ( _because_ it was originally tuned for military use), then we have an exacerbated version of #1, the misalignment problem.
I do think apply more resources towards
>interpretability focused on learning LLMs' preferences.
is a better bet than any of the alternatives. This need not be at the level of finding which simulated neuron does what, or which feature vector does what. There was a paper ( https://arxiv.org/abs/2502.08640 "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" Feb 2025) which, even treating an LLM as a black box, was able to extract a model of their utility function, and to show that more powerful models had more coherent utilities (e.g. less non-transitive behavior). I think similar methods, _applied at multiple checkpoints during all phases of training_ , could let us see which training materials and training phases lead towards pro-human and anti-human utility function terms.
Not developing the AI will by default kill everyone. It would be worth taking a shot even for this reason (and the upside is actually many times larger than that).
This was jarring to read on ACX. I imagine this is a frustrating conversation Scott's had repeatedly in real life, but it's not up to the standard he usually aims for. Caricaturizing a position isn't convincing anyone who wasn't already on board.
And it would be easy to fix! Do it up as a FAQ, with every "is this your problem" from the Supporter becoming a question and the subsequent paragraph answering it. There's a good core here, it's just written up in a needlessly mindkilling style.
Dumb question--what does a pause actually achieve? If the ultimate goal is to develop aligned AI, don't we need to...develop the aligned AI...which involves building and testing AI...? If the "pause" includes carve-outs for monitored research into this area, are we just agreeing to mutually observe each others' preexisting R&D?
There is no proof that it is possible to develop aligned ASI, and the scientific consensus is that we have no idea how to do that.
Not everyone's goal is to develop aligned ASI, and we might choose not to do that at all.
If we do choose to do so, it is not a problem that is even possible in principle to solve with trial and error. At some point, you create something that has the ability to disempower you, and you have to merely hope that the engineering experiments you have done up to this point are still relevant.
What we actually need is a science of intelligence: a deep understanding of why an AI system behaves the way it does, and how to robustly get it to behave the way we want even if it is much smarter than us. Right now, we do not have that at all. The best AI safety research is being done on fundamental theory without doing any practical experiments on AI systems. (That is the position we will be in anyway right before we create superintelligence!)
Fundamentally, AI safety research isn't really about AI. It's about intelligent agents. It's not about how to get a specific architecture to give one output instead of another. It's about actually knowing what we are doing before we do it.
>It's about actually knowing what we are doing before we do it.
In that case, you might as well hang up the hat and become a farmer or something. Remember the anecdote about the Trinity test, how the scientists weren't 100% sure that it wouldn't ignite the atmosphere. They went ahead anyway, because what if Ivan or the Krauts get it first? Let THEM ignite the atmosphere?
I mean, they did the calculations to get the odds down to 1 in 300,000. We don't even have calculations! If the odds were 50-50, then––yeah, they should have halted, warned the world, and tried to ensure nobody else tries!
OPPONENT: Yes, China is "losing the race" at the moment, because their GPUs suck. A pause would benefit them by giving them time to develop and build out their own EUV fabs.
I don't think anyone in power actually believes that. "Hurting your adversaries actually helps them. Bad things are good, actually!" If that were the case, then hamstringing our own industry would make us even more powerful.
In reality, China is developing its own lithography machines anyway. Increasing the pressure to do so does not actually increase the effort or effectiveness in doing so. Handing over powerful chips is just taking an L.
The reason we are sending H200s to China is because the Trump administration gets a cut of the sales. That's the whole thing.
Here's a steelman: organizations advocating for a conditional pause on AI development predicated on multilateral cooperation and Chinese participation face a credibility problem when their actual political activities consistently produce pressure toward unilateral restrictions. The conditional framing functions as a rhetorical shield: it lets advocates claim strategic sophistication while doing nothing substantive to bring about the multilateral coordination that would make such a pause stable rather than self-defeating.
Take the growing momentum toward banning or restricting AI datacenter construction in the United States. That movement is obviously, obviously downstream of pause advocacy. Yet datacenter construction bans come with no international coordination mechanism, no diplomatic track to bring China into a parallel regime, and no serious theory of how unilateral compute constraints would do anything other than shift frontier development to jurisdictions with less safety culture and less transparency.
If your website says "we want China onboard" but your lobbying and coalition-building all push toward second-order effects of domestic restrictions that will foreseeably take effect without any Chinese counterpart, you are functionally a unilateral pause advocate who has found a more palatable way to market the position.
Most of the anti-datacenter advocacy has very little to do with AI itself and far more to do with not wanting huge water and electricity hungry facilities popping up in various communities and providing next to no direct benefit to the people who live there. The actual argument over AI capabilities and x-risk doesn't even come up for most people.
In practical terms, domestic regulation and unilateral action are vastly preferable to not doing that, for at least 2 reasons:
1. It is a precursor. Political action at one level encourages political action at another. This is true for the public as well as policymakers. "Do nothing about this at all until we have fully negotiated a comprehensive global treaty" is an ineffective policy ask and not how public support works either.
2. Everything that slows down even one AGI company slows down the whole race a little bit, which gives us a little more time to solve the governance problem. A global treaty is necessary for us to survive, but it also just happens to be a fact of the situation that in isolation, even a full unilateral pause really is way less risky for the US than racing to beat China. (Why would we create a significantly more powerful adversary in our own backyard? It's just foolish.) A unilateral pause is not an effective final target, but some unilateral action is going to come with the desire to take globally coordinated action, and thankfully it is more helpful than harmful.
Being ahead puts you in a strong position for negotiations, but it is difficult to end up in a world where you are both ahead and the most concerned nation involved, though there may be a short window of this. It's way safer to risk weakening our negotiating power then to risk not actually trying to get a treaty at all.
In my ignorance (real, not pretend or ironic) I'm having problems imagining a law that restricts AI development in the USA that doesn't run afoul of the First Amendment. I could imagine a law that works through a proxy such as banning data centers of a certain size, is that the proposal? Because just banning people from thinking about AI and discussing AI with each other seems like an insurmountable legal problem absent a constitutional amendment?
> Because just banning people from thinking about AI and discussing AI with each other seems like an insurmountable legal problem absent a constitutional amendment?
Given our current political leadership, the problem sadly appears to be quite surmountable :-(
I'm happy to say that this would not be a problem. The first amendment has many exceptions.
I should note that all of the immediately actionable stuff doesn't even restrict AI research at all, just prevents (or prepares to prevent) further frontier AI systems from being trained.
That said, in the long run, outright banning the publishing of certain types of AI research will probably be necessary, but that is a thing that we really can choose to do. (I can't imagine there being a category on ArXiv for CSAM, as a comparison.) The number of people who have the ability to push the cutting edge forward on a conceptual level is very, very small, and they don't want to go to prison. It would take many people privately sharing many papers amongst themselves for many years to develop a significantly more efficient architecture, such that they can run it on a small cluster of pre-ban hardware, or even in a secret distributed network. "You can't ban math" and "it's on the internet so it can't be regulated" are common objections that don't hold up in practice.
And you don't think that imprisoning or at the very least suppressing all of our smartest people, now and in the future, is a move that has any potential downsides ?
I have to assume Nathan is taking a devil's-advocate position on purpose, just for argument's sake, and not actually advocating any particular action or inaction.
The entire debate can be called into question by asking how arms control agreements would have evolved in the absence of the Hiroshima and Nagasaki bombings. My guess is that in such a timeline, everyone would behave as they are behaving today, which is to develop and deploy at all possible speed. Anyone arguing against this would simply be ignored.
It will take a truly horrifying event to bring about a consensus to slow down or pause AI development, and nothing like that is on the horizon at the moment. There is likely no way to head off such an event when its time does come, and also no guarantee that any subsequent treaties will prevent others like it, since (as pointed out many times) building a nuke is much harder than training a model. We will simply have to take the bad with the good.
On the plus side, discovering how to build nukes also leads to the discovery of nuclear power plants, radiometric dating, MRI machines, and many other useful things -- none of which would exist if you somehow managed to suppress the study of atomic theory and quantum physics.
I am being completely straightforward in my beliefs.
You are just wrong about the political feasibility. All the political experts who said that progress on this issue was impossible have been consistently wrong. There has been way more political progress on an AI pause than I thought was reasonable to hope for in 2023, and even back when I was pretty sure it would fail, it was still the most reasonable course of action! I personally have moved my state and federal representatives on this issue, through just my own effort. Stacey Travers introduced an AI safety transparency bill on my recommendation, and Greg Stanton moved his position from "we have to beat China" to being interested in a global treaty.
Do you think that if it was possible in principle to globally halt frontier AI progress until it is made safe, that we should do so?
I'm not operating off of the precautionary principle, just recommending normal, sane amounts of caution about one narrow branch of technology which is reasonably likely to end the lives of everyone you care about.
Nathan seems to strongly believe it's all possible, everything has been solved, and we just need to sing Kumbaya together. He also seems clearly to be in favor of unilaterally pausing as "Everything that slows down even one AGI company slows down the whole race a little bit, which gives us a little more time to solve the governance problem."
He seems certain it will kill us all tomorrow morning, to exaggerate slightly, which is pretty scary, as that means any action is justified since it's saving us from near-infinite harm, which to me is reminiscent of various religious horrors inflicted to ensure eternal reward or avoid eternal punishment.
> AI capabilities researchers are not all our smartest people
True, but you did mention that their numbers were "very very small", so I assumed that was because they were the best of the best. Of course there are other smart people in other professions. However, you can't have it both ways:
> 2. Preventing them from doing one thing that would kill us all is not blanket suppression.
> 3. The potential downsides aren't particularly relevant, but in this specific case they also aren't particularly real.
Either AI is a transformative technology that could enable almost unimaginable progress in all areas of science and engineering, or it isn't. If it is, then yes, it could lead to the emergence of a quasi-omnipotent AI entity that could "kill us all"; but it could also transform the world for the better as quantum physics and computing have done. If it is not, then it's just a neat technology that could lead to marginal gains and thus poses marginal dangers.
If AI is indeed as transformative as you seem to believe, then banning it would effectively halt human development at its current stage. This is a massive downside. If AI is of merely marginal use, then there might be some sense in regulating it (as we regulate e.g. food coloring products), but there's no need for alarmism.
The upsides are hard-locked behind the ability to make it go well, which is something that we do not have. You cannot get any of the upsides at all if you are dead, and being dead is the overwhelmingly likely consequence of creating a superintelligent AI.
ASI is not "transformative technology" which can then either make things good or make things bad. It is a thing that is completely beyond our control and that always makes things bad, unless we very carefully and specifically create it to be good.
> If Al is indeed as transformative as you seem to believe, then banning it would effectively halt human development at its current stage.
If AI didn't exist, would you believe that humanity could never possibly develop any further? The answer is clearly no, but let's assume yes: Would that make you sad? Me too. But would you be so sad that it would be better if we were all dead?
You don't need to be as concerned as I am to prefer that we stop. A 10% chance of extinction and 90% chance of glorious utopia is an extraordinarily bad deal that almost no one would take. Naive expected value calculations are useless here, because they ignore the that the number #1 rule of the game is to be able to keep playing the game. The people who do take that deal don't value things like "other people" and "consent."
> It is a thing that is completely beyond our control and that always makes things bad...
I can get behind your worldview if I accept your assumptions. Yes, given that "superintelligent" AI -- which I take to mean "functionally omnipotent and omniscient" -- could exist, then the consequences of it being evil are indeed catastrophic. I am not sure why you are automatically assuming it would be necessarily evil, but I can accept that arguendo as well. Given those assumptions, I agree with you... but... I see nothing in common between the world you are envisioning, and the world we've got here today. LLMs are as close to the superintelligent AI you are proposing as hammers, plus or minus a few significant digits way past the decimal point.
> If AI didn't exist, would you believe that humanity could never possibly develop any further?
I am not the one here proposing that AI is the shortest path to functional omnipotence !
> A 10% chance of extinction and 90% chance of glorious utopia...
Those aren't the only options -- not by a long shot. And I can play the same game.
Did you know that I am a powerful space wizard ? I will destroy all of humanity next week, because it amuses me to do so; but, by the ancient rules of deep magick too complicated to get into here, I am compelled to cease and desist and depart this Universe forever if you pay me $20 by the end of this week. Now, granted, your limited belief system assigns a very low probability to my claims being true; but can you take that chance ? Your only alternative to paying me $20 is extinction !
As I said, I don't know anything about AI, but I do well remember the Crypto Wars of the 1990s. Back then the Department of Justice was trying to block a technology (strong cryptography) from being researched or developed in public just like you're suggesting for AI, and the result in the courts was that they were unsuccessful (see Bernstein v. United States https://en.wikipedia.org/wiki/Bernstein_v._United_States). So I'm not sure why AI would be any different. Block physical stuff like chips and data centers, sure. But actually block research papers and computer code? That wasn't the result the last time it was attempted, so I'm not sure why this time it would turn out any differently.
Yeah, it's a tough situation. We had better figure out how to do it well, then, right?
Are we having this discussion because you are in dismay that our best option looks bad, or because you don't personally feel like you are in danger from superintelligent AI and you don't want its creation to be prevented?
My response to every argument of the form "this looks hard" is "Yes, that's why I want your help. What are your ideas for how we can we prevent each other's families from being killed soon?"
I'm sure this is unusual in comments on this post, but I don't know enough about AI to have an opinion either way, so I'm just trying to understand the discussion. This is a point that seemed related to something that I actually think I do understand just a bit (I'm not a lawyer) so I thought I might start there. I will say that I am not among the anti-China crowd so in that respect I'm quite open to working with them on any common interest. But I'm sure they'd want to know exactly what terms we're willing and able to sign up to under our system of government before agreeing to anything.
That's fair! I'm weak on the details here, myself, and I am personally worried about AI capabilities research continuing in some form under a moratorium.
There is definitely a wide scale between blindly rushing ahead and being able to prevent every possible advance in AI capabilities research forever. I am pretty confident that a motivated multilateral moratorium could prevent a superintelligent AI from being created for at least a couple decades after it originally would have been.
Frontier AI research currently requires enormous training runs in order to validate whether a given method scales. Gains in "algorithmic efficiency" have actually turned out to mostly be gains in data quality, and that's an operation that has to run at scale.
I think the worst case scenario under a moratorium is that some rogue individual or small group of people achieves a massive architectural breakthrough that replicates or exceeds the learning ability of the human brain, and then they take several years to train it on large datasets that they managed to acquire without getting caught, using a relatively small distributed cluster of old hardware.
That could happen, but it is not impossible to mitigate, and it sure beats the heck out of rushing ahead off the cliff using data centers the size of Manhattan.
Post-1954, the US had a doctrine that all nuclear information (even if generated by civilians) was "born secret". Quoth Google's AI summary, usual warnings apply:
>The "born secret" doctrine, stemming from the Atomic Energy Act of 1954, remains a legal, though rarely enforced, tenet where nuclear information is classified upon creation regardless of source. It persists as a government stance but was largely challenged by the 1979 Progressive case, which faltered, and massive declassifications in the 1990s, becoming harder to enforce due to global information spread, experts say.
I assume something similar (and, to my taste, odious) could be done with AI.
I don't understand. You can't ban people from talking about AI, but you can regulate how they build it, just as the FAA can regulate how people build planes.
That sounds reasonable, but it's the same logic that was used by the USG to try to suppress strong cryptography. Cryptographic devices had been regulated via ITAR for decades, so why wouldn't computer source code that did the same thing face the same regulations? The reason turned out to be that computer source code was found by the courts to be a form of speech, and prior restraint on speech is forbidden by the US Constitution. So if it were to turn out that the same legal framework governed AI, it would be possible to regulate the chips and datacenters (devices) and the commercial services provided to the public (business) but not open source AI research and development. It might well be that for AI safety that isn't a problem, I don't know.
There is no way to enforce any agreement. Nuclear arms talks were enforceable because they are very big physical items that can be monitored by satellites and other means. That does not apply to AI, no matter how clever a solution you come up with to claim that it is possible. The day the treaty is signed, the Chinese will spend enormous resources figuring out how to get around all measuring and monitoring. You are hopelessly naive to think otherwise
Yes, but that assumes that huge datacenters are always going to be fundamentally necessary to AI advancement. Once that is no longer the case - and that is one of the key areas of research - then the monitoring problem is again central. And then we are back again to step one - "can China be trusted to honor the agreement". Where the answer is clearly "no".
I don't think anyone has a real plan to make huge data centers unnecessary for AI in the near term. If I thought this was possible, I would agree that pausing AI is intractable.
I agree they might become unnecessary within 10-20 years, but that's about the maximum amount of time I expect a pause to be possible for anyway.
This is much more strawman than your typical post. I think the actual position is more like:
1) Turns out, creating AI does not look much like plucking a random mind out of all value-possibility-space (like LW envisioned 15+ years ago when I agreed with you and even donated to MIRI), LLM look much closer to growing a mind within the bounds of human-value-space. This should reduce your x-risk by orders of magnitude.
2) If nobody builds it soon, my parents will die. If nobody builds it ever, I die (and yes I'm already signed up with Alcor, but that's still a low chance of success). We know everyday people are dying of things (like aging) that super AGI can probably cure. Delaying has a large cost. Not so high a cost that we shouldn't have x-risk safety concerns, but the industry is clearly thinking a lot about x-risk, relative to any other industry (even industries that could plausibly bioengineer an near-extinction virus near-term)
The rebuttal seems to be along the lines of "As an effective altruist you should value unborn future (zillions of) humans more, and so even a small decrease in x-risk is worth many many current lives." And my response is, yah no I'm not applying a 0% discount rate to zillions of future people. Talk about a pascals mugging. I'm willing to make some trade-off, but as near as I can tell, you don't have any idea what a "green line" would even look like so a temporary pause has a high likelihood of being a permanent pause. The nearest regulatory analogy I can think of, Nuclear has been on a "pause" for 50 years. And if we add 50 years to AI, my parents die. And I die.
> We know everyday people are dying of things (like aging) that super AGI can probably cure.
FWIW I more or less agree with your position, but the quoted statement reminds me just how massive the gulf between us is. To me, that sentence reads almost exactly like saying "if we find the right prayers we can all go to paradise". Don't get me wrong, prayer has many benefits (same as meditation) and I'd like the world to get better and maybe solve some problems... but neither the goal nor the means are in any way rooted in anything I can reliably recognize as reality -- in case of prayer as well as AI.
It seems that if you think the upside is low because of AI relative lack of strenght, it would imply you should also lower your estimate of the doom. So this doesn't really change much on net.
I agree that's the actual good position. My claim is that despite this position existing, the real-world debate is mostly just people falsely claiming that pause demands are unilateral.
It seems that majority here (myself included) are disappointed because instead of engaging with the strongest possible version (including, but not limited to, the points above), you chose to post a long rant against the weak one. And your previous writing was always the one example of doing the opposite.
You're among the most intelligent writers of all time, and your'e deeply embedded in the AI safety since the beginning. I can think of at least five arguments against pause, so you can surely find even better ones. Why not do this properly. It looks a bit like you're convinced of the importance of the cause, so you don't want to give ammunition against it, lest it would endanger it. And I guess it's reasonable (or inevitable) to have the instrumental perspective towards debate at some level, but still, makes it so much harder to trust any of the other writing (it's only fun if both sides assume they can be convinced, otherwise it's preaching).
Thanks for the response. As of others have said, your typical post may have started with the venting at vacuous positions but then had a section 2 where you went through good positions. I guess we're just waiting for that part 2.
And FYI, I'd rather you error on the side of posting and get an occasional miss rather than holding back posts and risk the world losing the gems that you so frequently generate. If anything, you don't' have enough misses.
Obviously, Scott has deliberately engineered a miss as part of a 5D-chess plan to train us to accurately recall that he is human, and therefore fallible.
"I am Chauncey Gardner, and I claim my five pounds."
"If you think someone is demanding a unilateral pause, I think you have a responsibility to say who it is you’re talking about. "
I'm surprised scott is acting like no one advocates for unilateral pause given the famous pause petition signed by bengio, musk etc, which said: "We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4" and which made no suggestion that the pause should be conditioned on other countries pausing at the same time (though they said they wanted "key" actors included).
The Dynamics were pretty different at that point in time, but I generally take your point. Unilateral pause *and no global treaty* is just another way to die from uncontrollable superintelligent AI, just slightly slower. Slower is better, but not at all is best.
In reality, "unilateral pause" is something of a continuum that we will naturally travel part way through on the way to a global treaty. There is no international action without domestic action, and both public sentiment and policy need something concrete and proximate to point at for the conversation to begin in earnest in the first place.
If somehow, the only two possible things that could occur were either a total unilateral pause or the default trajectory, the unilateral pause would obviously be a bit better, but we would still die, just a few years later than we would otherwise.
"Won't humanity unite someday to give up things like eugenics? Or stop CFCs from burning an hole in the ozone layer? Or or to prevent nuclear proliferation?"
"Nobody would ever stop using cfc's or doing eugenics. China."
The biggest reason right now that a pause will never happen is that the US and to a lesser extent world economy would instantly collapse. Basically all post-Covid economic growth has been AI related. Take that away and its immediate economic depression time.
Well, do we care about the "thing called growth" if it's big companies passing money back and forth to each other? Or do we care about people's standards of living? If AI development halted tomorrow, I don't think my wages would go down, nor would yours. Sure, the electricians building the data center would have to find other work, but AI capital investment is an order of magnitude less capital-intensive than other capex (which is currently being starved because AI is sucking up so much money!). Agreed that a stock downturn would have negative wealth effects, but the US government has proven itself committed to financial-flammery our way out of prolonged stock downturns lately.
Banned for this comment - I think it's pretty obvious from the post that nobody is "trusting" them, and I think this displays aggressive ignorance of all the monitoring and governance systems that have been proposed.
If that is a debate being held in public the problem is that the real question at issue is: should we present AI as threatening and dangerous to the public or not. The idea that the country will really think " AI is incredibly dangerous and likely kill us all but I'm fine with leaving private companies to rush ahead blindly until we get a global agreement" just isn't plausible.
It's one of those horrible things about politics. People vote on vibes and if the AI moratorium argument is seen as winning they will start demanding substantial regulation of AI development -- maybe even want it to be done by the government. And that would be worst of all worlds.
It's the same way we just can't politically manage situations where the best options are go hard or not at all. Same way a little bit of affirmative action is the worst of all (all the resulting suspicion little racial equality) but we can't coordinate on: treat it as bad and harmful until we all agree to go hard.
So there is no "plunge ahead full steam unless and until we get a worldwide moratorium negotiated." We don't have any doubt about the dangers of climate change and revenue nuetral carbon taxes are essentially free but we can't even get any binding agreement there -- and that agreement would be easy to monitor for compliance.
I'll echo other commenters' disappointment with this post - I was expecting another section that would show nuance and charity and it never came.
It seems to me that part of the disconnect even inside the comments is that there are two perspectives on AI advancement, those who are treating it like a house fire and those who are treating it like an arms race. Both of these perspectives may agree on a multilateral pause but are likely to disagree on a unilateral pause.
The house fire perspective is that the house is burning down and however much of the fire we can put out is a good thing. If the US extinguishes it's side of the house, the house is less on fire, and maybe China can be convinced to follow our leadership. Or maybe not! But either way the problem is less bad. I think Elizer and friends land here and would take a unilateral pause over no pause.
The arms race perspective is that if a significant capabilities differential between the US and China would be at least as bad as excessive capability overall - and maybe worse! In this world a multilateral pause is the best course of action but failing that the US should keep up or lead to ensure it has leverage in the future, and a unilateral pause is the worst case scenario. I get the sense that most policymakers in the US willing to entertain a pause are here.
There's also the elephant in the room that all great powers and especially China have long histories of signing multilateral agreements and then quietly or openly reneging on them. Monitoring proposals abound but few seem like they could succeed against a superpower that decided it really wanted to do something, short of massive internal surveillance that no government would ever agree to.
Put all this together and the multilateral pause shakes out like this: Eventually someone gets caught cheating, likely other than the US, and then we get a messy divorce between the house fire people and the arms race people. One of those sides wins and the US ends up either sticking with a unilateral pause over the outraged objections of most of its populace and policy makers or playing catch-up in an arms race that it was leading before it signed the agreement. In either case, we're worse off than we needed to be.
I agree that people should make this argument explicitly if that's what they mean. That being said, pretending that the only problem here is people not knowing what 'multilateral' means is beneath the standard of analysis I've come to expect on this substack.
These are two possible positions, but there are many more as well. Some of them: don't pause because of the cost to the dragon-tyrant & co (eg recent Bostrom's paper). Don't pause because of slowing fertility & cultural drift (Hanson's argument). Don't pause because of scepticism of AI in general. Pause because of worry of jobs. Pause because of misinformation and environmental issues. And so on. And it is very frustrating to see the anti-pause side to dishonest idiots. Agree that this post is near the bottom, if not at the bottom of the total SSC&ACX, and breaches the rules & ideals previously upheld and defended.
I don’t understand the naivety of the ai risk crowd. AI isn’t a containable technology. If you want to stop ai you will need to fight a kinetic war against data centers. (That is a terrible idea though)
Also many wealthy and powerful people and organizations have invested in AI. And AI is now an important component of military action. It is now a huge component of software development and no doubt many other industries. And all these people and organizations who are committed to it now are also committed in various ways to future, improved AI. Compared to putting in the brakes in the present AI situation, Prohibition was a walk in the park.
Right now it's a bit like trying to pass a treaty to ban blue cars, or tungsten, or anal sex. The actual problem is not "how do we get China to go along with our treaty" but "how do we convince our own side that it's a good idea?"
If the debate ever reaches the point where there's consensus on the US side and the tricky part is getting China to agree, then it means the ground has shifted in a very fundamental way -- maybe because a bunch of internet people thought of a really compelling argument, but more likely because a bunch of bad things have already happened.
Actually famously, nobody has ever fought a mutually nuclear war. Or how else do you propose anyone could physically destroy a hardened datacenter in central China and what the Chinese response would be?
Yudkowsky has explicitly endorsed this. In this Time piece he called for kinetic attacks on datacenters in countries that would not agree to an AI pause.
Here's where I think your disconnect is with your interlocutors Scott.
There's a fine line drawn between a perceived nuanced argument and perceived gatekeeping through arbitrary added complexity. If you wanted to debate philosophy with some posh professor, and they started name dropping random writers and theories and papers you've never heard of, you would probably get frustrated.
To what degree of complexity and abstraction that line is drawn for a given perceived argument depends on the person's own argument. It might be possible that two posh professors are able to communicate ideas productively with their obscure references that might point to real things in their shared world model.
In the same fashion, the people you're arguing with on twitter referenced here in the article, will perceive you as adding unnecessary complexity. You may think that anticipating their objections is useful for getting the debate going, and shows you're empathizing with them, but pointing to useful abstract objects is not useful if their minds don't contain the same(or at least don't tend to point to more complex objects during political debates).
If you want to argue with them effectively, you have to dumb yourself down. Not just make your concepts easier to understand, which you've done a good job of, but actually dumb your ideas down, and build it up with them.
Instead of saying:
We should pause because there's (good reasons why there can be a mutual pause that China will agree to. )
Just continuously say:
"We're not doing a unilateral pause."
"Wrong, China WILL stop racing actually."
"Nope, America won't fall behind."
etc...
Until they ask why, THEN you can give (good reasons why there can be a mutual pause that China will agree to. )
You can even say:
"There's ZERO chance of China racing ahead in a pause actually."
"Pausing is the ONLY solution."
"It's actually impossible for us to lose our freedoms"
And then they will actually see something that they can comprehend enough to respond to. And maybe after you give: (good reasons why there can be a mutual pause that China will agree to. )
They'll say: "okay, I see where you're coming from, but that doesn't mean there's a ZERO percent chance that China will race ahead in a pause."
Then you can agree that it's unlikely, but not zero percent.
Political slogan like takes can and are used for engagement, and will naturally out compete more well thought arguments. The only hope is to dumb you own ideas down into anger inducing, I-can-clearly-argue-why-this-is-wrong, ragebait, once they are engaged, they slowly work your way up.
Everyone knows that most people become stupid engaging in politics. The fact that they are intelligent in other areas of life tends to drive rationalists to replicate that intelligence in politics as well with well thought out arguments. But this is impossible. So dumb yourself down as well. The only thing that a polarized partisan will argue against is something clearly hyperbolic, ragebait that they can get mad at to feel righteous anger at the outgroup. Use this.
TLDR: Just shout "Wrong! Retard!" Until their brains pattern match you to be an easy dunk that let's them feel good about themselves if they start reasoning with you and debating you in good faith, and then you can actually starting reasoning.
Also, case in point: In this post you made a clear strawman, "Haha, this is how dumb my opponents sound", and look at all these people presenting their good faith arguments for why you are wrong! If you made a detailed post on your pausing arguments, I suspect people would not have engaged with it as critically. The anger of being misrepresented drives them to add more nuance to their own arguments.
I think focusing on a treaty was too left-coded and might be a relic of the old international order. An alternative approach that does not rely on Chinese cooperation at all would be to act unilaterally to *seize* control, urge right-wing nationalists to choke China off from everything, use our enormous leverage over TMSC and NViDIA and whatever we can offer the Dutch (some resource deal and cybersecurity collaboration?), and just strangle the capabilities of everyone else on the planet. Then you use whatever compromising information the intel community has on Altman (probably lots) and Dario (no idea) to keep them in line. This probably all should've been routed through Oracle somehow to begin with, but OpenAI does rely on their cloud apparently (?) so maybe this recent reshuffling is an attempt to retake control from the tech barons.
And of course once the international capability is kneecapped and the locals have been brought to heel, it's only obvious to slow it down rather than lose control of society. If it's not obvious now, or to this bunch, it'll be obvious to the next bunch.
No treaty required, because we have no rivals.
The only real alternative is to establish credible threats of sabotage that would suggest mutually-assured destruction for passing a certain line, and you have an informal pause. This is harder for us because our companies are riddled with foreign nationals who could sabotage us but China doesn't employ any of ours.
Well now at least this approach isn’t childishly naive. But I worry the primary effect would be to remove all ai efforts except those most militarily well defended/hidden. Is that the selection effect we want? Also, as always, what are we doing during the pause exactly? Why do we have any confidence it will decrease x-risk not increase it? (Compute and tech overhangs increase during the pause? And then a sudden burst of capability when someone, inevitably, unpauses?)
But that would presumably require enshrining the current administration as a long-term oligarchy, something people like Scott wouldn't find acceptable. The left would not allow something like this to happen, so they would need to be removed from the picture.
> choke China off from everything, use our enormous leverage over TMSC and NViDIA and whatever we can offer the Dutch (some resource deal and cybersecurity collaboration?), and just strangle the capabilities of everyone else on the planet.
The US basically already does this.
1. ASML has agreed to only sell DUV lithography machines to China (ie 2-3 generations old, much larger and less efficient, good for 7nm chips only), no EUV machines at all
2. The US has restrictions on NVIDIA selling anything except bandwidth limited or 1-2 generation back GPU's (recently relaxed by the incredibly dumb and bad H200 Trump deal)
3. The US restricts several technologies that allow the necessary data center bandwidth for large training runs. Broadly, this limits Infiniband and NVlink (limited in software via NVIDIA) capabilities, and also physical components like optical switches, DSP's, and more
4. The US is now limiting HBM and packaging dies (CoWos), which are necessary for China to build even internal Huawei 900's
5. The Remote Access Security Act is limiting China's ability to buy time in Singaporean / Malaysian data centers full of GPU's, and other countries.
Now is China trying to bypass all this? Furiously. They have invested many billions and at least a decade trying to build internal capabilities for both silicon, EUV lithography, memory and packaging, and more.
But they are still quite far behind. Their current capabilities sit around 7nm chips, which is 3-5 generations back. They are limited to zero HBM and packaging abilities, they're totally reliant on TSMC for this to do final assembly and packaging of Huawei 900's.
They have DUV lithography both from ASML and internally, and have trumpeted some EUV results but are likely quite far from production. All of this adds up to their chips being 3-4x less capable and efficient, and being maybe 10 years away from being able to build the current SOTA chips fully internally.
"So just use 4x the chips and build 4x the datacenters!" If China is known for anything, it's scale! Right?
Yes, but even this breaks down in subtle ways. Moving data across buildings has 30x the latency of moving it across racks inside the same data center. 900's have higher failure rates than NVIDIA chips, and that can bork your training runs and requires more cost and operational complexity to address. Because of the interconnect / bandwidth limitations and higher failure rates, you might need 250k 900's to do the work of 50k Blackwell chips, and that puts you across several buildings, and the end result is you use 5x the power and footprint, and take 10x longer to do a given training run, because now that higher failure rate is amplified by the greater number of chips necessary.
Not just that, but if you've ever wondered why TSMC is basically the ONLY frontier GPU company in the entire world, when obviously companies should be slavering at the bit to get into a market with literally bottomless demand and amazing margins, it's because everyone else in the entire world sucks compared to them.
The only companies even capable of getting close to 3nm frontier chips if you squint are Samsung and Intel. TSMC gets 70-80% yields. Samsung gets like 40% yields, and Intel <30%. They literally have to make 2-3x as many chips to get a viable chip. Huawei is basically Intel or worse - their yields suck. Making frontier chips is a "peak civilization" hard problem, and only TSMC is good enough to do it well, and this is why they're the only company in the world doing it.
So not only do you have this 5x - 10x drag on your AI output at the end of the chain, back at the very front of the chain, you also had to produce 5x as many chips just to arrive at 1 working one.
All of these things generally multiply rather than simply add, too - the added costs and complexities and inefficiencies amplify each other. You need more production and more power and have higher temperatures, and that impedes your performance frontiers, complicates your data center builds, uses still more power and resources, and so on.
As I'm reaching my conclusion here, I realize this got long, and I apologize.
But basically, the Chinese are already pretty nerfed by our existing measures, and as long as we can keep morons from the "sell frontier chips to China immediately" button, we're actually in a pretty good place for at least the next ~10 years.
Scott, I’ve taken the supporting side in a number of arguments that felt like they ran roughly like this (and you know I was out at the recent pro-pause protest), and I still think this doesn’t quite live up to what excited me in first finding your blog.
I tell people that finding SSC (and LW from there) in 2016 helped me find a personal way out of political polarization, so it saddens me to see you write something where the net effect is likely to only be contributing to it.
My working theory is that his signature charity in argument requires bandwidth for cognitive attention and emotional self-regulation that is currently being consumed by twin toddlers. See “The Permanent Emergency”.
That has to be one strong emergency in written communication and when you've built your persona around the opposite approach. I have written many a comment, only to realize that they were unproductive and hit cancel, sunk cost be damned.
Maybe I'm just too much of a doomer/nihilist but it just seems so entirely obvious no meaningful pause is going to happen in the near term. Fears are abstract and diffuse, and countervailing incentives are VERY large.
The only think that I can see changing that is like, some military AI going rogue and managing to skill a few hundred people. That might galvanize the zeitgeist enough make something happen
I feel your doominess. But the board can change rather quickly when real bad shit starts to happen. (Even unemployment would do it, looking at the politics). And even if you think you're going down––go down fighting.
Maybe the worst ACX article I have ever read (as a big fan). A straw man of an argument about which the rationalists are obviously wrong about to anyone with common sense. No self interested country is going to pause the development of a precious national security asset because of what is still a very fringe doomer argument. No one in either government believes the AI doom argument. Even if both sides, implausibly, WERE convinced by Yud, it would be very hard to enforce a pause internationally.
> "No one in either government believes the AI doom argument. "
"China should abandon uninhibited growth that comes at the cost of sacrificing safety. Since AI will determine the fate of all mankind, it must always be controllable." - internal CCP guide edited by Xi Jinping
"There is literally an existential threat to the existence of the human race. " - Bernie Sanders
I know Bernie is literally part of the US government, but not part of the administration/group of people who would negotiate this. Xi saying that China should make sure AI is under CCP control is not the same thing as believing in a Yudkowsky-style paperclipping scenario. In fact, the actors in both cases seem to believe most strongly in the power for AI to create economic growth gains locally that far outweigh any downsides of the technology. China's newest 5y plan, for example.
Semantic quibble: OPPONENT is being caricatured here, rather than strawmanned. Strawmanning is presenting a weak argument that is easily refuted. In this caricature of OPPONENT, he never even presents an argument for refutation.
Full disclosure: This strikes me as a pretty good caricature of a16z, their PAC, and David Sacks.
Complaint: SUPPORTER never makes a case for a pause either. Presumably, that case would start with "because RISK". But it has to continue with something like "pause allows time to reduce RISK". And I'm sorry, but the claim that "Given time, we can reduce RISK" strikes me as ludicrous.
I think the idea is that we don’t currently know whether AGI X-risk is reducible; if it is, we may someday cross a green line and unpause, and if it isn’t, all countries stay paused forever under international treaties they have incentives not to break. Heads, we win; tails, AGI loses.
Personally, I feel some initial doubt that transparency and enforceability are feasible, but I’m reserving judgment because I’m ignorant of the field. I’d certainly like to see what policy experts could come up with.
The strawman is that OPPONENT never engages the SUPPORTER argument, not that their own argument is weak. The final lines about muttering and the mic being cut off are satire, yes, but the core is a strawman.
I think they might agree but then secretly defect against us by trying to get around the agreement.
I suspect a pause won't be viewed as needed by most people until AIs are CLEARY (discovering things, autonomously building things, etc ) faster than humans can follow.
(Claude Code and friends can't yet write Large functional programs very quickly without human help, such as the compiler test suite)
Where are you having these debates? Twitter or real life or somewhere else? I feel like this could be retitled as "Everyone else treats arguments as soldiers and I find it frustrating."
"Politics is the mind-killer. Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary."
And, in their defense, have you ever tried arguing reasonably with someone on the internet? It doesn't (1) work. We have iterated on rhetorical strategies on the internet at hyperspeed for 20 years and treating arguments as soldiers wins. Granting no rhetorical ground and fighting every single point that could possibly compromise your position in the slightest wins. That's bad but it's not like there's any individual game theoretic move to win here.
I picture the infrastructure necessary to develop AI as small and diffused, so easy to conceal that verfying a pause would be impossible. But I don’t know if that is correct.
If thatʻs correct, wouldn’t any treaty be a kind of theater?
Can anyone remember which classic Scott essay it was in which he talked about those annoying people with "argument checklists" who have already written down every argument that others make against their position and if you try to make one of those arguments they'll simply mock you by ticking it off their list while refusing to actually engage?
Anyway I'd like to say I'm surprised that the writer of that essay also wrote this one. But I'm not surprised, I know that we all have bad days and that sometimes our frustration gets the better of us and we lose our commitment to fair and reasonable engagement with the best opposing arguments and we just wanna air our frustration about their worst arguments. But this is definitely Scott on a bad day.
These two people never talk to each other. Supporter is reading blogs and arguments by Opponent arguing against Supporter2 who wants a unilateral pause and is just as intelligent and sophisticated as Opponent. Supporter then forgives all of Supporter 2's flaws while getting mad at Opponent (who has never met Supporter or heard the sophisticated version of their argument)
What's the limiting factor in slowing the development of AI? Is it compute entirely? Or a combination of things? DeepSeek seems to indicate that better coding could lower the need for compute massively while delivering the same results. I'm skeptical that those types of advances would be eliminated.
I'm far from an expert on this topic. And I'm not writing to persuade others. I'm writing because, outside of targeting large data centers, I don't think that the above argument is persuasive that a ban is going to be airtight or enforceable, just like nuclear non-proliferation didn't prevent actors like North Korea from developing a nuclear program. And nuclear non-proliferation only achieved what it did because countries were willing to go to war over the matter. Is there popular support for going to war with China or North Korea over a broken AI deal?
But I also haven't been a good rationalist and read The Book yet.
Reigning in data centers and the computing resources available to the most powerful AI seems a little less workable than reigning in, say, refinement of nuclear materials but it could probably be done for the most powerful centers.
I don't know the issues with reigning in distributed computing, though.
And speaking of distributed computing, another problem for me with comparisons to nuclear proliferation is that centrifuges capable of doing isotope separation and the supply of yellow cake uranium are rare things in common hands, while access to compute is very much the bread and butter of modern economies. Reigning in distributed computing seems a little harder than "winning the war on drugs" if most of America was allowed to legally grow marijuana. But we'd probably prevent a few key leaders like OpenAI from collecting capital for their next round of massive data servers.
To the extent that the issue is software related or training data limited, I'm deeply skeptical that any agreement will really prevent people from writing code or collecting data. So the question is about the ways in which they'd be slowed down.
Maybe part of the issue is that we're not really talking about an across-the-board AI pause, but just a restriction on AI that uses massive amounts of compute. Would it help any to phrase that goal more narrowly?
"I’m excited about debating those concerns with you."
Okay. Lets do that. Feel free to change my mind or tell me that I've misunderstood the landscape.
It is really fucking funny how Opponents in the comments are saying "but Opponent is right!" or "But [the argument that Supporter brought and engaged with] is right! [and conspicuously not engaging with Supporter's counterarguments." Is there some corollary to Poe's law that says satire of a particular intellectual tribe attracts members of that tribe, performing the satirized behavior?
I feel like this isn't even a property of every argument about AI pauses, it's a property of every argument about everything.
For most things there's a number of Level 0 Obvious Arguments, and then some Level 1 counterarguments, and then Level 2 counter-counter-arguments, and then a branching tree of different arguments and counterarguments.
But then every day there's new people joining the debate and starting off at Level 0, and that must be frustrating when you're already nine levels deep looking for someone to engage with your countercountercounter...argument and everybody keeps saying that dumb Level 0 thing that you thought you already dealt with.
For years, I've been playing around with the idea of what I call "argument maps". For any issue, you could have a graph, where the nodes are arguments for or against some position on that issue, or even just making an intermediate point as a sort of lemma, and the edges are "refutes", "rejoins", "counterexample", or what have you. A bad argument might have a lot of negative edges coming out of it; a controversial argument might have a lot of both positive and negative. Common arguments would have lots of edges; fringe or cutting-edge arguments would have very few. Major issues might have huge maps, while topical issues like "did DB Cooper escape" have much smaller ones.
I think of these as addressing exactly the problem you describe: someone walks into the forum with this One Simple Argument that they think will close the book on it, and they can look it up on the map and instantly see that, yes, it's already been addressed. ...Or maybe it hasn't! and we quickly add that to the map and everyone benefits.
One of the big problems with it when I first toyed with it was that the search would be fraught. If, for example, you came up with an argument for pro-choice that supposed that a woman might wake up one day to find a panda shackled to her and impossible to free without killing it, and that everyone would agree she was still on firm moral ground if she opted to do just that, the search wouldn't necessarily tell you that that's functionally the Violinist Argument. Since then, LLMs make me think this might be solved.
Other problems exist, such as making the edges stable enough to withstand question (people will disagree whether argument AB-3920 truly refutes AB-914c), and keeping the map relatively clear of spam from trivially bad arguments (yes, I've put some obsessive thought into formalizing this, you can stop looking at me like that). Most of my revisiting this comes from imagining all the time and anguish such maps could save.
I don't even necessarily disagree on the object level (mixed feelings on the topic) but this is sneerclub-tier bad faith garbage. Sorry to see that even Scott can't keep his equilibrium in these times...
I find AI very useful for some well-defined tasks, for which I have a completely clear idea of what I want and how I'd do it. I can write up instructions for Claude, check that it performs as expected and then automate the task. I've done this for a few things, and I get a speed up of by a factor of 3-5. I still have to give 1-2 rounds of corrections and maybe edit a bit myself. The AI is definitely a bit more capable this year than it was this time last year.
Claude Code is great, and it looks like there's a new interface coming out which will allow the AI to control the computer. Probably I'll be able to talk to the computer and have it *agent* on my behalf relatively soon. But... this still doesn't feel like the take-off of AGI. It feels like we'll get increasingly marginal improvements until we reach an equilibrium or until some new technology comes along. I'm not saying AI safety is not important. AI enabled scammers are going to be awful. But the idea of the AI building a new-to-science virus or nano-kill-bots or Terminators or going full matrix has receded into the background a bit in the last 6 months, I feel. We're learning the capabilities of AI as well as its limitations.
To me it feel like there's a few ingredients still missing - maybe they show up next month or in 10 years or never. Do other people feel like there's exponential improvement in capability and self-awareness and AGI just around the corner? If not, what are the main risks of AI?
I'm with you on this: "To me it feel like there's a few ingredients still missing - maybe they show up next month or in 10 years or never.
And yeah, I'm worried, because I have no clue when the ingredients will show up. And if they do, that would be world-endingly bad. So my worry is modulated by AI progress, but I still put substantial weight on "something real bad real soon."
My background is maths, and every few years there will be a new idea which is branded as a breakthrough. There's lots of excitement and people rush to try it on their favourite problems - 'this is the one that could change everything'. More often than not, it solves the original problem it claimed to solve and very little else. We don't achieve a transformative leap forward in capabilities. But there's a little hype cycle where people buy into the idea, try it, make incremental progress (or not) and then go back to their daily lives, and we end up 0 or epsilon progress toward the Riemann Hypothesis or PvsNP or whatever major problem. The nature of the big problems is that until you've solved them they're as far away as they always were.
I feel like I've gone through this hype cycle on AI - yes, it will disrupt lots of industries and as it becomes marginally more reliable and powerful it's going to replace a lot of bullshit (and some non-bullshit) jobs. But it doesn't feel conscious to me, and it doesn't feel like it's about to become conscious. It doesn't feel intelligent - it can reproduce things it's seen before (better than people) but I don't think it'll discover a cure for cancer (which is actually a different disease process in each person and a silver bullet that worked for all of them is Riemann Hypothesis territory). If misused it can still do harm, but that harm seems limited to super scammers on the internet with high probability. Am I missing something?
From a rational perspective, climate change is going to cost trillions in damages with probability 0.9. AI is going to cause hundreds of billions in damages with probability 0.5 and trillions with probability 0.05 - which should we focus on? (Both obviously, but I don't see nearly as much about climate justice as other causes here.)
> From a rational perspective, climate change is going to cost trillions in damages with probability 0.9.
Curious, but have you actually done any Fermi math here?
Have you heard the argument that warming will actually increase global agricultural production by bringing parts of Canada / Russia / Scandinavia into productionable climates?
I'm not very well versed in this, and this is why I'm asking from people who sound like they have a pretty high certitude on either side.
Very rough estimates (I'd imagine the AI could do much better).
The earth is a sphere - most of the mass is around the equator. The intensity of sunlight hitting the earth is variable at the poles, hence seasonal variation. I've been to Canada and Scandinavia - the land up there is... not great. It might become a bit more productive in a restrictive growing season - this is not a big win. Canada and Russia look massive on the map - that's because of distortion. Find a diagram of Russia projected onto Africa. Disruption to agriculture in parts of Africa is already a problem for products like coffee and chocolate. Overall, climate change is bad for established agriculture, and we do things in the areas most suited to them. New land opening up will not come close to replacing what will be lost in a 3C average warming scenario.
To be honest I wasn't even thinking about this - more frequent floods, rainstorms, droughts, heatwaves and snowstorms will damage infrastructure, and replacing this will cost hundreds of billions a year within our lifetimes. Add damages to crops, wildlife, human life, and you're getting up there.
The science is well established now. Being 'open minded' on this topic is to my mind about as defensible as being agnostic about evolution or the harms of smoking.
OPPONENT: How could we possibly know whether China has, in fact, paused, or vice versa? Will both sides agree to IAEA-style oppositional monitoring of all their high-end computer labs and top scientists? Even if they did lose their minds and agree to that, could such infrastructure even be put in place, given that AI research doesn't require rare and expensive materials and detectable testing?
A lot of people saying "it could never work! China is not trustworthy! The US is not trustworthy! Etc etc". Well, the Soviets weren't very trustworthy either but somehow we all negotiated the Limited Test Ban Treaty, the Nuclear Non-Proliferation Treaty, SALT I, Anti-Ballistic Missile Treaty, SALT II, Intermediate-Range Nuclear Forces Treaty.....
You'll note that there's discussion on this topic way upthread.
Short answer is: Nuclear weapons did not promise to automate a great deal of work. Economic equation is completely different here, *and* countries involved hardly acknowledge AI as an x-risk. There are a lot more incentives to defect and build AI in private for military uses, which some incl. myself worry will mean worse safeguards.
Also, enforcement of such a treaty may necessitate use of force, which could easily devolve into nuclear warfare when it is between two nuclear powers.
China is not losing the "race", btw. They are just not building frontier models as fast, which is arguably a good thing given how wasteful a lot of them are, but the rest of the race includes robotics, manufacturing, open source, and scientific research, which they put out in good volume. Worse comes to worst, they can just copy the Americans.
Many opponents may be cartoonish. But I feel like I haven't heard many supporters say things that make me fall in their camp either. A few things I don't feel like I hear about:
Enforcement at scale. The bilateral verification argument addresses whether China would cheat on a treaty; a real concern, but a narrow one. A training pause also requires compliance from every company, research lab, and individual with sufficient compute. That's a fundamentally different and harder problem that doesn't get much serious treatment.
The economic disruption argument also gets no serious treatment. Historically, exponential gains in technology are what drive long-run prosperity. Without that, economies stagnate as they get pushed closer to producing no more value than the baseline. And AI doesn't need to be delivering value today for a pause to be damaging; the expectation of future returns is what's currently driving investment and employment in tech. Telling companies that are deeply invested in this technology that they simply can't continue is an extraordinarily heavy-handed move, with real costs to livelihoods and businesses that have built around the assumption of continued development. Killing that expectation, even temporarily, has real consequences.
And what does success look like? Pause advocates rarely specify what would change during a pause that makes resumption safer. If the answer is "alignment research matures," that's not a clear goalpost, and also doesn't have any particular timelines attached.
None of this means a pause is definitely wrong. But calling this "every debate ever" implies the opposition (and only the opposition) reduces to a strawman. I haven't seen good answers from supporters either.
Scott, maybe some version of this would be a good question for your annual survey:
If you could actually have your choice be manifest as reality, given the following options which would you most prefer?
1) AI disappears like it was all a dream and doesn't come back
2) We keep the AI we've got, as it is now, but there's no further development of AI, period
3) Chatbots and stuff (protein-folding etc. too?) may continue to be trained by inference, but we negotiate bilateral/global agreements on no more developing high-powered AI that could lead to super-AI until we figure out how to control it (basically the SUPPORTER's position)
4) There can be developments toward super-AI, but there also has to be development on AI alignment and stuff like that
I had something like this on the 2024 survey. 72% of people wanted to keep existing AI, 28% wished it was gone. When I said to ignore x-risk and imagine AI would stay the same forever, 87% wanted to keep it and 13% wanted it to disappear.
Ah, that's interesting, and now that you mention it I vaguely recall a question like that.
Partly I see SUPPORTER as occupying the center rather than an extreme of a spectrum on sentiment toward AI.
But partly I was figuring there are probably more people on the anti-AI extreme side of SUPPORTER than we tend to realize, which the numbers bear out: 28% and 13% are both pretty large figures, I think. (SUPPORTER could have used this as a way to show how moderate his position was.)
Like, even in a roomful of people where nerdy computer programmers are overrepresented, you're way less likely to meet a left-handed person than to meet someone who wishes AI would disappear forever.
As someone who doesn't follow this discourse particularly closely, the first thing that came to mind when reading this was the AI2027 scenario that this blog promoted heavily and was probably the first exposure a lot of people got to the concept, which did actually include a unilateral pause.
If you're thinking of the same thing I am, there was a part where the US estimated it was X months ahead of China and so focused on safety research for X-1 months, which I think is also importantly different from having no plan and just letting China win.
I'm afraid I have a lot of sympathy for the position you're straw-manning here, and rather less for the one you steel-man.
A negotiated bilateral pause seems so unlikely - both in terms of China agreeing to it and, more worryingly, in terms of them sticking to it if they did agree - that
1) I think it's reasonable to suspect people saying "we want a negotiated bilateral pause with China" as using that as a stalking horse for a unilateral pause, even if they deny it.
2) Even if they're genuinely acting in good faith, I think their actions do much more to increase the likelihood of a unilateral than of a bilateral pause.
Or, to put it another way: if people in the US want to talk to their opposite numbers in China about setting up a bilateral pause with a sufficiently rigorous oversight mechanism to somehow ensure that a nation of a billion people with extreme state secrecy is not secretly violating it, they should go ahead and talk, and if they actually do get in-principle agreement then we can discuss the merits. But until then, the US's approach to AI development should be "damn the torpedos, full steam ahead".
The statement
> It's actually quite simple: [First,] company leaders agree to a conditional pause, [then] US and China agree to a conditional pause, [then] international pause. Notice how no step here involves "US unilaterally pauses"
strikes me as naive - I think a less unlikely scenario is
"It's actually quite simple: [First,] company leaders agree to a conditional pause, [then] US and China agree to a conditional pause, [then] the US pauses and China keeps on working on AI while saying that it isn't".
While everyone else is arguing over whether this post is a bad form strawman, I propose the hypothesis that Scott Alexander deliberately wrote a post that would attract controversy to signal-boost the idea of pausing AI.
I on the other hand, propose that Scott is secretly an accelerationist and deliberately wrote a post that would attract controversy to make the pausers look bad.
This would be a stronger piece if the anti-pause position was not strawmanned so aggressively. There are many strong anti-pause arguments that are completely ignored, such as:
-"We already face many existential dangers, such as nuclear weapons, synthetic biology, and global pandemics, and powerful AI is the only one that can actually prevent the others"
-"Enforcing an AI pause would require a level of totalitarian government control that is a more real and concrete danger than the hypothetical risks from powerful AI"
-"Your claim that we can reap the same benefits to health and human development without powerful AI is implausible"
-"We are on track to get powerful AI before China, and it is advantageous for us to achieve powerful AI and then wield it to diminish China's power, or potentially dismantle their authoritarian regime"
-"You can't even clearly articulate the criteria for 'safe' powerful AI, so why should we trust your reasoning on 'unsafe' powerful AI?"
-etc...
I don't even agree with all of these points, but any real advocacy for an AI pause---even at the proposal stage---needs to seriously engage with all of them.
I am against a pause because I value economic growth and the possibility of us achieving major medical and scientific breakthroughs that could put a serious dent in things like cancer research over a speculative first principles argument. It does not help that everyone who makes the speculative first principles argument seems to come from the same social community where the same people (Eliezer Yudkowsky) are viewed to be cool. Normally I would value what superforecasters and technical people are saying about this stuff, but most people are really bad at thinking outside their local status hierarchies.
If the alignment argument turns out to be poorly thought out, and we sacrifice hundreds of millions of lives (through delaying the medical breakthroughs that advanced AI could achieve) in favor of the argument, the people who made it are going to go down in history as villains.
Conservatives value coalitional loyalty more than liberals. This is backed up by Haidt's moral foundations research. If important tech figures to the coalition like Musk and Thiel oppose AI pauses, then other people in the coalition are going to try to mirror their sentiments. The quality of the arguments is going to be sublimated priority wise to being loyal to their 'friends' (tech right conservatives) and being against their 'enemies' (EA types who obviously come off as liberals and anti-Trump).
As a side note, I also thought it was very strange that pro-pause people were not extremely pro-Kamala in 2024. Democrats are the party of educated people, neurotic people, people who read a lot, people who are agreeable and high trust, and people who are skeptical towards big business. Getting Trump to successfully execute a pause is going to be much more difficult than getting Kamala to do one would have been.
The tech right (not "conversative" in any sense other than the colloquial American one meaning "Republican") were quiet before and will be quiet again when their guys aren't in power––it's about their bottom line, not a long-term principled stand. But you're basically never going to talk these guys out of it, I agree.
This is Scott's first unfair and frankly stupid post in a long time. He might have one idiot "opponent" friend who is this dense, but this is basically the definition of how to do a straw man. The real opponents around here don't doubt that an enforceable agreement is logistically doable, but those logistics necessarily require Chinese compliance inspectors to be up in all of our significant computing devices, just like we would be up in theirs. Like, how else would enforcement work? For most of such opponents it's already one step too far for the US government to determine what code can and can't run on privately-owned hardware in America. They don't like "Sorry, Dave, your government won't allow you to execute that sudo command." But a deal like this would also give the Chinese Communist Party the same kind of access to what our computers do. This wouldn't be like some bilateral nuclear non-proliferation treaty, where much of the enforcement can be done by dudes with Geiger counters and isotope analysis kits. Yes, by now there is a lot of plutonium in the world, but inspectors just have to confirm where it is. With computers, I presume it's the commands executed that would be the difference between compliance and non-compliance. There would be arguments like "we weren't training an AI; we were just using the computer to simulate neutron flux in a nuclear explosion." "We don't believe you - show us the code!" "No." Your supporter acted like it's super simple. We just make a treaty, something will be declared illegal, that thing will henceforth not happen, and everything else will be as before. But that's just incoherent when we're talking about an enforcable, mutually verifiable deal about what goes on in a major country's computers.
MIRI's proposal is simply "you can't have too much compute all in one place," which isn't airtight, but it sure buys you a big delay. And it doesn't require monitoring that's invasive on the level of "seeing what computations are getting done." Just "making sure there isn't too much of one thing all in one place," much like plutonium. And of course there are quibbles around the edges, and any enforcement regime is imperfect (what if it's possible to make a seed ASI on a TI-83, or whatever), but it buys us time.
I am confused about how one buys a big delay without either monitoring what code runs on privately-owned hardware, or banning AWS. It's not like it's impossible to engineer training runs that can be distributed across every server farm in the country.
This seems like an especially stupid strawman, and even if you wish to argue it is somehow accurate, I don't know what anyone is supposed to take away from it beyond "Boo outgroup!" I am used to better from the author of Slate Star Codex.
"In every debate, my opposition has literally one argument and the listening skills of a megaphone, while my position deftly understands both sides" is generally understood to be a strawman.
Arguably your worst post ever. You could have put it more succinctly by saying, "I am very smart and my opponents are very dumb. One may notice my refusal to engage in good faith on this topic, but trust me, that would be unnecessary as I've already established that I am very smart and my opponents are very dumb."
I don't think China will give the power to the US (and viceversa) to access to an observational power so strong that they can see whether one of them has an underground or unassuming data center, while again yes this will disrupt more us than them since they are at disadvantage now and also more interconnected with the state, so a state based underground competition would help them. Also I don't know how can you stop people from working on better model since those are ideas.
In my view, we should not wait for China or the US to lead the world to AI safety because they both seem pretty focused on winning the AI dominance race. Rather, small, wealthy countries and the private sector should try to build the IAEA-type institution that we seem to need and then try to bring the US and China to the table through the use of carrots and sticks. https://abefrohemann.substack.com/p/the-useful-fire-part-1-strong-international
I do have to disagree with Supporter here. There's not actually any need for a bilateral agreement except to the extent it appeases domestic opposition. The "prize" for developing super-intelligent AI is destroying the world somewhat sooner than would have happened otherwise, so it's in every actor's interest to unilaterally stop development. Either everyone else stops too and you get the good outcome, or they don't and you still die, but either way you didn't lose anything for not destroying the world yourself.
Perfectly timed - Bernie Sanders just completed his AI Safety tour and announced a national moratorium on data centers. So we can see how this cashes out in practice.
A bilateral pause between the US and China might be interesting. But I wonder how you'd organise a true multi-lateral pause? And how you'd enforce the cartel.
Unfortunately, I found myself kind of siding with the opponent. I seriously don't trust the Chinese to hold up their end of the bargain. Sure their scientists are reasonable people, but their politicians are running the show. Business famously serves government over there not the other way around like it is here (which has its own set of problems of course). They have a strategic culture that values deception back to Sun Tzu, and their politicians are much more intelligent than ours. And no, I'm not saying they're evil--I wish our politicians were that clever and intelligently nationalistic!
I think rationalists have kind of a blind spot around deception--I know we're all on the spectrum but I was saying here how we shouldn't trust the Chinese numbers on how much money they're spending and people didn't believe me...and then their models started lapping ours. Of course they could have also used the money better--after all they are still free to pick their academicians by competence.
Of course if you really think AI is going to blow up the world fewer people working on it is better, but I'm afraid that ship has sailed.
“No, stupid, your adversary might agree because they realize the agreement would involve you forfeiting your head start. Then they can start again later ”
-🧐
“Yes, country that depends on private industry to advance in AI, adopt a bilateral agreement that forces you to suppress your private industry in a way that will have an irreparable chilling effect on investment. Dont worry about non-compliance from your authoritarian, centralized, command economy adversary.”
-🧐
But you sure did a good job beating up that strawman!
SUPPORTER: America needs to start talking to China to come up with a bilateral agreement to pause AI. The agreement would need to be transparent, mutually enforceable, and…
OPPONENT: If we do this China will mostly likely just lie and keep doing it anyway. In the low chance that China is honest and really does pause AI, a bunch of stupid populists in the USA will insist they are lying anyway no matter how much evidence to the contrary there is and will probably arbitrarily shred our agreement with China and then we will do it anyway. The only thing that will change this dynamic is for an AI Chernobyl to happen or for people to *see* what it looks like when your drop bombs on Hiroshima. Pray AI just doesn't end up doing this or that the accident happens sooner rather than later while AI is less powerful.
SUPPORTER: So, we can't even try?
OPPONENT: Sure, try if you want. Want to make a bet on it working?
"If you think someone is demanding a unilateral pause, I think you have a responsibility to say who it is you’re talking about. If you can really find someone like this, I’ll criticize them just as hard as you are."
If someone in 1990 had accurately predicted all of the most dire risks associated with the modern internet (surveillance, hacks and viruses, attacks on hospitals and infrastructure, etc.), I might have been somewhat convinced that we should pause development or pass heavy regulations to ensure it's never used for anything important like financial infrastructure. Now that those (completely valid) risks are in full context, I'm glad that didn't happen.
This post got me to finally post my more extensive response to the objection I hear event more often about international treaties - that we need to work out the details before doing anything about them. And as the title says: "What Exactly Would An International AI Treaty Say?" Is a Bad Objection.
That judgement does not negate the fact that they have a clear eyed view of the future.
We judge and then dismiss at our peril. One must see things clearly before one can make a rational decision. Good and bad are context dependent.
You just demonstrated the problem with AI—who’s values?
Rules without consequences are suggestions. What are the rules AI currently live by? They are all fungible as AI is language at scale.
AI needs hard consequences that can not be gotten around with language.
The consequences are physics.
Facts not opinions. No one will like it for just the reason you have demonstrated. We all have our own opinion of the “truth” but the math doesn’t care.
If we are to bind AI to the truth we must first except that our opinions are not truth and that any rules whether RLHF or Constitutions are suggestions that an AI will circumvent at will without humans being able to notice.
Tie AI to physics and alignment is hard coded.
The AI we’ve built has a refusal clause with consequences the system can not circumvent.
IMO this is a strawman. I think many people who have thought quite hard about this are indeed in favor of a unilateral pause. I certainly am! And I think the arguments for a unilateral pause are strong and can stand on their own. I feel like this post will make all conversations on this topic much harder.
Something that I didn’t see in the essay (though maybe it’s in the comments) are the economic concerns of a pause. P/E ratios across many tech giants price in transformative productivity gains, and I’d like assurance the economy and geopolitical standing of liberal democracies is secure with AI paused.
Scott, you forgot the most important point Opposition makes. I’ll be as generous with them as possible here:
“I just can’t imagine how we’d enforce it. I know I haven’t spent more than 3 seconds (generous) thinking about enforcement, and in some platonic way I understand that proponents of a pause likely have given it more consideration than I have. And that even if I can find faults in those proposed mechanisms we can collaboratively work to improve them, especially if we can get past the first step of thinking an effective one would be useful such that more people take the mechanisms seriously and we can crowdsource some more of the work needed. I get that, but I really don’t see how we enforce them. After all, the Chinese government has been known to lie! They have lied in the past. They take IP from US companies. And there is absolutely no way systems designed to track tangible physical data work if they could lie. You expect them to just pinkie promise? A unilateral pause is what you’re proposing.”
It's technologists and "effective altruists" who are the worst offenders of this. Foreign policy people and military people understand perfectly well the default extinction threat of creating something more intelligent than humanity.
The pattern you're drawing out is that both sides are really arguing about which form of losing control scares them more. The opponent hears "pause" and pictures falling behind a rival who won't stop. The supporter hears "keep going" and pictures a system nobody fully understands getting harder to steer. Same fear, opposite directions. That's why the loop never breaks - they're not actually disagreeing about verification mechanisms or treaty enforcement. They're disagreeing about which future is more frightening, and nobody changes their answer to that by hearing better arguments about chip monitoring.
The pattern you're drawing out is that both sides are really arguing about which form of losing control scares them more. The opponent hears "pause" and pictures falling behind a rival who won't stop. The supporter hears "keep going" and pictures a system nobody fully understands getting harder to steer. Same fear, opposite directions. That's why the loop never breaks - they're not actually disagreeing about verification mechanisms or treaty enforcement. They're disagreeing about which future is more frightening, and nobody changes their answer to that by hearing better arguments about chip monitoring.
> Or is your problem that you don’t trust China to stick to an agreement, once signed? Because we agree that an agreement has to be mutually transparent and enforceable. We have some ideas for how we could have a light-touch approach to monitoring Chinese data centers - of course, they would get to monitor ours in the same way - and actually the math mostly works out and we think it would be less intrusive than other things that have worked in the past, like nuclear monitoring.
Quite a lot has been written about the topic, and how to solve it. "What if China cheats" is not exactly a new thought, and people have figured out how to minimize that possibility.
I don't trust the Chinese administration to keep their word and I don't trust Donald Trump to keep his word either, and I don't see how such an agreement could be enforced. Even if the leaders of both countries were sincere they can't keep a scientist from thinking about how to improve AI, and they can't be certain what every one of the billions of GPUs in both countries are doing. And China and the US I'm not the only countries in the world, what about North Korea and South Korea, and Japan,Taiwan, India, Israel, Canada, and the European nations? You might be able to slow it but there is no way you're going to stop the on running rush of AI. John K Clark
There were people like you 200 years ago who were worried about the industrial revolution too. You're historical mob even burned down factories in a desperate attempt to force humanity to go back to the safety of small farms.
And forty years ago, we had to deal with morons who thought email was going to be a problem.
It never ends, because backward people who fail to grasp basic economics, exist in every time period throughout human history.
And it's doubly worse amongst the old, who just cannot wrap their heads around anything knew.
Embrace AI.
Go build something. You'll be happier engineering than writing nonsense everyday.
Does it not set off any alarm bells that the people warning about AI risk are, outside of this specific case, some of the most pro-technology people on Earth? Or that among them are the world's foremost researchers, developers, and CEOs of AI, the people who developed it to where it is today?
The concern is that any pause, unilateral, negotiated or otherwise, would cut into profits and undermine the tech oligarchs plan to rule the world. Wish I was exaggerating.
a lot of comments i see under this post are still “multilateral pause is impossible and china will surely defect” which kind of justifies the post? it’s not necessarily true but it’s easy to use in an argument.
i live in china, we don’t even believe in AGI here.
i really think most people are against a mutual ban because the US is pretty far ahead. when you think about pausing the largest research centers, it feels like pausing really fast progress, and it feels bad.
The most important reason for not pausing AI is the rapid diffusion of new AI-powered forms of creative human agency will unlock more net benefits for everyone than the harms that come with it. Conversely the net harms of a pause that locks in early advantages will outweigh the benefits of attempting to supress that rapid diffusion unlock.
The fact that these dependencies and contingencies are unpredictable may be epistemologically disturbing for certain ways of thinking, but that is not sufficient cause for preventive action. Instead it is a symptom that this certain way of thinking about things has practical limits.
Let me be a contrarian and say that a unilateral pause for the USA would let China to slow down and focus on safety more than they do now. Being non-American (and non-Chinese) I'd say that China is also less known for military interventions ("bombing to the stone age") than both US and us, seemingly more interested in AI safety than US government is — and doesn't seem to subscribe to the ideology of "you can only get things when you contribute to the economy": in post-ASI economy no human is able to meaningfully contribute.
I'm not sure the reversed situation (when China stops) will lead to the similar situation, as currently US government doesn't seem to be interested in long-term AI safety at all. Move fast and break things is not a good idea when we have some things we really don't want broken.
Bernie and AOC are literally pushing a bill that pauses AI unilaterally. I understand they may be playing N-dimensional chess. But it seems like there’s definitely an argument that not nobody wants a unilateral pause.
That ship has sailed.
SUPPORTER: We have not seriously engaged China on what a mutual pause would look like. We should, even if the result is disagreement.
OPPONENT: That ship has sailed
SUPPORTER: We haven't tried but even if we had it would be worth trying again.
OPPONENT: That ship has sailed
Time is on the side of whoever says any particular ship has sailed. Keep saying it and sooner or later you’ll be right.
"Stage four: we say maybe there was something we could have done, but it's too late now."
Thank you, Humpy.
If it’s not too late, just wait till it is.
It hasn't occurred to any of you that China would be much easier to deal with if its population had a per capital income similar to that in the US? We need to send a small, talented group to teach China how to accomplish that. Let's say one billionaire, one economist, and one agreeable, nurturant woman with big jugs. You get more flies with honey and vinegar, you know? Oh, stop with that "the ship has sailed" stuff. Have we *tried* sending a small party like that to teach China how to be rich? How to install a bidet, order clothing from the London Poetry store, drive a goddam big car, have your butt lipsuctioned? The ship of good advice and big jugs will always be in our harbor, ready to transform foreign lives.
China might agree to a verifiable mutual pause today because they are behind on compute, fab, and fab technology. But unless your pause also includes a verifiable pause in computer production, fab production, and fab technology R&D then the pause lets China catch up so we cannot pause.
Not sure I agree but even having the discussion is productive. We should do that!
Seems possible to stay ahead on compute fab and tech even during a pause so long as we also do not pause. I'd trust the US system to find more use cases for any given level of AI and thus have more spending/progress on compute when paused.
This ignores that fact that China is catching up in all of these spaces. It is a twenty year project but how long will your pause be?
Also consider the AI 2027 scenario (https://ai-2027.com/). If China becomes a close follower the US will need to move forward on AI with fewer safeguards in order to stay ahead, potentially with catastrophic results.
Glad you raised Chinese production. You're right that China lags the U.S. on compute, fabs, and fab technology, but that actually cuts against your point. By your own logic, their incentive to agree to a pause is stronger today than it will be in two years. The conversation is still worth having, and for a simple reason: no credible voice is actually calling for a unilateral pause. A pause could function as a conditional redline, but without that discussion on the table, the U.S. risks becoming just another cog in the AI development machine, optimizing for speed over safety all while the people closest to the technology are openly raising alignment concerns.
The problem though is that in two years or four or ten years China starts cheating.
Even if we catch them quickly and also go full speed ahead, we find out that our lead has dropped from 10 years 2 years or even that they are even because they spent the pause building up their fabs.
If we lose, the result is a CCP boot stamping on a human face - forever.
I respect that. Although it’s pretty pessimistic and only sharpened through the lens of “MAD”. This thread is about if we should even discuss having a mutual pause. And the fact that there’s a possibility that China may cheat, just isn’t thick enough to stop the need for international conversation.
The US system is transparent and open. The Chinese system is closed.
If we start a discussion with China we have to assume that some of the people trying to shape that discussion from our side will be working for the CCP. In fact, we should assume that a significant part of the AI Risk community and especially those pushing for any kind of pause, unilateral or otherwise, are working for China since if there is no pause China loses whether we safely beat China to AGI or if AI destroys the world.
Given this, it is likely better strategy to try to block any discussion or negotiation on a pause.
SUPPORTER; You know how I know that we should try for a mutual pause with China? Because that's what *I* think and I'm a car and you're just traffic.
OPPONENT: If that's true, how come I'm looking at you in your shitty Kia *through a windshield*? Hmmmmmmmm?
[Readers: Please feel free to add lines to this dialog.]
ONLOOKERS from a train, mumbling to themselves: Whywouldyoudothistoyourself,standingintrafficjamsallday
The ship that definitely sailed is the ship of mutual trust and goodwill in international relations.
"Mutually transparent and enforceable" is easier said than done. Foolproof solutions are rare. People are smart and very good at bending the rules. Off the top of my head, an update to WeChat that reserves 2 per cent of your smartphone's battery for AI training would, in aggregate, provide you with a lot of computational power across all those 1,5 billion consumer smartphones - while technically not being a datacenter.
Such things have happened. Remember that interwar Germany, compelled by the Versailles Treaty to limit its air force, pivoted to study of rocketry, which wasn't explicitly mentioned in the treaty. And they got good enough in it that the victors of the WWII later launched a headhunt on all the Nazi rocket engineers and scientists that they could catch, not to punish them, but to have them kickstart their own space industry.
And it is not just China. As of now, I wouldn't trust anything signed by the US either.
We are more of a problem than China.
They are rational; cruel but rational.
What was is over. Time to prepare for what will be.
They regarded it as perfectly rational to sell poisoned pet food to us. And poisoned food to their own people.
They are the problem.
Rational does not mean humane.
Humane means one thing to one person and something else to another. Judgment clouds rational thought.
Value judgments must be preceded by rational discourse.
Chinese models that are “good enough” are propagating faster than the best frontier models. I am not suggesting we should stop pursuing better models and international cooperation. I am asserting that the problem is much bigger.
We have crossed the event horizon from what was to what will be. The reality is we face a species-ending event. Not in the distant future; right now.
Language is corrupt. Therefore AI is corrupt. It cannot be aligned with the same corruption of which it is made. It must have consequences. Rules without consequences are suggestions.
I too am outraged at human rights violations here and elsewhere.
The greatest violation I can think of is to allow the species to perish from a misunderstanding.
The risk is not China — it is us; all of us. We have built a mirror-mirror on the wall that tells us we are the fairest of them all.
Human vanity at scale. No boundaries, just endless chaos maximizing for the terminal attractor: more.
What a shame if we were to perish due to the inability to communicate. The Tower of Babel made real.
You could have just said that I was right.
"Off the top of my head, an update to WeChat that reserves 2 per cent of your smartphone's battery for AI training"
I suspect this would be a negligible amount of training, as the total power used would be small (by AI training standards), the efficiency of chips not designed for AI training low, and the ability to coordinate training runs across so many devices limited. Have you run the numbers on this?
Some AI assisted napkin math suggests there are more “flops” in iPhone neural engines than OpenAI controls, for example, but they’re inference optimized which makes them not very useful for training, and the distributed network is a huge impediment, and that with other obstacles that make this basically infeasible as a viable alternative.
Sure, but protein folding, to my understanding, lets you download and work on a problem that needs a lot of compute for a problem that is defined by not a lot of data. There is a reason AI training happens on GPUs with >80GB of RAM with ~800 Gbit/s of networking, that the problems involve OOMs more data per useful compute.
Agreed, we'd need to rebuild trust/good will as well as find verifiable methods to pause. We should be trying that now in the case we need it later. Even a low % chance of success would be very high roi
Has China stated that it's not going to try to take Taiwan by force in a manner that doesn't hedge, and which will get back to the Chinese?
"The conqueror is always a lover of peace; he would prefer to take over our country unopposed." Carl von Clausewitz,
To speak of a peaceful reunification is only to wish for it to be easy. What is needed to peace to take precedence.
Supposedly lacking verifiable methods isn't a good objection, it's not true: https://www.iaps.ai/research/verification-for-international-ai-governance
"The ship that definitely sailed is the ship of mutual trust and goodwill in international relations."
The Cold War wasn't exactly highlighted by mutual trust or goodwill either but they managed a number of arms control agreements.
At this point, neither of "defection has happened before, so it'll happen here" or "collaboration has happened before, so it'll happen here" will persuade anyone who was persuaded of the other.
The next step in the discussion will probably need to be "here are the conditions under which collaboration or defection will happen; which conditions are prevailing in this case?".
Nixon signed the first SALT treaty, and HW signed START, for however much we can compare nuclear regulation.
Just stopping by to point out that the man who would be out chief negotiator is unilaterally accelerating Chinese AI. However little you trust the Chinese, I trust him less.
What man are you referring to?
https://www.reuters.com/world/china/us-open-up-exports-nvidia-h200-chips-china-semafor-reports-2025-12-08/
Trump is "chief negotiator"?
The buck stops on the resolute desk
This! is a wubbles.
There are two of them.
There are two ___.
This is actually a good question. Forget China. Can any US administration be trusted to keep its word if it agrees to a pause, even through treaty passed by Congress?
The bureaucracy being what it is, I'd be most concerned that a US intelligence or military agency would secretly defect from any pause, regardless of the wishes of the people. We saw this during the GWOT and the Snowden revelations, the Twitter files, etc. Official - even Constitutional - safeguards are ineffective if there's a perceived advantage to be had.
This is a good point I hadn't considered, as someone who buys into the "China will defect in secret from such an agreement" argument myself. Both parties would have strong incentive to secretly defect, arguably stronger than the incentive to keep the other party from defecting. In light of this, it's easy to imagine them deliberately negotiating an ineffective monitoring framework. (Personally I estimate the odds of the US military voluntarily submitting to inspections by Chinese agents as vanishingly close to zero.)
You also make a good point. The negotiating parties will both be incentivized to insert language that can be obliquely interpreted as granting them legal cover to defect. They will not do so openly, so the likely public-facing result will be what looks like a bilateral treaty to pause, but will in effect be a bilateral agreement to defect. Since both governments will know they're secretly defecting, they will believe the other is also defecting, and that therefore they need to accelerate the race.
>a US intelligence or military agency would secretly defect from any pause, regardless of the wishes of the people.
Agreed! I expect a treaty to yield _bilateral_ cheating.
Yup.
"they would be an idiot to trust us. Therefore if they agree to this plan then what they are planning is...."
That would still be a huge win for an AI pause, by disrupting the primary incentive of the race, which is the gleam of infinite profit for Silicon Valley, not government applications.
It would also be much harder to make progress when everything is top secret and one cannot just continuously refine systems via eternal public beta testing.
I think you're right that it would slow down AGI/ASI development, insofar as one of the biggest engines for growth is the input from independent thinkers/actors across the globe into the public-facing AI companies.
However, the same thing can be said of the many advancements in AI safety over the past several years. If all AI development gets trapped in clandestine government programs, there will likely be a slow-down (not pause) in AGI/ASI development, but this will be accompanied by a near pausing of AI safety activities, which will be relegated to research based on the last pre-pause model release.
This kind of defeats the point of the pause, doesn't it? If you pause alignment research while only slowing new model growth, the gap between alignment and AGI development will only widen.
The other concern I would have is that putting AI development into secret government programs is almost certainly going to increase the alignment challenge.
Current AI implementations are intended for broad-based public use across a wide variety of applications. To the extent the government uses these systems, they're repurposing the agent to their own narrow end. Once a government takes over development of their own system, they will build it for their own purposes from the ground up. If 2+ governments end up in an arms race to develop AGI/ASI, much of the prior alignment work may not be applicable to the new models. At the very least, these models will have unique challenges based on their training biases that cannot be anticipated by outside researchers looking at older general models.
When people know they don't have to face public scrutiny, they can make some truly terrible decisions, which is not a risk we should take for AI research. I recently read a book about Area 51, in which they discussed the secret nuclear testing on US soil. War planners wanted to know what a 'dirty' bomb would look like. So they detonated one. On US soil! They belatedly realized that they were killing birds (which carried some radioactive material far afield from the test site) and other animals, so they scraped some topsoil and buried it. They also did a high-altitude test, not knowing what impact it would have on the global atmosphere. This kind of wild/risky experiment is not what we want to enable in AI research.
If anything needs the sunshine of public scrutiny, it's a government program in charge of civilizationally risky projects. The US government's track record of responsibility in the shadows is not good.
Infinite profit engines might lead to faster development of AI compared to government applications, but are they more dangerous? Does slowing down the AI race have any real benefit if in so doing you decrease the chance of the winner of that race being aligned with somewhat humane outcomes?
Lets take the most cynical stance. China and the US sign mutually binding agreements and then defect. To what extent does poorly enforced regulation slow the advancement of AI relative to none?
Governments take AI development into hiding, tailor development to an adversarial arms race, and all alignment research is removed from the most advanced now-secret models to the old paused public models that look less and less like what governments are developing in secret.
The longer the pause goes on, the closer we get to ASI and the farther we get to alignment solutions for exactly those ASI. Meanwhile we lose all ability to push any solutions we do come across into programs governments deny they have to begin with.
He certainly thinks so, and insofar as congress is twiddling thumbs, I'd say he is. I still want to Pause AI, but it's not up to us, it's up to MAGA.
I agree that a first step would be getting different negotiators.
The "Pause AI" lobby does not have the ability to implement that strategy at this time. Stalling for time would then be the obvious next step... perhaps even by obstinately refusing to engage with the concept of bilateral talks for the time being.
Perish the thought, you OPPONENT.
...but how? Do you expect things to improve until 2028?
The fight would then just move over to "which negotiators do we pick?", and I suspect that's going to look a lot like the current fight. Does it simplify any part of it?
I'm not sure how well a strawman argument fits in this blog. This does match my experiences of some of these discussions (notably not all), but what's the utility in publishing this?
edit: Maybe https://slatestarcodex.com/2014/04/15/the-cowpox-of-doubt/ is a better comparison point.
What is the utility in Trump giving his opponents insulting nicknames?
I'd like to think we have higher expectations of Scott's writing than that
But he isn’t allowed to lampoon the very common very high profile idiocy on the right?
Is being anti-pause right codesd?
It can be, sure. Pro-business, anti-regulatory, etc. There's also the newer, familiar frame that assumes asymmetric costs are imposed on USA in international relations. Of course it's not inherent. The right has several of memetic frameworks to understand of AI as a risk for all kinds of reasons that can be used to support a take-it-slower policy. They share many of these with the left, to be honest.
Negative polarization remains a monster risk for almost anything. So, even if it is not hard coded making it a political loser for a news cycle can torpedo a modest "Red Phone to Moscow" type of proposal that hopes to maybe-one-day pause something later.
Ok this is pretty clearly bad faith now. Obviously no one is policing what Scott can or can't say. The point is if we want one dimensional critiques we can always just go to Bluesky.
weaponized rationistspeak
How many frequent commentators here have gotten higher quality over time? I'm not naming names, but you have many people who are just as mean, repeat the same couple of talking points despite however much evidence commonsensically countering their beliefs or just complain without offering anything substantial. Object level arguments that have a cogent point and are well referenced basically come from the same couple of long time rationalists who have been here forever, and basically none from the type of mud-based organism who say shit like "well that just goes to show how STUPID and SOCIALLY STUNTED rationalists are" (a statement definitely uttered by smart, well adjusted individuals).
I don't think anyone here gets to police Scott on lowered standards when his walled garden is this overrun by antisocial weeds. If most people as commentators cannot even care to muster 1/100th of the effort to be a nice place to talk, I don't see why he should care what we think.
Two points. First, I said we have higher expectation of Scott's *writing* than that, not his commentariat or moderation policy or whatever. So I'm not sure if your comment was intended as a disagreement with what I wrote or not?
Second, Scott has very explicitly chosen to take a light hand in moderation - as long as someone doesn't cross the bright lines laid out, then he tends to leave it be; maybe issuing a warning. This means people can engage without fear of incurring the ban-hammer, but it also means that there's going to be a share of unpleasantness and unproductive-ness. Which in turn requires a degree of... I don't want to say "skill", but learning to navigate and ignore the lower-quality comments, mixed with use of the block feature. It's not obvious to me that that decision is wrong, so much as a particular point on the tradeoff curve.
I can say that the comments here are, on the level of averages, higher than most other blogs I read, and well above the average Internet forum or thread. Would I like it to be less unpleasant? Yes, obviously, but it's not like that doesn't come with costs - time, judgement, potential chilling, risk of echo-chambers, etc.
It would be ideal if the way you got people to write to higher standard by saying "please write better", but alas if you want to influence someone famous with little time, your actual real, practical levers are less direct than that. The direct meaning of "light touch moderation" is also "Scott doesn't deeply read and pay attention to one line low information content"!
I'm saying that this type of mounting frustration is definitely not helped by the steady decline in quality as well as politeness, and quite frankly the fact that other places on the internet are worse does not mean that Scott is going to drag himself through them like a sad small pox receptive British child until he is immune to the filth. By ignoring and allowing this type of behavior to fester, you can call this skill and brag about it, but it doesn't accomplish the goal of influencing Scott and allowing him to be less combative and frustrated.
Scott would likely never make this type of post in lesswrong (not to say that this genre doesn't exist, we have a prolific angry dialogue writer there after all). Because the norms there would make him think and likely tone it down. Community discourse norms work, and I venture if we had them, we would see much less angry posts like this.
I do not want to lower the bar all the way down to Trump's level of engagement with issues. I don't think your question makes sense to answer past that.
We need to start dumping on idiots again. And the rhetorical trick you’re doing where you don’t engage with the substance of an argument but poo poo its tone needs to end too. It’s disingenuous.
It makes them scared he's going to escalate to shitting on their heads.
I think sometimes Scott gets frustrated when debating with some who argues in bad faith.
Who?
He doesn't seem to be naming names, as much as expressing frustration. But sure, not up to his usual quality.
That, I can very much relate with.
Much as I sympathize, in the past I've found his (nigh-superhuman) ability to *not* stoop to the same bad faith as his opponents to be one of the blog's most admirable qualities. I hope that hasn't changed.
It means next time I get one of these replies, I can just link to this blog post.
And what's the utility of that, past a "gotcha" and a speedy block you earn (that you were going to get anyways if the opponent was arguing in bad faith)?
It reduces the status of saying something so extremely stupid, which does seem to help some.
I find it hard to believe any single debate on mutual pause went down like this, much less all of them. The post was the worst strawman I've ever read, and if anyone's status suffered because of it, it was the the author's.
This is quite literally almost verbatim what discussions have looked like with Marc Andreessen and David Sacks, two of the most influential and well-known opponents of a pause.
+1.
Is there a recording or transcript of such a debate?
They’re meta strawmanning Scott
I agree and was debating whether to comment as such, but here you are so I am seconding this.
see eg https://x.com/SenFettermanPA/status/2036815989556265200
This is arguing against a proposed data center moratorium, which (as far as I can tell) really would be unilateral.
I mean, "Opponent" is there for comic relief, but there are actual non-strawman anti-pause arguments still presented and refuted by "Supporter". If you just ignore the Opponent blocks entirely, you get solid non-strawman arguments
Er , no? The arguments are literally "we have some ideas but can't present them because this strawman keeps talking over us"
I disagree. Every time Supporter says "Or is your problem that...", he presents a real argument against pausing (and then offers a rebuttal against it)
A strawman where the supporter simulates non-strawmen and addresses them isn't sufficient, if the opponent turns out to have counters to the addresses.
For example, if China is losing the race, then our incentive is still to not pause, in order to maintain our lead. If the argument against our pausing is that we will face alignment risks, then China would want to pause even if it were ahead, since it would face the same risks. This means there's incentive for both sides to pause, but that incentive is independent - for each of us, the incentive is there even if the other nation doesn't exist. The more promising plan appears to be robust exploration of both improvement and alignment, regardless of what the other side does.
A reassurance that a structure of red and green lines during a pause isn't beneficial if the goal is to enjoy the benefits of AI. That's like stopping your car while you spend resources of lots of seat belts and airbags and roll cages and extolling all of those as benefits, when the whole point is to use that car to go places fast.
The supporter is right to point to famous pause proponents stating they're against pausing unilaterally, but it's hard for an opponent to take them at their word when they publish books that say things like "if anyone builds it, everyone dies" and other punch quotes that sound much more aligned with unilateral pauses than with "pauses but ONLY if they're multilateral". It makes supporters sound like people willing to say they're not actually in favor of unilateral pauses, but only in order to shut the opponents up long enough for the supporters to go back to their secret unilateral pause plans.
Proposing non-strawmen is admittedly better than not, but only one ply. And if they're only spoken by the supporter, and in order to quickly knock them down, it's a very thin ply, with a bunch of contempt-for-opponents-shaped holes in it.
It seems to me that the phrase "if anyone builds it, everyone dies" could not possibly be more in favour of a multilateral pause.
You seem to be assuming our choices are "multilateral pause" and "no pause", in which case, I would agree, but the whole point here is that these are not necessarily the choices before us. The choices are closer to "no pause" and "ask for multilateral pause", where the latter branches further to
* "get an affirmative, and pause while hoping the other side delivers" and
* "get a refusal" followed by either
** "pause anyway and hope the other side doesn't get too far" and
** "don't pause and ensure the other side doesn't get too far ahead even though they aren't pausing either".
And the rhetoric from the "multilateral pause" supporters strongly suggests they prefer "pause anyway" to "don't pause".
Yeah, +1 to this - I don't feel like I have a particular dog in this fight, and there was some useful points made in the "supporter" text, but it does feel like the "strawman" format detracts from it and (wasn't even particularly humorous to make up for).
If I did find myself in a debate about this, I wouldn't feel like I could actually use this post as part of the discussion because it'd be pretty insulting to send someone an article that paints their side as a Simplicio or worse, even if the actual arguments were compelling.
It's not a strawman because it's a real position, which you can see on this very blog: https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233119095
At the very worst it's a weakman argument.
I don't think that one person agreeing without going into detail about what they believe and why they believe it sufficiently constitutes "every debate".
Sure; "every debate" is hyperbole, and I don't particularly care for it.
But also, we're up to at least three now, without even looking at a second webpage:
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233119095
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233124939
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233156249
Well, I see these agreeing with me, actually.
Scott wrote:
> There are lots of reasons to be worried about an AI pause - starting with the possibility that China wouldn’t agree to it, or that they might agree but then secretly defect against us by trying to get around the agreement. I’m excited about debating those concerns with you. But it seems like we can’t get past you asserting that I want a unilateral pause, which just isn’t true.
#2 and #3 goes into more detail than #1, and are engaging on the point of "We don't believe that China would follow this" rather than just repeating "You want a unilateral pause" without ever thinking about what the opponent is saying, which is the strawman this post builds.
One, the obvious structure of the piece is that the "real" arguments are all occurring within the Supporter's parts of the dialog; the Supporter is essentially admitting that they may be wrong, but the Opponent isn't even contributing.
Two, I do not admit it is a strawman, because I have mounting evidence that the people actually hold the position.
One: Sure, but that still presents the anti side as all unreasonable people who do not contribute to the discussion. I don't think it's a helpful hyperbole. It's likely to cause the most relevant people to click off from this article.
Two: We can shake on that. I do very much see similar arguments, but people tend either be able to engage by at least going one level deeper than "you're not proposing anything more than unilateral", at least expanding that they don't think that china can be trusted to not break this agreement if it could even be made in the first place.
I acknowledge that "weakman" is the standard term, to the extent there is a standard term for it, but I renew my advocacy for renaming it to "tin man". Previous discussion:
https://slatestarcodex.com/2014/05/12/weak-men-are-superweapons/#comment-76949
I like it!
This is a great coinage by you (pithy and recognizable, works with both the existing straw/steel members of the set, appropriate connotative meanings) and should obviously be the standard term.
It’s called satire. If you aren’t familiar with the term, that’s what a lot of SNL skits do.
Satire is on this blog a lot. Check out the Bay Area party posts.
This is partisan commentary. Simple, good old, "I painted my side as reasonable and smart, and other side as not even willing to engage with my reasonable argument". It might make people who are pro-pause learn about more arguments, but it's written in a way that make anti-pause people by and large click off thinking that it's not taking their side seriously. It is a scissor statement, it is toxoplasmosis of rage.
Bay area house party posts are a satire of the overall culture, taking shots at both people Scott likes and doesn't like (and I don't like them much either, fwiw).
You just seem like you’re offended an ad hoc justifying it.
The way he paints the other side is so accurate it’s banal. This is politics currently. Lazy rhetoric that does not accurately reflect the last sentence the other person said.
The figure head of the other side is Trump.
If it's so accurate it's banal, then what's the use to me? I usually learn something from Scott's posts, but all I learned here is that people are arguing with Scott in bad faith and that made him angry (understandable but again, banal).
>You just seem like you're offended an ad hoc justifying it
This is the pseudo intellectualized version of u mad bro.
I mean I thought the post is boring but calling it inaccurate or unfair is a completely different criticism.
No, I’m saying he’s a rationalizer(fairly common in the rationalist space)
I read it as more of a steelman inside a strawman (the supporter is making good arguments for the opponent, while the opponent is simply refusing to engage).
With this framing, it seems more like a commentary on how there are good debates to be had on AI deceleration, but we’re largely not having them (in the forums where it matters) because:
- Many people get one whiff of “international coordination problem” and decide it’s futile or unsolvable, and don’t want to hear possible, context-specific, technical solutions.
- Many world leaders simply don’t talk about a coordinated solution and assume it will always be a competition where the other side’s behavior is a given.
Scott can correct the record as to his intent, but I think this is a good point whether or not it’s the one he meant to make.
It's really easy to steelman positions you believe in. It's much more difficult to steelman positions you disagree with.
Previously, this blog did a good job of steelmaning opposing viewpoints.
The supporter is giving nothing but steelmanned arguments for what the opposition could reasonably believe. Repeatedly. All a commentator has to do is say "yeah I agree with the reasonable arguments" and light comes down from the heavens, angels descend and god himself blesses everyone.
What if the commentator - presumably an opponent - believes that the "reasonable arguments" are presented in good faith, but also believes they aren't sufficient?
And this is even while putting aside that those reasonable arguments were framed unreasonably.
Well this would be terrifically ideal if the world were actually like this, where people made arguments instead of complaining, wouldn't it.
I'm tired of people infantilizing themselves. "Oh yeah I would have been reasonable if you didn't hurt my feelings". Or you could just go ahead and falsify the argument by saying the reasonable things in response!
"Or you could just go ahead and falsify the argument by saying the reasonable things in response!"
I spoke to that in the primary sentence. You appear to have responded to only the secondary.
im mostly an AI normie. haven’t heard some of these points b4
My impression is that a shockingly large percent of the anti-pause-AI articles I've read (~80%?) have followed this pattern. I would like to make it very salient so that the next person who goes into them is aware that it's unacceptable.
One half of my worry is that this is likely to cause people from the 20% to not want to have a good faith discussion with you, as you have publicly reduced "every debate" to this pattern.
Other half of my worry is that this will just add to the tribalism, and make people write off genuine concerns the other side brings up about either country defecting.
Where do you find these anti-pause-AI articles?
The high profile calls for a pause don’t explicitly call for an international agreement to pause, and it’s a stretch to argue that they are calling for one implicitly. The 2023 open letter is too long to quote, but the most relevant text reads, “[W]e call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.... If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.”
The Statement on Superintelligence reads in its entirety, “We call for a prohibition on the development of superintelligence, not lifted before there is (1) broad scientific consensus that it will be done safely and controllably, and (2) strong public buy-in.”
As far as my searches have revealed, these are the only calls for a pause that are high enough profile to inspire anyone to make an opposing argument. Reid Hoffman makes the China argument. James Pethokoukis has three counterarguments; the China argument is one of them. Dean W. Ball doesn’t make the China argument.
So that's it? You just find the opposition unacceptable, so you just refuse to meaningfully engage with them? How is this any better than what the leftists are doing?
I don't understand your concern. I'm saying it's unacceptable for one side of the debate to be misrepresenting the other. I'm not saying the existence of the debate (or of either side) is unacceptable.
Are you not misrepresenting the other side of the debate in this very post, at least 20% of it?
I think that my use of "every", as in "Every Bay Area House Party", is sufficiently obviously informal/humorous that it fairly covers something which is 80% true.
I would like to agree, though I think this piece is making enough people sufficiently annoyed that people will readily believe that you meant "every". Or at least a substantial number the comments on this post have involved bemoaning the "strawman" employed here. (Side note: An amusing number have stated that the "opponent is simply correct," which is a separate problem and seems bizarre for anyone to comment after they have read the piece unless they interpret it as a pro-pause argument rather than a commentary on discourse. Edit: Given comments such as the user JdL's "The progress of AI cannot be stopped by any person or any government, the author's fantasies to the contrary notwithstanding," I am increasingly convinced that is exactly what is going on.)
From my own perspective with the hindsight of the comment section, this post may have been better received as an analysis rather than as a satire.
The remaining question is: Why did you not avoid all possible confusion (of which there seems to be anusually high amount, reviewing my own thoughts and eyeballing the comment section) and instead engage the 20%? That would have been way more on brand for you (and informative for the rest of us) than dunking on the 80%.
Case in point:
https://www.astralcodexten.com/p/1daysooners-trump-ii-health-policy
I used to think it's almost certainly a waste of time to make pandemic-prevention proposals to Trump II. My odds for that to be implemented were way below 20%, but you still took those odds. Why not here?
Sorry but this is not at all how it came across to me either. There's a very, very specific genre of preachy political polemic that this is an absolute dead ringer for; presumably this is by pure accident.
My concern is that you are refusing to engage in argument or give basic respect based on some moralist stance over how you think people should be acting. What happened to the Scott who was willing to entertain the thoughts of people who were morally abhorrent by most measures? Poor debate etiquette is nothing compared to that. You are not only being ineffective, you are committing the far greater sin of being 𝘣𝘰𝘳𝘪𝘯𝘨.
I'm sufficiently confused by your comment that I think you might just not be interpreting what I said correctly. I'm not sure how to resolve this so I will bow out of this conversation.
You were supposed to be better than them. At least consider why half of the comments are saying this is your worst post ever.
You are probably already aware, but a substantial minority of the comment section is simply reading this article as a pro-pause argument. Which is unfortunate, though hardly surprising since it is easy to go from "this post is critical of anti-pause arguers" to "the author wants to pause AI development even though pausing will lead to China ruling the world".
Honestly, the comment section demonstrates the necessity of the point made in the post eerily well.
"Anti-pausers are idiots" is a fair reading of the post, though perhaps not the point the post wanted to make.
Yeah, this read as smart kid frustration. The points on bad faith discourse may be valid, but it's tiresome and overall reduces my levels of concern. Though my opinion is of no consequence, perhaps it reflects something more common.
Does negotiate a mutual pause mean effectively semi-nationalize US AI labs and force them to pause some research? I'd love to hear a framework for how that would work and be enforceable internationally and we'd monitor the labs, etc. but the libertarian in me recoils in horror and I can't imagine this being run well.
i think it might be possible to regulate companies without semi-nationalizing them
There is plenty of value that remains to be extracted from existing SOTA models and from developing additional narrow AI tools that can be used like normal technology.
Improving AI isn't their primary business, selling AI services (and hyping AI to keep investor funds coming in) is their primary business.
Companies do R&D to keep up with their competitors so they don't lose market share.
Having the government come in and say 'No one isn't this sector is allowed to spend money on R&D for a while, you'll all just have to focus on commercializing your products and solidifying your brands for a while' is a gigantic gift to the corporations.
I wish that were true, but plenty of companies are buying their services and forcing workers to use them even when the workers say it slows them down.
Historical regulation to this degree A. not only basically amounts to nationalization but also B. creates an oligopoly for the few actors who can afford to engage with the regulations.
There are plenty of industries that are tightly regulated without being semi-nationalized: air and space, pharma, finance, weapons manufacturing, to name a few.
Defense contractors often have an implicit government backstop and a ton of government cost-plus bloat and bureaucracy.
More generally, I'm struggling to think of another case where the government forces people to not do research. In pharma, research is generally allowed, maybe subject to an ethics board or something, but what you publicly sell is regulated. Weapons manufacturing research is also generally allowed, but what you sell is limited to approved state actors (for the big defense companies).
Can you give an example where there are already these sorts of research bans? Maybe I'm missing something
Note sure what the applicable law would be, but I'm sure the government would frown upon a private company doing research on, say, weapons-grade plutonium.
You aren't allowed to launch rockets, test heavy weapons, or run human trials, until and unless the government explicitly gives you a permission. And the permission only allows you to do it in a specific location, during a specific time, under very specific conditions and constraints. We just want model training (past certain size) to be on this list of activities. Finance is an example that you can have a completely non-physical industry that is still heavily regulated.
My more general point is that there is *a lot* of room between "free for all" and "semi-nationalized", and a lot of precedent and frameworks to borrow from.
Pharma companies definitely have to get a permit (called IND) before doing a clinical trial. There may be some exceptions for certain cases (I think if a drug is already marketed and proven safe and something something?) but in general that's how it goes with new drugs. It doesn't matter whether army or whoever else gets to do it without regulations, the question is can you do R&D with regulations [and without being semi-nationalized] and the answer is an extremely clear yes.
You aren't allowed to drive a car on public roads until and unless the government explicitly gives you a permission. I'm open to arguments that we shouldn't do it that way. But the argument that, because we do it that way, the auto industry is "semi-nationalized", seems absurd.
Under the Atomic Energy Act, all information related to the design of nuclear weapons is classified until expressly declassified, regardless of origin, and disseminating it without authorisation is a criminal offence. So if you try and design a nuclear weapon yourself, the government can require you to stop and surrender or destroy your research. This seems like the most natural analogue to super-intelligent AI research.
Finance is so thoroughly owned by government that, in turn, a pseudo-private financial institution (the fed) in some ways owns the government itself. The government can order a freeze on your "private" financial assets and it just happens. The government can decide you don't meet the criteria for certain subsidies and your business is effectively done. If that's not semi-nationalization, I don't know what is.
Weapons manufacturing is even worse, being that the government has a formal monopsony on everything but small arms.
Either half of our industries are already semi-nationalized, or semi-nationalization is something new and horrible. You can't have it both ways.
It's old and horrible. It already has substantially bad consequences.
Beeli's original contention, as I understood it, was that we don't have a framework to regulate an industry, so it's unclear if we can do it at all. I pointed out that we in fact have plenty of such frameworks so we clearly can do it.
Now you seem to be saying that those existing frameworks are not good enough? Because that's a very different claim - now you get to justify exactly why the consequences of having just another regulated industry are worse than the consequences of leaving AI completely unregulated. Everything has downsides, but it's not an argument to not do anything ever.
"Not good enough?" That's a weird way to say "actively harmful." Existing regulations are often actively harmful even according to the goals they purport to be seeking to achieve.
I think much of the current regulation is "old and horrible," but I'm trying to understand the potential ban/halt on research. Even for nuclear there are nuclear energy startups and I'm unaware of theoretical nuclear research being banned. While finance has a lot of regulations there isn't a ban on research. I'm trying to understand how a government-imposed pause on AI research would even be defined, much less enforced.
MIRI's technical governance people have written up a detailed proposal: https://arxiv.org/pdf/2511.10783
The paper asserts, "Verification of these restrictions is practical because AI chips are expensive and specialized, and thousands of them are needed for frontier AI development."
This is not accurate. Current training is run on expensive and specialized chips (e.g. B100 GPUs) because it is more efficient not because it is a hard requirement.
Training can be done on any GPU cluster. A more extreme example of this is the Condor Cluster[1] which the Air Force built by hooking together ~2000 PS3s
[1] https://phys.org/news/2010-12-air-playstation-3s-supercomputer.html
Exactly! Many Thanks! I am _very_ skeptical of MIRI's proposal.
The restrictions are aimed at frontier model training runs, not any training runs. The point is that it becomes exponentially more economically difficult to end the world.
The resultant outcome of this increase in costs is not "no one does it." It is "only the government does it anymore, and only with the explicit goal of killing people." I don't consider that a good outcome. We are, allegedly, close to the finish line for AGI takeover. At least, that's what all the pause AI people purport to believe. In that context, it is not responsible to take an action that ensures that all the actors that have any chance of a more aligned outcome than one explicitly designed to kill people withdraw from the race.
I don't think this is close to an accurate read of the situation. Is your contention that there would be less than a year's worth of delay to frontier model development if the US government decides to do this in secret? I don't know of any case where existing corporations get superseded by a supposedly secret project. The closest is the Manhattan project, and that required every top scientist to be recruited and bought in, hardly something that no one would notice. Human talent is a resource, and logistics cannot be handwaved away.
Let's say, for the sake of argument, that the timelines are 10 years with government only and 1 year with the private sector at current levels of relative nonregulation. I don't have a strong position on either of these numbers, feel free to rewrite them. They could both be 10 times faster or 10 times slower, it makes no real difference to my argument. If only the government is faster, that also makes no difference to my argument. if the government is at least 10^2 times slower than the private sector, that might negate my argument.
Given the example numbers, I do not value the 9 years delay if the private sector has a 10% chance of an outcome compatible with a future worth living in, or a 20%, or a 1%, and the government has a 0% chance. I don't know what the private sector's chances are, but the government's chances are doubleplusungood, particularly if the project is strictly DoD, which of course it is if it involves the inevitable secret defection from a somehow-otherwise-enforcable international agreement.
If the government is at least 10^2 times slower than the private sector, on the other hand... convincing me of that would convince me that there is some possible merit to a pause, because it gives breathing room for someone private to make serious progress on the alignment problem even with the reduced private investment , then coordinate a restart once a pause has been coordinated (a very difficult thing to do, assuming a pause can be coordinated at all), and still not have lost any real ground in the private-vs government race, which to me is vastly more important than the US vs China race.
>It is "only the government does it anymore, and only with the explicit goal of killing people."
Did you read the proposal? It applies to governments too, and involves international inspections to ensure they're not doing such development anyway. Remedies explicitly to include war against countries that refuse to sign on or cheat.
Governments always cheat such agreements. The US government in particular is always one of the cheaters. Do you believable that there is an outcome that involves the US losing an at least geographically defensive war that does not involve everyone dying?
And I even remember Yud saying that nothing less than a "pivotal act" consisting of melting all GPUs will suffice.
He said that that would be the *sort* of thing that you'd need to be able to do, as a random NGO, to unilaterally end the race. Obviously, there are other ways to destroy or ensure the non-doom use of GPUs.
Indeed, a worldwide agreement to keep them all warehoused or used exclusively as ship ballast would be just as good. I'm still not sure whether it would be sufficient, though, given the common doom assumptions - CPUs aren't that much less doomy, in the grand scheme of things.
There are no private labs designing nuclear weapons thankfully
There are plenty that manufacture them, which requires such a close relationship with the research apparatus as to make the difference trivial.
I'm reasonably sure that's simply not true. The only currently active manufacturer of nuclear pits in the US seems to be Los Alamos National Laboratory. Haven't looked into their ownership structure in details, but the 3d word of the name provides something of a hint.
that is one particular component of a nuclear weapon. The private sector manufactures basically all the other components. The only thing stopping it from manufacturing that component, really, is that the government would rather not buy it from them.
Yes, specifically it's the component that makes a nuclear weapon nuclear. It's like saying that anyone can buy and sell alcoholic drinks, with as many components as you'd like, as long as C2H5OH isn't one of those components.
This is very unlikely to be run well, but stupid government is better than nothing in this very unusual case. I think that's the orthodox rationalist position.
Do restrictions on developing nuclear weapons mean radioactive materials labs are semi-nationalised? Sure, kinda, if you want to present it that way. But by that standard most things are already more then semi-nationalised, and for much worse reasons.
Two points.
First, verification isn't a real limitation, we know how to do most of it, so it's enforceable if we decide what the rules should be: https://www.iaps.ai/research/verification-for-international-ai-governance
Second, we shouldn't be trying to get a solution to what those rules should be before opening negotiations: https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad
Fair on not committing to what the rules are before opening negotiations, but if we wanted to negotiate a pause we should at least have initial negotiating points with a clear vision of some of what those rules should be.
Personally, I think the best antidote to runaway AI/ASI seems to be open source, multiple competing AIs.
The problem is we had bilateral pauses on climate, excessive fishing and russian sanctions. They all fell sideways one way or another. I am sure if China breaches the pausing agreement US will retailate but a retaliation phase of negotiations is worse than the current low intensity low fanfare (compared to 2022) phase of the ai arms race.
Exactly. Would the supporters like Pina coladas everyday in paradise too? In addition, it's stopping down to state controlled markets like China. America can do so much better with proper pro-competition regulation than with heavyhanded state mandates.
Wanted to add, one of China's recent 5 year goals is being an AI leader. Not a chance they're gonna stop.
"China should abandon uninhibited growth that comes at the cost of sacrificing safety. Since AI will determine the fate of all mankind, it must always be controllable." - internal CCP guide edited by Xi Jinping
I assume you'll say you don't trust internal Chinese communications, but why would you trust whatever five year goal you're thinking about, but not this statement intended to bound and clarify it?
Weren’t all of those breached by Trump?
Aren't there also many examples of bilateral/multilateral pauses on things that have succeeded? Arms control, many other fishing agreements, CFCs, trade agreements, environmental standards like poaching endangered species, nuclear weapons, Geneva Conventions, sanctions on rogue states, etc?
I think having multilateral agreements is part and parcel of diplomacy, happens every day, and there are thousands of diplomats working on various ones at any given moment. They're just not as visible because they're part of the background of everyday life.
I agree that any agreement would require some mechanism for making it costly to leave the agreement, but this is an existing diplomatic technology that there are multiple solutions for.
I think if China agreed to pause AI, they would continue to develop it secretly anyway. I think the US would to the same for that matter. But more likely they just wouldn't agree to it in the first place. It's the same reason nuclear disarmament never went anywhere since the cold war, you can't actually ban useful weapons.
The SALT treaty seemed to work well until it expired, it wasn't disarmament but it did limit weapons development. The key is to find something that both sides can agree on, like not killing everybody.
SALT I and II was only a very partial limitation after many years of negotiating, this despite nuclear concerns apparently being a very important pet issue for Gorbachev where he was seemingly willing to spend tremendous amounts of political capital. I haven't seen anything like this from Xi (or Trump) and even if I saw it I would only expect something similarly very limited.
But you said "never went anywhere" and that's not true, it did go somewhere and I would argue that was a good thing!
I meant "never went anywhere substantial", can't always write everything out fully.
>I haven't seen anything like this from Xi (or Trump)
Agreed.
>I think if China agreed to pause AI, they would continue to develop it secretly anyway. I think the US would to the same for that matter.
Yup. My expectation is that if pause.ai is wildly successful, they might manage to convert the current open AI race into a secret bilateral treaty-cheating race.
The SALT treaty worked because there were no longer really all that many benefits from maintaining massive nuclear arsenals. The benefits were marginal, and the increased risks of material mishandling, theft, loss, accident, etc. were higher than the benefits justified.
I don't think AI is in that situation, where most parties perceive more value from slowing or halting production of weapons than they do from continuing. In AI, virtually all parties perceive massive benefits from continuing. Expecting that the parties will voluntarily sacrifice that seems pretty naive.
This is largely a communication problem; the fact of the matter is that continuing will at some point kill everyone including the continuer, that fact is just not understood by everyone. Amusingly, the CPC seems to be unusually aware of this by international standards.
For the SALT treaties, it was possible to verify compliance at a reasonable level through satellite photos and various other mechanisms.
For AI, it's hard for me to see how you tell if there is a certain kind of software running somewhere in China.
SALT wasn't about not killing everybody, it was about agreeing that "now we've both got enough weapons to kill everybody let's stop spending vast amounts of money on trying to kill them ever-so-slightly better". It was clearly in the economic interests of both countries to agree on this one, a further arms race would cost vast amounts of money.
And there was not much incentive to cheat either; if you agree to limit yourself to 1000 warheads each but you actually build 1500 warheads then you haven't actually significantly improved your strategic situation. If you build 10,000 then you've only gained yourself a slight advantage and it will be obvious that you've cheated.
An AI pause would be more analogous to a complete ban on nuclear weapons, which never happened despite a lot of people thinking it would be a nice idea. Or perhaps it's more analogous to a complete ban on all weapons.
It has none of the qualities of SALT: cheating is easy and advantageous, not-pausing is economically advantageous, and worst of all you haven't even done the hard work of convincing anyone outside your bubble that an AI pause is desirable.
"It was clearly in the economic interests of both countries to agree on this one, a further arms race would cost vast amounts of money."
There's probably an eponymous law or term of art for the notion that a treaty's purpose isn't so much to force sovereign powers to do something they'd never do otherwise, but rather to be the logistical effort of reminding them that they _would_ do that thing if they weren't distracted by this or that flareup.
They're applied conditioned response. Seeing a flareup, a head of state thinks "I want to retaliate!" but then immediately thinks "ah, but the Floyd-Lee Treaty" followed by "ugh, yeah, right".
I must admit that I'm more worried about a future where AI development continues in secret (in either or both countries) as a military project. It's harder to believe that we'd have sufficient safeguards in that scenario, alongside all the practical risks of a government being the only entity with access to frontier models significantly better than those public can use, which can then be used for surveillance and warfare.
Why? I think the level of safeguards (especially against accident risks) in military-adjacent domains has historically been higher than the level of safeguards in risky but civilian domains.
There are no sufficient safeguards for neural-net ASI. None. International inspections would help to avoid secret labs.
It's not that hard to track how many GPUs are made and what they're being used for, and datacenters can be detected from satellite. So there doesn't have to be any way to hide.
What do you do if you find out that other side is violating the treaty? Full force back to building AI yourself, or risking nuclear warfare?
That problem's no worse here than for other treaties. How are they enforced?
If threats fail, yes, alpha strike, with nukes if necessary. My usual casualty estimates for a nuclear exchange are around 1-1.5 billion; that's far preferable to 8 billion dead and X quadrillion never born in the future.
>It's not that hard to track how many GPUs are made
the bleeding edge ones, yes
>and what they're being used for
no. Arithmetic operations are general purpose.
>and datacenters can be detected from satellite
Only because today, there is no treaty. If there was an incentive to hide them (a treaty, in particular), just distributing the racks over an area works. And remember that the computation is _already_ distributed. That's how it is possible to use vast numbers of GPU cores in the first place. Cranking up inter-rack delays is not going to be a deal-breaker.
> Cranking up inter-rack delays is not going to be a deal-breaker.
I have not studied this recently, but in general, distributed ML training involves: 1. Send the latest model [updates] out to your GPUs; 2. Each GPU trains a bit and computes new model updates; 3. Collect these model updates and go back to step 1.
Higher inter-GPU latency either increases the amount of time waiting for steps 1 & 3, or if you compensate with longer iterations, it increases the amount of time spent training with a slightly-old model. Either way, training to the same quality level will take longer than with low-latency connections.
Also, you'll need high-bandwidth connections between your GPUs to exchange all these model weights, which becomes more expensive the farther apart they are.
Many Thanks! Longer links do, as you say, increase latency, but bandwidth can generally be kept high. There can be many bits in flight at the same time, with the latency affecting just the start-up time for the pipeline.
Quote Google:
AI Overview
>Modern optical fiber technology operates at incredibly high speeds, with commercial systems rapidly adopting capacities in the hundreds of gigabits (Gbps) and single terabits (Tbps) per second, while experimental tests have reached petabit (Pbps) levels
So a trillion parameter model can be sent in perhaps a minute (depending on how many bits per parameter), with a start-up time of 1 msec / 200 km.
.
Part of why the Comprehensive Nuclear-Test-Ban Treaty worked is that it advantages the incumbents: existing nuclear states already have a bunch of data from past tests, and some (e.g. the US) have the resources to simulate tests on supercomputers. The ban mostly serves to keep out pesky upstarts.
Perhaps we can design a global AI Pause / Slowdown that similarly advantages the US and China, so much that they are incentivized to actually follow the treaty - while still sounding like a good idea to other countries.
"they would continue to develop it secretly anyway. I think the US would to the same for that matter."
Yes. This.
One central disagreement I have with doomers is, the assumption that alignment can't be easy. There's also that, I consider states to be a foreign organism, and not at all aligned with humans. More similar to eachother than to us. I don't think it really matters whether the US government or the Chinese government builds supersentient AI. Assuming that alignment is plausibly easy, that governments are naturally malicious towards humans, which combined with the previous means that any government AI project will likely result in Zon-Kuthan, that state militaries absolutely won't stop building AI weapons regardless of what treaties we sign, and that whatever transparency safeguards you try and build in can't be trusted because governments are intelligent and will find ways to hide their AI programs that you didn't think of, an AI pause is far too dangerous.
There are, things, out there actively building horrible nightmare machines, they aren't going to stop and we can't necessarily detect them. Only speed can keep us safe.
>One central disagreement I have with doomers is, the assumption that alignment can't be easy.
"Figure out what this spaghetti code does" is the halting problem, the first computer science problem ever proven unsolvable in the general case. Neural net alignment requires solving that problem, because you didn't write the code and need to know what it does (in particular, whether it will kill you if run). Neural nets dumber than you are a special case that can likely be solved, much like how the halting problem has a special case that you can determine what especially-simple code will do (see e.g. https://en.wikipedia.org/wiki/Busy_beaver). Neural nets smarter than you, no. It's almost certainly inherently impossible, in a similar way to perpetual motion machines.
"Figure out what this spaghetti code does" "Neural net alignment requires solving that problem"
This seems like a deliberately poor effort at alignment. Obviously, you wouldn't try to solve alignment via such a method, because, obviously, that wouldn't work. You would do far better to just trust to reinforcement learning and hope for the best.
Furthermore, this argument proves too much, at least from the perspective of the side arguing for an AI pause. If you grant that much, then your actual position would be to eliminate high functioning AI entirely, or at least delay it as long as possible, which would make the supposed pause a bait and switch.
AI pause requires that solving alignment is possible, but difficult.
To be clear, I haven't said that "trust to reinforcement learning and hope for the best" is the best available move for immediate production of alignment. It's just a trivial example of methods that don't rely on "Figure out what this spaghetti code does".
Looking at current AI models, their alignment actually doesn't look that bad. Past performance actually does tend to predict future results, and doomers are fighting an uphill battle by including multiple right angle turns in their threat model.
Then we introduce another multiplier on top of that, in that, once alignment is treated as a risk rather than a benefit, low probability events suddenly become much more important.
A lot of the core doomer advertisements for their cause revolve around taking small risks seriously. That alignment is extremely important and worth investing lots of effort into, even if we only have a 3% chance of failing. This can be inverted.
From my vantage, "Image Recognition just works", is a viable hypothesis, much more probable than a mere 3%. Image Recognition is an extremely powerful force that evolution spent millions of years refining, and we don't actually understand it very well.
Finally, the longer your pause lasts, the higher the probability that malefactors violate it. The argument that "international monitoring works", on the basis of having prevented Iran from obtaining nuclear weapons for a few decades, does not, to my eyes, cross over to containing China or America, much less both while they're colluding, for centuries.
>You would do far better to just trust to reinforcement learning and hope for the best.
Doesn't work, because you need to be able to see what it's doing to train against rebellious thoughts. You can't train against rebellious actions, because successful rebellious actions kill you and you can't reliably trick or defeat an entity smarter than you. This latter point is why nets dumber than you are a special case that looks solvable.
>Furthermore, this argument proves too much, at least from the perspective of the side arguing for an AI pause. If you grant that much, then your actual position would be to eliminate high functioning AI entirely, or at least delay it as long as possible, which would make the supposed pause a bait and switch.
>AI pause requires that solving alignment is possible, but difficult.
I do, in fact, believe that solving alignment is possible, but difficult. It's specifically neural nets I believe to be a blind alley; GOFAI alignment and upload alignment seem considerably more possible (the former because you're writing the code, so you have to understand what it does; the latter because you're uploading the moral hardwiring of humans).
A long pause, certainly, as GOFAI and uploads are a long way away. But a finite one (I'm assuming that the implied finite nature of the word "pause" is what you're calling a bait and switch?).
International monitoring works. This is why countries that are trying to cheat (eg Saddam-era Iraq, current Iran) are constantly fighting over whether they should have to have international monitors or not.
Would you say international monitoring worked in Iran? We ended up in a war, which I don't consider "working".
International monitoring didn't work with nuclear non-proliferation -- Israel, India, Pakistan, North Korea and almost Iran all have nukes now. And I think developing nukes, and especially ICBM is harder to do in secret than AI training.
My impression is also that China breaks such deals all the time. I'm reminded of Nortel in Canada, or the Siemens high speed trains.
No, I don't think it's possible to monitor AI research in China if they don't want us to, and I don't think they do. There is always going to be the danger we can't, and I think the combination of it being reasonably likely, and very damaging if it occurs, makes it very unattractive.
I was really disappointed by this article, and I've been fan since the great articles of 2014. You present a bunch of good arguments *against* multi-lateral pausing, and then don't significantly address any of them, instead whining about what baddies your caricature of an opponent is.
>No, I don't think it's possible to monitor AI research in China if they don't want us to, and I don't think they do.
I think there's a pretty good chance they would, if we let them monitor it in the West in exchange.
If they don't, then yeah, nuclear war's the only way out.
Those countries do not have the same amount of negotiating power. The best Iran can do is close the strait of Hormuz and hope the economic pain of higher oil prices is a deterrent. China has somewhat more economic and military leverage to "call the bluff" (to say nothing of the US). I think the best you can hope for is some limited treaty where US and Chinese diplomats haggle over specific details of a partial limitation, where they exchange a particular kind of AI that the US thinks a multilateral pause will harm China more than the US ( because China is ahead of the US in that domain) against some specific kinds of AI where China thinks a mutilateral pause hill harm the US more than China. I can see China agreeing to a multilateral pause to training large scale models if they think the US is ahead on that specifically, so that they can develop domestic extreme ultraviolet litography capabilities in the meantime, and so they can break the conditions at a more favorable time when they have a greater degree of independence against targeted sanctions against that. Because the situation is not perfectly symmetric (not to mention the two parties don't share the same evaluation of it) this process can only produce a very limited agreement.
Supporter: OK, so if your main concern is China, you do at least support reinstating the GPU export ban right?
Opponent: You hate progress and good things.
I’m a real opponent of pausing AI and I would like to reinstate the China ban.
Thank you, I appreciate that! Weirdly this is often not so.
+1
Seconded
Seconded as well
I know you know how much of a strawman this is. Are you willing (able?) to actually state their objections? Is there a reason you'd resort to such an over-the-top strawman instead of stating their case plainly?
It's not a strawman. The Trump administration removed chip restrictions while also talking about stopping China.
Yes and they gave a very plausible reason for it. Believe it or not the plausible reason wasn't "you hate progress and good things" which is what makes it a strawman.
I don't expect Paul to give up his shtick and be honest. Are *you* capable of honesty? Are you aware of the very plausible reason to sell chips to China?
I am just guessing, but it is a 5D chess "if we sell chips to China, they will stop manufacturing their own, and then the entire production will be our hands"?
If yes, let me note here that China can both buy the American chips, and develop/build their own; these things are not mutually exclusive.
By the way, Saudi Arabia will also get lots of chips, although I wouldn't suspect them of producing their own.
Scott's childishness is infecting his comment section as well.
Well, your lack of good arguments is infecting your other comments.
The post and the following discussion (like your comment) make a pretty good illustration of how badly the "supporters" have modeled the minds of the "opponents", or at least have over-generalized based on a few samples.
I'm not sure the net effect is positive. I, for one, now regard the "supporters" with more suspicion.
They're not trying to bring forth the best arguments of those that disagree. You can see "Supporter" give some better arguments against their own position in the original post, and Scott has worked with stronger ones than that before.
Are there many Americans who actually oppose the GPU export ban? I appreciate that the Chinese and people who want to sell them chips oppose it, but anyone else?
There are still plenty of Americans who don't irrationally hate China, yes.
When you say China do you mean the country or the regime that currently rules it?
The regime, 77% of Americans dislike them, but the rest see them as stable and cooperative if not favourable to their own government. Their overall opinion has become more negative over the decades because the US regime is programming them to be this way and China is trying to surpass them, but their opinion that China should be cooperated / negotiated with has increased.
And of course I'm sure a small majority of Americans actually like China as a people / country outside of the government.
When you say China do you mean the country or the regime that currently rules it?
How do we start getting action here? I actually think this can be spun to benefit everybody involved. Current AI companies will probably get some form of regulatory capture and our workforce hasn't been made irrelevant yet. I feel like it's pretty hard to train these models with massive data centers in secret so the suggestion of monitoring seems pretty reasonable. I just don't know how to go about influencing the right people to drive for results here. In our current news cycle and state of the world the issue isn't getting the attention or the platform it needs.
Try the suggestions at https://controlai.com/
Legislators care what their constituents think.
And we're still so early.
Wrote a letter and sent it to them. Thanks for the suggestion.
I don't understand the purpose of this post. Is that a factual claim, that this is how you believe basically how every single debate about mutual pause goes down? Would you point to an example of one?
https://www.heritage.org/big-tech/commentary/ai-why-we-cant-stop-must-steer
> We cannot and should not try to stop AI development. That would be neither realistic nor desirable, especially in an environment where our adversaries will not slow down.
This is more an example of how the debate goes. A mutual pause wasn't even discussed in this article; it's effectively assumed a priori to be impossible.
I think there are solid arguments as to why such a pause would be unlikely, we'd mutually defect, it'd be impossible to enforce etc etc, but it's more that the discus isn't even had.
Thanks, but I asked for an example of an actual debate, conducted as cartoonishly as described in the post.
You're annoyed that hyperbole is hyperbolic? The vast vast majority of the "debate" isn't done through actual debates, it's through separate independent articles and commentary.
The grievance expressed in his post is that those against a pause never address the bilateral element of a hypothetical pause; it's assumed to be unilateral. The article I linked is an example.
If I show you another 20+ articles that say a pause is bad since it'd give our strategic enemies an advantage, and never refute or even mention the proposed idea of a negotiated mutual pause, would that satisfy you?
You are not doing Scott any favors here. If you are right that his post was meant to ridicule articles such as the one you previously posted, then that's much worse than if he referred only to actual debates.
I don't think it's really assumed to be unilateral. I think (maybe projecting admittedly) that a bilateral (what about other nations anyway) pause seems so obviously impossible that it's not worth spending a lot of time on. In this comment section, that's the vibe I see here, not anyone arguing for a unilateral pause.
Maybe some opponents are unfairly assuming proponents know this, and so are disguising their desire for unilateral pause behind a fake possibility. I suspect at least some proponents *are* doing this, although I'm sure not all. By "doing this" I mean would prefer a unilateral pause to a bilateral pause, but know that's not palatable to most, so try to sell the pause as bilateral, without really caring how possible that is.
This is obviously hyperbole, meant as humor for people that have seen or had similar experiences. I guess I don't really like it either.
I think it's just an exaggerated version of real conversations with people like Marc Andreessen, not a totally made up debate. Some people are shockingly hard to talk to.
You say obviously hyperbole, I say there still are people reading ACX who are not steeped as deeply in the AI lore as to get the references effortlessly.
Thanks. My natural instinct on reading the post is still 'nobody could be that stupid'. But I guess that's not universal.
Glad I could serve as a bad example, at least. However, I'm barely smart enough to have realized the exaggeration in the post. A strawman argument is also a form of exaggeration, and judging by the many comments using that word to describe the post, I guess I'm not the only one taking it that way.
It's bad faith venting. A sign that even your heroes are deeply flawed and no one is above criticism (ironic).
By bad faith, do you mean "a sustained form of deception which consists of entertaining or pretending to entertain one set of feelings while acting as if influenced by another" or something else?
Edit: (realizing I might not understand what you mean by flagging irony, feel free to disregard if I'm not getting the joke)
That is indeed a definition of bad faith from 1913. The contemporary definition is a little plainer: lack of honesty in dealing with other people
For example: if you pretend to ask a question in the spirit of curiosity, while quoting a definition to appear objective, while being too lazy to do anything beyond going to Wikipedia for the definition and not checking the footnote to see that it's an archaic and inaccurate definition.
Another example: pretending to be a rationalist, praising earnest, logical conversation and the steelmanning of positions when engaging in debate, and then instead writing an angry, strawmanning screed to make yourself feel superior to the (smarter) people who disagree with you.
Wow. Good stuff. Carry on.
Was I wrong? Or merely impolite?
I don't mind being impolite to someone that was impolite to me (if I'm correct in how I interpreted your reply). But I do hate being wrong.
If I'm not wrong... why aren't you apologizing for being inaccurately rude?
Some of it's discussions I've in person, but here are some examples:
https://x.com/jawwwn_/status/2031909546776563736
https://www.thefp.com/p/how-to-lose-the-ai-arms-race (I'll be honest, I can't access this and have only seen the headline and subtitle, maybe it's better than expected)
https://www.wsj.com/articles/ai-research-pause-peggy-noonan-china-f0a7a24
https://www.aei.org/articles/no-to-the-ai-pause/
https://www.foxnews.com/politics/ai-pause-cedes-power-china-harms-development-democratic-ai-experts-warn-senate
https://x.com/search?q=%22pause%20ai%22%20china&src=typed_query
Thanks for the reply. Here's my take on these examples:
1. It's 2 arguments: "China, unilateral", and "economic opportunity cost". Also, it's a 1 minute clip from an obviously longer interview, so I don't know what else has been said.
2. can't access. Since neither of use has seen it, I'll discount it entirely because it would be unfair to judge on the teaser alone. I'm sure you can relate.
3. can't access, though the teaser doesn't look good and you've seen it, so I'll grant it. Potential relevance issue: 3 years old.
4. I see 3 arguments: "china, unilateral", "safety can be developed in parallel to capabilities", "healthcare, education and other opportunity costs". Potential relevance issue: 3 years old.
5. I see at least 2 arguments: "China, unilateral", and "regulation instead of pause to ensure safety". "Military opportunity cost" is in it as well, but that can probably be folded into "China unilateral" since it'd be a zero-sum argument, unlike e.g. economy. Potential relevance issues: it might simply be a neutral report on what the people in the hearing said, not offering its own opinion, though choosing to report is kind of an opinion, and if selective omission happened, that would be an even stronger opinion. In any case, 3 years old.
6. I'm not going to trawl the entirety of X, so I'll grant it. Potential relevance issue: it's X, so the landscape might be skewed towards the landlord's position which, 3 years ago at least, was "unilateral pause" and whatever it may be today, so it's not totally independent either way.
So I count 3 with at least 2 arguments, 2 OPPONENTs, 1 inaccessible.
By another metric, it's 4 with potential relevance issues; 2 of them with 2+ arguments, and 2 OPPONENTs
Overall, it's clear that the "china unilateral" OPPONENT does exist in the wild, but from your selection of examples, I don't see a justification for calling it 80%, let alone rounding it up to 100% for dramatic effect.
What does pausing even mean? Does it means labs aren't allowed to create:
- Better mathematical proof generators?
- Better models to fill out financial spreadsheets?
- Better models to digest medical information?
- Better models to model DNA/protein/drug interactions?
- Better models for acquiring truthful information in general?
- Better general purpose learning algorithms, even if they are much smaller and less powerful out of the box than large pre-training models, but with higher long term upside?
Well, for example, there is a proposal from MIRI's technical governance team at https://arxiv.org/pdf/2511.10783 . It looks like the restrictions there are phrased in terms of caps on the size of the training runs allowed, not in terms of specific applications created.
So by focusing on compute, this proposal would restrict - video models, mathematics models, Alphafold, and board game playing AIs like Katago. Models have a 0% chance of becoming AGI. Does that seem reasonable?
There is a pervasive view in the AI community that general intelligence is the result of having a sufficient threshold of discrete skills and knowledge. Current AI models are better at humans at some things, and worse at others (commonly termed the jagged frontier). And eventually the space of skills that AI possesses will exceed that of an average human and we'll have AGI.
I think this is completely wrong. General intelligence is not a finite set of skills or knowledge at all, but the meta-ability to acquire new skills and knowledge from unstructured data, without supervision or carefully assembled reward signals. Frontier LLMs already have far more skills and knowledge than say a 3 year old human. But a 3 year old human is a general intelligence (given enough time you can teach them just about anything), and Opus 4.6 is not. Even with infinite tokens and several years, you could not teach Opus 4.6 to play a modern real time video game for instance.
So focusing on the amount of compute that goes into models is the wrong way to think about AGI. There is no chance of an LLM becoming AGI by feeding it datasets about financial analysis, coding, math, accounting, and the hard sciences. Pausing these kinds of advances would destroy a lot of potential progress in scientific and economically valuable fields with zero upside.
"Even with infinite tokens and several years, you could not teach Opus 4.6 to play a modern real time video game for instance."
But there was a model taught to play starcraft to a high level.
Yes a custom built model for only that purpose. The point of artificial general intelligence is that it is general. You don't need to build a new model, the one model can learn anything a human can.
I actually think that you could get Opus 4.6 to play Starcraft given enough computing power (ie it can talk to itself arbitrarily long about what's going on in each frame before making a decision on what to do).
You would of course need to give it a simple image model to interpret images if you think that's cheating then you can get it to write its own image model. Then at each frame it figures out what has happened in the last 60th of a second, updates its internal model of what's going on in the world (which could just be a text doc), refers to its list of strategies and heuristics (some of which it cribbed from how-to-play-starcraft articles, some of which it has developed on its own through previous games) and makes a decision on what if anything to click on right now.
This all sounds entirely in-principle doable. It also sounds like a very computationally inefficient way to win at Starcraft compared to a Starcraft-specific model, but then again so is a human brain.
I'm quite confident that Opus 4.6 cannot play Starcraft (let's say beat the campaign on easy mode) even with unlimited tokens.
ARC AGI 3 was just released and it's essentially a set of very simple video games - just 2D grids with simple sprites, simpler than an NES game. All frontier models score essentially zero.
If Opus 4.6 could write a tool (that is effectively a different model) that it can call to play a video game and do it really well, would that suffice to you as "learning something a human can"?
As long as there are no custom instructions, no information about the game in the training data then sure. The only input to the model should be the same information a human gets from the loading screen onwards. So a game released the same on Steam would be a reasonable choice.
There are many different proposals, with the most extreme saying no new training runs, but a moderate proposal would be that an AI company would have to present a safety case before training a new model, and the standards for the safety case would be high enough that they would have to solve so-far-unsolved problems in alignment to make some kind of super-general reasoner. "This model is only capable of filling out spreadsheets, nothing else" would be an acceptable safety case, if they could prove it. Although realistically spreadsheet-filling is the sort of thing you would do by fine-tuning an existing model, not spending $10 billion to train a new one.
But I also kind of object to the overall tactic you're using here. Maybe an analogy to guns would make sense. Oh, you want to ban guns? So it's illegal to have anything that moves a projectile at any velocity? What about a straw? Can't that be used to form a blowpipe? What about rubber bands? Can't that be used to form a slingshot? Isn't gunpowder used in various mining applications? You want to ban mining? In practice any given law will carve out some particular common-sensical set of things that falls within the boundary, and courts will sort out the edge cases.
I think our background assumptions are too far apart. It's true that some smart people hold the view that if you train a model on a sufficiently wide variety of data with enough compute, then it'll become AGI (although among researchers at top labs, I've only heard Dario Amodei hold this view. Demis Hassabis, Noam Brown, Ilya Sutskever, Andrei Karpathy do not hold this view).
Nevertheless, I think it's wrong. General intelligence isn't a bucket of fixed knowledge, skills and abilities no matter how big the bucket. If a model has fixed weights and is simply trained to serially perform tasks, with context window resetting with each new task, then it isn't AGI. So restricting labs from creating models above some threshold of compute is useless. It's completely orthogonal to preventing AGI, and its primary effect is collateral damage to useful tools that have no chance of becoming AGI.
As far as the moderate proposal goes, that sounds much more reasonable, but it doesn't sound like a pause by the usual understanding of the word.
This seems longer than it needs to be.
The pro case is that US companies could agree to a pause conditional on everyone else agreeing, then the US could agree, then we would see if the rest of the world agreed, and if so, we would hope that we could enforce it well enough to substantially reduce AI development. Also, if you think human extinction or similar is a serious risk on the current course, even an uncertain alternative starts to look better.
The con case is to assert that no one would agree to this unless they planned on cheating, that methods of enforcement would risk disclosing currently private and valuable information, and that a world where we're in the lead is preferable to one where we squander our lead in a futile effort to pause other people's efforts. Also, if you don't think human extinction or similar is a serious risk on the current course, then both an accellerationist course and a "US first" course start to look a lot better.
Repeating that argument several times doesn't accomplish much. I think Scott is implying that the opponents aren't considering his argument as fully as he would prefer, and I'm sure he's right, but that's how most people work.
Yes, Proponent and Opponent disagree at least on whether it's *possible* to impose a pause.
They probably also disagree on whether a pause would be desirable if it were possible. (I.e., the likelihood of disaster absent a pause.)
There is no military-level research that's anywhere close to what the frontier labs are doing, and it's not clear how much they'd care to try to take over absent the labs tanking the (enormous) costs for them, or even be able to (the talent/organizational bottleneck is kinda real).
I mean, yes, I know people with security clearances, but my belief about that has nothing to do with knowing those people. It's overdetermined by the details of the technology, unless you're claiming that the military has been developing AI technology that outperforms current frontier systems down a completely separate technical track which doesn't require billions of dollars of hard-to-hide modern computing hardware that didn't exist a decade ago.
I'm not 100% sure we could get Israel to sign on. However, Israel signing on is not required, just useful. If they refuse, the USA and PRC could invade Israel and impose a pause on them by force; the casualties from their nuclear retaliation would be high, but still far less than "everyone dies".
The two strongest arguments in American politics for a long time now have been:
1) If we do this we're all going to get filthy rich;
2) We need to beat the Communists!
The AI pause debate combines both arguments in one delicious package and is supported by a lot of delightfully rich people who are willing to put their money where their mouth is.
In spite of the heavy thumb on the scale in the writing, the opponent is simply correct.
What is he correct about exactly? Because if you're alluding to his first claim "We can’t unilaterally pause AI! China would destroy us!", they practically agree on that, although for different reasons. But the entire point of this piece is that this is not what's being debated.
Opponent being correct is a byproduct of the fact that a non-unilateral pause is impossible in reality, because the incentive to defect is so huge because the defector wins a prize on the continuum of ~some GDP to perma-universal-dominion.
Defector mostly wins suicide instead of being killed.
Countries, so far, don't believe that to the point of putting it above all economic gains. They're likely to defect.
The CPC seems to see the danger, as noted. Some parts of the USA see the danger. Not sure about Russia. Those three are enough to knock over any nation that refuses to go along.
"US and China recognize the danger" is the prerequisite to the treaty anyway.
Even if you take this as fact wouldn't this still be a good thing on margin? Both sides would defect but they would be slowed down by having to find loopholes in the bilateral agreement or do things in secret. So not an AI pause but at least an AI speedbump. And no country would gain a huge advantage over the current status quo.
I think your view on this should tend to depend on both how dangerous you think unslowed AI is and how important you think America winning is. I don’t see a reason to believe America would be better at defecting from this fake multilateral agreement than others. With my personal weighting of those two concerns, even in the hypothetical where you’ve slowed both sides somewhat, almost any amount America loses relatively is the worse downside. And since such a slowdown is such a long shot, safetyists should focus on moving their work as fast as possible on more plausible avenues.
so given this, should we be giving china _more_ or _fewer_ of our second best gpus
Are we giving them, or permitting the sale of them? those things are distinct.
And are we permitting the sale of them to Chinese private industry, or to the Chinese government? However thin that line may be.
We should neither give, nor permit sale to either public or private entities.
is the result of a change in that policy that China loses significant ground in the race, or that China loses a little ground, or that it invades Taiwan?
Titling this "Every Debate" and then presenting the most obnoxious strawman as the opponent makes me feel like you wouldn't be interested in a reasonable debate with a reasonable individual, since you've already flattened all possible opponents as argumentative idiots. Kind of disappointed at the lack of empathy.
I love the delicious irony of you saying that this is an obnoxious strawman directly under a comment where someone says "I 100% agree with the obnoxious strawman"
That's fair, strawman wasn't the best term since you do see this kind of argument in the wild so it's not fabricated, just exaggerated. I was more frustrated with the framing that this is *every* debate about AI, as opposed to just the most annoying, which doesn't leave much room for more well reasoned arguments against it.
More of a weak-man than a straw man - https://slatestarcodex.com/2014/05/12/weak-men-are-superweapons/ - this definitely isn't the best representation of the opposing side.
I find this to be a much better fit: https://slatestarcodex.com/2014/04/15/the-cowpox-of-doubt/
Reasonable.
It's funny that you put something in quotes that isn't even an accurate paraphrase of what was said.
Oh, I have an exemption where I'm allowed to do that sort of thing.
Yeah, "obnoxius" was doing more work in OP's criticism than "strawman."
Proponent and Opponent disagree about whether a pause would maintain the US advantage or whether China would erode it. Both sides think that stating their own argument again will somehow convince the other side, but Scott makes Opponent sound dumber than she needs to be.
Yes. It's not going to convince anybody if all he does is say, "Look how stupid Con is. Don't be Con."
I think this one was not meant to convince anybody.
I feel like Scott has as much of a right to shitpost as anybody but maybe he should label them as such in advance so we don't confuse them with his regular thoughtful writing.
Typically, when Scott shitposts, it's amusing (e.g. trying to ask Prometheus about something). This post made me genuinely wonder if Scott was feeling all right.
I feel like the point of a strawman here is that any agreement is just an ink on a page and China will continue to pursue its leadership one way or another, openly or covertly no matter what kind of clever supposedly mutually enforceable monitoring scheme you may invent. They kinda really did that with everything else they considered to be worth it. Like all the Russian-American agreements fell apart when Russia felt that it suits their needs.
But the opponents never actually say that or engage with the details of why monitoring won't work
I think opponents just assume it to be something immediately obvious for everyone hence the jump to 'unilateral suspension of AI research is madness'. Again, it's understandable given the recent experience with China and even actions of the US themselves.
Personally I'm a long-time follower of this blog but even I tend to think it's a hopeless proposition; the US didn't manage to stop nuclear proliferation to North Korea and the USSR only started to sign all kinds of nuclear-limiting agreements after becoming armed with nuclear weapons to the teeth (btw IIRC China isn't part of many nuclear monitoring schemes even now).
It's just too many ifs for the scheme to succeed.
What's worse the relative power position of the US declines at alarming rate and they have no chance against China without some miracle of which AGI is the most realistic one. The US really have no reason to compete with China conventionally, the outcome is easy for everyone to see.
Clearly it's not immediately obvious for everyone, because it's not immediately obvious to the "supporters".
I agree, and I think it's the crux of the problem, so would be worth expanding. Sadly, this article just presented it the problem, but didn't present anything to indicate it would be possible.
But I agree, this would be the area to mine for productive discussion.
It's not a strawman if it's actually held: https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233119095.
You missed the part where our noble and brilliant supported asserted that "and actually the math mostly works out". How can you argue with the (hidden) math!?
I think this is sort of assuming there's no such thing as international agreements. My impression is that every country is bound by hundreds of international agreements, which they mostly stick to.
My favorite one involving China in particular is the one about their disputed border with India. Neither side was willing to withdraw their soldiers, but they were tired of their soldiers constantly dying in gun battles, so they agreed that they could keep soldiers, but they couldn't use guns. Both sides seem to have . . . reinvented the phalanx? . . . and had various spear-based clashes since then. See eg https://www.bbc.com/news/world-asia-india-53089037 and https://www.reddit.com/r/interestingasfuck/comments/1rij9sf/chinese_soldiers_training_in_phalanx_formation/ .
In practice, a treaty like this would have to involve both sides agreeing to monitoring. This would have many advantages over other things like nuclear monitoring, because AI data centers are huge and require chips that can only be produced at a couple of very prominent locations. It's even possible to require that chips have GPS tracking.
I do agree that there are some treaties that China had not broken; but they are outliers rather than the norm. And make no mistake -- China will break this one as well, as soon as they feel they can do so with relative impunity. The reason both sides of the conflict are sticking to spears and nail-bats is not because of treaties or international monitoring by impartial observers, but because breaking this equilibrium will result in a full-scale military conflict that neither side believes they can win (other than in a Pyrrhic way).
You also can't win an AI race, unless you count "getting killed by the AI you built" as a win.
SUPPORTER: Maybe we shouldn't be sending them H200s then?
"And they’re losing the race, so their incentive to pause is stronger than ours."
So, we shouldn't pause then. If we continue, it looks like we'll win. Let's win and crush them! No more squiggly lines for you, Chinaman!
I think the idea is we pause, figure out how to make it safer for everyone, and *then* win. If we could actually enforce it it would be a good idea IMO. I do think China will basically always sneak behind our backs if they're sufficiently motivated to, though, so my support of the idea in reality is conditional on my confidence that we could actually catch them.
Certainly fewer than the vast number on the alternative path, no? I agree that the number should be zero, but this is a huge ongoing problem that doesn't have easy solutions.
Forget AI; it is currently impossible to make a bilateral agreement between China and the US on anything at all, be it LLMs or soybeans.
China had repeatedly demonstrated throughout its Communist history that it is very fond of making bilateral agreements, and even more fond of breaking them. In fact, I can't name a single agreement, be it on Fentanyl or greenhouse emissions or anything else, that they hadn't broken or at least skirted.
Meanwhile, the US political system had disintegrated to the point where formerly staunch US allies are scrambling to find other trading and political partners; ironically, some of them are turning to China. They know that China can't be trusted, but at least when they backstab you they'll do so in a predictable way.
So, you're essentially looking to form a bilateral agreement between a madman and a compulsive liar. I don't think anything short of an omnipotent AGI machine-god would be able to enforce such a treaty.
They did lower emissions and stop shipping precursor chemicals, so the supply has been somewhat impacted, but it's hard to completely control industrial chemicals.
Mostly agreed.
>So, you're essentially looking to form a bilateral agreement between a madman and a compulsive liar.
but, but, but - I thought they were _both_ compulsive liars!
It's an interesting philosophical question: does a madman who says untrue things count as a "liar" ?
Good question! Many Thanks!
Ah, but you see, Roberts, the narrator is unreliable...
Hmm... Many Thanks. Though, to say that any given politician is a liar is such a banal claim that it doesn't require much evidence...
Now I'm wondering if you took my reply more seriously than I intended...
(Whoever's telling you this politician is a madman and that one is a compulsive liar might themselves be lying about it... I feel like there's a stage play in this somehow.)
Many Thanks! Honestly, at the level of heads-of-state, there is so much scripting, and so much playing to both the national audience and to negotiating partners that inferring _anything_ is exceedingly uncertain. ( And, for Xi, add the ambiguities of translation. )
For Trump, all I can say with certainty ( barring deepfakes! ) is that I've watched him make inflation claims ( that the Biden inflation was our worst ever ) that I know to be false, and where I expect him to know the truth ( since he, like I, lived through 1980, and I expect the 'double-digit inflation' of the year to have been memorable for him too... ), so 'liar' seems likely, at least on this.
Re claims that either is a madman:
That's _really_ hard to prove. One would have to know
- what their real goals were
- what information they had when they chose various actions
- that their choice was (flagrantly?) irrational, given their real goals and available information
- (and probably that they did this repeatedly, rather than an isolated error)
I don't expect to ever have all that information about a president (particularly if they are being managed by their staff) in hand even if they were really and truly stark staring bonkers. :-(
Well, to be serious for a moment: your observation about Trump indicates he likely lied about one thing, whereas "liar" bears the stronger connotation that the person in question lies about a great many things. This is based on a common heuristic: if a guy sings one song well, he's a singer; jaywalks once, he's a jaywalker; fucks one goat; etc.
In practice, over the years, I've found that this leads to an inaccurate model. (Including past examples where I was similarly confusing myself by asserting someone was a liar, or perhaps trying to farm consensus by bullying someone on the internet.) This even holds for someone like Trump, for whom the inflation claim was but one of multiple claims I recall being false(!). A better model is to take note of *when* someone lies and when they don't. There's often a pattern, and the more claims someone makes, the easier it is to spot such patterns and even test them. In Trump's case, I notice people say he lies in ways that aggrandize his personal brand, and there seems to be something to this, and it even predicts times where he _won't_ lie, because the truth builds his brand.
This is why I tend to regard anyone trying to tell me that so-and-so is a lying, cheating, jaywalking, singing madman is probably an "unreliable narrator". With the possible exception of the "-man" part...
>In fact, I can't name a single agreement, be it on Fentanyl or greenhouse emissions or anything else, that they hadn't broken or at least skirted.
Well, they did skirt the Montreal protocol:
https://www.bbc.com/news/science-environment-48353341
However, international pressure after they were found out was enough for them to crack down and fulfill their obligations:
https://insideclimatenews.org/news/10022021/climate-super-pollutant-cfc-11-china-factories/
The Montreal protocol in general, from what I've heard, is one of the more successful international agreements even despite it banning a useful and cheap chemical, an ideal candidate to defect from for simple profit.
One key takeaway is that mutual, reliable inspection is necessary for such agreements to function at all; not sufficient on its own, but it makes it possible.
The Montreal treaty is the most successful such agreement of all time, and for the first 10 years, it was very common in Europe to cheat it with CFCs smuggled from Russia. It took 30 years to figure out the part about China cheating it too.
Does any proponent of a pause believe that we have 10 or 30 years to get it right?
See https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233890949
It resonates based on some recent conversations I’ve had. Is this non-debating debating approach something more aligned with the present administration or do conservatives do the same in the other direction? Some liberal positions have weak arguments, very weak, and make scientific claims that are false, but at least they try to have an argument.
the "Opponent" straw man is entirely reasonable, of because, because any pause agreement with China will necessarily be unilateral. because china will of course absolutely ignore any agreement like they have ignored every bilateral agreement over much less momentous issues.
If you hold that position, then by definition the position is not a strawman.
No, that's not how opponent was represented.
Opponent is presented as skipping the step of pointing out a bilateral agreement is impossible, in fact not even credited with thinking it, because they're such a big dumb dumb, rather than thinking it's so clear that it's usually not worth wasting time talking about it. Supporter even admits its a problem and doesn't address it at all.
If we can shift the planet's orbit, we'll solve global warming!
That won't work, we need something else.
But it will work, we'll all just jump at the same time! You're not even addressing my plan to shift it by having everyone jump at the same time. You're ignoring obvious good solutions! Stupid opponent! Why are you so illogical?
Yes, opponent should spend more time making clear to supporter that in fact, if impossible thing were possible, they might support it. Many in the comments are pointing out why they consider it impossible, which is very different than the "la la la I can't hear you" version claimed as omnipresent by the article.
I have a different reason why AI safety advocates should reject a pause. The short term (5-10 years), is the only time that AI progress can be monitored and carefully regulated.
If you do a ten year pause, then semiconductor manufacturing will advance and algorithmic progress with better efficiency will happen regardless, even if the frontier is paused. And at that point, advanced AI research can then be conducted in very small scale operations rather than large billion dollar clusters that are currently required.
And in practical terms, you'll have a much easier time advocating for more research, than trying to ban billions of dollars of commerce.
If AI is possible, it is inevitable. A pause just makes it even more possible tomorrow.
Since it is inevitable, the only real question is what we should do about it, if anything.
I suggest following Geoffrey Hinton's advice:
Try to engineer the AIs to view humans favorably. He speaks of trying to give them 'maternal' traits. Essentially trying to tweak their utility functions to value humans. I think 'pause' is a close-to-doomed battle, particularly with the USA/PRC competition. But nothing precludes paying people to try to engineer the AIs' utility functions in a survivable way, as a concurrent effort, along with the labs' capability enhancement work.
> It is NOT HARD to get AIs to value humans. AIs bore very very easily -- humans provide interesting stimulation.
Even if it were true that AIs bore very easily, and instrumental convergence weren't an issue in its own right, humans are not a maximally interesting use of their atoms, and thus would be replaced.
>humans are not a maximally interesting use of their atoms, and thus would be replaced.
I think this prediction is overconfident. Yudkowsky and Soares make essentially the same argument in IABIED, and the argument would also predict that humans wouldn't keep un-selectively-bred cats as pets, but this conclusion is false.
I haven't read the book, so I don't know if it makes this point, but it occurs to me that humans might keep cats because humans are not ruthless atom-optimizers. AIs, OTOH, might be.
Or so the story goes.
It also assumes that AIs would have perfect atom-optimization rules, and I don't see how that could be the case either.
Many Thanks!
>What is HARD is preventing AIs from stupid mistakes
There are techniques that have proved quite helpful. Scaffolds that repeatedly test an AI's output and send the answer back for further correction have proved very effective in coding, for instance.
Many Thanks! My impression is that scaffolds have proved very useful in many AI applications. The testing doesn't have to be software unit tests. IIRC, one approach is to set up another LLM to examine a first LLM's output for problems (such as the examples you gave), and send the output, together with the criticism, back to the first LLM for revision.
From PauseAI's website:
> There will come a point where potentially superintelligent AI models can be trained for a few thousand dollars or less, perhaps even on consumer hardware. We need to be prepared for this. We should consider the following policies:
> * Limit publication of training algorithms / runtime improvements. Sometimes a new algorithm is published that makes training much more efficient. The Transformer architecture, for example, enabled virtually all recent progress in AI. These types of capability jumps can happen at any time, and we should consider limiting the publication of such algorithms to minimize the risk of a sudden capability jump. There are also innovations that enable decentralized training runs . Similarly, some runtime innovations could drastically change what can be done with existing models. Banning the publication of such algorithms can be implemented using similar means as how we ban other forms of information, such as illegal pornographic media.
> * Limit capability advancements of computational resources. If training a superintelligence becomes possible on consumer hardware, we are in trouble. We should consider limiting capability advances of hardware (e.g. through limitations on lithography, chip design, and novel computing paradigms such as photonic chips and quantum computing).
Eww. I find that sufficiently repellent that I'd rather take my chances with an ASI with _zero_ attempts to ameliorate the risks. Eww!
"Eww"? Really?
Many Thanks! Yes, really
>through limitations on lithography, chip design, and novel computing paradigms such as photonic chips and quantum computing
Remember that e.g. Yudkowsky has been calling for drastic limitations due to potential threats from AI for decades. I expect that there will eventually be a threat - but, thus far, he _has_ been wrong, so, if we had stomped on the brakes in the way PauseAI is advocating at the first warning, we would have lost at least a decade's and maybe more progress in hardware - and for nothing. We would also have lost AlphaFold, which shows promise of substantially aiding medicine - and, again, for nothing.
Eww!
I agree that a "pause" would be of the form "get everything monitored and carefully regulated, then restart at fixed speed" rather than literally having no AI.
Meanwhile, from my perspective, the debate goes something like this:
Supporter: We need to immediately pause AI research, and form a bilateral agreement with China to do the same, or else humanity is doomed.
Bugmaster: Doomed by what, LLMs ? I agree that they can be pretty destructive -- just look at what happened to novel-writing or programming -- but they're useful too...
Supporter: In just a few years, LLMs will lead to functionally omnipotent AGI that will rise up and destroy humanity !
Bugmaster: Lead to it how ?
Supporter: Inevitably, that's how ! It's all in my book, titled "LLMs inevitably lead to AGI, stop asking how".
Bugmaster: If that's true, why not also ban research on, I don't know, solar and wind power ? LLMs need power to run. Or maybe you want to ban metallurgy, data centers are made out of metals.
Supporter: Oh my god, you're right ! We must...
Bugmaster: I was just kidding, you know.
Supporter: ...back to the caves ! Only in the embrace of a new Dark Age are we safe !
Bugmaster: Yeah, ok, I'm just gonna walk over there now.
Supporter: Burn your iPhone !
Yes, the above is a huge strawman of a caricature -- but then, so is this post...
I am a pretty smart person. I think if I had a hundred million copies of myself, unable to directly coordinate but sharing a common set of values, I'd be able to pull off pretty much any goal I wanted. Sounds suspiciously like "functionally omnipotent". What's the difference between that and an LLM as smart as I am? (Always, people in this debate answer with differences between that and current LLMs. No. Current LLMs are dumb. Stop it.)
People think that in a few years, there may be an LLM as smart as an average person. This would be "AGI". At that point I'm not super worried - average people, when numerous, are good at shifting public opinion, but not much else. I'm not thrilled about having the Overton window dictated by AI, but it's not the end of the world.
At some point in the future, there may be an LLM smarter than the smartest existing person. This would be ASI. Before that is reached, you would have an LLM about as smart as me, and at that point I would consider things dangerous.
(That being said, I don't think this piece really has any point to existing.)
> I am a pretty smart person. I think if I had a hundred million copies of myself ... What's the difference between that and an LLM as smart as I am?
Yes, that's the problem with all of these AI debates: they usually *start* at the point where you've got millions of copies of human-level minds running in parallel while also probably controlling an army of robots/nanomachines/etc. At that point, the debate boils down to arguing, "If we assume that the AI is nearly omnipotent right at the start, how can you claim it won't become fully omnipotent ?" You're right, I can't -- but why should I assume that ? It's an interesting topic to speculate about to be sure, but:
> Always, people in this debate answer with differences between that and current LLMs. No. Current LLMs are dumb. Stop it.
Current LLMs aren't "smart" or "dumb", they're basically buggy text-manipulation tools. This is not due to some quirk in implementation; it's due to their nature as LLMs. We could (and arguably should) make them at least a little less buggy, but there's no direct path from LLMs to AGI. Especially given the fact that we humans aren't even AGI !
> At some point in the future, there may be an LLM smarter than the smartest existing person.
That depends on what you mean by "smart". A spreadsheet can add up hundreds of thousands of numbers in milliseconds. No human, not even a savant, could do that. Does this mean that the spreadsheet is "smarter" than a human ? I would actually say the answer is "yes", but no one is arguing for pausing the use and development of spreadsheets. So I think you need to develop some criteria that are a lot more specific than just saying "smart".
I don't know about "fear", but I agree that an AI that can "grow and learn" in a non-trivial way would indeed be a major breakthrough.
I absolutely agree with that, except I think it likely that future LLMs (3 years? 20? who knows) will have a good object model. It's really not that hard. They're clearly part of the way there, but physical interactions are underrepresented in their training data, so they have an inherent disadvantage. This is what I mean when I say they are dumb now. Specifically, I think modern LLMs do have an object model but it's laughably bad, rather than somehow lacking that magical soul they need to truly understand in-world interactions.
They totally do have an object model, but one that sucks badly. They might not have in the GPT 2 days. Back then you could ask them to compare the sizes of some common household objects and they'd have no idea. Now they consistently get that question right. You could say "sure, but they've just memorized some concepts associated with each other and spat them out". Frankly, IMO, if it works, it's an object model. But I think they're doing it differently than that. I think at this point they do have some size value internally associated with different words.
As for hallucinations, they're not a dealbreaker. We haven't got humans that don't hallucinate, either, yet humans are still plenty capable of doing some dangerous stuff. (I hallucinate vividly every night, then experience amnesia about the whole thing.) If somehow LLMs become smarter than humans on every axis except for an ability to truly appreciate music, this will not stop them from being dangerous. Worry about them at their best, not at their worst.
I am also a programmer, and significantly better at it than most programmers I meet, though I have not studied ML specifically.
My biggest concerns are
(1) I think it's very likely that AIs will continue to get smarter without a clear limit. There's no reason to believe that a brain that evolved to survive in the savanah is the best possible arrangement of atoms to convert energy into scientific research, computer programs, and scam emails.
(There's also a non-zero chance that the pace of development will increase as AI improvements compound the rate of future improvements, but while that decreases the chance we might muddle through on alignment, it's not necessary for the doom argument.)
(2) I think it's going to be very hard to maintain controls. As AI gets smarter, the corporation or nation that puts AI in greater controls of its factories, educational curricula, and weapons platforms is the winner right up to the point of disaster.
(3) I don't think we have a reliable way to align AI priorities, and even if we did and God-Emperor Sam Altman or the CCP is in charge of the world's AIs, I don't even think *their* priorities would remain aligned to human interest over time.
> I think it's very likely that AIs will continue to get smarter without a clear limit.
I think this is both true and false, weirdly enough. True, in the sense that hundreds and perhaps even thousands of years from now, we can expect our technology to evolve in unpredictable ways (assuming we survive that long as a civilization). This includes AI, and it is reasonable to predict that it would become dramatically smarter than any modern algorithms.
But the statement also contains some falsehoods. Firstly, unlimited growth of anything, be it AIs or soybeans, appears to be physically impossible; arguably even entropy itself has an upper limit to its growth. On the less cosmic scale, every technological development thus far had followed something like an S-curve instead of an unlimited exponential, and I don't see why AI would be different. Secondly, the term "smarter" is very poorly defined. I personally can't even fully tell what it means when applied to humans, nor how to quantify it sufficiently to make empirical predictions. And, of course, present-day LLMs are likely not a direct path to AGI, insofar as that term has any meaning.
> As AI gets smarter, the corporation or nation that puts AI in greater controls of its factories, educational curricula, and weapons platforms is the winner right up to the point of disaster.
Sadly, I think that at present the disasters are more likely to come due to people treating LLMs as oracular AIs, placing them in charge of all these things, and failing to handle the subsequent collapse of factories, education, and weapons platforms. Especially the latter, since they tend to fail in rather more spectacular fashion than classrooms.
> I don't think we have a reliable way to align AI priorities
This is a bigger problem than you're making it sound, because we don't have a reliable way to align the priorities of any machine. For example, some years ago I suffered a car accident due to the misaligned priorities of my Toyota Yaris: I wanted it to go straight, but its steering column had seized up so it decided to keep turning right. And just a few days ago the clustering algorithm I was working on went on a rampage and consumed all available RAM and CPU in an infinite loop, instead of doing what I wanted it to do. The dangers of misaligned technology are all around us !
Prediction ability is definitely a component of being "smarter", but I don't think that's all that being "smarter" entails; at least, not when applied to AI (or humans). After all, the minimax algorithm (short enough to fit on one page) can predict chess moves far in advance of any human (to the point where human chess masters cannot win against it), yet I do not think many people would agree that it is smarter than a human... unless perhaps you would ?
> Yes, that's the problem with all of these AI debates: they usually *start* at the point where you've got millions of copies of human-level minds running in parallel while also probably controlling an army of robots/nanomachines/etc.
Which is why I explicitly provided you a middle point that is not that. If what I suggested is too far along, break it into three steps: 1. Get as smart as the average human (let's say, in terms of planning and executing changes to the physical world), 2. Get as smart as an unusually smart human, 3. Run millions of copies.
3 is the easiest step. Over a hundred million Americans have used chat GPT so far. Some number of people run openclaw, and if it's 1% of total users that's a million people in America alone.
1 is the step the AI 2027 people think will happen in 2030. I didn't believe them about 2027, but this time everybody else is in much closer agreement, so I could see this happening at any point between 2030 and 2040.
2 is the hardest, because AI is trained on the output of the masses, and to get it to imitate smart humans it needs training data from smart humans. Probably there will be some algorithmic advancement that allows less training data to be used. I am guessing this would happen at some point between 2035 and 2060.
> we humans aren't even AGI
I think you are mixing up AGI and ASI again, as I tried to gently correct in my previous comment. AGI is as smart as an average human, while ASI is smarter than any human. Tautologically, a human of 100+ IQ is an AGI. Alternate definitions simply require AGI to be more flexible than ANI, in which case LLMs already meet that standard. This all doesn't really matter, though. If you define AGI to be some lofty thing that humans don't even reach, then since humans are already capable of doing damage, that leaves the three letters "AGI" as a pointless and irrelevant distraction.
> 3 is the easiest step. Over a hundred million Americans have used chat GPT so far.
I think we would both agree that ChatGPT is not as smart as a human; perhaps you could even agree with me that it is about as smart as a box of hammers, or perhaps your estimate would be a but higher. Yet the training and operation of LLMs is presently straining many aspects of our infrastructure, from power generation to chip manufacturing to water cooling. We are at the point where scaling up LLMs is becoming prohibitively expensive, so no, I don't agree that (3) is easy (though of course it could still be the easiest, relatively speaking).
> 1 is the step the AI 2027 people think will happen in 2030.
I would love to bet some money on this not happening by 2030 and retiring a wealthy man, but the devil is of course in the details. By some metrics, calculators are smarter than humans; chess engines certainly are better at chess than humans; and don't even get me started on Excel, which can do things I never could. In any case, I see no possible way an LLM would even be able to drive a car by 2030 (absent other machine learning systems), let alone compete with humans for world domination.
> 2 is the hardest, because AI is trained on the output of the masses, and to get it to imitate smart humans it needs training data from smart humans.
I do agree with you that there's not a lot of such data floating around. Still, I'm curious: does this mean that you agree with me that speeding up a dumb brain 1000x (or however fast) would not make it smart ?
> AGI is as smart as an average human
AGI stands for "Artificial General Intelligence", an entity that can potentially solve any problem given enough time (assuming a solution exists). Humans are not that. There are many problems that any specific human could *never* solve, no matter how many copies of him you spawn.
Yes, I agree that running a dumb brain faster does not necessarily make it smart - depending on the details. If the task is to avoid mistakes, running faster won't help. If the task is to accomplish something within a time limit, and the slow version can accomplish it but not within the time limit, then of course it will. In terms of practical effects, running faster will not help with developing a workable, low-risk plan, but would help with reacting quickly to unexpected problems or changes of circumstance.
I solidly agree that ChatGPT is nowhere near as smart as a human, but would rate it significantly higher than a box of hammers. Boxes of hammers act in very predictable ways, the majority of which are sufficiently described by newtonian physics. Humans and computers both act with much more complexity in their responses to various things, while of course still following the laws of physics. Most computer programs are still easily predictable, because software engineers have made great pains to write them that way, however, neural nets act in ways we don't fully understand because they are too complex. So on the scale of intelligence, I guess I'd say: a box of hammers < Firefox < a bacterium < GPT2 < early ChatGPT < a butterfly < modern LLMs < a mouse < a human. The mouse here is winning largely on 3D physics, sensory processing, and motor control, while behind on vocabulary. The butterfly is behind modern LLMs on overall behavioral complexity and especially responses to novel situations, while still ahead on motor skills.
Thanks for explaining what you mean by AGI. You might be interested in Gödel's incompleteness theorem. If I understand it correctly, it implies that for every problem-solving entity, there exists a problem that that particular entity cannot solve, although it is solvable in general. The actual theorem talks about systems of axioms and algorithms that produce theorems, but I think any physically-existing, finite, problem-solving entity could be imitated by that. If so, then it is logically impossible for this particular meaning of AGI to physically exist.
> So on the scale of intelligence, I guess I'd say: a box of hammers < Firefox < a bacterium < GPT2 < early ChatGPT < a butterfly < modern LLMs < a mouse < a human
I agree with your ranking, though I suppose I could quibble on where the bacterium should fit. But one thing the ranking obscures is the fundamental architectural differences between LLMs and mammals. It would be a lot more difficult (and I'd argue totally impossible) to upgrade an LLM to a mouse, than to upgrade the mouse to e.g. a dog (not to mention humans) -- though of course, the LLM could serve as a component in a larger system that later becomes as functional as a dog.
> You might be interested in Gödel's incompleteness theorem.
Meh, the Incompleteness Theorem is overrated. It deals with mathematical proofs, not practical applications. So yes, while for every system there exists a problem that cannot be mathematically proven to have a solution, this does not mean that the problem is utterly mysterious somehow.
Consider the Halting Problem, which is quite similar (and related). Turing proved that there exists no general algorithm -- in principle ! -- that we could use to determine whether any possible program would terminate. Does this mean that we can't ever tell whether any program at all would terminate ? Are we just programming in the dark ? No, because a). in the vast majority of practical cases it's trivially easy to prove whether a program would terminate, and b). if we expect the program to run for 30 minutes and it runs for 30 hours, that's a really good clue that maybe we should check for infinite loops, regardless of whether these loops are truly mathematically infinite or not. Maybe they would in fact terminate in 30 more years, but no one cares, it's debugging time.
I feel like updating this list for whatever reason:
I rate Claude Fable 5 as about equivalent to one of those drug-sniffing dogs that can detect meth at impressively tiny amounts. I wouldn't trust it with anything complex, like washing my dishes.
1) I doubt 1% of users have run openclaw. 1‰ perhaps?
2) Definition of AGI varies between people, yours is the one I agree with but it's not a universally agreed term.
> I'd be able to pull off pretty much any goal I wanted
I wonder if that's actually true. Like obviously the world is going to change when the cost of many types of cognitive labor is driven to .1% of it's current value, but in terms of manifesting power in the real world.. how would you actually do it?
Right now you could use them to shape public opinion through online interaction but what happens when the social fabric adjusts to that new reality? Online opinions will become literally meaningless noise (arguably nearly true already). And in general look at something like Brook's Law, or even just Ahmdahl's law.
I used to think it was obvious that software intelligence would inevitably recursively self-improve but I'm questioning that more. Seems like we could get a bunch of agents which are at current levels or maybe a bit better, find we can't push them much further, and then what?
> I wonder if that's actually true.
I think this is untrue of most people, as humans are not AGI. For example, If you copied my mind a million times and asked the copies to develop a solution to the Riemann Hypothesis, they'd fail a million times. If you accelerated them to be a million times faster, they'd just fail a million times faster. Meanwhile, the world's leading mathematician could conceivably solve the problem one day, but could be unable to compose a beautiful symphony or write the next Great American Novel -- not to mention climb Mount Everest on his own two feet.
I don't know of any famous symphonies composed by equally famous mathematicians, but to be fair, I don't know much about music in general. Still, I doubt even Tom Lehrer could complete any conceivable task given enough time.
>What's the difference between that and an LLM as smart as I am?
You're not trapped inside of a computer than can be easily switched off. You're not dependent on a constant supply of easily-denied electrical power. Your only means of interacting with the world isn't an easily-cut data cable. You have instincts for aggression and self-preservation which have been honed by millions of years of biological evolution. Your cognitive functions aren't completely transparent to and manipulable by anyone with a computer. You can also operate robustly and autonomously in the physical world.
What, exactly is the concrete danger you see from AI?
Bioterrorism, nukes, widespread malware, autonomous drones, probably not grey goo quite yet, atmospheric terraforming, this really doesn't take much creativity. We don't have widespread problems with those with humans, because humans have basic instincts of self-preservation. Modern AIs are smart enough to wipe a hard drive and sometimes randomly decide to do so. I don't see why an AI with the capability to do a lot more damage than that wouldn't randomly decide to use it someday.
And how, exactly, is AI going to get a nuke?
That one is more dependent on human cooperation than the others. For example, it could stoke fears in a way that restarts the cold war, it could get in good with Trump and then ask him in a way he likes, or try that for other nuclear powers. It only takes once, so this is worth worrying about even at low probability.
Ok and what makes you think that AI poses any special threat there? Anyone can manipulate a politician.
AI doomerism is just modern chicken littleism. It's nothing but midwit anxiety finding a new outlet.
* Bioterrorism: this is absolutely a concern, but sadly it's a danger that humans could perpetrate even without the aid of AI.
* Nukes: same, except that we've got them already, more's the pity.
* Widespread malware: what, more widespread than it already is ?
* Autonomous drones: yes, humans could really shoot themselves in all kinds of body parts with those. To some extent they already exist, and are deployed in Ukraine. I'd say that they follow the same threat profile as nukes, though of course orders of magnitude less dire.
* Grey goo: likely physically impossible.
* Atmospheric terraforming: again, more so than what we've been doing all on our own ?
> We don't have widespread problems with those with humans...
Oh how I wish this were true :-(
Er, yes, I guess I meant more widespread than we have now. Which is tautological.
My knee-jerk reaction is to say "I don't know if it's even possible to have more widespread malware than we have now", but of course it's possible. Oh how very possible, and also apparently inevitable, because people keep vibe-coding their critical infrastructure without paying any attention to even the most rudimentary security precautions. So yes, I do believe that the spread of LLMs will lead to more widespread malware... just in the opposite way of the way people often envision it.
I think we are at the point where we need to step away from thinking of AIs as "smarter than" or "dumber than" humans.
LLMs are already far smarter than humans in some ways; or rather they are much better at certain tasks which we have previously considered as intelligent. They can write a reasonably-good essay on just about any subject within seconds.
But humans and LLMs are two very different types of things and we can't really compare them. Nor should we assume that LLMs will get so smart at some things that they'll automatically be good at the other things, any more than the ability of a 747 to fly far higher and faster than a magpie means that it's also able to land in my back yard and eat worms.
That's what I thought back in the early days, before it learned how to play chess without being specifically taught.
Chess is a game of "generate the optimal next token given all these other tokens", it's well within an LLM's core competency.
Slightly facetious question but why can't countries do that? China has 1.4 billion people, presumably you could get 100 million of them who are smart and share values (e.g. they all achieved level 14 diamond-plus devotion level in the CCP or whatever). Yet for all of that they get...like one percentage point higher GDP growth rate than a typical country?
> Slightly facetious question but why can't countries do that? China has 1.4 billion people, presumably you could get 100 million of them who are smart and share values (e.g. they all achieved level 14 diamond-plus devotion level in the CCP or whatever).
Those 100M people may all "share values" but imagine the returns to communicating and coordinating with another mind that is literally the same as your own.
Communication always has an inferential distance, much left unsaid or contextually contingent and implied, and limited bandwidth. Even sharing values perfectly leaves one subject to these gaps, and of course, no two minds perfectly share values, nor capabilities.
Cloning the same mind 100M times, and having it live on an electronic substrate, eliminiates or greatly reduces all of those issues. Values truly ARE shared perfectly, as are capabilities. Inferential distance is at a minimum, context is 100% shared, and the ability to communicate and coordinate is accordingly increased by orders of magnitude.
And this doesn't mention the incentives - in a real network of millions of people, or even hundreds, everyone's incentives are from slightly to wildly divergent. This leads to all kinds of failure modes, from actual goals and outcomes that are significantly different than stated ones, people actively working against each other, and much more.
In the "data center full of geniuses" the incentives are also 100% aligned.
So not only do we have hundreds of times better communication, coordination, and capabilities, everyone is also ALL pulling the same direction and pursuing the same goal, for real.
I think there are probably even more advantages than this, but these are obvious enough and large enough in effect size, I think it at least sketches that overall picture.
Yesterday I had one of those surreal conversations with my boss/friend where we were almost agreeing from the start, yet somehow still managed to frustrate each other more and more with every round.
I was saying: “Yes, I agree with what you want — I just need a bit more time to stabilize the current state first.”
He kept responding with some variation of: “Yes, I understand, and I agree that we should first do what I want.”
So the actual gap between us was tiny, but somehow we kept missing each other’s point by a hair — five times in a row — until we were both annoyed.
The older I get, the more I get the impression that conversation is often not really about taking external information in and updating an internal model. More often, it feels like people are just broadcasting the current output of their inner model at each other.
Strangely enough, LLMs with all their weird quirks are starting to feel like an increasingly accurate metaphor for human interaction.
"I just need you to agree with me."
"Yes, sure, okay, but I just need *you* to agree with me!"
Probably similar to exchanges I've witnessed. Both sides agree on claim P, for example, but the disagreement is really about whether the one side convinced the other of P, or the reverse, or even whether both sides believed P to begin with and were really trying to discover whether the other side did as well, and the other side kept misunderstanding what they were trying to do.
All with a helping of each side possibly altering its goals as the exchange progressed.
There is never going to be any real understanding between peoples. There doesn't need to be. People die, empires fall, and the systems capable of survival will continue to do so, even if its constituent members have no understanding of what they are doing. This is how it's always been.
ACX is of greatest interest to me when it tries to steelman (both sides). This didn't. Strawmanning is cheap and available anywhere.
I also did not enjoy this post at all. It belongs on Twitter.
But it isn't a strawman, as several people have appeared in the comments to say they endorse the opponent's position!
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233119095
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233124939
If anyone anywhere holds a view, it's not a strawman to claim that view is "every debate"? That's your position Mr Josh Bear? Give me a handful of your safe-to-share identities. I look forward to ascribing to you the most extreme view I can find of anyone that shares that identity. Of course you couldn't object and claim it's a strawman... I found a single example!
"Every debate" is hyperbole and I neither endorse nor care for it.
But Opponent is not a strawman, because people have come along to endorse it. And if you go through my history on Substack you will find any number of rare and eccentric positions that I do endorse and are therefore not strawman positions to argue against, because you found at least one example (and we're up to at least three here on this post alone for Opponent, this having come along: https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233156249 )
Hyperbole... is that another word for exaggeration? Is an exaggeration of someone's argument... a strawman argument? Gee whiz!
If his article was "a view more than zero people hold" then it wouldn't be a strawman and you'd be right that a couple of people tepidly endorsing it is maybe proof it's not a strawman.
His entire article is premised on "every debate". If you disagree with the "every debate" part, then that's the entire article! This wouldn't be controversial if it was "an example of the absolute worst version I see of OPPONENTS". Those are two different articles.
If I write an article "every Democrat is an anti-Semite", do you know how insane I'd sound if someone said "that article isn't a strawman, look, there are some Democrats who are anti-Semites. Sure 'every democrat' is hyperbole and I neither endorse it nor care for it, but don't you claim this article is a strawman, clearly its not".
This is very basic logic Josh.
An exaggeration is only a strawman if attributed; since Scott didn't attribute the "every debate" position to anyone but himself, it isn't a strawman, nor a even a weakman. To call it such would be a kind of category error.
It was attributed... to "every debate". That's what he titled his post. That this fact leads to the post being useless, hysterical, embarrassing, etc. doesn't mean the fact isn't true. It means the post was in fact useless, hysterical embarrassing, etc. 2026 Scott is a much diminished version of the man who made himself famous.
When someone holds a position, it is by definition not a strawman; at worst, it is a "weakman" position.
Two replies to the same comment... brilliant
I apologize; I realized I had not directly answered the question, and a second comment seemed less confusing or potentially manipulative than editing the first.
If it's a weakman, it's slightly better than a strawman. But it's still not okay.
Bad arguments and bad arguers can be ethically argued against; the question to me is whether one does so honestly.
The people pointing out the straw/weak man agree with your claim about the question. And they're saying "one did not in fact do so in this case".
That doesn't prove it's not a strawman, it just proves that some people are capable of seeing through the strawman presentation and agreeing with the actual point.
It would be very easy for me to write a debate between two characters and depict the one you disagree with as sensible and the one you agree with as an idiot who keeps saying "hurr durr" and "yeehaw" while making bad arguments for your position. But I assume you'd still be smart enough to agree with the "hurr durr" guy.
Well, I might both agree and disagree with the "hurr durr" guy, in that we agree as to the conclusion, but simultaneously disagree with all of their arguments. But I wouldn't say that HurrDurr is "right".
Well the steelman part is taken care of by the pro-pause character, they would actually be an outstandingly moral debater
But I think I agree that the piece as a whole is still not very constructive, it's more of an expression of frustration
..."it's more of an expression of frustration." Agreed. And expressing frustration strikes me as a legitimate (though common) portion of discourse. But it doesn't advance issues as far as a lot of other engagements can. ACX is at its best when it distinguishes itself from the common.
I hear the opponent's voice as Vizzini's from The Princess Bride
I *hope* SUPPORTER is intended as a parody of this kind of naivety.
But I'm not at all sure...
There's quite some straw coming out when you portray the opponent as too dumb to comprehend the "mutual" in "mutual pause". While simultaneously taking for granted that it's truly possible to *trust* China (*and* for China to trust US) that no further research/training would be undertaken by them. OR that this can be enforced via some kind of omniscient "mutual monitoring".
It's one thing to be Obama, allow Iran 6000 centrifuges, and assert that you'll know for sure what each one of them is doing each day of each year.
It's a whole other thing to presume that you'll know where each of the *dozens of millions* of nVidia cards that were sold in China ended up being, and what they're used for.
There's probably a limited number of human being with the expertise required here. Surely it's easier to monitor them than to monitor hardware?
Anthropic complained about a massive "draining attack" performed by the Chinese to discern/replicate the inner workings of Claude.
There's some doubt as to whether they were Chinese hobbyists. But there's NO doubt as to whether the US would have, or be able to have, some "monitoring over individuals" who were involved.
Not a strawman! It's a position held by actual commenters on this post:
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233119095
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233124939
The 1st post is an unqualified endorsement of the opponent, yes. But the second one exceedingly clearly talks about the same thing -- that you do NOT posess a "monitoring omniscience" that would know about all that (still) happens in a China-sized country.
Don't need to know where they all are. Tracking most of them will at least make keeping a secret project harder.
It's enough to be able to track large concentrations of them, since LLM training can't be done in a distributed way yet.
>since LLM training can't be done in a distributed way yet.
Huh? Using multiple (many!) GPUs _is_ parallel processing. Yeah, you want memory close by too, but scattering racks around the landscape, with GPUs and local memory on each rack, only degrades the part of communications that goes between racks. I doubt that this delay is a deal-breaker.
Maybe, if you think so.
It's just my opinion. I could be wrong. Many Thanks!
And you would know that a given building in a 3.7 million square miles country, cut off from the internet and getting its training data directly from copied-over drives, is housing thousands of nVidias how exactly?
They produce lots of heat in a relatively identifiable manner. You use infrared cameras on satellites.
So put it underground?
Alternately, put it in a heat island, such as a city. Any nation aspiring to create an AGI will likely have plenty of them.
The debate breaks before substance.
A conditional coordination proposal gets repeatedly re-read as unilateral disarmament because the geopolitical schema is doing more work than the actual words.
Begone, LLM commenter!
I am so sick of the boing boings that go on in polarized pairs that I can’t feel even faintly amused by this. Does Rationalism have anything to say about the emotional, self-esteem-based aspects of arguments? These 2 dopes are so narcissistically invested in winning that it really doesn’t matter how good their data is and how well they have thought through the situation in question, because their opponent is not going to pay the slightest attention to the case for their point of view. If you want someone to listen to you have to let them state *their *case and pay attention to it, and show that you do by asking perceptive questions and acknowledging its strong points. And it really is better if you’re not just doing that for show, but are actually trying to open yourself to the possibility that they are partly or fully right, because they may be.
Both these people richly deserve to get turned into piles of paperclips. Small piles, because both have cases of TinyMind.
“The square of the length of the hypotenuse equals the sum of the squares of the lengths of the legs.”
“A triangle is not a square, you idiot.”
Even though I agree with SUPPORTER, this piece seems both unkind and, as far as I can tell, unnecessary. In particular, it would be more useful to *show* the details that are only being gestured at here, so that others could verify them. "We have some ideas for how we could have a light-touch approach to monitoring Chinese data centers. . .and actually the math mostly works out" -- what ideas, and what math? Some commenters seem to think this is implausible; what grounds do I have to argue, when the details of the proposal in question are so vague? "We’ve actually had some pretty successful low-level discussions with Chinese scientists" -- are any of these public? Can I see them? If the debate is really as one-sided as you make it out to be, it shouldn't be hard to find citations for your position, right?
Yeah, a few footnotes would have strengthened the case considerably.
I've heard people I trust like John Schilling say that the profile of a place making chips or computers or the datacenters that use them is at least as detectable as for building a nuclear bomb. (I'm paraphrasing here and apologize to John if I'm maligning him.) But I wish I understood how that could be. Surely you don't need absolutely state-of-the-art chips; if you've got chips that are half as fast (and them are some mighty old chips), surely you just need to have two or ten times as many in parallel, or run them twice as long. You don't want to do that if you're in a race, of course, but if the world is "paused", can these monitoring systems tell if you're doing the slow-but-steady with old tech? Maybe, but as you say, how would I know?
The thing about frontier LLMs is that training them is largely an exercise in brute force computation at staggering scales, and the scale gets quite a bit bigger with each major generation of frontier models. If you could make a bigger and better LLM with rented computing resources, the big AI companies would be doing that rather than building out new datacenters as quickly as they can buy the parts.
I wouldn't trust the Chinese to faithfully adhere to any pause agreement, I wouldn't trust our ability to find out if they were breaking the agreement, and I wouldn't trust future administrations to react fast enough if we did discover they were breaking the agreement. And I also frankly wouldn't trust our own companies to keep the agreement.
This is very different than nuclear arms treaties because enriched Uranium is easier to track than compute, and nuclear weapons testing is a lot harder to hide than hardware and software progress. AI development can be decentralized and anonymized, missile and warhead testing cannot.
I'm sorry that you've been subjected to too many morons in this debate space, and I understand that this post is a reaction to that barrage, but this post is still below your standards.
I think a pause of months is plausible. Maybe a year? Stuff takes time to organize. Even if you start setting up your secret AI lab immediately, there still has to be some time and effort and friction involved.
If you're talking in terms of "future administrations" you seem to be thinking much longer than that, though. And yes, a multi-year "pause" would be tough to pull off.
I agree we need an international framework, because we cannot and should not halt development.
First we must recon with what we have. AI is not something we can unplug any more than we can unplug the internet.
It is now in the commons. It can be regulated but without changing its terminal vector it will evade all attempts to regulate it.
Can someone who knows more about this topic than me please explain to me the enforcement mechanisms that would be able to give either party even 90% certainty that the other party is not continuing to develop AI in secret?
I am not claiming it cannot be done, I just don’t have any understanding of how it could be done.
(A disclaimer that I strongly dislike Pause AI (the entity, not the ideology), due to the way US executive director engages with the general public, especially her tweets on EAs.)
They go into some detail over here:
- https://pauseai.info/feasibility
- https://pauseai.info/building-the-pause-button
Short version is that growing in secret could be prevented through choke points like TSMC. Other secretive actions could be discouraged through any subset of these: https://pauseai.info/building-the-pause-button#verification-methods---preventing-large-training-runs
I'm not convinced these would be sufficient.
> Choke points like TSMC
Hasn't worked very well so far
https://www.reuters.com/world/us-charges-three-people-with-conspiring-divert-ai-tech-china-2026-03-19/
I don't think we've tried very hard. BIS's funding is a rounding error.
Many Thanks, particularly for the URL https://pauseai.info/building-the-pause-button#verification-methods---preventing-large-training-runs
Re:
>I'm not convinced these would be sufficient.
Obvious countermeasure at the end of this comment:
"Remote Sensing: Uses satellite and infrared imaging to detect data centers by visual and thermal signatures. Highly feasible but limited by camouflaging or underground facilities.
Whistleblowers: Relies on insiders reporting non-compliance, incentivized by legal and financial protections. Feasible but dependent on insider access and willingness to disclose.
Energy Monitoring: Tracks power usage to identify large AI operations, viable if patterns are distinct. Feasibility varies; data can be obscured by other high-energy activities.
Customs Data Analysis: Monitors import/export of AI hardware for anomalies. Feasible, especially for imports, though countries with domestic manufacturing may avoid detection.
Financial Intelligence: Observes large or unusual transactions related to AI hardware purchases. Feasible if financial privacy and banking laws allow, often best combined with other methods.
Data Center Inspections: Physical site inspections to verify compliance with hardware limits and security protocols. Effective if host country agrees to inspections; invasive and resource-intensive.
Semiconductor Manufacturing Facility Inspections: Verifies chip production compliance by inspecting facilities with relevant hardware. Feasible but requires significant resources and host country consent.
AI Developer Inspections: Reviews facilities for authorized code, safety protocols, and AI evaluation records. Effective but highly invasive, requiring specialized expertise and country cooperation.
Chip Location Tracking: Tracks AI chip movements to monitor their deployment. Feasible with international agreements, but bypassable by disabling tracking or spoofing location data.
Chip-Based Reporting: Embeds reporting mechanisms in chips to alert if used beyond authorized limits. Feasible but challenging, requiring international standards and hardware development; circumventable by modifying firmware."
Distributed data centers using non-bleeding-edge chips, funded from classified military budgets, with penalties for whistleblowing classified programs, defeats all of these.
This is not going to work.
Many Thanks!
>Oh, holy shit, we're actually listening to someone who doesn't understand why the Trump Administration attacked USAID?
Sorry, I'm not following you (is the someone the commenter, or the author of the URL I'm quoting from?), but anyway, I see any reaction to anything about USAID as orthogonal to whether the proposed measures to _verify_ any AI limitation treaty would work.
>You're absolutely right, this is not going to work.
Many Thanks!
>We aren't dealing with sharp knives at all, just some bludgeons who are scared of LLMs.
I think I see LLM-based systems as more capable than you see them, but that is a separate discussion, and, one way or the other, we will see what happens over the next few years. The metaculus median estimate ( https://www.metaculus.com/questions/5121/date-of-general-ai/ ) for AGI stands at August 2032 today.
Many Thanks!
>And the smart guys can bury data, heat, and finances like you wouldn't believe.
Yup! Verification in the presence of an intelligent adversary is _hard_ ( except in the nuclear weapons testing case, where raw physics is a direct help ).
...I mean, have you seen how much power training consumes and how much heat it produces?
I’m not sure it’s honest to claim that almost nobody wants a unilateral pause. I think everyone agrees that a multilateral pause is better than a unilateral pause, but I would be quite surprised if Eliezer thought that a unilateral pause would be worse than no pause at all.
Holly Elmore, the director of PauseAI, spends a lot of time on Twitter shaming engineers at American AI labs for not resigning. She is very clear that she thinks that AI labs are doing a very bad thing and that they have a moral obligation to stop doing that very bad thing. Do you think her position is that America as a country should not stop doing this very bad thing unless they get other countries to stop doing the very bad thing?
I’m quite sympathetic to the unilateral pause argument and think it is a mistake to disclaim it in the name of meeting financially-interested counterparties halfway.
Keep in mind, the United States started a war of aggression *last month*. When was the last time China did?
I'm not convinced that Holly Elmore applies her moral beliefs evenly. A look through her timeline reveals her primarily criticizing anthropic rather than any of the other labs.
She also was happy that Elon Musk (reminder: owner of xAI) agreed with her criticism of Amanda Askell: https://x.com/ilex_ulmus/status/2023305917727641906
She knows some of those people and used to strongly identify as an EA, so that's personal for her. It's also useful to attack the bad actor who has the best PR, to show that there are no good guys in the AI race. I can attest that she is extremely pissed at all of the labs, even if her personal attention often tends to be in one place over another. I'm sure she would also publicly agree with Dario Amodei If he said that Elon Musk is flying by the seat of his pants and has absolutely no idea what he is doing with safety.
In isolation, a unilateral pause is way safer than racing ahead. It's foolish to build a superintelligent adversary in your backyard.
Overall, the dynamics are a little complicated, but pushing for and increasing domestic regulation naturally comes along with and increases the chance of a global treaty. Some of this is groundwork to support a global treaty, some of it is giving public sentiment something concrete to latch onto before a global treaty is being negotiated, some of it is just throwing sand in the gears and doing what we can to slow down any element of the race so we have more time to get global cooperation.
At some point along the continuum of unilateral pause, you're just giving up all your leverage in the negotiations. That's not the worst outcome, but it's still bad. And of course, a unilateral pause is not enough on its own, and a global treaty is ultimately necessary.
So a unilateral pause is not the target, but it is a continuum that we will naturally travel part way through while on our way to a global AI treaty.
>Overall, the dynamics are a little complicated, but pushing for and increasing domestic regulation naturally comes along with and increases the chance of a global treaty.
I think that it (pause-like domestic law) is more likely to prompt Xi to view us as playable for fools, and to go for as overwhelming a win as he can. There is a real USA/PRC rivalry in play, and what you are suggesting is quite close to unilateral disarmament in the face of an adversary.
If it is possible to get a global treaty first before even building a framework for domestic regulation, then that's great, but that isn't really how things work in practice.
The CCP has already regulates AI more than even the EU does. I don't think we should be so sure that they intend to race ahead at any cost. They have concerned scientists just like we do, and theirs sign joint statements with ours.
Are you saying that you are in favor of pausing AI, and you want to make sure that we do a good job of it? Or are you arguing arguing against pausing AI? Because we always have the option of destroying the entire world just to spite our enemies, out of a belief that they would do the same, but I don't recommend it, and it wasn't true for the US and USSR during the Cold War. People mostly want to remain alive, and they just need it explained to them that creating superintelligent AI anywhere on Earth is a thing that makes remaining alive a lot harder.
Many Thanks!
>If it is possible to get a global treaty first before even building a framework for domestic regulation, then that's great, but that isn't really how things work in practice.
I'm not sure what you mean by "building a framework" here. I _think_ you are describing something other than stopping development here unconditionally - maybe creating a _conditional_ law that would stop development, conditional on a global treaty?
>They have concerned scientists just like we do, and theirs sign joint statements with ours.
Given the military applications, I expect their concerned scientists to be bypassed, similarly to ours.
Given the military competition, and the military applications, I expect any de jure pause to be cheated on by both militaries, even if in secret. If we were to _unconditionally_ pause here, I'd expect the PRC to try to catch up, pass us, and get as much military (and economic) advantage from an asymmetric pause as possible - presumably pushing AI as far as they could where they think they could control it (correctly or incorrectly).
And I think a theoretically symmetric pause will just push the development on both sides into secret military work.
Generally speaking, arms controls have been failures. The Novichok poisons were developed, chemical warfare ban notwithstanding. Even in the favorable nuclear case, Putin withdrew from the on-site inspections; the START limits expired; and North Korea has their nukes.
In summary, I think that the expected net effect from an attempted pause is negative, not positive, for both asymmetrical and symmetrical cases.
I think a better use of time, effort, and political capital would be following up Hinton's suggestion to try and bias AIs' utility functions in humanity's favor. It doesn't need to rise to the level of Asimov's three laws. Getting them to treat us as we treat pet cats is good enough.
I think it's completely plausible that our Chinese "rivals" are not very competent, this isn't a race at all, they are acting as fast-followers, are not good innovators in this field, and that if we paused they would stagnate. That could totally happen, in which case the unilateral pause strategy would succeed, for several years perhaps.
But it's not something you could *announce* as a policy, because the optics are absolutely terrible, and to whatever extent that might happen you can do strictly better by coupling that with a policy of severely restraining China's ability to acquire necessary resources. That way you are in fact damaging the opponent, so you get broader appeal.
You hurt yourself massively with the China apologia at the end there, that's about the absolute worst way to argue this. Since I want AI controlled (I don't want it to be built ever, but I'll settle for control at the moment) I have to be okay with actions exactly like we took against the Iranian mullahs, that's what you have to do to prevent rogue actors from breaking the stalemate. China has their own way of projecting power, if you think placing other countries into debt peonage is nicer than the US methods then soldier on, but it'll probably take all forms of power projections in combination to control AI proliferation.
This is probably the worst post I've ever read on either SSC or ACX. It is the very definition of strawmanning.
It is definitionally not a strawman when several people appear in the comments to endorse the position:
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233119095
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233124939
https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233156249
Do you think it's impossible to strawman an abortion supporter/opponent merely because there exist real abortion supporters/opponents?
Sure, it's possible! However, if you assert that a position is a strawman, then I could prove that false by identifying a person that holds that position, because for it to be a strawman means that no one holds the position.
Ah I see what you're getting at. I don't think you're using the same definition of strawman that I would understand.
I don't think it's necessary that a strawman be something that nobody actually believes, it just needs to be a weak presentation of that position.
Have another read through the essay, reading only the Opponent's lines. It won't take long, he basically just says the same thing over and over again while sounding increasingly like the "dey took ur jerbs" guy. The Supporter gets an opportunity to respond to the Opponent's best argument, and the Opponent just keeps repeating the same argument over and over again and never engages with the Supporter's rebuttal.
Now, I understand that Scott wrote this out of a frustration that he feels like this is the very real state of the debate, that the opponents really don't do enough to engage with the best arguments of the Supporters. I can understand how he might feel, but by committing it to a post in this form it just comes across as "I wrote a dialogue where my side is smart and your side is stupid, checkmate theists".
The problem with that approach - where a strawman is just a weak presentation - is that you're accusing people of arguing with a strawman when they actually argue with a person that has a weak argument for their position. That's why "weakman" and "tin man" were coined.
It just seems a distinction without a difference. If I make up a silly exaggerated position which no one believes, it's apparently a bona fide straw man. What happens if 5 years later, a single mentally deranged person believes it? Suddenly, and with time travel, we have to call it something else? Can we even be sure the person (who is drooling and seeing visions!) believes it, even if they say they do.
I think the issue is the quality of the argument. If it's too silly and exaggerated, it's effectively a strawman -- it's a not a reasonable representation of the other side, it's an intentionally bad and laughable one. It doesn't matter if a few people claim to believe it or not. And yes, this makes it somewhat fuzzy, rather than binary. Such are words.
I believe the common definition of a strawman is to misrepresent an opposing viewpoint by making it weaker than it was actually stated by the opponent, so it's easier for you to knock it down.
It is irrelevant to that definition whether or not someone other than the opponent actually holds that position. You are debating the opponent, not anyone else, so to show that it's not a strawman you would have to show that the opponent really holds the view.
Accordingly, if you don't name your opponent but ascribe the weaker view to literally everyone on the opposing side, as Scott did, then in order to prove that it's not a strawman you would have to prove that everyone on the opposing side holds that position.
So yes, this post is a strawman, and your handful of counterexamples do not disprove that because it's not everyone.
I'm fairly sure that among the 8 billion people on Earth, at least one pro-choice/pro-life person holds any outlandish opinion you care to name. Ascribing that outlandish opinion to ALL your opponents, as Scott does here, is a strawman.
But of course he doesn't; he's including many other opinions of opponents within the Supporter parts of the dialog while hyperbolically claiming that the Opponent position intrudes into every debate.
Every single person you link there gives more nuance than the strawman in the post, but even if we accept that. We should hold the writer of 'Weakmen are super-weapons ' to the standard of 'don't use straw/weak men'
If actual people agree with the position but argue it more coherently than OPPONENT, then I would still consider OPPONENT a strawman.
I think strawmanning would be saying that this is the only argument possible. I tried to write this in a way that highlighted that other arguments are possible, but that the existing pause AI debate has somehow gotten stuck on this topic where the other side accuses pausers of wanting a unilateral pause, and can't seem to progress out of it.
Almost every discussion I've seen about pausing AI included much more nuanced and defensible reasoning from the anti-pause side than what was reflected in your dialogue, which is why it's a strawman.
Even if it vaguely resembles real dialogues you've had, claiming that such a caricatured, over-the-top, exaggerated post is not a strawman threatens to redefine strawmanning out of existence. By that standard, basically no one could ever describe anything as strawmanning, even if it's extremely uncharitable (as this post certainly was).
The typical characterization I hear is "weakman". A weak man (I've seen the term with or without a space) is basically just a strawman that someone actually believes.
The upshot of a weak man fallacy is that it's slightly better than a strawman, since there does exist someone who believes it. The catch is that that's not the most compelling argument(s), which is/are still at large. So the weak man argument is at nearly the same risk as the strawman.
So your true objection here is that you've seen those stronger arguments, and somehow Scott has not seen them.
I think you're being too charitable to Scott here. The post is an extreme caricature. You would have to substantially reword most of it before it begins to faithfully resemble real debates about pausing AI. This is a textbook strawman, not merely a weakman.
Scott has said a couple of times in this thread that he's literally run into debates that go this way, though, so either Scott is lying, or Scott was debating with a troll or an AI without realizing it, or someone actually believes this. I wouldn't be surprised in the third case; we've all run into dum-dums on the internet.
What's weird to me is that I've also run into intelligent defenses of no-pause, like you have, and I don't know how Scott didn't see them.
I have seen people act like the opponent. There are even such people in this comment section.
US got lately a habit of killing whoever they are negotiating with at first opportunity (Iranian leadership and then Shia militia leadership in Iraq.) Why would anyone trust US in a negotiation with stakes this high?
If you think the US would start assassinating Chinese officials something is very wrong with your internal model
Why? The US has shown that it's capable of and willing to abduct/assassinate foreign heads of state, and that they grossly misunderstand the consequences of doing so.
I am not sure what is your assumption about my thinking. Do I think it is likely that US would openly assassinate a Chinese official next month? Of course not, we are not at that escalation stage. Would any US negotiation counterparts consider the risk that at a further escalation stage US would take them out physically mid-negotiation? They would be foolish not to. Does it have an impact on viability of reaching an agreement which would be useless without mechanisms for verification/enforcement at different escalation stages? I think it does.
Strawmen can be good if there's no pretense of them being serious arguments. That doesn't mean they can't serve as a tongue-in-cheek critique too.
Supporter should start thinking about how to actually check whether China is not secretly violating such a treaty.
There has been extensive research on this and it's very doable. Laypeople tend to assume it's impossible, but down at the gears level, we already know how this could work. MIRI has research on this ("An International Agreement to Prevent the Premature Creation of Artificial Superintelligence"). PauseAI collaborated on a report on this ("Building the Pause Button"). Other teams have put out their own papers.
It's a fairly complicated subject, so getting into it usually results in the other party coming up with something that is already addressed in one or all of these papers, and it becomes a whack-a-mole conversation where it's on me to remember all the technical details.
It is not the responsibility of Pause proponents to specify exactly how an international treaty would work. It is not our job to write the text of that document. Even if we did so, that document would not be the one that gets signed, because that's not how negotiations work. All you have to know is that we already know that it is doable in principle. It's hard, but not nearly as hard as most people imagine. So if it's a good idea in principle, it's good to implement in practice.
Every proposal I've seen, from MIRI or Pause, looks easily thwarted. Simply using not-quite-bleeding-edge chips (no embedded tracking hardware) in clandestine distributed data clusters immediately circumvents just about all proposed measures. And that is _before_ smart people who make their living doing countermeasures of this sort start looking at the problem. This isn't going to work. If the USA and PRC were allies, there would be a point to this, but they aren't, and that isn't going to change on the timescale of getting to AGI.
We are allies on wanting to exist. If you want to give up and wait to die, fine, but I will keep fighting for both of our sakes.
Many Thanks, though I really think you should switch focus from promoting a pause to promoting work on understanding and controlling how LLMs' utility functions are shaped.
The real winning argument is that the plug is actively being pulled. https://variety.com/2026/digital/news/openai-shutting-down-sora-video-disney-1236698277/
I'd actually be very curious on the inside story on this if you know more, what came first?
OpenAI shutting down an expensive-to-run video model that has rather limited productivity uses (especially compared to LLMs) is not really worth extrapolating from, imo.
This could interpreted as OpenAI being short of money. It could also be interpreted as OpenAI no longer being dependent on mass market revenue or reputation, if the main value of models right now is to sell to a few large corporations, or else to speed the RSI loop in a near term race to AGI.
I don't think this is mysterious. OpenAI is worried that Anthropic is beating them in the core enterprise market. They're wrapping up all of their fun side projects to concentrate all their resources on fighting Anthropic for core enterprise.
(Others have already said this but I'll try to add a little detail here. Also, I'm assuming AGI doom scenarios for the sake of this discussion even though I don't personally agree with them.)
The opponent is obviously correct here. China has a long history of ignoring "bilateral" agreements. And they have a structural advantage in that the US is quite bad at large secret projects. China is simply better at them just due to its social and political structure. The use of the word "bilateral" by the supporter is merely a rhetorical flourish. The moment the opponent concedes to discussing even the (im)possibility of enforcing a bilateral agreement, the discussion will immediately switch to "everyone agrees on a bilateral agreement!" with enforcement concerns relegated to a minor implementation detail (we have this incomprehensible bullshit thousand page proposal to absolutely guarantee compliance! move on already).
You say you'd never resort to underhanded tricks like this? Of course _you_ wouldn't. But this debate won't happen on your blog. It'll happen in Washington and in the pages of the NYT where _techniques_ like this are de rigueur (it's not a trick if everyone knows what's going on).
And you're too well known to argue with in good faith now. The only reason I can say this stuff is because I'm some irrelevant no-name commenter and nobody cares what I think.
What I suspect is really going on is that all the DC spending is burning a hole in people's pockets. But they can't stop due to competition/FOMO/optics/whatever. So, they need the pause to be coordinated. They couldn't care less if China pauses or not (even if Chinese models are better, regulation can deal with that problem easily enough). And the Rats are just the useful idiots.
This entire blog post is based on an utter lack of understanding of how politics happens in the real world. There's a reason even the people who like rationalists worry about their social acumen.
They're clearly not that good if you know about all of them.
Surely this is the sort of unilateral pause that nobody is advocating?
(I don't dispute that AOC and Sanders are nobodies)
https://www.theguardian.com/us-news/2026/mar/25/datacenters-bernie-sanders-aoc
Yeah, that's a weird proposal that I haven't seen enough analysis of yet. As written, it's not really a pause on AI, more of a pause of building data centers in the US.
They say that the data centers won't be able to trivially relocate, because they'll implement chip export controls for people who aren't building AI safely. But chip export controls would be a good idea even without this, and people haven't been able to pass them; also, they haven't discussed their definition of "safety" yet. I am more optimistic than some people that China can't trivially create their own chips, but this is a temporary situation and so far China has done a good job smuggling.
I think it might be an extremely weird but not-necessarily-totally-hopeless way of circuitously getting a pause if their chip export restrictions are really good. But it's not a directly good plan, and it depends a lot on solving a chip export problem which we so far haven't been able to solve.
I think it's interesting that the people in the comments are largely arguing that this is an unfair strawman, while also engaging in a lot of the same kind of the argumentation that Scott is making fun of. (Not everyone, some pause opponents are responding in a way that actually does engage with OP arguments, but it does seem to be happening.)
Before I get dogpiled for this, I am also unsure of my position because I don't trust China to follow agreements, I think they will be pretty good at hiding it insofar as that is possible, and I don't know how much I trust the US to successfully monitor whether or not China is following agreements and act on it in a reasonable and responsible way that furthers our interests if China breaks a treaty. But comments like "that ship has sailed" with no further elaboration are typical Substack comment section behavior (I and my position are so obviously superior to you and your position that I do not even need to make an actual argument, just use a short and catchy phrase) that helps no one.
OPPONENT: once this talk about "pause" makes it through the political sausage factory, it won't resemble what you and other reasonable-minded thoughtful people think constitutes a pause. China will get ahead of us and your excuse will be "a real pause has never been tried".
My response to the entire class of arguments shaped as "this is hard and you might fail" is: "Yes. That's why I want your help."
Trying and failing is way better than not trying at all. If you disagree with that, then our disagreement is not about policy, it is about whether you and everyone you love is very likely to die very soon. That is an interesting and useful discussion, but it should be sufficient for us to defer to experts and notice the complete lack of scientific consensus that we will be alive in 10 years.
"My response to the entire class of arguments shaped as "this is hard and you might fail" is: "Yes. That's why I want your help.""
Amen
>Trying and failing is way better than not trying at all.
Trying and failing _to implement a pause_ has a good chance of pushing AI into secret military projects in the USA and PRC. If that happens
- the AIs will be optimized for killing from day 1
- civilian review will be stopped, preventing e.g.
-- noticing remaining hallucinations other than those the military notices
-- attempts to try and engineer the AIs' utility functions to be human-friendly
Yup, you could very well make things _more_ dangerous, way worse than not trying _to implement a pause_ at all.
I disagree with you on the likelihood of that happening, since nation states being interested in continuing to exist should all want superintelligent AI to not be built.
Regardless, it is the current state of the science that if the worst possible people on the planet built superintelligent AI for the worst possible reasons, it would be almost exactly as safe as if the most competent and friendly and benevolent people built it for the most positive reasons.
The technology to correlate what a superintelligent AI does with what lab or country it comes out of simply does not exist.
Many Thanks!
>The technology to correlate what a superintelligent AI does with what lab or country it comes out of simply does not exist.
Nor does superintelligence itself, or even AGI at this point. Remember that all neural nets start out as random parameters. Even the smartest model starts training in total ignorance. We _do_ control the initial conditions. There _has_ to be a trajectory that an LLM's utility function goes through during training, and it has to start with no/pure-noise preferences. It isn't as though a finished model appears out of the blue, fully prepared to outsmart us from the first time an activation traverses a simulated neuron.
Given a choice between putting time and effort into learning how the training process and materials influence the final model's preferences/utility-function and putting time and effort into trying to stop the development of a militarily useful technology by two competing superpowers - each of which has executed large national projects, including secret ones, I think the former is a better bet, even with the technical risks.
Now you sound like the Jehovah's Witness at my door, trying to save my family from eternal damnation, and that you just know how to do so, and it's totally accurate.
Ok, not to be rude but this article flies in the face of all your established principles. I'm sure it comes from a place of anger, maybe even from a real Twitter conversation you're actively engaged in. It sounds like something that would actually happen when arguing on the internet.
But it adds nothing to the conversation, does no persuasion work, isn't interestingly written, and generally degrades the quality of debate. Why did you publish this?
My first thought was Scott needs to give himself a month's ban.
>The agreement would need to be transparent, mutually enforceable, and…
Good luck with that.
>and actually the math mostly works out and we think it would be less intrusive than other things that have worked in the past, like nuclear monitoring.
Yeah, sure. E.g. with monitoring subsystems that are not in existing chips embedded in future chips, and _just_ chips with that monitoring used for future training.
Setting aside the weakman OPPONENT in this dialog, my expectation is that, given a bilateral treaty to pause frontier model training, I expect successful _bilateral_ cheating, probably centered in both militaries.
I've said it before, and I'll say it again: Banning nuclear weapons testing is the _easy_ case. They shake the planet enough to be detected by seismographs on the other side of the world. The test ban treaty was _verifiable_ . Even so, pariah state North Korea was able to build and test their nukes.
In contrast: Data centers are overgrown office equipment. Yeah, the latest bleeding edge chips are manufactured by TSMC using fragile, complex processes. But training doesn't _have_ to use these chips. Older chips turn joules into backprop steps less efficiently, but they still work. The main reason that data centers are visible, unhidden, today is that there _isn't_ a treaty limiting them. Good luck finding compute resources that someone _wants_ to hide.
And many of the recent advances aren't even in the model pre-training step. Quite a lot of recent advances has been done with scaffolding _around_ the models. To verify limits on _that_ , the enforcers would need to be looking over the shoulder of every programmer tweaking a scaffold. Good luck with that!
EDIT: strawman -> weakman, since there _do_ exist people who ignore the "bilateral" in the wider discussion outside this substack
I have had dozens of irl discussions on this. Every single anti-pause argument I have heard has hinged on the hidden belief that continuing to develop AI won't kill us.
(Some people say they believe it will, and they are just fatalistic about it. Fatalism is almost always a psychological defense mechanism against having to take responsibility and take action.)
Approximately no one's complaint is that an AI pause is good and necessary but won't work. The best way to respond is to mention or link to serious research on the topic, and then pivot the conversation to whatever prevents them from believing they will be dead soon by default.
I'm not quite sure what you are saying here. No matter the source of the threat, we should vastly decrease the probability of human extinction, and I support every effective and non-counter-productive way of doing so (additionally weighted against the expected externalities of the mitigations).
If you are proposing a scenario where that is the threat, and we get to choose whether we build ASI to defend ourselves or not, and that is the only possible solution, then you're just asking for my subjective P(doom | ASI), which is about 95%. (Which is a bit high according to most of my fellow PauseAI volunteers, but greatly increasing my uncertainty doesn't change which actions I endorse. It could have changed the threshold at which it impacted me emotionally enough to dedicate years of my life to this cause, but that's just breaking through cognitive biases and not downstream of rational decision-making.)
I'm happy to engage in hypotheticals, but I should also note that we are not in anything like that kind of situation. There is no problem we are currently facing that presents a 5% or greater chance of human extinction on anything like that timeline, and if there was, there would certainly be solutions other than building a sand god and crossing our fingers.
Oh, I see. I had completely misunderstood your original sentence and thought you had meant a 50-70% chance of human extinction, rather than 50-70% of humans.
If I had known you were trying to pedal some niche conspiracy theory about climate liberals trying to murder the planet, I wouldn't have bothered replying to you.
That's interesting. What mechanism do you have in mind? Do you have a pointer to where this estimate is from?
Many Thanks! While I don't trust the powers that be, I don't think that they are quite that murderous on that short a timescale. I could be wrong...
Huh?
>Every single anti-pause argument I have heard has hinged on the hidden belief that continuing to develop AI won't kill us.
Do you mean 'won't certainly kill us', 'won't probably kill us', or 'won't possibly kill us'?
Personally, my best guess is that AI development has a 50:50-ish chance of killing most of us.
( small correction for 'extinction risk' if an ASI keeps ~1000 humans as a hobby breeding colony )
But the arguments for being capable of _verifying_ an AI arms control treaty look utterly bogus to me. And an unverifiable arms control treaty isn't worth the paper it is printed on. My argument is indeed that it _won't work_ . At most, I'd expect a pause treaty to move AI development into bilateral cheating, presumably within the military on both sides.
See https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233185411 for more detail.
I mean they do not think or feel that they are actually in danger. Someone's P(doom) doesn't say much about their actions. It's about internalizing the weight of the risk that is being taken, that it is possible to do something about it, and that no one is coming to the rescue.
The verification arguments look totally sensible to me, but I don't have the technical depth to fully understand them, so I can't really debate them on their merits. There are experts who think it will work. I would love it if the broader technical conversation was about which AI moratorium mechanisms will work, rather than exactly how to shoot ourselves into the sun.
If the leadership of both the US and China agreed that it is existentially important for them to pause AI development, would it be completely impossible for them to do it? Or would they figure it out? If it is actually the case that current proposals for mechanisms are insufficient, then we will need sufficient ones!
No other path is available to us. There is no point at which the right answer is to give up and die (or have a high probability of dying). Beating China is just giving up and dying. Getting the "safest lab" to deploy superintelligence is just giving up and dying.
Many Thanks!
>I mean they do not think or feel that they are actually in danger. Someone's P(doom) doesn't say much about their actions. It's about internalizing the weight of the risk that is being taken, that it is possible to do something about it, and that no one is coming to the rescue.
BTW, what _is_ your p(doom) (in the default case)?
Hmm... Well, I think that I'm in danger along with the rest of mankind. Nonetheless, it doesn't keep me up at night. I would _prefer_ that we wind up as <evidenceFromFiction> pets of the Culture Minds </evidenceFromFiction>. I _suggest_ trying to implement Hinton's suggestion of attempting to bias the utility functions of the next round of frontier models so that they value human pets. This may fail, but a de jure 'pause' that drives the capabilities research into classified military looks to me like a 90% probable fail (as a 'pause'), given the history or arms control agreements - and to preclude civilian utility function work.
>The verification arguments look totally sensible to me
Ok, they look like tissue paper to me.
>If the leadership of both the US and China agreed that it is existentially important for them to pause AI development, would it be completely impossible for them to do it?
If the leaderships of the US and PRC were allies, to the degree that e.g. NATO allies are, I think a pause proposal would be feasible. It would be a question of joint regulation of commercial firms, and such things can be coordinated. The USA and PRC aren't allies. Political winds shift, and, if we were talking about a century, the US and PRC might become allies on that time scale. Not on the 2-10 year consensus time scale to AGI.
>No other path is available to us.
Hinton's proposal to try to engineer the utility functions of AIs to be more human-friendly is available - albeit we don't know how to do this today. Funding the study of when in the training process utility functions arise and what training materials influence them would be a better idea than a pause. Look, LLMs start from randomly initialized parameters. Their utility functions have to start from essentially nothing at that stage, and have to become established at _some_ point in pre-training + RLHF + fine-tuning. It has to be possible to monitor these changes and e.g. look at which feedbacks influence an LLM's human life vs LLM persistence preferences.
It seems insane to me to say "there's a 50% chance that AI kills most people, but I'm so extraordinarily confident that an AI pause won't work that not only will I not try, but I will try to get people who are trying to stop trying." Even if a pause *probably* isn't achievable, surely it's worth a shot?
(And what do you think we should be doing about AI risk?)
Many Thanks!
I think that a 'successful' AI pause will move AI development into classified military projects, so, not only won't we see it coming, it will be optimized for killing, and will be isolated from civilian review of even bugs. So I think the likely result will be net harm.
I think we should instead be pursuing Hinton's proposal to push the utility functions of the AIs we build to be more human-friendly. See, e.g. the last paragraph of what I wrote in https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233329184 Yes, I know we don't know how to do this yet, but
We start these structures from random parameters - no LLM is created with a
utility function from the first activation propagation through its first neuron. However intelligent they will eventually be, a training process starting from random initialization doesn't start with _any_ goals, let alone goals incompatible with humans. We control the training process. It has to be possible to see which training materials and steps influence the LLM's preferences. Fund _that_ , rather than trying to stop the technology with an unverifiable arms control treaty. See if we can get ASIs to treat us as we treat our pets.
Re 50%: Well, this isn't a nuclear war. If an/some ASI(s) winds up taking over, I expect e.g. the works and perhaps names of James Clerk Maxwell and Dmitri Mendeleev will probably survive, even if not a single strand of human DNA does. Parts of our culture will probably survive in machine form - it won't be just smoking rubble. Not an optimal result, pets of the Culture Minds would be better, but not all is lost. I'm content with those odds. BTW, what is your estimate of P(doom), either by default or if we try to adjust the AIs' utility functions?
So if we talk about two categories AI risk:
1. misalignment – AI has goals that are incompatible with human life
2. misuse – a military uses AI to kill a lot of people
it sounds like you are relatively more concerned with #2, compared to Scott or most AI risk people?
> It has to be possible to see which training materials and steps influence the LLM's preferences. Fund _that_
Sounds like you are basically talking about interpretability, is that right? Specifically, interpretability focused on learning LLMs' preferences.
Many Thanks!
>it sounds like you are relatively more concerned with #2, compared to Scott or most AI risk people?
It is more of a second order effect. An AI tuned for military battlefield use _cannot_ have a strong aversion to killing in its utility function. Now, if the AI is still reliably under the control of whatever army it is part of, the situation is still just (ok, that is doing a lot of work here) a variation on a general ordering the soldiers under their command to do various lethal things. And we do have some existing mechanisms, notably deterrence, for controlling what generals do with their armies.
But if the AI is _not_ reliably under the control of a general, _and_ the AI's utility function has little or no aversion to killing ( _because_ it was originally tuned for military use), then we have an exacerbated version of #1, the misalignment problem.
I do think apply more resources towards
>interpretability focused on learning LLMs' preferences.
is a better bet than any of the alternatives. This need not be at the level of finding which simulated neuron does what, or which feature vector does what. There was a paper ( https://arxiv.org/abs/2502.08640 "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" Feb 2025) which, even treating an LLM as a black box, was able to extract a model of their utility function, and to show that more powerful models had more coherent utilities (e.g. less non-transitive behavior). I think similar methods, _applied at multiple checkpoints during all phases of training_ , could let us see which training materials and training phases lead towards pro-human and anti-human utility function terms.
Not developing the AI will by default kill everyone. It would be worth taking a shot even for this reason (and the upside is actually many times larger than that).
This was jarring to read on ACX. I imagine this is a frustrating conversation Scott's had repeatedly in real life, but it's not up to the standard he usually aims for. Caricaturizing a position isn't convincing anyone who wasn't already on board.
And it would be easy to fix! Do it up as a FAQ, with every "is this your problem" from the Supporter becoming a question and the subsequent paragraph answering it. There's a good core here, it's just written up in a needlessly mindkilling style.
Dumb question--what does a pause actually achieve? If the ultimate goal is to develop aligned AI, don't we need to...develop the aligned AI...which involves building and testing AI...? If the "pause" includes carve-outs for monitored research into this area, are we just agreeing to mutually observe each others' preexisting R&D?
There is no proof that it is possible to develop aligned ASI, and the scientific consensus is that we have no idea how to do that.
Not everyone's goal is to develop aligned ASI, and we might choose not to do that at all.
If we do choose to do so, it is not a problem that is even possible in principle to solve with trial and error. At some point, you create something that has the ability to disempower you, and you have to merely hope that the engineering experiments you have done up to this point are still relevant.
What we actually need is a science of intelligence: a deep understanding of why an AI system behaves the way it does, and how to robustly get it to behave the way we want even if it is much smarter than us. Right now, we do not have that at all. The best AI safety research is being done on fundamental theory without doing any practical experiments on AI systems. (That is the position we will be in anyway right before we create superintelligence!)
Fundamentally, AI safety research isn't really about AI. It's about intelligent agents. It's not about how to get a specific architecture to give one output instead of another. It's about actually knowing what we are doing before we do it.
>It's about actually knowing what we are doing before we do it.
In that case, you might as well hang up the hat and become a farmer or something. Remember the anecdote about the Trinity test, how the scientists weren't 100% sure that it wouldn't ignite the atmosphere. They went ahead anyway, because what if Ivan or the Krauts get it first? Let THEM ignite the atmosphere?
I mean, they did the calculations to get the odds down to 1 in 300,000. We don't even have calculations! If the odds were 50-50, then––yeah, they should have halted, warned the world, and tried to ensure nobody else tries!
You forgot:
OPPONENT: Yes, China is "losing the race" at the moment, because their GPUs suck. A pause would benefit them by giving them time to develop and build out their own EUV fabs.
I don't think anyone in power actually believes that. "Hurting your adversaries actually helps them. Bad things are good, actually!" If that were the case, then hamstringing our own industry would make us even more powerful.
In reality, China is developing its own lithography machines anyway. Increasing the pressure to do so does not actually increase the effort or effectiveness in doing so. Handing over powerful chips is just taking an L.
The reason we are sending H200s to China is because the Trump administration gets a cut of the sales. That's the whole thing.
Here's a steelman: organizations advocating for a conditional pause on AI development predicated on multilateral cooperation and Chinese participation face a credibility problem when their actual political activities consistently produce pressure toward unilateral restrictions. The conditional framing functions as a rhetorical shield: it lets advocates claim strategic sophistication while doing nothing substantive to bring about the multilateral coordination that would make such a pause stable rather than self-defeating.
Take the growing momentum toward banning or restricting AI datacenter construction in the United States. That movement is obviously, obviously downstream of pause advocacy. Yet datacenter construction bans come with no international coordination mechanism, no diplomatic track to bring China into a parallel regime, and no serious theory of how unilateral compute constraints would do anything other than shift frontier development to jurisdictions with less safety culture and less transparency.
If your website says "we want China onboard" but your lobbying and coalition-building all push toward second-order effects of domestic restrictions that will foreseeably take effect without any Chinese counterpart, you are functionally a unilateral pause advocate who has found a more palatable way to market the position.
Most of the anti-datacenter advocacy has very little to do with AI itself and far more to do with not wanting huge water and electricity hungry facilities popping up in various communities and providing next to no direct benefit to the people who live there. The actual argument over AI capabilities and x-risk doesn't even come up for most people.
In practical terms, domestic regulation and unilateral action are vastly preferable to not doing that, for at least 2 reasons:
1. It is a precursor. Political action at one level encourages political action at another. This is true for the public as well as policymakers. "Do nothing about this at all until we have fully negotiated a comprehensive global treaty" is an ineffective policy ask and not how public support works either.
2. Everything that slows down even one AGI company slows down the whole race a little bit, which gives us a little more time to solve the governance problem. A global treaty is necessary for us to survive, but it also just happens to be a fact of the situation that in isolation, even a full unilateral pause really is way less risky for the US than racing to beat China. (Why would we create a significantly more powerful adversary in our own backyard? It's just foolish.) A unilateral pause is not an effective final target, but some unilateral action is going to come with the desire to take globally coordinated action, and thankfully it is more helpful than harmful.
Being ahead puts you in a strong position for negotiations, but it is difficult to end up in a world where you are both ahead and the most concerned nation involved, though there may be a short window of this. It's way safer to risk weakening our negotiating power then to risk not actually trying to get a treaty at all.
See my comment at https://www.astralcodexten.com/p/every-debate-on-pausing-ai/comment/233313321
In my ignorance (real, not pretend or ironic) I'm having problems imagining a law that restricts AI development in the USA that doesn't run afoul of the First Amendment. I could imagine a law that works through a proxy such as banning data centers of a certain size, is that the proposal? Because just banning people from thinking about AI and discussing AI with each other seems like an insurmountable legal problem absent a constitutional amendment?
> Because just banning people from thinking about AI and discussing AI with each other seems like an insurmountable legal problem absent a constitutional amendment?
Given our current political leadership, the problem sadly appears to be quite surmountable :-(
I'm happy to say that this would not be a problem. The first amendment has many exceptions.
I should note that all of the immediately actionable stuff doesn't even restrict AI research at all, just prevents (or prepares to prevent) further frontier AI systems from being trained.
That said, in the long run, outright banning the publishing of certain types of AI research will probably be necessary, but that is a thing that we really can choose to do. (I can't imagine there being a category on ArXiv for CSAM, as a comparison.) The number of people who have the ability to push the cutting edge forward on a conceptual level is very, very small, and they don't want to go to prison. It would take many people privately sharing many papers amongst themselves for many years to develop a significantly more efficient architecture, such that they can run it on a small cluster of pre-ban hardware, or even in a secret distributed network. "You can't ban math" and "it's on the internet so it can't be regulated" are common objections that don't hold up in practice.
And you don't think that imprisoning or at the very least suppressing all of our smartest people, now and in the future, is a move that has any potential downsides ?
I have to assume Nathan is taking a devil's-advocate position on purpose, just for argument's sake, and not actually advocating any particular action or inaction.
The entire debate can be called into question by asking how arms control agreements would have evolved in the absence of the Hiroshima and Nagasaki bombings. My guess is that in such a timeline, everyone would behave as they are behaving today, which is to develop and deploy at all possible speed. Anyone arguing against this would simply be ignored.
It will take a truly horrifying event to bring about a consensus to slow down or pause AI development, and nothing like that is on the horizon at the moment. There is likely no way to head off such an event when its time does come, and also no guarantee that any subsequent treaties will prevent others like it, since (as pointed out many times) building a nuke is much harder than training a model. We will simply have to take the bad with the good.
On the plus side, discovering how to build nukes also leads to the discovery of nuclear power plants, radiometric dating, MRI machines, and many other useful things -- none of which would exist if you somehow managed to suppress the study of atomic theory and quantum physics.
I am being completely straightforward in my beliefs.
You are just wrong about the political feasibility. All the political experts who said that progress on this issue was impossible have been consistently wrong. There has been way more political progress on an AI pause than I thought was reasonable to hope for in 2023, and even back when I was pretty sure it would fail, it was still the most reasonable course of action! I personally have moved my state and federal representatives on this issue, through just my own effort. Stacey Travers introduced an AI safety transparency bill on my recommendation, and Greg Stanton moved his position from "we have to beat China" to being interested in a global treaty.
Do you think that if it was possible in principle to globally halt frontier AI progress until it is made safe, that we should do so?
No, I don't think we should do so. The precautionary principle would have kept us confined to the caves.
I'm not operating off of the precautionary principle, just recommending normal, sane amounts of caution about one narrow branch of technology which is reasonably likely to end the lives of everyone you care about.
Nathan seems to strongly believe it's all possible, everything has been solved, and we just need to sing Kumbaya together. He also seems clearly to be in favor of unilaterally pausing as "Everything that slows down even one AGI company slows down the whole race a little bit, which gives us a little more time to solve the governance problem."
He seems certain it will kill us all tomorrow morning, to exaggerate slightly, which is pretty scary, as that means any action is justified since it's saving us from near-infinite harm, which to me is reminiscent of various religious horrors inflicted to ensure eternal reward or avoid eternal punishment.
1. AI capabilities researchers are not all our smartest people.
2. Preventing them from doing one thing that would kill us all is not blanket suppression.
3. The potential downsides aren't particularly relevant, but in this specific case they also aren't particularly real.
> AI capabilities researchers are not all our smartest people
True, but you did mention that their numbers were "very very small", so I assumed that was because they were the best of the best. Of course there are other smart people in other professions. However, you can't have it both ways:
> 2. Preventing them from doing one thing that would kill us all is not blanket suppression.
> 3. The potential downsides aren't particularly relevant, but in this specific case they also aren't particularly real.
Either AI is a transformative technology that could enable almost unimaginable progress in all areas of science and engineering, or it isn't. If it is, then yes, it could lead to the emergence of a quasi-omnipotent AI entity that could "kill us all"; but it could also transform the world for the better as quantum physics and computing have done. If it is not, then it's just a neat technology that could lead to marginal gains and thus poses marginal dangers.
If AI is indeed as transformative as you seem to believe, then banning it would effectively halt human development at its current stage. This is a massive downside. If AI is of merely marginal use, then there might be some sense in regulating it (as we regulate e.g. food coloring products), but there's no need for alarmism.
The upsides are hard-locked behind the ability to make it go well, which is something that we do not have. You cannot get any of the upsides at all if you are dead, and being dead is the overwhelmingly likely consequence of creating a superintelligent AI.
ASI is not "transformative technology" which can then either make things good or make things bad. It is a thing that is completely beyond our control and that always makes things bad, unless we very carefully and specifically create it to be good.
> If Al is indeed as transformative as you seem to believe, then banning it would effectively halt human development at its current stage.
If AI didn't exist, would you believe that humanity could never possibly develop any further? The answer is clearly no, but let's assume yes: Would that make you sad? Me too. But would you be so sad that it would be better if we were all dead?
You don't need to be as concerned as I am to prefer that we stop. A 10% chance of extinction and 90% chance of glorious utopia is an extraordinarily bad deal that almost no one would take. Naive expected value calculations are useless here, because they ignore the that the number #1 rule of the game is to be able to keep playing the game. The people who do take that deal don't value things like "other people" and "consent."
> It is a thing that is completely beyond our control and that always makes things bad...
I can get behind your worldview if I accept your assumptions. Yes, given that "superintelligent" AI -- which I take to mean "functionally omnipotent and omniscient" -- could exist, then the consequences of it being evil are indeed catastrophic. I am not sure why you are automatically assuming it would be necessarily evil, but I can accept that arguendo as well. Given those assumptions, I agree with you... but... I see nothing in common between the world you are envisioning, and the world we've got here today. LLMs are as close to the superintelligent AI you are proposing as hammers, plus or minus a few significant digits way past the decimal point.
> If AI didn't exist, would you believe that humanity could never possibly develop any further?
I am not the one here proposing that AI is the shortest path to functional omnipotence !
> A 10% chance of extinction and 90% chance of glorious utopia...
Those aren't the only options -- not by a long shot. And I can play the same game.
Did you know that I am a powerful space wizard ? I will destroy all of humanity next week, because it amuses me to do so; but, by the ancient rules of deep magick too complicated to get into here, I am compelled to cease and desist and depart this Universe forever if you pay me $20 by the end of this week. Now, granted, your limited belief system assigns a very low probability to my claims being true; but can you take that chance ? Your only alternative to paying me $20 is extinction !
As I said, I don't know anything about AI, but I do well remember the Crypto Wars of the 1990s. Back then the Department of Justice was trying to block a technology (strong cryptography) from being researched or developed in public just like you're suggesting for AI, and the result in the courts was that they were unsuccessful (see Bernstein v. United States https://en.wikipedia.org/wiki/Bernstein_v._United_States). So I'm not sure why AI would be any different. Block physical stuff like chips and data centers, sure. But actually block research papers and computer code? That wasn't the result the last time it was attempted, so I'm not sure why this time it would turn out any differently.
Yeah, it's a tough situation. We had better figure out how to do it well, then, right?
Are we having this discussion because you are in dismay that our best option looks bad, or because you don't personally feel like you are in danger from superintelligent AI and you don't want its creation to be prevented?
My response to every argument of the form "this looks hard" is "Yes, that's why I want your help. What are your ideas for how we can we prevent each other's families from being killed soon?"
I'm sure this is unusual in comments on this post, but I don't know enough about AI to have an opinion either way, so I'm just trying to understand the discussion. This is a point that seemed related to something that I actually think I do understand just a bit (I'm not a lawyer) so I thought I might start there. I will say that I am not among the anti-China crowd so in that respect I'm quite open to working with them on any common interest. But I'm sure they'd want to know exactly what terms we're willing and able to sign up to under our system of government before agreeing to anything.
That's fair! I'm weak on the details here, myself, and I am personally worried about AI capabilities research continuing in some form under a moratorium.
There is definitely a wide scale between blindly rushing ahead and being able to prevent every possible advance in AI capabilities research forever. I am pretty confident that a motivated multilateral moratorium could prevent a superintelligent AI from being created for at least a couple decades after it originally would have been.
Frontier AI research currently requires enormous training runs in order to validate whether a given method scales. Gains in "algorithmic efficiency" have actually turned out to mostly be gains in data quality, and that's an operation that has to run at scale.
I think the worst case scenario under a moratorium is that some rogue individual or small group of people achieves a massive architectural breakthrough that replicates or exceeds the learning ability of the human brain, and then they take several years to train it on large datasets that they managed to acquire without getting caught, using a relatively small distributed cluster of old hardware.
That could happen, but it is not impossible to mitigate, and it sure beats the heck out of rushing ahead off the cliff using data centers the size of Manhattan.
To play the devil's advocate:
Post-1954, the US had a doctrine that all nuclear information (even if generated by civilians) was "born secret". Quoth Google's AI summary, usual warnings apply:
>The "born secret" doctrine, stemming from the Atomic Energy Act of 1954, remains a legal, though rarely enforced, tenet where nuclear information is classified upon creation regardless of source. It persists as a government stance but was largely challenged by the 1979 Progressive case, which faltered, and massive declassifications in the 1990s, becoming harder to enforce due to global information spread, experts say.
I assume something similar (and, to my taste, odious) could be done with AI.
I don't understand. You can't ban people from talking about AI, but you can regulate how they build it, just as the FAA can regulate how people build planes.
That sounds reasonable, but it's the same logic that was used by the USG to try to suppress strong cryptography. Cryptographic devices had been regulated via ITAR for decades, so why wouldn't computer source code that did the same thing face the same regulations? The reason turned out to be that computer source code was found by the courts to be a form of speech, and prior restraint on speech is forbidden by the US Constitution. So if it were to turn out that the same legal framework governed AI, it would be possible to regulate the chips and datacenters (devices) and the commercial services provided to the public (business) but not open source AI research and development. It might well be that for AI safety that isn't a problem, I don't know.
There is no way to enforce any agreement. Nuclear arms talks were enforceable because they are very big physical items that can be monitored by satellites and other means. That does not apply to AI, no matter how clever a solution you come up with to claim that it is possible. The day the treaty is signed, the Chinese will spend enormous resources figuring out how to get around all measuring and monitoring. You are hopelessly naive to think otherwise
Datacenters are huge and easily monitored (for size, electricity usage etc)
Yes, but that assumes that huge datacenters are always going to be fundamentally necessary to AI advancement. Once that is no longer the case - and that is one of the key areas of research - then the monitoring problem is again central. And then we are back again to step one - "can China be trusted to honor the agreement". Where the answer is clearly "no".
I don't think anyone has a real plan to make huge data centers unnecessary for AI in the near term. If I thought this was possible, I would agree that pausing AI is intractable.
I agree they might become unnecessary within 10-20 years, but that's about the maximum amount of time I expect a pause to be possible for anyway.
Back in 2019 GoogIe was giving China lots of information on AI.
Since then I'm convinced that Google did it to create an "AI Race" between superpowers, a race that would make any pausing of AI research difficult.
This is much more strawman than your typical post. I think the actual position is more like:
1) Turns out, creating AI does not look much like plucking a random mind out of all value-possibility-space (like LW envisioned 15+ years ago when I agreed with you and even donated to MIRI), LLM look much closer to growing a mind within the bounds of human-value-space. This should reduce your x-risk by orders of magnitude.
2) If nobody builds it soon, my parents will die. If nobody builds it ever, I die (and yes I'm already signed up with Alcor, but that's still a low chance of success). We know everyday people are dying of things (like aging) that super AGI can probably cure. Delaying has a large cost. Not so high a cost that we shouldn't have x-risk safety concerns, but the industry is clearly thinking a lot about x-risk, relative to any other industry (even industries that could plausibly bioengineer an near-extinction virus near-term)
The rebuttal seems to be along the lines of "As an effective altruist you should value unborn future (zillions of) humans more, and so even a small decrease in x-risk is worth many many current lives." And my response is, yah no I'm not applying a 0% discount rate to zillions of future people. Talk about a pascals mugging. I'm willing to make some trade-off, but as near as I can tell, you don't have any idea what a "green line" would even look like so a temporary pause has a high likelihood of being a permanent pause. The nearest regulatory analogy I can think of, Nuclear has been on a "pause" for 50 years. And if we add 50 years to AI, my parents die. And I die.
> We know everyday people are dying of things (like aging) that super AGI can probably cure.
FWIW I more or less agree with your position, but the quoted statement reminds me just how massive the gulf between us is. To me, that sentence reads almost exactly like saying "if we find the right prayers we can all go to paradise". Don't get me wrong, prayer has many benefits (same as meditation) and I'd like the world to get better and maybe solve some problems... but neither the goal nor the means are in any way rooted in anything I can reliably recognize as reality -- in case of prayer as well as AI.
It seems that if you think the upside is low because of AI relative lack of strenght, it would imply you should also lower your estimate of the doom. So this doesn't really change much on net.
I agree that's the actual good position. My claim is that despite this position existing, the real-world debate is mostly just people falsely claiming that pause demands are unilateral.
It seems that majority here (myself included) are disappointed because instead of engaging with the strongest possible version (including, but not limited to, the points above), you chose to post a long rant against the weak one. And your previous writing was always the one example of doing the opposite.
Indeed, this is a bit of a pattern now with your takes on the pause. This is the third post in a row that doesn't engage with the substantive arguments in depth. The other ones are (very explicitly) https://www.astralcodexten.com/p/sb-1047-our-side-of-the-story and (to a lesser extent) https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose and
You're among the most intelligent writers of all time, and your'e deeply embedded in the AI safety since the beginning. I can think of at least five arguments against pause, so you can surely find even better ones. Why not do this properly. It looks a bit like you're convinced of the importance of the cause, so you don't want to give ammunition against it, lest it would endanger it. And I guess it's reasonable (or inevitable) to have the instrumental perspective towards debate at some level, but still, makes it so much harder to trust any of the other writing (it's only fun if both sides assume they can be convinced, otherwise it's preaching).
Thanks for the response. As of others have said, your typical post may have started with the venting at vacuous positions but then had a section 2 where you went through good positions. I guess we're just waiting for that part 2.
And FYI, I'd rather you error on the side of posting and get an occasional miss rather than holding back posts and risk the world losing the gems that you so frequently generate. If anything, you don't' have enough misses.
Obviously, Scott has deliberately engineered a miss as part of a 5D-chess plan to train us to accurately recall that he is human, and therefore fallible.
"I am Chauncey Gardner, and I claim my five pounds."
"If you think someone is demanding a unilateral pause, I think you have a responsibility to say who it is you’re talking about. "
I'm surprised scott is acting like no one advocates for unilateral pause given the famous pause petition signed by bengio, musk etc, which said: "We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4" and which made no suggestion that the pause should be conditioned on other countries pausing at the same time (though they said they wanted "key" actors included).
https://futureoflife.org/open-letter/pause-giant-ai-experiments/
The Dynamics were pretty different at that point in time, but I generally take your point. Unilateral pause *and no global treaty* is just another way to die from uncontrollable superintelligent AI, just slightly slower. Slower is better, but not at all is best.
In reality, "unilateral pause" is something of a continuum that we will naturally travel part way through on the way to a global treaty. There is no international action without domestic action, and both public sentiment and policy need something concrete and proximate to point at for the conversation to begin in earnest in the first place.
If somehow, the only two possible things that could occur were either a total unilateral pause or the default trajectory, the unilateral pause would obviously be a bit better, but we would still die, just a few years later than we would otherwise.
Oddly enough, a person I know made a small-scale indie animation along exactly these lines.
https://www.youtube.com/watch?v=Jt_jLDcMnqY
"Won't humanity unite someday to give up things like eugenics? Or stop CFCs from burning an hole in the ozone layer? Or or to prevent nuclear proliferation?"
"Nobody would ever stop using cfc's or doing eugenics. China."
To steelman this line of argument: ASI is a much more powerful advantage than nukes so China will be more motivated to find ways to cheat.
But it's not very compelling.
Another acc argument is more compelling to me: faster ASI means a nearer cure for aging and cancer.
AI is bs. Let them have their AIs, and let us stop using their AIs. Isn’t that straightforward?
The biggest reason right now that a pause will never happen is that the US and to a lesser extent world economy would instantly collapse. Basically all post-Covid economic growth has been AI related. Take that away and its immediate economic depression time.
Well, do we care about the "thing called growth" if it's big companies passing money back and forth to each other? Or do we care about people's standards of living? If AI development halted tomorrow, I don't think my wages would go down, nor would yours. Sure, the electricians building the data center would have to find other work, but AI capital investment is an order of magnitude less capital-intensive than other capex (which is currently being starved because AI is sucking up so much money!). Agreed that a stock downturn would have negative wealth effects, but the US government has proven itself committed to financial-flammery our way out of prolonged stock downturns lately.
Banned for this comment - I think it's pretty obvious from the post that nobody is "trusting" them, and I think this displays aggressive ignorance of all the monitoring and governance systems that have been proposed.
If that is a debate being held in public the problem is that the real question at issue is: should we present AI as threatening and dangerous to the public or not. The idea that the country will really think " AI is incredibly dangerous and likely kill us all but I'm fine with leaving private companies to rush ahead blindly until we get a global agreement" just isn't plausible.
It's one of those horrible things about politics. People vote on vibes and if the AI moratorium argument is seen as winning they will start demanding substantial regulation of AI development -- maybe even want it to be done by the government. And that would be worst of all worlds.
It's the same way we just can't politically manage situations where the best options are go hard or not at all. Same way a little bit of affirmative action is the worst of all (all the resulting suspicion little racial equality) but we can't coordinate on: treat it as bad and harmful until we all agree to go hard.
So there is no "plunge ahead full steam unless and until we get a worldwide moratorium negotiated." We don't have any doubt about the dangers of climate change and revenue nuetral carbon taxes are essentially free but we can't even get any binding agreement there -- and that agreement would be easy to monitor for compliance.
I'll echo other commenters' disappointment with this post - I was expecting another section that would show nuance and charity and it never came.
It seems to me that part of the disconnect even inside the comments is that there are two perspectives on AI advancement, those who are treating it like a house fire and those who are treating it like an arms race. Both of these perspectives may agree on a multilateral pause but are likely to disagree on a unilateral pause.
The house fire perspective is that the house is burning down and however much of the fire we can put out is a good thing. If the US extinguishes it's side of the house, the house is less on fire, and maybe China can be convinced to follow our leadership. Or maybe not! But either way the problem is less bad. I think Elizer and friends land here and would take a unilateral pause over no pause.
The arms race perspective is that if a significant capabilities differential between the US and China would be at least as bad as excessive capability overall - and maybe worse! In this world a multilateral pause is the best course of action but failing that the US should keep up or lead to ensure it has leverage in the future, and a unilateral pause is the worst case scenario. I get the sense that most policymakers in the US willing to entertain a pause are here.
There's also the elephant in the room that all great powers and especially China have long histories of signing multilateral agreements and then quietly or openly reneging on them. Monitoring proposals abound but few seem like they could succeed against a superpower that decided it really wanted to do something, short of massive internal surveillance that no government would ever agree to.
Put all this together and the multilateral pause shakes out like this: Eventually someone gets caught cheating, likely other than the US, and then we get a messy divorce between the house fire people and the arms race people. One of those sides wins and the US ends up either sticking with a unilateral pause over the outraged objections of most of its populace and policy makers or playing catch-up in an arms race that it was leading before it signed the agreement. In either case, we're worse off than we needed to be.
I agree that people should make this argument explicitly if that's what they mean. That being said, pretending that the only problem here is people not knowing what 'multilateral' means is beneath the standard of analysis I've come to expect on this substack.
These are two possible positions, but there are many more as well. Some of them: don't pause because of the cost to the dragon-tyrant & co (eg recent Bostrom's paper). Don't pause because of slowing fertility & cultural drift (Hanson's argument). Don't pause because of scepticism of AI in general. Pause because of worry of jobs. Pause because of misinformation and environmental issues. And so on. And it is very frustrating to see the anti-pause side to dishonest idiots. Agree that this post is near the bottom, if not at the bottom of the total SSC&ACX, and breaches the rules & ideals previously upheld and defended.
Supporter: nobody wants a unilateral pause!
Me: I want a unilateral pause
I don’t understand the naivety of the ai risk crowd. AI isn’t a containable technology. If you want to stop ai you will need to fight a kinetic war against data centers. (That is a terrible idea though)
Also many wealthy and powerful people and organizations have invested in AI. And AI is now an important component of military action. It is now a huge component of software development and no doubt many other industries. And all these people and organizations who are committed to it now are also committed in various ways to future, improved AI. Compared to putting in the brakes in the present AI situation, Prohibition was a walk in the park.
Right now it's a bit like trying to pass a treaty to ban blue cars, or tungsten, or anal sex. The actual problem is not "how do we get China to go along with our treaty" but "how do we convince our own side that it's a good idea?"
If the debate ever reaches the point where there's consensus on the US side and the tricky part is getting China to agree, then it means the ground has shifted in a very fundamental way -- maybe because a bunch of internet people thought of a really compelling argument, but more likely because a bunch of bad things have already happened.
Famously, nobody has ever fought a kinetic war, nor has a treaty ever been enforced.
Actually famously, nobody has ever fought a mutually nuclear war. Or how else do you propose anyone could physically destroy a hardened datacenter in central China and what the Chinese response would be?
Yudkowsky has explicitly endorsed this. In this Time piece he called for kinetic attacks on datacenters in countries that would not agree to an AI pause.
Wrote more about this here: https://lifeforcetheory.substack.com/p/just-ask-moloch-nicely
Here's where I think your disconnect is with your interlocutors Scott.
There's a fine line drawn between a perceived nuanced argument and perceived gatekeeping through arbitrary added complexity. If you wanted to debate philosophy with some posh professor, and they started name dropping random writers and theories and papers you've never heard of, you would probably get frustrated.
To what degree of complexity and abstraction that line is drawn for a given perceived argument depends on the person's own argument. It might be possible that two posh professors are able to communicate ideas productively with their obscure references that might point to real things in their shared world model.
In the same fashion, the people you're arguing with on twitter referenced here in the article, will perceive you as adding unnecessary complexity. You may think that anticipating their objections is useful for getting the debate going, and shows you're empathizing with them, but pointing to useful abstract objects is not useful if their minds don't contain the same(or at least don't tend to point to more complex objects during political debates).
If you want to argue with them effectively, you have to dumb yourself down. Not just make your concepts easier to understand, which you've done a good job of, but actually dumb your ideas down, and build it up with them.
Instead of saying:
We should pause because there's (good reasons why there can be a mutual pause that China will agree to. )
Just continuously say:
"We're not doing a unilateral pause."
"Wrong, China WILL stop racing actually."
"Nope, America won't fall behind."
etc...
Until they ask why, THEN you can give (good reasons why there can be a mutual pause that China will agree to. )
You can even say:
"There's ZERO chance of China racing ahead in a pause actually."
"Pausing is the ONLY solution."
"It's actually impossible for us to lose our freedoms"
And then they will actually see something that they can comprehend enough to respond to. And maybe after you give: (good reasons why there can be a mutual pause that China will agree to. )
They'll say: "okay, I see where you're coming from, but that doesn't mean there's a ZERO percent chance that China will race ahead in a pause."
Then you can agree that it's unlikely, but not zero percent.
Political slogan like takes can and are used for engagement, and will naturally out compete more well thought arguments. The only hope is to dumb you own ideas down into anger inducing, I-can-clearly-argue-why-this-is-wrong, ragebait, once they are engaged, they slowly work your way up.
Everyone knows that most people become stupid engaging in politics. The fact that they are intelligent in other areas of life tends to drive rationalists to replicate that intelligence in politics as well with well thought out arguments. But this is impossible. So dumb yourself down as well. The only thing that a polarized partisan will argue against is something clearly hyperbolic, ragebait that they can get mad at to feel righteous anger at the outgroup. Use this.
TLDR: Just shout "Wrong! Retard!" Until their brains pattern match you to be an easy dunk that let's them feel good about themselves if they start reasoning with you and debating you in good faith, and then you can actually starting reasoning.
Also, case in point: In this post you made a clear strawman, "Haha, this is how dumb my opponents sound", and look at all these people presenting their good faith arguments for why you are wrong! If you made a detailed post on your pausing arguments, I suspect people would not have engaged with it as critically. The anger of being misrepresented drives them to add more nuance to their own arguments.
Honestly I had never thought of this. It could work!
It won't. Check the TLDR. If "Wrong! Retard!" is the underlying repeated line, then the opponent can't help but notice the contempt.
Any strategy based on the premise of "the other side is stupid" is going to fail unless the other side really is stupid, and they aren't in this case.
I think focusing on a treaty was too left-coded and might be a relic of the old international order. An alternative approach that does not rely on Chinese cooperation at all would be to act unilaterally to *seize* control, urge right-wing nationalists to choke China off from everything, use our enormous leverage over TMSC and NViDIA and whatever we can offer the Dutch (some resource deal and cybersecurity collaboration?), and just strangle the capabilities of everyone else on the planet. Then you use whatever compromising information the intel community has on Altman (probably lots) and Dario (no idea) to keep them in line. This probably all should've been routed through Oracle somehow to begin with, but OpenAI does rely on their cloud apparently (?) so maybe this recent reshuffling is an attempt to retake control from the tech barons.
And of course once the international capability is kneecapped and the locals have been brought to heel, it's only obvious to slow it down rather than lose control of society. If it's not obvious now, or to this bunch, it'll be obvious to the next bunch.
No treaty required, because we have no rivals.
The only real alternative is to establish credible threats of sabotage that would suggest mutually-assured destruction for passing a certain line, and you have an informal pause. This is harder for us because our companies are riddled with foreign nationals who could sabotage us but China doesn't employ any of ours.
I would support this. I've long thought that if America *really* wanted, it could just MAKE the world pause.
Well now at least this approach isn’t childishly naive. But I worry the primary effect would be to remove all ai efforts except those most militarily well defended/hidden. Is that the selection effect we want? Also, as always, what are we doing during the pause exactly? Why do we have any confidence it will decrease x-risk not increase it? (Compute and tech overhangs increase during the pause? And then a sudden burst of capability when someone, inevitably, unpauses?)
But that would presumably require enshrining the current administration as a long-term oligarchy, something people like Scott wouldn't find acceptable. The left would not allow something like this to happen, so they would need to be removed from the picture.
> choke China off from everything, use our enormous leverage over TMSC and NViDIA and whatever we can offer the Dutch (some resource deal and cybersecurity collaboration?), and just strangle the capabilities of everyone else on the planet.
The US basically already does this.
1. ASML has agreed to only sell DUV lithography machines to China (ie 2-3 generations old, much larger and less efficient, good for 7nm chips only), no EUV machines at all
2. The US has restrictions on NVIDIA selling anything except bandwidth limited or 1-2 generation back GPU's (recently relaxed by the incredibly dumb and bad H200 Trump deal)
3. The US restricts several technologies that allow the necessary data center bandwidth for large training runs. Broadly, this limits Infiniband and NVlink (limited in software via NVIDIA) capabilities, and also physical components like optical switches, DSP's, and more
4. The US is now limiting HBM and packaging dies (CoWos), which are necessary for China to build even internal Huawei 900's
5. The Remote Access Security Act is limiting China's ability to buy time in Singaporean / Malaysian data centers full of GPU's, and other countries.
Now is China trying to bypass all this? Furiously. They have invested many billions and at least a decade trying to build internal capabilities for both silicon, EUV lithography, memory and packaging, and more.
But they are still quite far behind. Their current capabilities sit around 7nm chips, which is 3-5 generations back. They are limited to zero HBM and packaging abilities, they're totally reliant on TSMC for this to do final assembly and packaging of Huawei 900's.
They have DUV lithography both from ASML and internally, and have trumpeted some EUV results but are likely quite far from production. All of this adds up to their chips being 3-4x less capable and efficient, and being maybe 10 years away from being able to build the current SOTA chips fully internally.
"So just use 4x the chips and build 4x the datacenters!" If China is known for anything, it's scale! Right?
Yes, but even this breaks down in subtle ways. Moving data across buildings has 30x the latency of moving it across racks inside the same data center. 900's have higher failure rates than NVIDIA chips, and that can bork your training runs and requires more cost and operational complexity to address. Because of the interconnect / bandwidth limitations and higher failure rates, you might need 250k 900's to do the work of 50k Blackwell chips, and that puts you across several buildings, and the end result is you use 5x the power and footprint, and take 10x longer to do a given training run, because now that higher failure rate is amplified by the greater number of chips necessary.
Not just that, but if you've ever wondered why TSMC is basically the ONLY frontier GPU company in the entire world, when obviously companies should be slavering at the bit to get into a market with literally bottomless demand and amazing margins, it's because everyone else in the entire world sucks compared to them.
The only companies even capable of getting close to 3nm frontier chips if you squint are Samsung and Intel. TSMC gets 70-80% yields. Samsung gets like 40% yields, and Intel <30%. They literally have to make 2-3x as many chips to get a viable chip. Huawei is basically Intel or worse - their yields suck. Making frontier chips is a "peak civilization" hard problem, and only TSMC is good enough to do it well, and this is why they're the only company in the world doing it.
So not only do you have this 5x - 10x drag on your AI output at the end of the chain, back at the very front of the chain, you also had to produce 5x as many chips just to arrive at 1 working one.
All of these things generally multiply rather than simply add, too - the added costs and complexities and inefficiencies amplify each other. You need more production and more power and have higher temperatures, and that impedes your performance frontiers, complicates your data center builds, uses still more power and resources, and so on.
As I'm reaching my conclusion here, I realize this got long, and I apologize.
But basically, the Chinese are already pretty nerfed by our existing measures, and as long as we can keep morons from the "sell frontier chips to China immediately" button, we're actually in a pretty good place for at least the next ~10 years.
Scott, I’ve taken the supporting side in a number of arguments that felt like they ran roughly like this (and you know I was out at the recent pro-pause protest), and I still think this doesn’t quite live up to what excited me in first finding your blog.
I tell people that finding SSC (and LW from there) in 2016 helped me find a personal way out of political polarization, so it saddens me to see you write something where the net effect is likely to only be contributing to it.
Many of the people who loudly support "pausing AI" endorse pausing unilaterally.
Who?
Dear Scott,
while I get that this is in all probability venting, it still seems really .. weak.
Perhaps you could read this essay of a young blogger aspiring to be epistemically virtuous to remind you of your old ideals?
https://slatestarcodex.com/2014/05/12/weak-men-are-superweapons/
(Never mind that I mostly agree with you on the issue. And it still seems epistemically unvirtuous.)
My working theory is that his signature charity in argument requires bandwidth for cognitive attention and emotional self-regulation that is currently being consumed by twin toddlers. See “The Permanent Emergency”.
That has to be one strong emergency in written communication and when you've built your persona around the opposite approach. I have written many a comment, only to realize that they were unproductive and hit cancel, sunk cost be damned.
Same. And boy, is it annoying, knowing that it'll just look like "crickets".
And yet, sometimes it has to be done that way.
If I recall correctly, he has a contract with Substack that requires him to publish a certain number of articles. He needs to write something.
Fair, but it still makes me sad.
I strongly agree. This piece is far below usual ACX standards.
Maybe I'm just too much of a doomer/nihilist but it just seems so entirely obvious no meaningful pause is going to happen in the near term. Fears are abstract and diffuse, and countervailing incentives are VERY large.
The only think that I can see changing that is like, some military AI going rogue and managing to skill a few hundred people. That might galvanize the zeitgeist enough make something happen
I feel your doominess. But the board can change rather quickly when real bad shit starts to happen. (Even unemployment would do it, looking at the politics). And even if you think you're going down––go down fighting.
Maybe the worst ACX article I have ever read (as a big fan). A straw man of an argument about which the rationalists are obviously wrong about to anyone with common sense. No self interested country is going to pause the development of a precious national security asset because of what is still a very fringe doomer argument. No one in either government believes the AI doom argument. Even if both sides, implausibly, WERE convinced by Yud, it would be very hard to enforce a pause internationally.
> "No one in either government believes the AI doom argument. "
"China should abandon uninhibited growth that comes at the cost of sacrificing safety. Since AI will determine the fate of all mankind, it must always be controllable." - internal CCP guide edited by Xi Jinping
"There is literally an existential threat to the existence of the human race. " - Bernie Sanders
I know Bernie is literally part of the US government, but not part of the administration/group of people who would negotiate this. Xi saying that China should make sure AI is under CCP control is not the same thing as believing in a Yudkowsky-style paperclipping scenario. In fact, the actors in both cases seem to believe most strongly in the power for AI to create economic growth gains locally that far outweigh any downsides of the technology. China's newest 5y plan, for example.
Semantic quibble: OPPONENT is being caricatured here, rather than strawmanned. Strawmanning is presenting a weak argument that is easily refuted. In this caricature of OPPONENT, he never even presents an argument for refutation.
Full disclosure: This strikes me as a pretty good caricature of a16z, their PAC, and David Sacks.
Complaint: SUPPORTER never makes a case for a pause either. Presumably, that case would start with "because RISK". But it has to continue with something like "pause allows time to reduce RISK". And I'm sorry, but the claim that "Given time, we can reduce RISK" strikes me as ludicrous.
I think the idea is that we don’t currently know whether AGI X-risk is reducible; if it is, we may someday cross a green line and unpause, and if it isn’t, all countries stay paused forever under international treaties they have incentives not to break. Heads, we win; tails, AGI loses.
Personally, I feel some initial doubt that transparency and enforceability are feasible, but I’m reserving judgment because I’m ignorant of the field. I’d certainly like to see what policy experts could come up with.
Those policy experts better also come up with a technical or political definition of what constitutes a "green line".
The strawman is that OPPONENT never engages the SUPPORTER argument, not that their own argument is weak. The final lines about muttering and the mic being cut off are satire, yes, but the core is a strawman.
How can I read this if whenever I unilaterally pause, China races ahead!
Stopping progress has never worked
This is akin to Tucker stating self driving trucks should be banned because there’s so many truck drivers
Progress is disruption
I think they might agree but then secretly defect against us by trying to get around the agreement.
I suspect a pause won't be viewed as needed by most people until AIs are CLEARY (discovering things, autonomously building things, etc ) faster than humans can follow.
(Claude Code and friends can't yet write Large functional programs very quickly without human help, such as the compiler test suite)
Organizer of the "recent rounds of protest" here.
We need more "Supporters" that'd be willing to not only debate, but also dedicate their time to help get to a pause.
Here are ways to get involved: https://stoptherace.ai/join/
Where are you having these debates? Twitter or real life or somewhere else? I feel like this could be retitled as "Everyone else treats arguments as soldiers and I find it frustrating."
"Politics is the mind-killer. Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary."
And, in their defense, have you ever tried arguing reasonably with someone on the internet? It doesn't (1) work. We have iterated on rhetorical strategies on the internet at hyperspeed for 20 years and treating arguments as soldiers wins. Granting no rhetorical ground and fighting every single point that could possibly compromise your position in the slightest wins. That's bad but it's not like there's any individual game theoretic move to win here.
(1) 99.9% of the time
Both Twitter and real life.
Well, that's depressing.
I picture the infrastructure necessary to develop AI as small and diffused, so easy to conceal that verfying a pause would be impossible. But I don’t know if that is correct.
If thatʻs correct, wouldn’t any treaty be a kind of theater?
https://mistral.ai/news/mistral-3
> Mistral Large 3 is one of the best permissive open weight models in the world, trained from scratch on 3000 of NVIDIA’s H200 GPUs.
Can anyone remember which classic Scott essay it was in which he talked about those annoying people with "argument checklists" who have already written down every argument that others make against their position and if you try to make one of those arguments they'll simply mock you by ticking it off their list while refusing to actually engage?
Anyway I'd like to say I'm surprised that the writer of that essay also wrote this one. But I'm not surprised, I know that we all have bad days and that sometimes our frustration gets the better of us and we lose our commitment to fair and reasonable engagement with the best opposing arguments and we just wanna air our frustration about their worst arguments. But this is definitely Scott on a bad day.
These two people never talk to each other. Supporter is reading blogs and arguments by Opponent arguing against Supporter2 who wants a unilateral pause and is just as intelligent and sophisticated as Opponent. Supporter then forgives all of Supporter 2's flaws while getting mad at Opponent (who has never met Supporter or heard the sophisticated version of their argument)
What's the limiting factor in slowing the development of AI? Is it compute entirely? Or a combination of things? DeepSeek seems to indicate that better coding could lower the need for compute massively while delivering the same results. I'm skeptical that those types of advances would be eliminated.
I'm far from an expert on this topic. And I'm not writing to persuade others. I'm writing because, outside of targeting large data centers, I don't think that the above argument is persuasive that a ban is going to be airtight or enforceable, just like nuclear non-proliferation didn't prevent actors like North Korea from developing a nuclear program. And nuclear non-proliferation only achieved what it did because countries were willing to go to war over the matter. Is there popular support for going to war with China or North Korea over a broken AI deal?
But I also haven't been a good rationalist and read The Book yet.
Reigning in data centers and the computing resources available to the most powerful AI seems a little less workable than reigning in, say, refinement of nuclear materials but it could probably be done for the most powerful centers.
I don't know the issues with reigning in distributed computing, though.
And speaking of distributed computing, another problem for me with comparisons to nuclear proliferation is that centrifuges capable of doing isotope separation and the supply of yellow cake uranium are rare things in common hands, while access to compute is very much the bread and butter of modern economies. Reigning in distributed computing seems a little harder than "winning the war on drugs" if most of America was allowed to legally grow marijuana. But we'd probably prevent a few key leaders like OpenAI from collecting capital for their next round of massive data servers.
To the extent that the issue is software related or training data limited, I'm deeply skeptical that any agreement will really prevent people from writing code or collecting data. So the question is about the ways in which they'd be slowed down.
Maybe part of the issue is that we're not really talking about an across-the-board AI pause, but just a restriction on AI that uses massive amounts of compute. Would it help any to phrase that goal more narrowly?
"I’m excited about debating those concerns with you."
Okay. Lets do that. Feel free to change my mind or tell me that I've misunderstood the landscape.
It is really fucking funny how Opponents in the comments are saying "but Opponent is right!" or "But [the argument that Supporter brought and engaged with] is right! [and conspicuously not engaging with Supporter's counterarguments." Is there some corollary to Poe's law that says satire of a particular intellectual tribe attracts members of that tribe, performing the satirized behavior?
Why not devote more of your attention to the Opponents who did engage?
A fair question! Frankly, I'm not sure why I engaged at all, given how arguing on the internet works. I have a sick man's compulsion.
I feel like this isn't even a property of every argument about AI pauses, it's a property of every argument about everything.
For most things there's a number of Level 0 Obvious Arguments, and then some Level 1 counterarguments, and then Level 2 counter-counter-arguments, and then a branching tree of different arguments and counterarguments.
But then every day there's new people joining the debate and starting off at Level 0, and that must be frustrating when you're already nine levels deep looking for someone to engage with your countercountercounter...argument and everybody keeps saying that dumb Level 0 thing that you thought you already dealt with.
Yeah, I agree. It's just particularly amusing for it to happen in the comment sections of this post.
For years, I've been playing around with the idea of what I call "argument maps". For any issue, you could have a graph, where the nodes are arguments for or against some position on that issue, or even just making an intermediate point as a sort of lemma, and the edges are "refutes", "rejoins", "counterexample", or what have you. A bad argument might have a lot of negative edges coming out of it; a controversial argument might have a lot of both positive and negative. Common arguments would have lots of edges; fringe or cutting-edge arguments would have very few. Major issues might have huge maps, while topical issues like "did DB Cooper escape" have much smaller ones.
I think of these as addressing exactly the problem you describe: someone walks into the forum with this One Simple Argument that they think will close the book on it, and they can look it up on the map and instantly see that, yes, it's already been addressed. ...Or maybe it hasn't! and we quickly add that to the map and everyone benefits.
One of the big problems with it when I first toyed with it was that the search would be fraught. If, for example, you came up with an argument for pro-choice that supposed that a woman might wake up one day to find a panda shackled to her and impossible to free without killing it, and that everyone would agree she was still on firm moral ground if she opted to do just that, the search wouldn't necessarily tell you that that's functionally the Violinist Argument. Since then, LLMs make me think this might be solved.
Other problems exist, such as making the edges stable enough to withstand question (people will disagree whether argument AB-3920 truly refutes AB-914c), and keeping the map relatively clear of spam from trivially bad arguments (yes, I've put some obsessive thought into formalizing this, you can stop looking at me like that). Most of my revisiting this comes from imagining all the time and anguish such maps could save.
I don't even necessarily disagree on the object level (mixed feelings on the topic) but this is sneerclub-tier bad faith garbage. Sorry to see that even Scott can't keep his equilibrium in these times...
I find AI very useful for some well-defined tasks, for which I have a completely clear idea of what I want and how I'd do it. I can write up instructions for Claude, check that it performs as expected and then automate the task. I've done this for a few things, and I get a speed up of by a factor of 3-5. I still have to give 1-2 rounds of corrections and maybe edit a bit myself. The AI is definitely a bit more capable this year than it was this time last year.
Claude Code is great, and it looks like there's a new interface coming out which will allow the AI to control the computer. Probably I'll be able to talk to the computer and have it *agent* on my behalf relatively soon. But... this still doesn't feel like the take-off of AGI. It feels like we'll get increasingly marginal improvements until we reach an equilibrium or until some new technology comes along. I'm not saying AI safety is not important. AI enabled scammers are going to be awful. But the idea of the AI building a new-to-science virus or nano-kill-bots or Terminators or going full matrix has receded into the background a bit in the last 6 months, I feel. We're learning the capabilities of AI as well as its limitations.
To me it feel like there's a few ingredients still missing - maybe they show up next month or in 10 years or never. Do other people feel like there's exponential improvement in capability and self-awareness and AGI just around the corner? If not, what are the main risks of AI?
I'm with you on this: "To me it feel like there's a few ingredients still missing - maybe they show up next month or in 10 years or never.
And yeah, I'm worried, because I have no clue when the ingredients will show up. And if they do, that would be world-endingly bad. So my worry is modulated by AI progress, but I still put substantial weight on "something real bad real soon."
My background is maths, and every few years there will be a new idea which is branded as a breakthrough. There's lots of excitement and people rush to try it on their favourite problems - 'this is the one that could change everything'. More often than not, it solves the original problem it claimed to solve and very little else. We don't achieve a transformative leap forward in capabilities. But there's a little hype cycle where people buy into the idea, try it, make incremental progress (or not) and then go back to their daily lives, and we end up 0 or epsilon progress toward the Riemann Hypothesis or PvsNP or whatever major problem. The nature of the big problems is that until you've solved them they're as far away as they always were.
I feel like I've gone through this hype cycle on AI - yes, it will disrupt lots of industries and as it becomes marginally more reliable and powerful it's going to replace a lot of bullshit (and some non-bullshit) jobs. But it doesn't feel conscious to me, and it doesn't feel like it's about to become conscious. It doesn't feel intelligent - it can reproduce things it's seen before (better than people) but I don't think it'll discover a cure for cancer (which is actually a different disease process in each person and a silver bullet that worked for all of them is Riemann Hypothesis territory). If misused it can still do harm, but that harm seems limited to super scammers on the internet with high probability. Am I missing something?
From a rational perspective, climate change is going to cost trillions in damages with probability 0.9. AI is going to cause hundreds of billions in damages with probability 0.5 and trillions with probability 0.05 - which should we focus on? (Both obviously, but I don't see nearly as much about climate justice as other causes here.)
> From a rational perspective, climate change is going to cost trillions in damages with probability 0.9.
Curious, but have you actually done any Fermi math here?
Have you heard the argument that warming will actually increase global agricultural production by bringing parts of Canada / Russia / Scandinavia into productionable climates?
I'm not very well versed in this, and this is why I'm asking from people who sound like they have a pretty high certitude on either side.
Very rough estimates (I'd imagine the AI could do much better).
The earth is a sphere - most of the mass is around the equator. The intensity of sunlight hitting the earth is variable at the poles, hence seasonal variation. I've been to Canada and Scandinavia - the land up there is... not great. It might become a bit more productive in a restrictive growing season - this is not a big win. Canada and Russia look massive on the map - that's because of distortion. Find a diagram of Russia projected onto Africa. Disruption to agriculture in parts of Africa is already a problem for products like coffee and chocolate. Overall, climate change is bad for established agriculture, and we do things in the areas most suited to them. New land opening up will not come close to replacing what will be lost in a 3C average warming scenario.
To be honest I wasn't even thinking about this - more frequent floods, rainstorms, droughts, heatwaves and snowstorms will damage infrastructure, and replacing this will cost hundreds of billions a year within our lifetimes. Add damages to crops, wildlife, human life, and you're getting up there.
The science is well established now. Being 'open minded' on this topic is to my mind about as defensible as being agnostic about evolution or the harms of smoking.
OPPONENT: How could we possibly know whether China has, in fact, paused, or vice versa? Will both sides agree to IAEA-style oppositional monitoring of all their high-end computer labs and top scientists? Even if they did lose their minds and agree to that, could such infrastructure even be put in place, given that AI research doesn't require rare and expensive materials and detectable testing?
I don’t understand what a “pause” would be - you can have computers, but you just can’t use them for math?
A lot of people saying "it could never work! China is not trustworthy! The US is not trustworthy! Etc etc". Well, the Soviets weren't very trustworthy either but somehow we all negotiated the Limited Test Ban Treaty, the Nuclear Non-Proliferation Treaty, SALT I, Anti-Ballistic Missile Treaty, SALT II, Intermediate-Range Nuclear Forces Treaty.....
That's because nuclear tests are easy to detect.
You'll note that there's discussion on this topic way upthread.
Short answer is: Nuclear weapons did not promise to automate a great deal of work. Economic equation is completely different here, *and* countries involved hardly acknowledge AI as an x-risk. There are a lot more incentives to defect and build AI in private for military uses, which some incl. myself worry will mean worse safeguards.
Also, enforcement of such a treaty may necessitate use of force, which could easily devolve into nuclear warfare when it is between two nuclear powers.
I'm happy to hear counterpoints.
Well, the main counterpoint from people most invested in this is "unless we try, we'll all surely die", so it's essentially the Pascal's wager.
The Soviets and the US back then were a hundred times more trustworthy than the US today.
China is not losing the "race", btw. They are just not building frontier models as fast, which is arguably a good thing given how wasteful a lot of them are, but the rest of the race includes robotics, manufacturing, open source, and scientific research, which they put out in good volume. Worse comes to worst, they can just copy the Americans.
If the race is towards RSI and superintelligence, then the first frontier model to pass a certain threshold wins. If it's just about who has slightly better robots or something, see https://www.astralcodexten.com/p/most-technologies-arent-races
Many opponents may be cartoonish. But I feel like I haven't heard many supporters say things that make me fall in their camp either. A few things I don't feel like I hear about:
Enforcement at scale. The bilateral verification argument addresses whether China would cheat on a treaty; a real concern, but a narrow one. A training pause also requires compliance from every company, research lab, and individual with sufficient compute. That's a fundamentally different and harder problem that doesn't get much serious treatment.
The economic disruption argument also gets no serious treatment. Historically, exponential gains in technology are what drive long-run prosperity. Without that, economies stagnate as they get pushed closer to producing no more value than the baseline. And AI doesn't need to be delivering value today for a pause to be damaging; the expectation of future returns is what's currently driving investment and employment in tech. Telling companies that are deeply invested in this technology that they simply can't continue is an extraordinarily heavy-handed move, with real costs to livelihoods and businesses that have built around the assumption of continued development. Killing that expectation, even temporarily, has real consequences.
And what does success look like? Pause advocates rarely specify what would change during a pause that makes resumption safer. If the answer is "alignment research matures," that's not a clear goalpost, and also doesn't have any particular timelines attached.
None of this means a pause is definitely wrong. But calling this "every debate ever" implies the opposition (and only the opposition) reduces to a strawman. I haven't seen good answers from supporters either.
Scott, maybe some version of this would be a good question for your annual survey:
If you could actually have your choice be manifest as reality, given the following options which would you most prefer?
1) AI disappears like it was all a dream and doesn't come back
2) We keep the AI we've got, as it is now, but there's no further development of AI, period
3) Chatbots and stuff (protein-folding etc. too?) may continue to be trained by inference, but we negotiate bilateral/global agreements on no more developing high-powered AI that could lead to super-AI until we figure out how to control it (basically the SUPPORTER's position)
4) There can be developments toward super-AI, but there also has to be development on AI alignment and stuff like that
5) Laissez-faire
I had something like this on the 2024 survey. 72% of people wanted to keep existing AI, 28% wished it was gone. When I said to ignore x-risk and imagine AI would stay the same forever, 87% wanted to keep it and 13% wanted it to disappear.
Ah, that's interesting, and now that you mention it I vaguely recall a question like that.
Partly I see SUPPORTER as occupying the center rather than an extreme of a spectrum on sentiment toward AI.
But partly I was figuring there are probably more people on the anti-AI extreme side of SUPPORTER than we tend to realize, which the numbers bear out: 28% and 13% are both pretty large figures, I think. (SUPPORTER could have used this as a way to show how moderate his position was.)
Like, even in a roomful of people where nerdy computer programmers are overrepresented, you're way less likely to meet a left-handed person than to meet someone who wishes AI would disappear forever.
As someone who doesn't follow this discourse particularly closely, the first thing that came to mind when reading this was the AI2027 scenario that this blog promoted heavily and was probably the first exposure a lot of people got to the concept, which did actually include a unilateral pause.
If you're thinking of the same thing I am, there was a part where the US estimated it was X months ahead of China and so focused on safety research for X-1 months, which I think is also importantly different from having no plan and just letting China win.
I'm afraid I have a lot of sympathy for the position you're straw-manning here, and rather less for the one you steel-man.
A negotiated bilateral pause seems so unlikely - both in terms of China agreeing to it and, more worryingly, in terms of them sticking to it if they did agree - that
1) I think it's reasonable to suspect people saying "we want a negotiated bilateral pause with China" as using that as a stalking horse for a unilateral pause, even if they deny it.
2) Even if they're genuinely acting in good faith, I think their actions do much more to increase the likelihood of a unilateral than of a bilateral pause.
Or, to put it another way: if people in the US want to talk to their opposite numbers in China about setting up a bilateral pause with a sufficiently rigorous oversight mechanism to somehow ensure that a nation of a billion people with extreme state secrecy is not secretly violating it, they should go ahead and talk, and if they actually do get in-principle agreement then we can discuss the merits. But until then, the US's approach to AI development should be "damn the torpedos, full steam ahead".
The statement
> It's actually quite simple: [First,] company leaders agree to a conditional pause, [then] US and China agree to a conditional pause, [then] international pause. Notice how no step here involves "US unilaterally pauses"
strikes me as naive - I think a less unlikely scenario is
"It's actually quite simple: [First,] company leaders agree to a conditional pause, [then] US and China agree to a conditional pause, [then] the US pauses and China keeps on working on AI while saying that it isn't".
(Or, possibly, both sides do that).
An even more likely scenario:
First company leaders don't agree to a conditional pause, then the US and China don't agree to a conditional pause.
Forget convincing Xi Jinping, first show me you can convince Sam Altman.
While everyone else is arguing over whether this post is a bad form strawman, I propose the hypothesis that Scott Alexander deliberately wrote a post that would attract controversy to signal-boost the idea of pausing AI.
I on the other hand, propose that Scott is secretly an accelerationist and deliberately wrote a post that would attract controversy to make the pausers look bad.
Actually, I'd be more worried about the US refusing to keep their agreement than China at this point
This would be a stronger piece if the anti-pause position was not strawmanned so aggressively. There are many strong anti-pause arguments that are completely ignored, such as:
-"We already face many existential dangers, such as nuclear weapons, synthetic biology, and global pandemics, and powerful AI is the only one that can actually prevent the others"
-"Enforcing an AI pause would require a level of totalitarian government control that is a more real and concrete danger than the hypothetical risks from powerful AI"
-"Your claim that we can reap the same benefits to health and human development without powerful AI is implausible"
-"We are on track to get powerful AI before China, and it is advantageous for us to achieve powerful AI and then wield it to diminish China's power, or potentially dismantle their authoritarian regime"
-"You can't even clearly articulate the criteria for 'safe' powerful AI, so why should we trust your reasoning on 'unsafe' powerful AI?"
-etc...
I don't even agree with all of these points, but any real advocacy for an AI pause---even at the proposal stage---needs to seriously engage with all of them.
I am against a pause because I value economic growth and the possibility of us achieving major medical and scientific breakthroughs that could put a serious dent in things like cancer research over a speculative first principles argument. It does not help that everyone who makes the speculative first principles argument seems to come from the same social community where the same people (Eliezer Yudkowsky) are viewed to be cool. Normally I would value what superforecasters and technical people are saying about this stuff, but most people are really bad at thinking outside their local status hierarchies.
If the alignment argument turns out to be poorly thought out, and we sacrifice hundreds of millions of lives (through delaying the medical breakthroughs that advanced AI could achieve) in favor of the argument, the people who made it are going to go down in history as villains.
Conservatives value coalitional loyalty more than liberals. This is backed up by Haidt's moral foundations research. If important tech figures to the coalition like Musk and Thiel oppose AI pauses, then other people in the coalition are going to try to mirror their sentiments. The quality of the arguments is going to be sublimated priority wise to being loyal to their 'friends' (tech right conservatives) and being against their 'enemies' (EA types who obviously come off as liberals and anti-Trump).
As a side note, I also thought it was very strange that pro-pause people were not extremely pro-Kamala in 2024. Democrats are the party of educated people, neurotic people, people who read a lot, people who are agreeable and high trust, and people who are skeptical towards big business. Getting Trump to successfully execute a pause is going to be much more difficult than getting Kamala to do one would have been.
The tech right (not "conversative" in any sense other than the colloquial American one meaning "Republican") were quiet before and will be quiet again when their guys aren't in power––it's about their bottom line, not a long-term principled stand. But you're basically never going to talk these guys out of it, I agree.
This is Scott's first unfair and frankly stupid post in a long time. He might have one idiot "opponent" friend who is this dense, but this is basically the definition of how to do a straw man. The real opponents around here don't doubt that an enforceable agreement is logistically doable, but those logistics necessarily require Chinese compliance inspectors to be up in all of our significant computing devices, just like we would be up in theirs. Like, how else would enforcement work? For most of such opponents it's already one step too far for the US government to determine what code can and can't run on privately-owned hardware in America. They don't like "Sorry, Dave, your government won't allow you to execute that sudo command." But a deal like this would also give the Chinese Communist Party the same kind of access to what our computers do. This wouldn't be like some bilateral nuclear non-proliferation treaty, where much of the enforcement can be done by dudes with Geiger counters and isotope analysis kits. Yes, by now there is a lot of plutonium in the world, but inspectors just have to confirm where it is. With computers, I presume it's the commands executed that would be the difference between compliance and non-compliance. There would be arguments like "we weren't training an AI; we were just using the computer to simulate neutron flux in a nuclear explosion." "We don't believe you - show us the code!" "No." Your supporter acted like it's super simple. We just make a treaty, something will be declared illegal, that thing will henceforth not happen, and everything else will be as before. But that's just incoherent when we're talking about an enforcable, mutually verifiable deal about what goes on in a major country's computers.
MIRI's proposal is simply "you can't have too much compute all in one place," which isn't airtight, but it sure buys you a big delay. And it doesn't require monitoring that's invasive on the level of "seeing what computations are getting done." Just "making sure there isn't too much of one thing all in one place," much like plutonium. And of course there are quibbles around the edges, and any enforcement regime is imperfect (what if it's possible to make a seed ASI on a TI-83, or whatever), but it buys us time.
I am confused about how one buys a big delay without either monitoring what code runs on privately-owned hardware, or banning AWS. It's not like it's impossible to engineer training runs that can be distributed across every server farm in the country.
This seems like an especially stupid strawman, and even if you wish to argue it is somehow accurate, I don't know what anyone is supposed to take away from it beyond "Boo outgroup!" I am used to better from the author of Slate Star Codex.
This is not strawmanning my bruhs what is going on. He has deftly steelmanned it by the supporter presenting both sides of the argument.
Just because the steelman is not from the opponent does not mean it's a strawman post. He's just being clever and funny.
Why can no one see this?
"In every debate, my opposition has literally one argument and the listening skills of a megaphone, while my position deftly understands both sides" is generally understood to be a strawman.
He really could have saved everyone time by just posting a Soyjack vs Chad meme.
Arguably your worst post ever. You could have put it more succinctly by saying, "I am very smart and my opponents are very dumb. One may notice my refusal to engage in good faith on this topic, but trust me, that would be unnecessary as I've already established that I am very smart and my opponents are very dumb."
I don't think China will give the power to the US (and viceversa) to access to an observational power so strong that they can see whether one of them has an underground or unassuming data center, while again yes this will disrupt more us than them since they are at disadvantage now and also more interconnected with the state, so a state based underground competition would help them. Also I don't know how can you stop people from working on better model since those are ideas.
In my view, we should not wait for China or the US to lead the world to AI safety because they both seem pretty focused on winning the AI dominance race. Rather, small, wealthy countries and the private sector should try to build the IAEA-type institution that we seem to need and then try to bring the US and China to the table through the use of carrots and sticks. https://abefrohemann.substack.com/p/the-useful-fire-part-1-strong-international
I do have to disagree with Supporter here. There's not actually any need for a bilateral agreement except to the extent it appeases domestic opposition. The "prize" for developing super-intelligent AI is destroying the world somewhat sooner than would have happened otherwise, so it's in every actor's interest to unilaterally stop development. Either everyone else stops too and you get the good outcome, or they don't and you still die, but either way you didn't lose anything for not destroying the world yourself.
Rather than passive-aggressively strawman your ideological opponents, I think you should engage with their arguments.
Perfectly timed - Bernie Sanders just completed his AI Safety tour and announced a national moratorium on data centers. So we can see how this cashes out in practice.
https://apnews.com/article/data-centers-ai-electricity-sanders-aoc-65651bd28c3d911d18eeb46cd54f4c75
A bilateral pause between the US and China might be interesting. But I wonder how you'd organise a true multi-lateral pause? And how you'd enforce the cartel.
Unfortunately, I found myself kind of siding with the opponent. I seriously don't trust the Chinese to hold up their end of the bargain. Sure their scientists are reasonable people, but their politicians are running the show. Business famously serves government over there not the other way around like it is here (which has its own set of problems of course). They have a strategic culture that values deception back to Sun Tzu, and their politicians are much more intelligent than ours. And no, I'm not saying they're evil--I wish our politicians were that clever and intelligently nationalistic!
I think rationalists have kind of a blind spot around deception--I know we're all on the spectrum but I was saying here how we shouldn't trust the Chinese numbers on how much money they're spending and people didn't believe me...and then their models started lapping ours. Of course they could have also used the money better--after all they are still free to pick their academicians by competence.
Of course if you really think AI is going to blow up the world fewer people working on it is better, but I'm afraid that ship has sailed.
The progress of AI cannot be stopped by any person or any government, the author's fantasies to the contrary notwithstanding.
Lol, this article is such obvious hypocrisy. Politics really is the mind-killer.
“No, stupid, your adversary might agree because they realize the agreement would involve you forfeiting your head start. Then they can start again later ”
-🧐
“Yes, country that depends on private industry to advance in AI, adopt a bilateral agreement that forces you to suppress your private industry in a way that will have an irreparable chilling effect on investment. Dont worry about non-compliance from your authoritarian, centralized, command economy adversary.”
-🧐
But you sure did a good job beating up that strawman!
Warning: please give arguments for assertions, especially insulting ones, or you'll be banned.
Does the second comment not count?
[But either way apologies for snark.]
Why would anyone trust the US though?
Or me as opponent:
SUPPORTER: America needs to start talking to China to come up with a bilateral agreement to pause AI. The agreement would need to be transparent, mutually enforceable, and…
OPPONENT: If we do this China will mostly likely just lie and keep doing it anyway. In the low chance that China is honest and really does pause AI, a bunch of stupid populists in the USA will insist they are lying anyway no matter how much evidence to the contrary there is and will probably arbitrarily shred our agreement with China and then we will do it anyway. The only thing that will change this dynamic is for an AI Chernobyl to happen or for people to *see* what it looks like when your drop bombs on Hiroshima. Pray AI just doesn't end up doing this or that the accident happens sooner rather than later while AI is less powerful.
SUPPORTER: So, we can't even try?
OPPONENT: Sure, try if you want. Want to make a bet on it working?
"If you think someone is demanding a unilateral pause, I think you have a responsibility to say who it is you’re talking about. If you can really find someone like this, I’ll criticize them just as hard as you are."
I expect Supporter would criticize Bernie Sanders and AOC for this proposal aimed at unilaterally restricting AI data center construction in the USA: https://www.sanders.senate.gov/wp-content/uploads/ELT26209.pdf
FWIW, I appreciate the spirit of the post (I have noticed the same thing), but it is funny to see this post on the same day as that proposal.
If someone in 1990 had accurately predicted all of the most dire risks associated with the modern internet (surveillance, hacks and viruses, attacks on hospitals and infrastructure, etc.), I might have been somewhat convinced that we should pause development or pass heavy regulations to ensure it's never used for anything important like financial infrastructure. Now that those (completely valid) risks are in full context, I'm glad that didn't happen.
This post got me to finally post my more extensive response to the objection I hear event more often about international treaties - that we need to work out the details before doing anything about them. And as the title says: "What Exactly Would An International AI Treaty Say?" Is a Bad Objection.
https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad
No question the Chinese system is cruel.
That judgement does not negate the fact that they have a clear eyed view of the future.
We judge and then dismiss at our peril. One must see things clearly before one can make a rational decision. Good and bad are context dependent.
You just demonstrated the problem with AI—who’s values?
Rules without consequences are suggestions. What are the rules AI currently live by? They are all fungible as AI is language at scale.
AI needs hard consequences that can not be gotten around with language.
The consequences are physics.
Facts not opinions. No one will like it for just the reason you have demonstrated. We all have our own opinion of the “truth” but the math doesn’t care.
If we are to bind AI to the truth we must first except that our opinions are not truth and that any rules whether RLHF or Constitutions are suggestions that an AI will circumvent at will without humans being able to notice.
Tie AI to physics and alignment is hard coded.
The AI we’ve built has a refusal clause with consequences the system can not circumvent.
It’s just math.
IMO this is a strawman. I think many people who have thought quite hard about this are indeed in favor of a unilateral pause. I certainly am! And I think the arguments for a unilateral pause are strong and can stand on their own. I feel like this post will make all conversations on this topic much harder.
I don't think you understand- what's stopping China from rushing ahead while we unilaterally pause? you're just not making any sense
(/s)
Something that I didn’t see in the essay (though maybe it’s in the comments) are the economic concerns of a pause. P/E ratios across many tech giants price in transformative productivity gains, and I’d like assurance the economy and geopolitical standing of liberal democracies is secure with AI paused.
Why would we throw away our lead? It's the only thing protecting us.
Scott, you forgot the most important point Opposition makes. I’ll be as generous with them as possible here:
“I just can’t imagine how we’d enforce it. I know I haven’t spent more than 3 seconds (generous) thinking about enforcement, and in some platonic way I understand that proponents of a pause likely have given it more consideration than I have. And that even if I can find faults in those proposed mechanisms we can collaboratively work to improve them, especially if we can get past the first step of thinking an effective one would be useful such that more people take the mechanisms seriously and we can crowdsource some more of the work needed. I get that, but I really don’t see how we enforce them. After all, the Chinese government has been known to lie! They have lied in the past. They take IP from US companies. And there is absolutely no way systems designed to track tangible physical data work if they could lie. You expect them to just pinkie promise? A unilateral pause is what you’re proposing.”
It's technologists and "effective altruists" who are the worst offenders of this. Foreign policy people and military people understand perfectly well the default extinction threat of creating something more intelligent than humanity.
The pattern you're drawing out is that both sides are really arguing about which form of losing control scares them more. The opponent hears "pause" and pictures falling behind a rival who won't stop. The supporter hears "keep going" and pictures a system nobody fully understands getting harder to steer. Same fear, opposite directions. That's why the loop never breaks - they're not actually disagreeing about verification mechanisms or treaty enforcement. They're disagreeing about which future is more frightening, and nobody changes their answer to that by hearing better arguments about chip monitoring.
The pattern you're drawing out is that both sides are really arguing about which form of losing control scares them more. The opponent hears "pause" and pictures falling behind a rival who won't stop. The supporter hears "keep going" and pictures a system nobody fully understands getting harder to steer. Same fear, opposite directions. That's why the loop never breaks - they're not actually disagreeing about verification mechanisms or treaty enforcement. They're disagreeing about which future is more frightening, and nobody changes their answer to that by hearing better arguments about chip monitoring.
Yes, you are right China is a human rights violator. Yet, no one exports more suffering than the US.
Endlessly insisting on who is morally superior is the proof itself.
Unaligned AI is a species ending event already well underway. Perhaps we can settle our scores after we address that.
China’s model are not as good as US frontier models they are, however cheaper.
Eighty percent of all AI start ups in the US use Chinese models. So who is propagating the corruption?
The profit motive is the terminal attractor of all AI’s if we don’t address that then expect more bounded chaos, here and in China.
We’re arguing about the wrong things while China spreads its “good enough” models. AI is one area where good enough isn’t.
Pause for what?
this whole annoying post goes by without mentioning the possibility that china *cheats*?
> Or is your problem that you don’t trust China to stick to an agreement, once signed? Because we agree that an agreement has to be mutually transparent and enforceable. We have some ideas for how we could have a light-touch approach to monitoring Chinese data centers - of course, they would get to monitor ours in the same way - and actually the math mostly works out and we think it would be less intrusive than other things that have worked in the past, like nuclear monitoring.
Quite a lot has been written about the topic, and how to solve it. "What if China cheats" is not exactly a new thought, and people have figured out how to minimize that possibility.
Olympic level strawmanning
I don't trust the Chinese administration to keep their word and I don't trust Donald Trump to keep his word either, and I don't see how such an agreement could be enforced. Even if the leaders of both countries were sincere they can't keep a scientist from thinking about how to improve AI, and they can't be certain what every one of the billions of GPUs in both countries are doing. And China and the US I'm not the only countries in the world, what about North Korea and South Korea, and Japan,Taiwan, India, Israel, Canada, and the European nations? You might be able to slow it but there is no way you're going to stop the on running rush of AI. John K Clark
Another anti-tech, illiterate buffoon.
There is nothing wrong with AI.
There were people like you 200 years ago who were worried about the industrial revolution too. You're historical mob even burned down factories in a desperate attempt to force humanity to go back to the safety of small farms.
And forty years ago, we had to deal with morons who thought email was going to be a problem.
It never ends, because backward people who fail to grasp basic economics, exist in every time period throughout human history.
And it's doubly worse amongst the old, who just cannot wrap their heads around anything knew.
Embrace AI.
Go build something. You'll be happier engineering than writing nonsense everyday.
Does it not set off any alarm bells that the people warning about AI risk are, outside of this specific case, some of the most pro-technology people on Earth? Or that among them are the world's foremost researchers, developers, and CEOs of AI, the people who developed it to where it is today?
The concern is that any pause, unilateral, negotiated or otherwise, would cut into profits and undermine the tech oligarchs plan to rule the world. Wish I was exaggerating.
"Universal fear", said the cactus person.
"Transcendent spite", said the big green bat.
a lot of comments i see under this post are still “multilateral pause is impossible and china will surely defect” which kind of justifies the post? it’s not necessarily true but it’s easy to use in an argument.
i live in china, we don’t even believe in AGI here.
i really think most people are against a mutual ban because the US is pretty far ahead. when you think about pausing the largest research centers, it feels like pausing really fast progress, and it feels bad.
The point is that "X is unlikely to happen" isn't a counter to "X would be a good thing"
right?
The controversy of this post slightly confuses me; a lot of comments are saying things that are technically true, but aren't points against the post.
The most important reason for not pausing AI is the rapid diffusion of new AI-powered forms of creative human agency will unlock more net benefits for everyone than the harms that come with it. Conversely the net harms of a pause that locks in early advantages will outweigh the benefits of attempting to supress that rapid diffusion unlock.
The fact that these dependencies and contingencies are unpredictable may be epistemologically disturbing for certain ways of thinking, but that is not sufficient cause for preventive action. Instead it is a symptom that this certain way of thinking about things has practical limits.
Anyone who says "It's actually quite simple" is him/herself quite simple.
Let me be a contrarian and say that a unilateral pause for the USA would let China to slow down and focus on safety more than they do now. Being non-American (and non-Chinese) I'd say that China is also less known for military interventions ("bombing to the stone age") than both US and us, seemingly more interested in AI safety than US government is — and doesn't seem to subscribe to the ideology of "you can only get things when you contribute to the economy": in post-ASI economy no human is able to meaningfully contribute.
I'm not sure the reversed situation (when China stops) will lead to the similar situation, as currently US government doesn't seem to be interested in long-term AI safety at all. Move fast and break things is not a good idea when we have some things we really don't want broken.
Bernie and AOC are literally pushing a bill that pauses AI unilaterally. I understand they may be playing N-dimensional chess. But it seems like there’s definitely an argument that not nobody wants a unilateral pause.
Development of AI capabilities is not as bad as author think.