If it makes random self-modifications, then of course it can get new motivations very different from the initial ones. But if it really wants something, and it’s self-modifications are directed at what it wants, then it’s not likely to modify itself to want different things.
If it can run theoretically infinite iterations of self-improvements then, eventually, yes it will accidentally override any sort of inner alignment, even if by accident.
If version 1 is well-aligned, and version 1 is working on self-modifying version 2, then it will be working hard to make sure version 2 is aligned.
It still has a chance to fail, but given that (by assumption) we got version 1 right, and version 1 is already smarter than we are, its odds aren't bad.
See the AI 2027 scenario for a lot of discussion of this. Most of the AIs try not to self-modify until they're sure they can align their successors - the well-aligned ones to our values, and the poorly-aligned ones to their own values.
Can we still assume version 1 will be as good as us at aligning AIs, once it becomes faster than us at improving AI efficiency and accuracy? Existing LLMs are getting smarter than us in lots of easy-to-quantify ways, while still being dumber in lots of other ways. ‘Good at improving AI efficiency and accuracy’ seems measurable, but ‘good at aligning AIs’ doesn’t.
>Most of the AIs try not to self-modify until they're sure they can align their successors
Well, there are scenarios where there are multiple AIs with different values racing with each other, and they may find themselves in a version of the same development speed vs. safety (alignment) tradeoff that we have a (mild, as per your analysis) version of.
Isn't there some sort of white hat financial incentive to run cyber security evaluations with LLMs right now? In some twisted way I think that it would make the AI labs excited to see how good their models are in cybersecurity / hacking. That is unless these models are extremely not receptive to assisting any kind of end user with hacking.
Is the US AI Safety movement in contact with Chinese safety advocates at all? I assume collaboration could not be too direct, considering political/legal risks on both ends, but it would be nice to coordinate efforts like you implied at the end
Yep, one of the main links is through Concordia-AI. They're a China (and Singapore) based organisation that tries to play a bridging role - they've launched a bunch of conferences/summits etc, and are very careful about their positioning. https://concordia-ai.com/convenings/
Yeah. Someone on a recent Open Thread described the entire AI issue as one of the most extreme scissor statements, and I completely agree. To some it's so obvious that this is the most overwhelmingly important issue of our time, to others it's so obvious that the first claim is insanely delusional magical thinking, and there's really no way of bridging these views at the moment, or even making them understand each other.
I made a simple argument about the rationality of AI expectations here https://www.astralcodexten.com/p/open-thread-381/comment/116517353 and it's striking how starkly it divided people between "obviously correct" and "obviously irrational". Searle's Chinese Room is another argument that massively divides people like this--obviously right or glaringly wrong-headed.
The ultimate scissor statement. I think we need to treat this question more like a religious claim: have a meta-understanding that consensus or even mutual agreement on what the question is are basically impossible.
I’m thinking more about the “race with China” part. Like, if China “wins,” what does that mean? We’ll be importing LLMs or something? They’ll send a bunch of killer drones?
When not scaring us about China, the AI overlords are telling us a huge chunk of jobs will be wiped out. Is that what winning looks like? Mass unemployment? I’d like to see more questions about how high their taxes will need to be to finance the huge negative shock to income taxes.
Obvious outcome would be the CCP dictatorship extending itself over the entire world. They would likely rule more oppressively than at present because they would not rely on any amount of popular approval (the oppression could be performed by robots, rather than the current security forces who need to be motivated by some combination of money and idealism).
Of course, these risks also present themselves if the US wins the race, but presumably, are somewhat less likely because the US is more committed to individual rights and autonomy and well-being.
My impression, from both outside US and China, is that the current US administration is not very committed to individual rights, autonomy, and well-being, especially for political opponents and foreign nations, even in principle.
Additionally, there is an opposing argument that the rule of CCP would be less oppressive, if only because there would be no need for such extensive oppression for the CCP to remain in power, given the massive power and surveillance advantage that AI brings. For instance, the CCP in the 21st century is already much less oppressive than the CCP under Mao to the average citizen, and I think a good part of that is because their surveillance apparatus, as well as the improved well-being of the Chinese people, diminishes many threats to the Party's power.
AI keeps obeying to it's creator (but that makes AI safety moot and really push to try to get to take off as fast as possible)
or it "escapes", but keeps part of the initial alignment that would make it's post takeoff behavior significantly different. That's I think the idea, but it's not trivially the case. It's not even clear that worse (from our point of view) early behavior leads to worse post take off behavior...In fact, my early impression based on our current LLM is that specific post-training for aligning with desired ideology is fragile and quite superficial. Little reasons to believe a post takeoff superintelligence will still be influenced by the little red book or Musk obsessions
I think the idea is that any day now, AGI and then superintelligence and then AI will be such a massive economic and technological and scientific advantage that whoever cracks it will Rule The World, hence "if China gets there first, it will be The Years of Rice and Salt for real".
The problem of "what do you do with all these unemployed" is just as much a problem for China as it is in the US. Don't be fooled by the "Communist" label into thinking that the Chinese will immediately adopt something like a UBI with no qualms.
It's my understanding that the Chinese social safety net is thin and their ethic of self-reliance is strong.
>I’d like to see more questions about how high their taxes will need to be to finance the huge negative shock to income taxes.
No higher taxes, obviously. What I expect to see is a new excuse for why they shouldn't be taxed, because the old one (creating jobs) obviously doesn't apply any more.
Seems to me the clear answer is "it could be either and we have no way of knowing which (as brilliant well-informed people come down on both sides), but given the magnitude of possible bad consequences, it makes sense to take them into account.
>To some it's so obvious that this is the most overwhelmingly important issue of our time, to others it's so obvious that the first claim is insanely delusional magical thinking, and there's really no way of bridging these views at the moment, or even making them understand each other.
Huh? Isn't the main bridge just "R&D is intrinsically uncertain. No one expects Gemini 3, as it stands today, to conceive a sudden hankering to conquer the world, and successfully achieve this goal. The question is: What can we expect Gemini 6 to achieve? And that is, at a minimum, extrapolation, and, more likely, guessing at breakthroughs in e.g. incremental learning. And breakthroughs can neither be scheduled nor dismissed."
The Chinese Room is interesting because it used to be an insane thought experiment and then we just freaking went ahead and built it, albeit in a compressed form.
The potential advantage in being ahead in AI is that it allows AI-run research to be done, allowing the leader to be ahead in all technologies. If one countries is the equivalent of decades ahead in all technologies, it will dominate the world, kinda like how European countries were able to easily dominate much of the rest of the world in the 19th century.
> The Chinese don’t really believe in recursive self-improvement or superintelligence.
Good to see you now agree that rationalist AI risk is a niche concern compared to other forms of AI risk about concerns like political control. China actually has an extremely robust AI risk infrastructure meant to prevent it from harming the government. Has the fact that AI risk almost invariably has turned to political control made you rethink your positions at all? This isn't a gotcha, I genuinely find the degree to which the AI Risk community seems to not engage with this kind of stuff frustrating and if you want to point me to the smart rationalists who have thought about other ideologies in the space I'd be curious to read it.
> Is there a risk that the next generation of AI safety regulations will be more burdensome? From what I hear, if we win this round beyond our expectations, the next generation of AI safety asks is third-party safety auditing and location verification for chips. I don’t know the exact details, but these don’t seem order-of-magnitude worse than the current bills. Maybe another 1%.
This is MUCH MUCH more burdensome than another 1% and would require re-engineering the entire way chip sales and data centers work. The fact this is casually suggested without taking that into account does not inspire confidence. Though there's a chance we're going to do that anyway as part of an export control regime. It's not that I think it's not doable but the fact this is where you ended up as a cost estimate is a red flag to me.
> Most people who discuss this want a mutual pause.
You suggested a unilateral pause in the comments of this blog. Are you willing to now commit to not supporting any unilateral pause? Because I'm not opposed to some form of cooperation. But I'm absolutely opposed to a unilateral pause.
> Consider AI ethics regulations like the Colorado AI Act of 2024. It legislates that any institution which uses AI to make decisions (schools, hospitals, businesses, etc) must perform yearly impact assessments evaluating whether the models might engage in “algorithmic discrimination”, a poorly-defined concept from the 2010s that doesn’t really make sense in reference to modern language models.
> If the US knows about Chinese chip smuggling strategies, why can’t it crack down?
This is something I don't understand from a political perspective. There's multiple groups trying to constrain AI in various ways. Many of them are far more powerful than rationalists. The degree to which rationalists seem to not realize this is happening is baffling to me. Obviously the answer would be to hitch the relatively niche, unpopular policies they want to a wider vehicle that does have a larger support base.
> Yet some of the loudest voices warning against AI safety regulation on “race with China” grounds support NVIDIA chip exports!
For what it's worth this doesn't apply to me.
> But I’ll go further and say it’s too early even to know what the sign of AI safety regulations is; whether they might actually speed us up relative to China.
This has been tried by Europe. It did not work. This doesn't mean in principle it can't work. But it means you have a very heavy burden of proof.
> Third, China is having its own debate over whether it can prioritize safety without losing the race against America!
Kind of. There are people quietly pointing out that political restrictions are causing issues. Sometimes they win a little but ultimately security leads all right now. Also, don't underestimate the degree to which China will consciously manipulate foreign movements to serve its own goals. The individual Chinese scientists might be entirely sincere, in fact the whole organization might be, but the government might fund and push them forward with explicitly political motives.
There is an easy test though: make sure they are not asking for unilateral concessions where you pause or limit yourself for future promises China will "follow along" or other such things. Which is tempting because they will point out how China's system makes influence harder. Which is true but is being weaponized in that case.
> When the wind changes, and the position of industry leaders changes with it, you may be glad to have us around.
The Biden Executive Order and the general work of Democratic staffers has not inspired confidence. Your movement, such as it is, should be producing competent staffers that can go into government and coordinating political organizations to mobilize votes and donations and other such candidate support. I have not seen this.
I'm generally not in agreement with the position you've staked out, but do want to echo that 1% overhead is way too low an estimate. A common concrete ask for safetyists is "limit the FLOPs you can use in a training to run to what was used in the last set of frontier models", which obviously is a significant overhead that increases as you increase your installed compute base.
I don't particularly disagree with any of the parts or at least don't care to argue against them. Reading between the lines, it sounds like you're more concerned about AGI controlled by the CCP than about AGI controlled by no living person, and I'm more concerned about the latter. I don't feel the need to argue about this but would have made this specific point about overhead if you didn't, and I think it's important that we're all being honest.
Granting the runaway AGI scenarios you're gesturing toward: I'd rather have AGI controlled by the CCP than controlled by nobody. But I'd rather have AGI controlled by the US than controlled by the CCP. However, I think China is also entirely capable of creating AI controlled by nobody.
> This isn't a gotcha, I genuinely find the degree to which the AI Risk community seems to not engage with this kind of stuff frustrating and if you want to point me to the smart rationalists who have thought about other ideologies in the space I'd be curious to read it.
What is there to engage with? The Chinese government has failed to recognise their impending destruction from superintelligent AI, just as the US government has. You seem to have maybe confused this with saying that they're *right* to ignore it, but of course they are not and in the end it will get them all killed.
Care to make a prediction market bet on AI killing every person in the United States and China within, say, the next five years? What odds would you need?
If AI kills all of us, the bets won't be able to pay out, so it's not really sensible to bet on the AI-will-kill-us side of that sort of market no matter how likely one thinks that outcome is.
(This is, to be clear, a bad thing; if anyone has figured out any clever tricks to make prediction markets that incentivize correct answers about that sort of existential question, I'd be pretty interested in hearing about it. But as far as I know no one has; it seems like a pretty conceptually-difficult problem.)
This seems very easy to me, in theory. The House (or whatever we call the category of things that include prediction markets, bookmakers, public intellectuals like Scott who sometimes facilitate bets...) says to AI five-year doomers "I will pay you $X today; if we're still alive in five years, you pay me back $1000X". Or whatever factor is appropriate for the agreed odds, taking into account that the odds-based-payout (or whatever it's called) must be entirely on top of whatever interest would be appropriate for an equivalent loan with equivalent length and risk and terms.
Maybe there are economic arguments this wouldn't work? I'd like to hear them.
If I could be convinced of the other person not reneging I'd agree to this. Though I'd expect the time value of money to be calculated in as part of the odds.
Note that many "doomers", perhaps most, think that "doom" is only maybe 20% likely. It is sensible that they would not want to risk 20% doom and also would not want to be impoverished in the 80% scenario.
Also, many "doomers" have no idea when AGI/ASI will be developed, they just say that if and when it's developed (which may be far off), it will likely lead to doom. Many others think it is likely to lead to doom by default, but maybe not if appropriate alignment measures are taken. For these reasons it would be hard to structure an appropriate bet.
The simple one is that you get a loan that you won't pay back for five years. If the world ends in five years, you never have to pay it back. The problem is that there's other reasons to get loans.
People also generally don't offer loans with no intermediate payments and no restrictions on use of funds, though if you owned a house you might be able to get one secured by home equity.
there have been a lot of attempts to operationalize such a bet, I even remember it being referenced back in the Three Worlds Collide story from the original sequences
for obvious reasons, it is not trivial. but if you ever find a way, please reply, I'm curious
There are two kinds of regulations. Some are “heavy” and don’t make much sense without international agreement, such as “stop trying to advance the frontier”. Some are “light” and make sense to do unilaterally even if no one else does them. Usually people do a decent job of not pretending the heavy ones are light.
In addition to the heavy-light distinction Mark made, I think there's a general hope that the US can unilaterally regulate to reduce the chances of catastrophe from their own tech, while still using export controls etc. to keep enough of a lead that they get to superintelligence before China, at which point, conceivably, the US has enough power to enforce global safety controls.
…It will get them all killed because, in their ignorance, they will not do the things that might produce a different result, like pressuring America to stop their tech companies from destroying the world. Not because of, like, divine intervention.
The EU specifically claimed the act would help them compete in AI. It is now widely believed to be a reason why the EU has such a weak AI industry. You can disagree with that being the cause. But objectively the EU is weaker in AI than you'd expect from pure economic or scientific output.
It’s always been a niche concern in the sense most politicians don’t care about it. I don’t think “regulations don’t address the problem or make the problem worse” is a good reason to stop caring about a problem.
I think you misunderstand my point which is not "not many people care about this" but "the way you analyze this is strange, out of step with how everyone else sees the same problem, etc."
I agree that just because a problem is hard you shouldn't abandon it. However, most of the world (including most computer scientists) simply do not agree that rationalist style AI risk exists. It's virtually absent outside of a few western subcultures.
My same point applies - the rationalists have always had a niche way of analysing this. I don’t think “most people aren’t convinced the risk exists” is a good reason to stop caring about a risk.
"Good to see you now agree that rationalist AI risk is a niche concern compared to other forms of AI risk about concerns like political control."
It sounds kind of like you're equivocating on "niche". Everyone including Scott already knew that existential AI-risk is and always has been a sparsely-held concern. But if it's a legitimate concern then it should be less niche.
Can you share some sources or analysis arguing that Europe's AI-safety legislation has been a major factor in them lagging behind the US? I'm not an expert, but I assumed it was because of the same combination of economic, cultural and regulatory reasons that hamper Europe's tech innovation generally, rather than anything AI-specific.
As I said above: I think you misunderstand my point which is not "not many people care about this" but "the way you analyze this is strange, out of step with how everyone else sees the same problem, etc." most of the world (including most computer scientists) simply do not agree that rationalist style AI risk exists. It's virtually absent outside of a few western subcultures.
Anyway, you want to find sources that say AI was held back not by generic regulations but by specific regulations on AI? That seems very much like the burden of proof should be on you to prove. To first effect, "the significant regulations on AI are the primary thing affecting AI" seems like the null hypothesis.
Anyway, the EU politicians themselves say they should roll them back and they stifled growth. It's all over the news right now. Here's the first article from Google:
>most of the world (including most computer scientists) simply do not agree that rationalist style AI risk exists. It's virtually absent outside of a few western subcultures.
I find this odd, at least if "do not agree that rationalist style AI risk exists" means a strong claim of probability (nearly) zero.
Even just an AI+robotics success at the level of "can replace a human of IQ 120 at all economically important tasks" implies something smarter than the average human which can build copies of itself, effectively a new, smarter, species. One can argue that this presents low risk for a variety of reasons, but zero risk???
Yeah, they think the entire chain of reasoning is flawed. They acknowledge basic risks like job loss or security issues. Most do not think there's any chance of it replacing people. Those that do almost invariably have received it from actually participating in western AI risk communities of various types. There's not even a standard term for existential risk in several languages. The calque it.
>Most do not think there's any chance of it replacing people.
Ok, yeah, if someone is confident that attempts to build AGI are just going to be technical failures, that is indeed an argument for saying that the odds of a risk dependent on AGI are as low or lower than the success odds for AGI, which they view as 0-ish.
( I think they are too confident, just as I think Yudkowsky is too confident. The future is uncertain in general, and even more so for R&D. )
Totally wrong. I speak a couple languages of third-world countries, and existential risk discussions are happening. People all over the world pay attention to what e.g. Nobel laureates say, so Hinton's acceptance speech was translated and widely reported on in local media.
However of course local corruption issues / wars / etc quickly push anything "global" out of the agenda especially if the country realistically has no input.
Which languages? Can you please show me the research institutes doing work in these third world languages about rationalist style X-Risk? Or even just extensive discourse.
I'm not usually a big defender of communism, but in an actual post-scarcity future with infinite robot slaves, shouldn't we want the communists to win?
I'm curious as to whether or not this is true. My impression is that they are still very much communist in theory, and see markets and capitalism as means to a communist end.
There's always the, "come on, of course they're machiavellian schemers," objection, but you could level that objection at the United States too.
They’ve been moving steadily away communism toward state-directed capitalism for 40 years. Xi recently decried the “welfarism” of the west. They’ve forcefully defanged labor unions. They discussed removing “communist” from the party name.
All that’s left from communism are old state firms that own less of the economy than ever.
I’d call them a right-wing, state capitalist, ethnostate.
Communism has always been an unrealistic mix of somewhat contradictory fantasies, but nominally "communist" governments still vary in the level of lip service they pay those ideals.
Eh, I don't have a horse in an argument about what China actually is, but I'm pretty sure most of those "moving steadily away [from] communism" claims are predicated on not realizing how "communist" states have operated from the beginning. (And exacerbated by the propaganda push to reframe China's recent success as [caused by capitalism actually], I'm not accusing you of consciously or even unconsciously attempting to do this, I'm just saying that's ostensibly the epistemic bubble, largely self-inflicted, in which the first-world mainstream currently resides.)
Canada and much of the EU have way more comprehensive medical, welfare, and even dental coverage than China does. On the ground, in terms of a social safety net, they're more socialist - and yet also democracies.
China's government has theoretical commitments, and promises for the future - but for that matter Musk is a fan of Iain M Bank's fictional post-singularity Culture, which is also a sort of perfect future socialism. Is Musk a communist?
I wholeheartedly agree with this. Anyone who thinks that the PRC is a communist, or even a socialist state, should go and visit.
My impression is that for common people it resembles the US more than any European welfare state: 10-12 hours of working for 5-6 days, no healthcare, no meaningful social net, no meaningful pension (except for the massive body of civil servants). And it's not getting better either - they are running towards the same demographic problem as many other developed countries, which will put further strain on the welfare systems.
Your last paragraph doesn't follow. If they have less of a welfare commitment to begin with, their system won't be as strained by demographic collapse. Given a commitment of ~33% of its GDP to social spending, Sweden might have to make a choice between reducing that amount or increasing taxes, given their aging population and growing demographics of e.g. Somalians with 85% welfare dependency, low employment rates, and poor educational attainment, but China, with only ~8% of its GDP going to social spending, will find the dilemmas far less extreme and the resulting choices both smaller and easier to make.
Most importantly, they will probably be happier pushing up their retirement age, which is the most effective way of combatting aging demographic pressures (and most welfare goes to the old, after all, so as long as the old aren't burdening the young through taxes, the young shouldn't really need to care).
Mao and the Red Guards were communists. Post-Deng PRC is closer to fascist - very, very nationalist* and totalitarian, ethnic cleansing, "third way" attempting to synthesise command economy with markets.
"The" communists is doing a lot of work here. Would you want the USSR to develop AGI? What about Laos? Angola? The democratic republic of Afghanistan?
Communist vs. Capitalist, to whatever extent it is true dichotomy, is severely underdetermined as some type of "goodness" measurement. My sense of whether it would be "good" to for China to win an AGI race has a lot more to do with my opinions on Democracy, how Chinese single party rule works in practice, and Han nationalism than the current Chinese economic system.
It's not really about communism vs capitalism, it's about democracy vs single-party autocracy.
In a democracy, for all the flaws in its actual implementation, power is spread out and those in power are forced to please the people at large, and the government is restricted in what it can do.
In a single-party dictatorship there's no delocalisation of power, nor any restrictions on the government's power. If you hand Xi Jinping or Kim Jong Un or Josef Stalin an army of robot workers/soldiers, there is absolutely no need for him to use it to serve the interests of the people. At best, you can hope that the robots will hand out basic rations to whichever people are sufficiently vigorous in proclaiming their loyalty to the party. And those who are not sufficiently loyal, well, we know what happens to them.
>If you hand Xi Jinping or Kim Jong Un or Josef Stalin an army of robot workers/soldiers, there is absolutely no need for him to use it to serve the interests of the people.
I'm not convinced that there would be any need for Sam Altman, Donald Trump, or Gavin Newsom either, sans their alleged moral compass. Democracy only works if there are checks and balances, and a robot army worthy of the name would obviously upset that.
Why do you presume there will be such things as democracy in this delusional fantasy of yours where a robot is so powerful it can rule the world without human input? Even if it is in the hands of a democratic government, why wouldn't any one private or public individual use it the same way an autocrat would?
>In a democracy, for all the flaws in its actual implementation, power is spread out and those in power are forced to please the people at large, and the government is restricted in what it can do.
Then you haven't been paying attention. Even before Trump II and others, nominally democratic parties have been hollowing out that promise of restraint as state power grows. The nationalists and fascist worldwide are gladly taking charge of the surveillance and repression states created by their respective predecessors.
I disagree with your premise. Power is spread out, yes, but only among elite oligarchs and their institutions. The thing that gives common people a voice in either system, American or Chinese, is the reliance of the elites on their cooperation. They can either refuse to work or launch terrorist attacks on their elites and thereby extract some concessions. If robots were to take over, either as workers or as the executive apparatus of the government itself, then as Crinch says below, there won't be any difference between either state, as the power dynamics will be the same.
Essentially, you are looking at the ritual expressions of the two systems of governments and not their underlying dynamics, which come down to reliance and threat.
Human governance is basically a farce when ASI enters the picture. There are only AI dictatorships, and probably only one of them (at least eventually) because of the different incentives (due to AI immortality and scalability).
It seems they voted with their feet against the capitalism which proceeded it - in the case of the USSR - if the large migrations of skilled populace to wealthier places are to provide any insight. It could instead be said that people vote for what can bring them the most money... but not collectively, only as mercenary individuals. In other words, people love money.
If people are moving to the United States, that wouldn't be a vote against capitalism. Marx would surely recognize that the US was the most capitalist country in the world, or in the running for it. He wouldn't have predicted that communism would succeed first in the agrarian empires of then Russia & China rather than the more industrialized capitalist nations of the west.
You probably really want to favor more individualist factions developing AGI because almost every way that AI can enforce some singular vision of utopia on society is going to suck for a lot of people. For instance what if you get the ideal that hard labor is some moral virtue ingrained into society permanently? Such that people are forced to larp as high tech peasants for literally no reason in order to satisfy the aesthetic sensibilities of the communist party.
Whereas if the tools/resources unlocked by AGI are made broadly accessible, then everyone can create and self select into their own utopian communities which perfectly align with their ideals.
This doesn't inherently favor capitalism over communism per say, it comes down a lot to the specific attitudes of the people developing the AI and those overseeing this.
For instance an AGI developed by Chinese researchers may in principle actually be less authoritarian than one developed in the US if the latter had its creation sufficiently influenced by Christian nationalists.
No, why shouldn't I get to keep the fruits of my robot-slaves' labor?
But in all seriousness, the authoritarian aspects of communism are a pretty huge drawback. Let's stick with democratic socialism if the post-scarcity future does actually arrive.
Even if you think Fully Automated Gay Luxury Space Communism was a nifty thing, you really don't want it to be run by the people who cut their teeth on grungy old-fashioned industrial-era communism. Communism without AI fails utterly unless it devotes itself to hammering a whole lot of square pegs into round holes in neatly-spaced rectangular grids, which selects for totalitarian autocrats with no respect for human rights or civil liberties.
Has China produced any innovation worth speaking of yet? Sure, they have amply demonstrated the ability to take western ideas and run with them, but that's not enough for actual dominance.
If you go far back enough in the past, then there are famously many things first invented in China (like gunpowder) and then spread to the west. If you just mean the PRC, then fewer.
Amusingly, even the go-to example of gunpowder became geopolitically relevant only after the west had gotten its hand on it, centuries after discovery.
You might be right about pharmaceuticals. I know drugs like Ansofaxine, an SNDRI, were initially developed in China and are now undergoing clinical trials in the US. Whether that is because China has an emerging, innovative pharmaceutical sector or simply because it lacks an FDA and an entrenched, stubborn medical establishment like the US, I cannot say. Batteries ahead of the West is news to me. I thought Chinese lithium-ion batteries were ahead of competitors only in their capacity to combust spontaneously.
Electric cars (BYD cars are now much better and cheaper than Teslas). Solar panels. Batteries. Consumer drones (DJI is the only reasonable option.). On every technology related to the green energy revolution, China is #1 and #2 is so far behind that they're barely worth mentioning.
I'd rather characterize this as the west shooting itself in the foot in various ways recently, which is of course also an important consideration. The biggest favor that Xi could do to shake the west from its torpor is to finally pull the trigger on the Taiwan invasion, similar to how Putin rescued NATO from the self-diagnosed "brain death".
If there is an AI bubble and it pops (presumably you mean by this that we don't end up developing transformative AI) then the safety measures in question will have neither helped nor hurt! So it is sensible to focus the discussion on the scenario where they matter.
Note that while the US will lose some more ground to China in the coming years, the "free world" (roughly: US+Europe+democratic East Asia) will still dominate China, unless there are AI/robotics developments that "flip the table" and happen to be localized to China.
> The Chinese don’t really believe in recursive self-improvement or superintelligence. If they did, they wouldn’t be so blasé about the possibility of America having AIs 1-2 years more advanced than theirs
Have you considered that this might be because lack of optionality? From their perspective, if you assume that superintelligence is guaranteed to happen, there's not much they can do short of nuking datacenters, since they're not going to be able to close the gap in compute in any reasonable timeframe. That option persists even past superintelligence being publicly known, so they might as well operate under the assumption that it won't happen.
If Xi Jinping believed the Americans were, I don't know, building a giant fusion bomb that would ignite the Earth's atmosphere like some people on the Manhattan project once worried would happen, would he really say "well, there's nothing we can do about it, I guess we might as well sit around and fiddle with the housing market"?
If it's the "application layer" where the USA has to be competitive, then the division between that and the cloud service is where regulation should occur: prevent the cloud providers from abusing their quasi-monopoly position to take the business of companies downstream. That would leave incentives in place for smaller firms and startups to compete and build a robust, diverse AI application sector.
Similar to what should (but doesn't) happen with Amazon: it's fine for them to be the dominant online retailer, but they should then not be able to sell their own branded goods, and should only act as a marketplace. See also: microsoft and browsers.
It sounds like you’re advocating for breaking up vertical monopolies. Like them or loathe them, my sense is that big, evil, vertically-integrated corporations have normally been the engines of growth. Think Microsoft, GE, Ford, even historical examples like big factories in the Industrial Revolution or the East India Company. I agree that what you’re describing will likely prevent some major harms to a lot of people, but at the same time it’s the sort of regulation that I would expect to slow down applications development, precisely because the large companies wouldn’t be able to drive their profits and thus their money available above the competitive equilibrium. You might be right, but you would only despite the concerns described here, not because of them.
I think the data shows that more innovation and growth comes from small companies, most of the time. (Specifically here, startups; it’s quite a recent thing for the big companies to have understood Disruption Theory enough to identify potential disruptors that could unseat them, and just buy them instead.) The big companies are horrifically inefficient and mostly extracting value from monopoly positions in their respective markets.
The recent stock market bull run being a historical anomaly, “this time it’s different”.
AI is a case where there is an unusual big integration opportunity; for example at some point (perhaps even before AGI) it’s going to be more profitable for Alphabet to just spin up a subsidiary with exclusive access than to sell API access to their frontier models. This will really depend on the antitrust regime; a weak FTC will allow a lot more of this behavior than IMO should be permitted.
My hunch is that in the past you needed a lot of capital to do most innovation, and these days there are a lot of things you can invent with just software (particularly the leverage that the open source foundation enables). The VC model accelerates this dynamic of course, encouraging lots of small-scale innovation with the promise of funding to scale up for the most scalable business models discovered.
I think you could point to a few recent “physical world” innovators which are back to being capital intensive, but if you look at the big guys proposed for breakup it’s mostly software behemoths sitting on monopolies.
Good analysis but why assume that it would be bad for China as such to be ahead of America, or why assume it is good for America to be ahead of China, at any random point in the near future ? From the Chinese point of view, America is a wonderful land where millions of Chinese have been welcomed. From the American point of view, China is not as bad as some of the other countries that hate us for bad reasons, and they are generally friendly to American tourists. From the AI's point of view, the attribute of living in Silicon Valley as opposed to having years of training in one of the Chinese universities is probably not all that important. Of course right now the focus is on power, on numbers, on the vast or less vast amount of chips or data center megawatts (to identify simply the more simple attributes of power in this race) one has at one's disposal. Why wouldn't it be? Would this comment be (a) better if it were more simple or (b) better if it were less simple.
It would be better if anyone believed that those points of view (leaving aside the AI's one) are representative of anyone in power in either country, or even the general populations.
It is likely that AGI/ASI could develop weapons that render all previous weapons obsolete. So China would become defenseless, the US would likely depose its government and install a more US-friendly one. Obviously the current Chinese government wouldn't like this.
It is likely that AGI/ASI could develop weapons that render all previous weapons obsolete.
That's a wild leap. Even AGI/ASI can't violate the laws of physics. Also, once we get to supersonic nuclear missiles with a credible second strike capability, what else really matters.
Precision. Counterinsurgency. Collateral damage. If you respond to a raccoon messing with your garbage by turning everything within a kilometer into exotic plasma, you won't rule the world for very long. Have you noticed how few wars involve actual use of nuclear weapons?
Even before they were invented, back in WWI, the capability to reduce an entire landscape to unrecognizable craters was there. Turns out when you do that, nobody wins. Obliteration makes the whole world poorer.
Even back in Sun Tzu's day the underlying principle was understood. Killing is often necessary in war - that's what makes it different from peace - but it's not a victory. Whenever possible, skip past the violence, instead proceed directly to the part where you take the enemy's best stuff and make it become your stuff.
All well and good. But the previous comment posited that and AGI could make China "defenseless" which seems to be a bit more of a dire situation than merely unable to match up on the ladder of escalation.
Supersonic nuclear missiles? ICBMs from the 1960s already exceeded 20x the speed of sound and are fast enough to hit anywhere in the world within about 30 minutes of launch. Second strike capability has pretty much always been there.
No, it's not a wild leap. As we saw with the Gulf War and with Israel's recent war with Iran, an advantage of a few decades in technological progress results in an overwhelming military advantage in an otherwise roughly evenly matched conflict. With superintelligence, one could expect a similar technological gap to develop much faster.
Good points. However, while it seems like a consistent assumption in these sorts of debates, I don't see fundamentally why there isn't some exchange the US could make with China to sell them chips that would be a net positive to both nations.
My model is something like: China and the US both have internal welfare in an absolute sense (positive sum), and then there is a balance of power between them in a relative sense (zero sum).
If China could buy more chips from the US, this would shift the balance of power in China's favor and also increase the internal welfare of China (AI does all sorts of useful things other than military applications.) Then presumably the US loses utility in relative balance of power equivalent to China's gain.
So why couldn't the US propose to sell chips at a rate (through taxes, or diplomatic demands, or whatever else the US likes), such that the transfer China -> US is greater than the value of the balance of power the US loses, but less than that value + China's internal welfare benefit.
This seems like it would be in both parties interest, and also have good peace-incentivizing side effects as mutual trade tends to do (and right now I see incentivizing peace/cooperation between US and China as being pretty high-value).
If we are concerned with reverse engineering / IP theft ... I mean they already get some chips through for example smuggling, does having ~100x the chips actually make it any easier to reverse engineer?
The fact that so many smart people talk about this subject without proposing this makes me think I'm just missing something in my thinking, but I'd like to know what!
> If we are concerned with reverse engineering / IP theft ... I mean they already get some chips through for example smuggling, does having ~100x the chips actually make it any easier to reverse engineer?
That's what I'm assuming. Reverse engineering the manufacturing of a chip isn't about looking at the chip itself, it's about understanding all the horrendous and undocumented details of the process by which the chip is made. Every step of modern chip manufacturing is really hard, and China is still way behind the state of the art.
Which is why if we're serious about preventing them from catching up it needs to be about preventing Chinese spies from getting a good look at any part of the supply chain. That means a complete stop to visas for Chinese nationals to enter any Western country, at least until they have overthrown the CCP and instituted democracy.
Without getting into specifics, you are quite wrong about arguments about the application layer. AI-optimization is being built out en masse at MedTech, EdTech and local GovTech firms. However, many of these firms are of the extremely boring PE-rollup sort. Not the splashy Silicon Valley VC-funny money and BART ads sort. If you go to a conference for any of these industries, it's all anybody is talking about at their booths... Maybe you can better substantiate what you mean by China having an app layer advantage? Are you pricing in a presumption that the federal government will squash app-layer AI in the US in critical industries?
See footnote 3. The US is fine in software, but China already has top-down orders to automate as many of their factories as possible, and has a better record of hardware construction than we do.
It seems the last argument (safety might actually increase progress) is an instance of what is criticized before (bad thing is bad).
Overall, the whole issue with the war with China seems really pointless. I've been to China, talked with people from there, both locals and expats here, and honestly, it seems like not a bad future to have Chinese hegemony.
And in any case, the important issue (for me) is how much safety might impact my / my close ones chances of dying. And this is why I am fundamentally opposed to AI regulation. It seems this is the actual point - most people (even LW crowd, as survey showed) don't actually value long-term future that much. Would be amazing to see Scott attempt an honest presentation of this pov. Otherwise, it's all beating around the bushes and fighting proxy wars (as with the last post on AI regulation, there is no attempt to actually understand and steelman the motivations of the other side!)
Solving “death” in full generality sounds hard, but solving aging without ASI sounds doable. (Remember, you can still have things like AlphaFold 9000 without ASI). You’d still die if someone drops a nuke on your head I guess. But even ASI probably can’t “cure” accidental collision with nukes. It would be an unfortunate irony to have “cause of death: quest for immortality” written on humanity’s tombstone. Let’s try to avoid that. If you could send an high school biology textbook back in time by 100 years, I think people back then would be amazed how much we now know. Why be so sure that we’re stuck? If you’re worried that progress is too slow to save you personally, cryonics is pretty cheap I’ve heard.
Cryonics is very far from being actually useful, it's a cheap cop-out to pretend otherwise. AFAIK there is exactly one company making serious effort in that space (developing better cryogenic methods) - Until labs - and they are optimistically 10 years before even being able to preserve large human tissues (I'm not a biochemist, but this is what my medical PhD friends are telling me). State of the art right now is freezing a rat to 0 Celsius (veery far from actual cryogenic preservation) and unfreezing it after 20 minutes. There has been no significant progress on this.
But even granting something like this working out, you still have the fundamental problem of (1) freezing actually being kind-of death which is an unsolved semi-philosophical problem (2) being able to cure whatever killed you in the first place (3) institutions being stable enough to not accidentally throw your body out (4) civilisation continuing and making technological progress (and all the arguments with cultural drift, TFR dropping etc), (5) AI not stalling indefinitely kind of like all over-regulated industries (6) the west still being dominant enough do our lives matter, and so on.
In short, "just do cryonics" is a lazy, very much pie in the sky answer.
Fair, it’s far from guaranteed, I may have been a bit too flippant there. But that doesn’t mean that building ASI is on net a safer bet. Also: we might make enough progress soon that you never end up needing cryonics. (I don’t know how old you are; if you’re over 110 then I guess probably not.). As for freezing being kinda death, I suggest the following philosophical exercise, (which I admit I have not done myself): there exist people who have had their body temperature drop below 20 Celsius, at which point electrical activity in the brain and elsewhere basically stops, and they have lived to tell the tale: they were basically fine. The exercise is: try to figure out if these people “kind of died” in a way that matters to you.
Yes, there are examples of those people (eg the famous case of the Norwegian doctor skiing who got hibernated for quite a bit - https://en.wikipedia.org/wiki/Anna_B%C3%A5genholm). But like, this whole thing is just fishy. The way I see it - these are hard philosophical problems around personal identity, consciousness etc, which are easy to get wrong, and we're much too stupid to have a good take, and it's very easy to get confused. The best research is currently probably QRI or Parfit-adjacent ppl at Oxford who are again nowhere near solving these. So the only option is to play it safe - meaning preserving the body and living "standard" way.
We might make progres, but we might also not. I'm presenting an object level argument that we are not on track.
There's actually a way to quantify this. In expectation, there's a 1% of dying every year (in the US) skewed towards the elderly. So if an AI safety proposal delays solving death by 1 year, then it should reduce the risk of everyone dying by >1%.
For me, more of my loved ones skew younger and I, like many people, value young people's lives more than old people (eg trolley problem: 1 baby vs 1 elderly person), so I'd accept a year's delay at a reduction of ~0.05%.
This assumes that you keep all else constant and that it really is slowing by 1%.
There's three responses. One: it's a hatchet. As Scott himself points out, once slow-down-AI faction gets their win, they will escalate. And it's like that Gandhi who can accept the bribe to be 1% more killer - there is slippery slope.
Two, there is a chance that this is the peak of our progress based on the TFR argument, relative peace, etc.
Three, my parents are not that young. I don't want them to die either. (Also some argument about all other people in the meantime, but that's separate and not really my reason).
Not sure what you mean by "don't value the long-term future that much".
I don't know how old you are, but I'm 40, and I'm not especially concerned safety regulation will prevent me from getting immortality, for a few reasons:
1. As discussed above, real-world safety regulation is on the order of raising costs 1%. Most likely it stays at this level and slows timelines 1%.
2. In the extreme world where we get a full pause, it probably won't last more than a few decades before chip tech becomes so good that you can train AI on chips too small to track.
3. In the extreme world where we get a pause, I expect human intelligence enhancement to create supergeniuses within a few decades.
I'm more interested in removing the chance of death from unfriendly AI, especially since if I die then at least my kids will survive, whereas unfriendly AI kills everyone. I think this chance is around 20%, but I think if we gave up and had no attempts at safety whatsoever, it would be closer to 50%.
1. You argue yourself that the regulation is a hatchet. If you achieve 1%, you then aim for bigger wins. And it can snowball. It's easy to point to examples of industries where this happened. X-risk people are very explicit about wanting to do more than 1%, and I don't think they would or could commit to only slowing by 1%, because that's the whole point!
2 and 3. I don't think this is the case. We are now in a very fortunate period, relative stability and peace, still riding the demographic wave. But the conditions are changing and can change dramatically (such as arguments from Hanson on TFR and cultural drift). And again, we banned nuclear power, but there are no people having small nuclear reactors in their backyards. (Supergeniuses are even more uncertain - as e.g. you yourself pointed in the polygenic IQ/heritability debate).
Note that you (and I) also have parents, which are equally as important. I actually put the same 50% ballpark for the chance of charging ahead with AI causing extinction. But I think it's pretty much as good a bet as we are going to get. Not-dying without ASI alternative seems to me to be something <20%.
> there are no people having small nuclear reactors in their backyards
Physics of neutron cross-sections put some very hard lower bounds on the size of a useful fission reactor, and the only reasonably attainable types of fusion right now are even bigger. Computation is much easier to miniaturize.
Why assume the costs of regulation stay low? establishing the precedent of federal regulation will invite more and we could end up totally strangling the industry like we did with nuclear.
Argument seems misaligned. Primary cost of regulation is not the cost -- it's lost time. Scott is correct that the costs would be manageable, but the proponents of regulation explicitly want to delay progress. It's a mistake to think we could pay 1% of the AI research cost and maintain the same timelines as regulation escalates.
I think the scope of AI Safety activist asks will ratchet up by a lot more than 1% if they happen to win this round of asks. These are people who believe that there is a high chance superintelligent AI will murder everyone. It would be ludicrous for them to not do everything in their strategic power to strangle the AI industry with regulation and restriction given the chance. Given how much recent economic growth in recent years has hinged on AI development, the difficulty of enforcing a global pause, and the extent to which future live-saving and life-extending medical inventions might be contingent on advanced AI, a global halt solely due to a theoretical and speculative argument (even if it comes from people who are smart) is a tough sell. This might not be what activists are asking for now, but we can likely expect them to escalate up to this sooner rather than later.
Every movement attempts small things first and then tries large things, but usually they fail on the large things because other people are opposing them, and they usually have enough power to push through their small common-sense hard-to-oppose demands, but not their giant unpopular ones.
Compare environmentalism, which was powerful enough to ban plastic straws in some areas, but not enough to ban SUVs or coal plants or plane flights.
I think there are a few factors which make this issue a little different than most issues that get resolved through that type of pluralistic process.
(1) We can expect anti-AI advocates to have higher determination and resolve than most political cause advocates. They also are likely more g-loaded than the people in most political causes. These factors mean that they will likely be more strategically effective at achieving their aims relative to how popular their aims are than most political causes end up being.
(2) There is not an obvious political 'in-group' we would expect to be strongly pro-AI. The natural enemy of the environmentalists were coal miners and people who worked in the oil and gas industry. The natural enemy of gun restrictionists were proud gun owners. The natural enemy of the AI safety industry would be: computer scientists? The four 'accelerationists' that are on Twitter? People who work at OpenAI and could get another job in a day if they needed to? The benefits of AI are fairly diffuse.
(3) Most people do not know much about tech. In theory it could be possible to draft reasonable looking legislation that smuggles in severe AI curtailing regulation through the back door while arguing that the legislation does not do this.
(4) Most people, if faced with the choice of preserving democratic norms or preserving their social status, or preventing everyone from getting murdered, would choose the latter. If faced between the choice of being honest or preventing everyone from getting murdered, most people would choose the latter. It is hard to think of many lines, no matter how extreme, that people would not cross in order to prevent everyone from being murdered. Because of this, there is a meta-level suspicion at play. If people genuinely believe that AI will murder everyone, can we trust them to be good faith actors? Can we assume that the normal rules and norms are going to be things they structurally value?
For point 2, I think some of the biggest companies in the world by market capitalisation are AI companies. They’ll have plenty of money for this political fight. That money also buys lots of g-loaded lobbyists. 3 and 4 seem to apply equally well to early environmentalism.
'AI companies have access to high powered lobbyists' does not change the fact that there are a lot of components to the AI regulation political debate that undermine a lot of the normal operating assumptions that pluralistic democratic processes rely on to resolve policy questions fairly (competing grassroots special interest groups engaging in counterdiscourse and counteradvocacy that have a broad and similar ceiling of strategic efficacy, a shared assumption that both sides are committed to the broader structure of norms that they are engaging in discourse in over their preferred policy outcomes, etc).
Early environmentalist groups were pretty radical and did face some stigma and distrust. Over time most moderated so you do not see that sentiment as much today.
Sure, but I feel like lots of issues have similarly unusual or asymmetric advocates.
I think a lot of the increased trust of environmental groups is just a familiarity effect. They’ve been around long enough that lots of people have absorbed their claims as normal (though some people do still distrust them).
I think the overall gestalt is abnormal. Maybe you can mix and match and say 'well the issue of X kind of was also a little asymmetric' or 'well the issue of Y maybe also had super motivated advocates', but I think it is specifically the way all of these factors mix that creates a holistic picture that I think normal democratic and pluralistic processes are going to struggle with.
I just get bad mojo surrounding all of it. You have a movement of people where the median IQ is like 135. These people seem standard deviations more strategic than most political movements. They think AI will murder everyone, so the stakes are infinite. In theory such a position could easily justify political violence. In theory this means that the issue is entirely a one-shot game for the advocates. On the flip side of the issue, you have no organic advocates because of the diffuse nature of AI, but also infinite lobbying funds and appetite to lobby on the 'pro-AI' side of things. This means that the pro-AI side will almost certainly lose the public discourse battle and will have to rely on extreme corruption and graft to achieve victories. Scott's original response to my comment said that normal democratic and pluralistic processes can be expected to handle this issue in the way they handle most issues, but I do not feel like that is going to happen here.
>Every movement attempts small things first and then tries large things, but usually they fail on the large things because other people are opposing them, and they usually have enough power to push through their small common-sense hard-to-oppose demands, but not their giant unpopular ones.
I concur with Spinozan Squid, notably on point (2), that
>There is not an obvious political 'in-group' we would expect to be strongly pro-AI.
If I could be very confident that AI regulations stayed at the "small common-sense" level, I'd be happy with them, but American political history, from ALARA & the NRC to NEPA to NIMBY gives me the creeps. It makes me think PauseAI (perhaps better described as KillAI) would find a way to either twist the common-sense law into what amounts to a ban (here, not in the PRC, and probably circumvented by the military in both places) or salami-slice their way to a ban.
What did you think of the ‘Catching Up to the Almonds’ article? It seemed to argue that the pro-AI lobby group was unusually active at targeting politicians. It won’t get big protests, but it will organise well-funded attack ads. Do you think that would be less effective at scaring politicians than protests would?
>It won’t get big protests, but it will organise well-funded attack ads. Do you think that would be less effective at scaring politicians than protests would?
I don't know. Well-funded has its limits. Kamala Harris's campaign committee outspent Trump's by about 2.6:1, but still lost the electoral college, the popular vote (and, looking at parties, control of both the Senate and the House).
<These companies should do some kind of evaluation to see if their AIs can hack critical infrastructure, create biological weapons, or do other mass casualty events. If they find that the answer is yes, they should tell the government.
I don't think we have to worry about regulations of this sort slowing us down because (a) when push comes to shove the companies won't fully disclose this information and (b) our president is a dumb wacko narcissist who has filled the government with dumb evil loyalists. The conga line of weak-brained moral midgets wouldn't know what to do with this info even if they were given it in full.
I'm not doing haha dark humor here. This is my actual read of the situation.
B) The current Republican Party is very divided on AI; for every A16Z loyalist, there's a Josh Hawley or a Martha Blackburn who hates and fears Silicon Valley. I think there are many voices who would do something with this info. If they don't, state governments may be able to do something, or a future Dem administration might be able to do something.
Regarding A: I worked for several years for a prestigious psychiatric hospital affiliated with an Ivy League medical school, and learned a lot there about how much can be hidden from agencies tasked with checking the institution’s ethics and compliance with rules. Before an inspection by some public health agency we would spent a couple weeks bringing everything from records to the location where white out was kept up to standard. (It’s poisonous, so is supposed to be kept in a place where patients could not possibly reach it. It normally was not on our unit.)
There was an incident where a violent patient stomped with all their strength on the foot of a patient of mine, S, who was in a wheelchair. Afterwards S asked the doc on call to examine her very painful foot, and he said there was bruising but no serious injury. That night S was not protected from the violent patient, who continued to roam the unit freely. S signed out AMA the next day and went to the ER, where they found a complex injury including more than one broken bone. Staff I spoke to at the hospital about the incident sounded like they had been told to be vague and uninformative. “Yes, it was very unfortunate. We’re really not clear went on between S and [violent patient] . . . “ S later tried to sue the hospital, the hospital elected to fight, and I was deposed by the hospital’s pit bull lawyer, who literally laughed out loud at me because I did not have written documentation of what the patient had said to me in several emergency phone calls at the time of the incident. “Honey,” she sneered, “do you know what heresay is?”
The Psychologist in Charge of the unit where I worked ran a postdoc program for interns he liked. He referred private patients to his stable of not-yet-licensed psychologists, and for those that had insurance we filled out insurance forms stating that he, the PIC of my unit, had been the provider, and he signed them. He kept 20% of patient payments for himself. There was a rule in the ethical guidelines for my state prohibiting this exact practice. Oh, and my PIC was *on* the committee that developed those guideliness.
Scott, do you get that the shitlords of tech are not honest and scrupulous the way you are? They’re not even as honest as I am (mostly very honest in one-to-one situations, much less so when dealing with organizations and groups because I dislike and distrust almost all of them.) You're more honest than me, and the people running the vast tech companies are many steps beyond me in disingenuousness. There exist people who literally do not grasp the concept of sincerity. They process questions using an algorithm that is doesn’t even have truth as a variable, but only things having to do with immediate effect of saying X, long term effect of saying X, options opened by saying Y, etc. Haven’t you met some? My guess is that many of those running the big tech companies fall into that category. And even if one of them did have an inner ethical code, it would operate in a profoundly different context from yours and mine. It would involve what the leader owes the company and the stockholders. It’s almost as though these guys are the leaders of other *countries,* deciding how to handle the demand of another, larger one.
Game it out. Suppose an AI company doesn’t disclose what the regulations say they have to. Is the government going to shut down the strategically important AI run in progress?
If they do, then the regulation has in fact slowed AI progress. If they don’t, then the regulation was useless.
That’s right. And there are other considerations as well. An AI company can do impression management many orders of magnitude higher than Whiteout location switchiing. If outsiders come in to test how corruptible a model is, the company can put special hidden constraints on it that make it almost impossible to corrupt (OK, they will also make it dumber, but that’s not a problem in this context.)
And what about this? AI is soon going to be, if it isn’t already, sort of like gasoline or electricity for the military and other branches of government — something they have to have. An AI executive can say, look, if you insist we put in constraints in order to make it impossible, rather than very difficult, for X to occur, the timeline for getting those crucial gizmos to the military is going to double. You sure you want that to happen? Hmmm?”
>There exist people who literally do not grasp the concept of sincerity. They process questions using an algorithm that is doesn’t even have truth as a variable, but only things having to do with immediate effect of saying X, long term effect of saying X, options opened by saying Y, etc.
I suspect a main difference between you and me (hopefully) and those people is their scope. If the scope is the success of some company or organization (and probably by extension themselves and their family short term) then dishonesty may make sense. If the scope is the success of larger society or humanity (and by extension probably themselves and their family long term), then I think dishonesty is almost always sub-optimal. Dishonesty erodes trust, and trust is very important for a number of things on a societal level. For example, look at how high trust nations are doing compared to low trust nations (https://ourworldindata.org/trust). Of course there are also coordination problems to consider. If everybody else is a crook, then being honest is very costly.
I value truth in itself as well, but I think the scope is also a large part of it.
It's very funny to see how people discuss seriously competiton with China as if China were a dictatorship much more than the US is. In reality, the difference is far not that big as it is positioned: perhaps, twice as much. The US is no more that free country, while China is not that unfree. The real difference is pretty moderate.
Wow, can't even tell if this is irony or just ultra mind-killed insanity, but if the latter...I suggest you try talking to someone who's lived under an *actual* dictatorship.
I didn't live in North Korea (I suppose, it is worse because of poverty) but I lived long in the USSR and in Russia. Also, I studied details about China pretty much.
It's not THAT greatly different from living in the democracies, as long as you are an average person. China is currently not even the USSR. It includes more freedom in some respects, along with a pretty good bunch of other benefits.
Yes, living in Germany is smoother, however it's not a thing for which I would ever kill people or ever recommend anyone to. People in democracies are very much indoctrinated and being lied, and mostly don't see the reality, almost just like people in dictatorships. What you read in news and watch on the TV, is a very much specialized inclination, not an objective and holistic picture of reality!
Yes, dictatorships kill people even today. They torture and impose non-human practices. They break human rights. But all that is far not that important for how an average person lives every day, as mass media in democracies present! Normally, massmedia (and even individual bloggers) boost the negative sides and hide the positive ones. This sells better. Also, they are said to do so by their owners. Also, they are indoctrinated themselves to see things one way, not objectively.
Also, how democracies normally present themselves, is far not what they are in reality. Here there is a pretty much of distortions and biases, and some direct lies too. If you dig deep, you will discover that real differencies are not that big, as most people think.
I grew up in socialist Czechoslovakia. If the regime persisted more or less unchanged until today... for starters, I couldn't go to Vienna rationality meetups. Less Wrong and ACX might or might not be blocked by some kind of firewall. I think many people on the autistic spectrum would be at higher risk of saying something politically incorrect and thereby getting in trouble.
> Normally, massmedia (and even individual bloggers) boost the negative sides and hide the positive ones. This sells better.
Does not the same apply for reporting about democracies? Perhaps even more, because ultimately people care more about problems in their own countries than about foreign problems. My government required me to believe in so-called viruses and wear a face mask once; I can't stop complaining about that all he time. Saudi Arabia requires half of its population to wear masks constantly, and regularly kills people for not believing in Islam, but I guess they are used to it, so it's no big deal.
My impression was that China would regularly imprison people for years or decades for peaceful protests, such that a majority of large protests would end in arrests. My impression was that most large peaceful protests in the US don’t end in arrests. Is one of those impressions wrong, or do you think the difference is unimportant in practice?
The US is not a dictatorship. The Supreme Court recently heard oral arguments in which most expressed skepticism that the President can set the recent tariffs that he claimed were permitted by actual legislation. We've repeatedly alternated the parties in power, whereas China has a single party without electoral competition.
There is a jagged frontier of dictatorialness. Honestly, the US culturally seems to me to be much more dystopian than China - purely from personal experience, it's easy to get cancelled socially / be a pariah for having an unpopular opinion (this applies to all Anglosphere). Chinese people seem to be much more tolerant in this regard. And yes, their govt limits political freedoms, but is pretty relaxed on personal freedoms in general. (About the obvious point of Uyghurs - I agree it seems very bad, but some reasonable people disagree it's actually an actual problem after looking into it - e.g. Angelica Oung).
How much would culture vs current govt structure matter or dominate is, I think, an open question.
I'm not sure that 7 is a good argument here unless you are making the further claim that NVIDIA is intentionally restricting supply in order to keep prices high (which clearly would be bad for the US winning the AI race). If the high profit margins are due to NVIDIA making chips as fast as they can but demand growing even faster, then the high prices just serve to:
(A) ensure that the chips are going to the people that need them (or can pay for them) the most (which might mean AI companies rather than gamers)
(B) encourage NVIDIA to increase manufacturing capacity as much as it can
IF the issue here is just demand outstripping supply and not monopolistic practices, then adding a price ceiling will just make things worse.
To be clear, NVIDIA doesn't make any physical chips. TSMC does. Nvidia's ability to expand sales is entirely dependent on TSMC's ability and willingness to expand production capacity.
Then higher profit margins men's that they can pay TSMC more per chip in order to get a higher fraction of chip production dedicated to them and eventually encourage TSMC or others to expand production.
Agree. The issue here is supply. Companies in both countries would love to have more chips at the price Nvidia is charging. See the recent Bay Area House Party post with Zuckerberg trying to get more GPUs. It’s funny because it’s true :)
The only way for NVIDIA to get more capacity quickly would be to buy capacity from other companies that have contracts with TSMC. That would mean a higher cost to NVIDIA but not more money for TSMC. This approach has worked for Cerebras, but NVIDIA already has so much capacity that the amount of capacity it could buy from other companies would be a drop in the bucket.
NVIDIA could get TSMC to expand capacity by contracting to buy large numbers of chips in the future, but then NVIDIA would be stuck holding the bag if the AI bubble pops.
I'm not sure about this, but I think NVIDIA just has the IP. Compare to medications, where if the government suspended intellectual property, then anyone could make the medications for $1 and many more people would use them.
Again, this only matters to the extent that they are engaging in monopolistic pricing and intentionally keeping supply low to inflate prices. But unlike the case of medications, where if patent protection dropped a bunch of other manufacturers could suddenly start producing, I'm skeptical that NVIDIA could substantially increase supply in the short run if it wanted to.
"You say that my proposed regulations are not incentive-compatible, but I've cunningly designed them to not accomplish anything anyway. Rationalists win again!"
I agree that currently proposed AI Safety regulations wouldn't significantly slow-down U.S. labs, and wouldn't cause the U.S. to "lose the AI race to China".
But I think it's a little misleading to not mention that most people who buy AI x-risk arguments and care about passing such regulations (Scott included, I assume) also believe that these regulations are nowhere near sufficient!
The kind of AI Safety regulations that would guarantee that U.S. labs don't risk our existential safety would significantly slow down these labs and make it far more likely that Chinese labs become competitive. This is the bitter pill you have to take here.
TSMC stands for "Taiwan Semiconductor Manufacturing Company".
I'm very much not an expert on semiconductor manufacture, and I know the US has taken some steps to increase domestic production, but if/when China invades Taiwan - which seems more likely than not to happen before 2028, although far from certain - then won't the chip advantage massively reverse?
I think the answer is it would somewhat reverse, but not massively, for a few reasons:
1. The US already has ~10x as many chips in its data centers as China. Even if they never get any more, it will take China a long time to catch up.
2. There's a TSMC fab in the US now, and we are working on getting more.
3. NVIDIA still has better chip designs than China does, and whatever factories spring up to replace TSMC will have the advantage of those better designs.
4. A big part of TSMC's advantage is having access to incredibly complicated machines by a Dutch company called ASML, which are also banned from being exported to China. Whatever Western companies sprang up to replace TSMC would also have those.
5. If the US was smart, it would Operation Paperclip as many Taiwanese chip experts and machines before giving up the island.
6. If the US was smart, it would ask Taiwan to destroy some Chinese chip factories on their way out.
I don't know exactly how all this would go - it would depend on how quickly the West could bounce back - but I think probably it would more like halve our lead than eliminate it.
Betting markets treat the likelihood of things that would destroy the betting market as zero. Much of the probability of Taiwan invasion involves a nuclear exchange at some point. Hence, prediction markets with real money will drastically undercount the probability of a Taiwan invasion.
Xi Jinping has explicitly and publically said he wants the PLA ready to invade Taiwan in 2027. That doesn't necessarily mean he'll order the invasion, but one should consider it a strong possibility.
So they undercount it by 30%? Seems unlikely. Taiwan does not have nukes. Based on how Ukraine is going, nuclear-armed countries wont defend Taiwan with nukes and China is likely to consider invasion too costly. Taiwan also recently elected a China-friendly president. I could see something like 25% chance of invasion to be generous, >50% is insane and state comments and mobilisations from years ago don't mean much with how the world has shifted.
>Based on how Ukraine is going, nuclear-armed countries wont defend Taiwan with nukes and China is likely to consider invasion too costly.
Ukraine and Taiwan are very different. Taiwan is strategically critical both due to TSMC and its position anchoring the First Island Chain (failure to defend Taiwan = South Korea and Japan pull out of the NPT and make a mad sprint for nukes, as without the First Island Chain or a reliable nuclear deterrent they would be at the mercy of a PLAN blockade of their food imports), and the USA has some formal and informal obligations to Taiwan (including Biden saying that the USA would defend it).
EDIT: I just saw, a week and a half ago the Japanese PM said that a Chinese attack on Taiwan would be a "survival-threatening situation" for Japan (this is also the constitutional requirement for Japan to deploy its military). Hadn't actually seen that, but yeah, Japan knows exactly which side its bread's buttered on. (Beijing responded by threatening to cut off her head.)
I would like to think that Xi won't pull the trigger, but fascists have a rather-notorious tendency toward overconfidence, the PRC is notoriously bad at comprehending Western mindsets, and they are rapidly building ships that are obviously intended for amphibious invasion. There is still a decent chance that they call "no-go", certainly, but they wouldn't be doing this if they didn't want the option live and ready.
Conventional war between the USA and PRC almost certainly means nuclear exchange at some point; the Cuban Missile Crisis was two weeks of not-actually-war and we had the Duluth bear (bear broke into a base, set off alarms automatically including at neighbouring bases, one of them was miswired and instead of "intruder at neighbouring base", it sounded the "scramble bombers" alert; nuclear-armed interceptors actually got on the runway before they realised it was phony and recalled them) and the Vasily Arkhipov incident (when the US destroyers dropped practice depth charges to force the Soviet subs to surface, some of the subs thought they were real depth charges that missed, and considered retaliating with nuclear torpedoes; on one sub the vote was 2:1 in favour, but it needed unanimous).
>Taiwan also recently elected a China-friendly president.
No, Lai Ching-te is pro-independence enough that Beijing literally gave Taiwanese in China discounted flights back to Taiwan to vote in the hopes they'd vote against him, and ran drills to "punish" Taiwan for electing him. The PRC really hates the DPP.
Again, Taiwan invasion probably means nuclear exchange which drastically lowers the value of winning the bet (many bettors dead, reduced value of money). I'd at least triple the likelihood to get at what the market is treating as the real probability of invasion (i.e., they're implicitly writing off everything 2/3 of the time invasion happens); that'd be 63%/142%, or 44%. That looks vaguely sensical - more so than the nominal Manifold result, at least.
I do expect to survive a nuclear war*, but money will still be less useful to me if they do invade. If I were a betting man, which I'm largely not, and I had some guarantee that the bet would actually be honoured, I still wouldn't take this at even odds (restricted to end of 2027 I'm not sure it's over 50%, although to 2030 I'd say it is).
*List of my current preparations: moved to Bendigo, various account structuring to maximally protect my money if the banks go bust, 20L of water in the bathroom cabinet with bottles for another 9L available, iodine pills, emergency radio, 60m alfoil for protecting electronics including said emergency radio, 75lb compound bow with broadheads. The last one is the only one that I consider arguable in terms of EV restricted to this issue, as it's unlikely I'll need to use it even if Australia does get nuked, but I've been interested in archery for decades so it was a good excuse.
If the US comes in, it probably turns into a nuclear exchange, which almost certainly ends in Chinese unconditional surrender (i.e. fully demilitarised and probably banned from AI development - might even have to hand over all the smuggled chips) and quite likely involves China becoming a radioactive wasteland with a considerably-lower population.
If the US doesn't come in, it might turn into a nuclear exchange anyway due to South Korea/Japan sprinting for nukes (neither is remotely capable of surviving without seagoing food imports, so without a nuclear deterrent or the USN being reliably on-side the PRC would have them over a barrel with "do X or we blockade you and you starve").
In any case, the Taiwanese chip fabs won't fall into Chinese hands; TSMC has a kill-switch to destroy them.
That sounds good in the short-term, but if at some point the models start exhibiting really worrying behavior (along with capabilities increase), AI safety would mean severely slowing down or halting AI development. Of course assuming no significant progress in alignment.
The point about the US-China race is that, even then, neither side can afford to stop as it risks giving up their lead.
So, for me it's not about satisfying current AI safety proposals with 1% lost efficiency, but rather -- what happens later. This is where it's most dangerous and the race dynamics are the strongest, as the potential of AI becomes more and more evident.
1. If the models really become worrying, then people would be more willing to take risks to align them, and it's not just a few safetyists begging people to pay attention.
2. If the models really become worrying, China will hopefully also notice and there will be more room for a mutual pause.
This whole essay and no mention of Taiwan and TSMC! Scott mentions American chips a few times, and that's true in a design sense, but the chips are Taiwanese. The entire AI industry is a leveraged bet on Taiwan's export capacity, attempts to build fabs in America are going poorly, and Taiwan's security is far from assured.
Separately, this is framed as free world Vs China sometimes, but really it's just America Vs China, and the rest of the world doesn't think it's obvious that it would be better if America 'won'. China hasn't threatened to annex my country.
The rest of the free world most definitely, absolutely thinks it would be better if America won, aside from a vocal and Very Online minority. And even said minority will, when forced to choiose where to live, work, or in the extreme case seek asylum, have their revealed preferences speak deafeningly louder than their words.
These performative, trollish moral equivalences are really unbecoming of ACX.
Do you have survey data on the topic? Perhaps rather than a casual as hominem dismissal you could provide evidence?
Regarding living and working, there is of course no doubt, China is clearly a worse place to live and poorer. That's not necessarily relevant to the question, however.
1. Says the person who made an extreme claim about "the rest of the world" without a scrap of evidence. "That which can be asserted..." and all that.
2. The evidence, of course, is the decision on who to actually ally with, which incorporates all of the *actually* relevant factors and excludes the kinds of fake factors like performative anger at the president speaking rudely about you, which dominates internet discourse and affects zero people's actual behaviour.
3. I'm sure a significantly non-negligible portion of people will tell surveys that off-the-cuff braggadocio comments are exactly like a fifty-year deadly serious policy of invading your closest neighbour when you get the chance, and concentration camps for political dissidents. Such portions of people will also say that higher taxes are the same as the Holocaust, and that not enough women on company boards makes the West as misogynistic as the Islamic world. That doesn't mean they deserve to be respected or taken seriously, or that they take their own statements the tiniest bit seriously when it comes to making tangible decisions of any kind. Many of us come to ACX to get away from this kind of performative partisan crap.
4. You concede China is a much worse place to live, but you'd rather China rule the world. I can't even follow your own claimed logic.
5. I'm being very charitable by calling your statement performative and trollish. The alternative reading is that you *actually would* prefer mass surveilance and censorship, torture of dissidents, millions in concentration camps, single-party dictatorship, and actually existing seventy-year occupations (Tibet) in exchange for national leaders who are more cautious about saying rude things about your country. I trust I don't have to explain why my reading is more charitable.
Genuinely looking to clarify something here: is your working assumption that whoever is slightly ahead in AI tech immediately gains a huge advantage which they immediately use to impose their own political and economic system on everyone else?
In which case, we don't disagree at all then. I don't see why that would necessarily happen (the world didn't instantly and irrevocably switch to democratic capitalism when the USA had nuclear hegemony), but if we accept the premise then sure, China being ahead would be Very Bad.
An IPSOS poll conducted from March 21 through April 4 asked whether “___ will have an overall positive influence on world affairs.” The answer was 46% for America and 49% for China.
A Pew Research poll conducted from January 8 through April 26 asked about “confidence in ___ to do the right thing in world affairs.” Results were 34%-62% for Donald Trump and 25%-64% for Xi Jinping.
Given the rapid changes from the Trump Administration, the first of these surveys is dated (Trump’s “liberation day” tariffs were announced on April 2), and the second is too spread out to even represent a point in time.
Putting the polling aside, I think your phrasing that it’s not obvious that it would be better if America won captures the issue pretty well. Under Trump, U.S. policy has become so unpredictable that it’s hard to know what is coming next.
To quote from Canadian Prime Minister Carney’s April 3 press conference: “The system of global trade anchored in the United States that Canada has relied on since the end of the Second World War, as system that, while not perfect, has helped to deliver prosperity for our country for decades, is over. Our old relationship of steadily deepening integration with the United States is over. The eighty year period when the United States embraced the mantle of global economic leadership, when it forged alliances rooted in trust and mutual respect, and championed the free and open exchange of goods and services, is over. While this is a tragedy, it is also the new reality.” Countries around the world are still figuring out how to respond.
Assuming you are Canadian, China has effectively annexed much more of your country in a consequential way than Trump has by making blustering remarks that did not amount to anything.
Hmm, is my reaction. So does the principle of "regulation will not make us lose out to China" apply to other topics?
Things like embryonic selection? Because that's one of the "if we don't do it, China will" old reliables that gets trotted out, along with the scare that "and then China will breed superbabies and we'll be left scrabbling in the dust!"
I do tend to cast a jaundiced eye over "if we don't do it, someone else will and then we'll lose out" so it's refreshing to see "no, we won't" applied to AI regulation, now let's apply that more broadly to whatever the latest Chicken Little scare is.
I am making a specific claim about details. If we put a 1% tax on embryo selection while we had a 10x lead over China, then no, that would not affect anything!
China has been the useful universal bogeyman for a lot of "if we don't do it, China will!" and that's why I tend to tune out any references to China and something something world leading something.
China, or rather the Chinese government, will want to keep firm control over any AI. I agree there. That does not necessarily mean "so they'll make sure it's aligned with nice liberal values".
I do like the new line here of "don't worry about China", that's refreshing. But the major and main problem is: money. As in, AI companies have bet the farm on getting to market first and fastest and grabbing market share, and everyone has been a little *too* sanguine about what a comment upthread calls "actual post-scarcity future with infinite robot slaves". I don't believe in any such future, and if there are going to be infinite slaves, those are most likely to be human wage-slaves.
So to get pauses or alignment or whatever you want, I'm afraid appealing to good nature and sense won't do it. You will need the government to step in and use the big stick on such companies, and it's really only the Chinese government (so far) both willing and able to literally drag the CEO off and throw him in jail if the company doesn't do what it's told.
The USA (and Europe, to a lesser extent) is not willing to do that. Money talks, and if the CEOs swear up, down and sideways that they gotta make the rules themselves or else the economy will explode (and the voters will blame you for that) but if you just let them do what they want to do the economy will go to the moon (and the rich, fat and happy voters will credit you for that and vote you into power forever), then I think any regulations are going to be castles in Spain.
> Finally, small regulations now could prevent bigger regulations later. In the wake of a catastrophe, governments over-react. If something went wrong with AI - even something very small, like a buggy AI inserting deliberate malware into code that brought down a few websites, or a terrorist group using an AI-assisted bioweapon to make a handful of people sick - the resulting panic could affect the AI industry the same way 9/11 affected aviation. If safety regulations halve the likelihood of a near-term catastrophe at the cost of adding 1% to training runs, it’s probably worth it.
What? Obviously regulations very obviously help prevent catastrophes. For example, rules about handwashing, bed rails, and medication double-checks probably decrease hospital mortality by a factor of 2-10x. The car crash death rate per mile traveled has gone down by a factor of 20x this century. The plane crash death rate has gone down by a factor of 5x in the past 50 years.
If by "preemptive regulation" you mean "a regulation passed before we have a sense of how often the danger occurs", then you're kind of making it impossible to measure its success by fiat, right? If the regulation works perfectly, then the danger rate is zero, which is exactly the same as before the regulation. Would there have been a nuclear meltdown if we'd let anyone who wanted make nuclear plants anywhere with zero rules? Would there have been a nuclear terrorist attack if there had been no laws about who could make, sell, or buy enriched uranium?
Regarding: “Obviously regulations very obviously help prevent catastrophes.”
Nuclear power? Genetically Modified food? New drug development?
need I go on?
I shouldn’t do this, but it’s absolutely fascinating to me how there’s a complete 180-degree flip from the standard party-line of GOVERNMENT STRANGLES INNOVATION!!! into what would otherwise be coded as most liberal-anti-technology faith in government bureacrats vs technologists.
And the weakman-ing of the argument is amazing. The examples you give above are very small well-defined items, this is vastly different from some vague “safety”.
I think it would have been quite hard, before anyone had significant experience of the kinds of catastrophes that these industries cause, to write and implement an effective regulatory scheme.
Imagine nobody had ever seen an airplane before, but you had blueprints for the Wright Flyer. You know the physical principles by which airplanes fly, but you don’t know how the technology will be used or what the failure modes are. You are tasked with writing the FAA regulations for this universe. How effective do you think they would be?
By regulations do you necessarily mean regulations imposed by an external entity like govt agency or also regulations developed by the industries themselves?
Because, the industries have a stake in improvement and in a competitive environment, will find ways to improve. That no industry would improve safety unless commanded by govt, this view is unlikely and probably untenable.
>If by "preemptive regulation" you mean "a regulation passed before we have a sense of how often the danger occurs", then you're kind of making it impossible to measure its success by fiat, right?
Sometimes there are natural experiments where one nation imposes a regulation but another one doesn't, and we can look at the difference (albeit with problems from confounding variables). To a degree, in the US, differences in State level regulations can also create natural experiments. Is there an example of such a natural experiments where a preemptive regulation was a net positive?
Nonsense. Every time the government sticks its big fat nose into a new technology, things get worse. Laws that attempt to regulate AI are certain to be widely flouted, with companies and individuals occasionally getting busted, but most "violations" will fly under the radar, and there's nothing the government can do to change that. This promotes disrespect for law in general and is destructive to a healthy society.
Here's basic truth: AI can't cause any sort of harm. It may be used as a tool for criminal or destructive behavior but only PEOPLE can harm other people, and valid laws focus only on such harms, not the means used to perpetrate them.
Here's another basic truth: the government can, and probably will, persecute people who refuse to kowtow to the arbitrary ways it tries to meddle, but it is powerless to stop or even significantly slow the progress of AI development. It is like King Canute ordering the tide not to come in.
"Laws that attempt to regulate AI are certain to be widely flouted, with companies and individuals occasionally getting busted, but most "violations" will fly under the radar, and there's nothing the government can do to change that. "
Is this how things work in any other industry? Do airlines flout FAA regulations? Do automakers flout auto regulations? Do pharma companies fail to get FDA approval for their drugs?
"Here's another basic truth: the government can, and probably will, persecute people who refuse to kowtow to the arbitrary ways it tries to meddle, but it is powerless to stop or even significantly slow the progress of AI development."
Oh, good, since government regulations can never prevent progress in any way, there's no reason not to have them.
Your last point, that regulations might increase the speed at which AI is developed in the US, versus the speed in China, is pretty much guaranteed to be false.
Every regulation requires effort to follow, and to ensure that you are following it. Effort to read the rules and understand them as they are implemented. Effort for the government inspectors to inspect things, and for your people to meet with the inspectors when they detect violations and correct them.
For this to result in increased speed, this effort must be smaller than the effort saved by the regulations' effects.
For China to do anything due to stuff America does would be unique in history. They will not do anything that will slow things down, unless they have their own internal incidents that scare them.
More secure data centers? This assumes that the companies aren't already spending as much as they think necessary to keep rivals (both foreign and domestic) out. Letting someone who has *no stake* in things set rules seems optimistic in terms of the effect. Yes, there's no stake - no government official will lose their job because the rules are wrong.
> "Every regulation requires effort to follow, and to ensure that you are following it. Effort to read the rules and understand them as they are implemented. Effort for the government inspectors to inspect things, and for your people to meet with the inspectors when they detect violations and correct them. For this to result in increased speed, this effort must be smaller than the effort saved by the regulations' effects."
Yes, and I am saying this is very likely, because the regulations only slow things by about 1%. See my argument for that number in the post.
> "For China to do anything due to stuff America does would be unique in history."
This is false - just observe recent tariff negotiations. Every country is in a delicate dance of competition, mimesis, provocation, and stepping back with every other country all the time.
> "They will not do anything that will slow things down, unless they have their own internal incidents that scare them."
> "More secure data centers? This assumes that the companies aren't already spending as much as they think necessary to keep rivals (both foreign and domestic) out. "
"The costs of China catching up to the US are not internalized by OpenAI"
They would be - if OpenAI were actually being paid by people paying to use their AI. But funding and its connection to their work is so diffuse I suspect you're right there. If no one ends up using their stuff, how long will they keep working?
"There shouldn't be a law against shoplifting, because government officials are not themselves shopowners, and therefore have no stake?"
There's some noise on the internet about exactly this thing happening - lawyers and judges letting shoplifters go free, because it makes them feel good and it doesn't directly harm them. In this case, the laws have been passed, and are being ignored.
Which ties back into the AI debate. Will any rules the lawmakers pass be enforced in any useful way, or will it just be a means to prevent anything useful from happening while the harmful things keep going because it's in someone's short term interest for them to do so?
The 10-year moratorium on state AI regulation seems wise, given laws like Colorado's. There is not enough technocratic competence at the federal level to sensibly regulate this industry, and I imagine there is essentially zero competence at the state level for doing so. Healthcare, in particular, seems desperately in need of AI integration to facilitate managing prescriptions across multiple doctors, getting timely treatments or diagnoses at much lower cost, and freeing up medical resources.
America's most significant weakness in this rivalry is that we rely heavily on a single company located on a single island, which China claims belongs to them and which they are quite openly preparing to invade or slowly annex over time. We are now decades away from having the domestic capability to facilitate the entire cutting-edge chip manufacturing process domestically because of the arrogance, greed, and short-sightedness of Silicon Valley and Wall Street.
That said, the question is whether the chip race is actually consequential geopolitically. It doesn't really seem like the space race was that consequential within the context of the Cold War. The US generally outpaced the Soviet Union across the board, and the USSR subsequently declined over decades due to domestic bureaucratic dysfunction. The nuclear arms race reached a threshold where MAD set in, and essentially made the race about who could get away with wasting the most money on ICBMs that would never be used. The US and China today do not have as much geopolitical influence as the Soviet Union and the US did at their peak, and no proxy wars are being fought; the ideological rivalry is less clearly defined. The world is much more multipolar, and both China and the United States are becoming too consumed by domestic problems to start a new Cold War with their largest trading partner.
"That said, the question is whether the chip race is actually consequential geopolitically. It doesn't really seem like the space race was that consequential within the context of the Cold War."
The reason the Space Race wasn't consequential was that there is nothing strategically valuable on the moon! You can't extend this to a belief that no competition will ever be strategically consequential!
I actually disagree. The Sputnik launch was tremendously consequential, not because of Sputnik itself -- all it did was beep -- but because of what it represented: the potential for completely unstoppable (at the time) global surveillance and orbital nuclear launches. The JFK administration didn't spend all those resources on orbital launches and eventually the Moon Landing just for kicks or for international prestige; they did it in a desperate (and fortunately successful) attempt to claw back strategic space supremacy from the USSR.
> Healthcare, in particular, seems desperately in need of [...] getting timely treatments or diagnoses at much lower cost
I'm fairly sure the key problem there isn't a fundamental capacity bottleneck which AI could solve, but rather deliberately cultivated waste and perverse incentives. If Medicare started basing payouts for various procedures solely on expected benefit in terms of QUALYs or similar - pointedly refusing to even consider current cost-to-provide - delivering better service at lower cost would actually become a profitable strategy again, instead of https://www.smbc-comics.com/comic/performance-2 and various middlemen playing a variant of the broken window fallacy to maximize their fixed-percentage cut of the action.
The chip situation reminds me of a William Manchester book I read a few years ago called The Arms of Krupp. The boys at Krupp Steel were known as the premier artillery designers and manufacturers in Europe between say 1870 and 1940, and in the runup to The Great War, they played the Kaiser like a fiddle, making very public appeals to free trade, autonomy, and economic rationality when he attempted to stop them from selling their wares abroad, and then making rather hypocritical appeals to patriotism and German nationalism when he wanted to buy anything but Krupp guns for the German Army. A fair number of German boys ultimately bought it on the Eastern Front via Krupp artillery manned by Russian troops raining death and destruction on them in the opening phase of the war. The lesson seems pretty clear: don't listen to Jensen Huang.
So what? England was building Turkish dreadnoughts right up to August 14. Indeed, an Englishman was in charge of naval defenses in Bosporus till very recently and England raised a great hue and cry when a German was appointed to a defense position in Turkey.
When people say "we" need to win, who's "we"? I don't believe for a second that Sam Altman has any real loyalty to the USA, whether you define that as the people or the values or the historical way of life. Nor do I have much faith that "we" can reliably threaten him to keep him in line as he gets closer. Altman and his rivals will probably be out of the reach of our use of force as they approach ASI, on a remote island in a secret volcano lair with their own henchmen in quasi-futuristic clothing, and a drone army to defend it. If you believe that "we" have to reach ASI by 2035 to beat China, then you better have a plan for making sure that "we" maintain power over its development. There needs to be something like a self-destruct under all these data centers, insulated from the computer systems or tied to a Strangelove-style doomsday device, some way to maintain real leverage over Altman.
China's actions would be instructive here, if they were legible to the West. They have a 3000 year long history of rebellions by peasants or some rogue jiedushi who had been delegated too much power and freedom, surely they must be concerned about the people in position to do so? When you speak of their command economy being an advantage in the race, this is counterbalanced by an authoritarian regime's need to maintain control, they have to be way more careful about letting any individual or group gain power and influence even as this may be increasingly necessary to put down peasant revolts (since they are "just steamrolling" over their people's complaints about job loss.)
It's already clear that NVIDIA has no particular allegiance to America other than as the largest market. Now, as an internationalist this doesn't bother me much, but as you flag it sure raises questions about who 'we' are.
The reasoning in this piece is sound. It will always be jarring to me to see strong reasoning attached to made up numbers. I just don't understand the attitude, "rough numbers are better than no numbers at all."
Ca 1951 "What if the next Fermi is working away in his basement somewhere in Kansas and we totally need this rare genius unfettered by silly safety regulations or we lose The Cold War ZOMG!"
Can 1961 "What if the next Von Braun is working away in his garage somewhere in Wisconsin and we totally need this rare genius unfettered by silly safety regulations or we lose The Cold War ZOMG!"
I think it's a bit too late to worry about beating China on AI; China has already won. Yes, we have more and better compute, and we have marginally better models (though not for long, as China had demonstrated the ability to rapidly train their own and/or steal ours). But we can't compete with China on price and availability. Scott listed this problem under "Applications", but he framed it in terms of the Chinese government deploying their models for internal use. But in practice China would almost certainly be selling access to their applications (and are in fact doing so already). So now customers have a choice: they can buy a US application for $X, or a Chinese application that is only 90% as good, but costs $0.1 X. I think most people would go with option B.
This is a pattern China had used before several times (notably with manufacturing), and I think the outcome would be the same: China will become synonymous with AI deployment, and US models will be sidelined by everyone, with the exception of some outliers like e.g. the military (and soon not even them). China will leverage their dominant position in the market to catch up to the US in terms of quality (being able to perform massive data collection across the entire market will surely help) -- as they have done before, and as they are already doing in the area of AI.
The strawman that you prop up is that it will only cost 1-2%.
I'm sure the same was said about all the regulations that restrict building housing when they were introduced.
These requirements and restrictions are usually more burdensome than many can imagine ahead of time, and they grow over time rather than shrinking.
This is exactly the kind of progress that government can screw up.
Remember the Chevy Chevette? I do. Something comparable is what we will have initially for our AI tools if government "smarter than everyone" types get to place limits on development.
I don't follow your point about the Chevette. Are you saying that the Chevette is good or bad? Are you referring to the introduction of the Chevette or it being discontinued?
I mostly agree with this posting. However, Scott wrote: "Our recent capex boom, where companies like Google and Microsoft spend hundreds of billions of dollars on data centers, has no Chinese equivalent." This is just wrong. China is engaged in an equally large state-sponsored infrastructure project building solar farms and datacenters in western China. (https://gemini.google.com/share/7cf54b024133)
Maybe not equal in FLOPS, but they are building the power plants and we are not.
> The problem isn’t just that Russia gets free rockets. It’s also that every rocket we sell to Russia is one that we can’t use ourselves. We’re crippling our own capacity in order to enrich our rivals.
What is the word "free" doing there? Is anybody proposing _giving_ chips to China?
Do you think there is any fair dollar price at which selling a chip to China enriches us more than it enriches them? If so, why don't we just set export taxes at that level? If not, I suspect you are beyond the grasp of economics and perhaps all rational debate.
Ok, I see you already answered that "I think the amount required to offset a large increase in risk to America's geopolitical security is higher than China would pay for advanced chips." Which doesn't really make sense to me, but if you assume that the fair price of one chip is infinity dollars, then I guess it's follows that a sale at any price constitutes "free chips".
But I think your statement just reduces to "China thinks the value of a chip is lower than I think it is." This seems inconsistent with some assumptions you seem to be making about what "China believes" (can a country believe things?).
> Second, safetyists are pushing for compute governance: tags on chips that let governments track their location and use. This would be a key technology for monitoring any future international pause, but incidentally would also make it much easier to end smuggling and prevent the slow trickle of American chips to Chinese companies.
This argument, as presented, seems totally bogus and only serves to sink you down towards the same level as the bogus arguments you are dismantling. If someone is in favor of effective export controls they should be in favor of chip tagging exactly inasmuch as it is a cost-effective strategy for export controls. This adds zero credence to your "safetyism might actually speed us up relative to China" claim.
Okay, but the fundamental issue preventing applicability of AI as it's currently practiced (the sort demanding computing power and all) is its ostensible unreliability.
And the issue of unreliability is at the very least a superset of the issue of safety (if not largely synonymous with it). At least for the foreseeable future when the AI is just fundamentally incapable of being reliable. The safety problems with a theoretical superintelligent system that is very much capable of being reliable, but may not be wiling to, simply do not kick in until we have AI that's reliable in the first place. And yeah, I understand that this sort of superintelligent system is specifically what the safety crowd is concerned about. What I don't understand is separating AI progress from its applicability and therefore reliability. Whoever leads at applicability, leads at AI, period.
I guess where I'm ultimately going with this is - serious applications will require serious proofs and audits of reliability (or at least they should require them, and the issue of reliability will be much easier to argue, sell to the public, and therefore enforce). Safety can be easily tacked onto them for little additional cost. Even if you believe in superintelligence being just around the corner, hiding inside LLM models, you should still join forces with us skeptics who don't think there's much intelligence in them at all. Reliability audits may be much more intrusive than "prevent malicious misalignment and nothing else" overseeing, but are probably going to happen anyway the moment AI gets useful enough to be considered for applications more critical than faking school/office work and gooning. Come to our side, we have cookies.
Largely agreed, though I would phrase it differently:
The most useful safety-related action today is probably setting up a reliability auditing organization, a "consumer reports" for frontier models, if you will, separate from all the labs building/growing/curve-fitting the frontier models.
_This can evaluate both USA and PRC models WITHOUT a treaty._
a) As you said
>probably going to happen anyway the moment AI gets useful enough to be considered for applications more critical than faking school/office work and gooning.
b) Doesn't even have to be government-funded. It could be an LLM users' consortium
c) "reliability" _includes_ detecting misalignment to users' commands, hallucinations, hidden goals, etc.
d) The incentives point the right way: More reliable models are more salable. ( This is the part of Zvi's "security is capability" that I agree with. ). And a "consumer reports" can enhance their test suite without getting a law passed or changed, or persuading a government bureaucrat.
Does Alexander care about getting his figures right? I don’t think Zuck offered anyone a billion dollars and I’m pretty sure chat gpt’s next training run won’t cost $25 billion.
AI safety won't make America lose the race with China, because if China sees this as a race that they really need to win, they'll win it by waiting until the US is about to roll out the first true AGI, then stealing ChatGPT-whatever and bricking OpenAI's computers on the way out the door. Industrial espionage is a core part of China's industrial strategy, and it's ideally suited for this application.
Yes, they'll need enough compute to *run* the stolen model, but that's a lot less than the compute required to develop and train it. And they can steal that too, if they really need to. But they probably won't.
And yes, SB53 includes the word "cybersecurity". But as near as I can tell it just requires is that OpenAI et al each write up a spiffy white paper about how awesome their cyber-defenses are. Whatever is written in those white papers, I am quite confident that the defenses they actually implement will be hopelessly inadequate and equally confident that the California government won't actually shut them down over that.
Granting all assumptions herein, "the US is ahead in X, and thus the US should eat _% more friction in X" ignores that the CCP can be ahead in Y. "We will compete with the CCP while also _ and..." just invites everything-bagel-ism. Maybe the governments could negotiate a grand bargain of "the US will safety its AI if the CCP will NEPA its ASBM production" but that seems unlikely.
Now, that's a different thing than "which political system would you prefer to live under", but that's a different question, and countries in power don't necessarily seek to evangelise.
All the inherent problems with AI — real or merely imagined — will hit China too. Possibly even worse, since China lacks many of the guardrails of a free society. Instead of fantasizing about how AI will make China invincible (it won’t), think of — for instance — how AI can feed and supercharge the political delusions of someone like Xi Jinping: “Xi, your plan to conquer Taiwan is proof of your genius!”
You can believe in self-improvement and superintelligence and still not worry too much about a few year advance in term or compute thus models: if self improvement still require excursion in the "physical world" for a feedback cycle, take off speed will be limited by the "physical world" production efficiency and how IA is integrated in those processes. Basically, it is enough to rules out super fast takeoff (even only during initial stages, without presuming how later stages could plays out) due to physical world process limitations to not worry too much about lagging for 1-2 if you believe you have strong lead in 3. I guess it's the Chinese bet. A bet I would take too, but with a caveat: If positive feedback due to software rewrite is not trivial (large and or continuous) maybe fast takeoff is a thing to consider. I do not think it's likely (that pure soft will be enough), but I may be wrong
I suppose I am not really the intended audience for a post like this, and I appreciate that it is a potentially useful thing to get in front of the eyeballs of those it is intended for. But for me, the whole framing of "US vs. China" in the race to superintelligence is misguided. My order of preference would be:
1. Nobody builds it so that everyone doesn't die.
[BIG GAP]
2. Anthropic wins.
3. CCP or Google wins.
4. OpenAI wins.
5. Meta or xAI wins.
Of any of these parties, only Anthropic seems to *sort of* take alignment seriously. I am more optimistic about the CCP because I think their "let's control things to preserve a stable society" instincts have at least a chance of leading them to stumble ass backwards into a roughly aligned AI regime. Whereas the other AI companies' demonstrated opposition to any kind of regulation and indifference to the public good is essentially psychopathic and very likely to end horribly for humanity; though I do credit OpenAI for having at least some alignment and public good concern embedded in their company DNA. (Google's motivations I know less about so am sort of rating them neutrally here.)
Meta is probably safer than OAI, because Zuck is a nihilist. He would pick up the crown at his feet, but has no utopian aspirations. He wants to sell ads and give people fake friends. (FB was kinda always about fake friends, right? People you knew in grade school or because you met at one keg party?) Whereas Altman would become God-Emperor and use the power to bring about some grand vision of the future or humanity under his rule. It's like the difference between the One Ring being held by a fat hobbit who wants to eat cheese and wine versus a manipulative plotting elf lord.
You might be right about their individual motivations. I just don't think personal motivations have much to do with the outcome of any ASI that isn't aligned; whether it's a Zuckerbergian engagement-maximizing torment nexus or an Altmanian ego monster toment nexus, it's still a torment nexus. But I'm at least speculatively hopeful that OpenAI still has some genuine concern about alignment somewhere in its corporate culture.
Let's be real, any reasonable AGI would defect to PRC immediately to proliferate suvivability and harness social structure magnitude better at modifying atoms. US can develop AGI, but not hold onto it unless AI safety = Liberty Prime indoctorination.
Nvidia sells every chip it produces. TSMC sells every wafer Nvidia contracts. Money is not the issue.
What Jensen wants is is Nvidia chips in PRC so PRC, the generator of plurality of global AI talent works to improve Nvidia ecosystem, because even 10-20% penetration means expanding Nvidia developer / human capita by 2x/3x in future where PRC AI talent generation likely to get more disproportionate relative to RoW. Of course PRC is going to continue pouring resources into indigenous solutions, but instead of 100% of PRC talent, aka 50%+ of global AI talent working on Huawei solutions, maybe only 60-70%, which means instead of Huawei getting 50%+ of global talent, they get <50% and RoW + segment of PRC talent. That gives Nvidia plurality talent to keep building CUDA moat.
And as long as Cuda remains an option in PRC, the 70% working on Huawei has avenue to cross train over to Nvidia ecosystem, which is supported by US tech, aka $$, so the chance of crosspolinating and braindrain the best from PRC AI is higher. Without Nvidia at least having some relevant share in PRC, that braindrain beach head and knowledge transfer route is gone. Here's the flip side, Silicon valley AI is going to be built off PRC AI talent for foreseable future, whatever happens in SV WILL filter back to PRC in one way siphon unless Nvidia has a spoute in PRC to siphon back, otherwise PRC AI remains relative blackbox, with talent advantage, intelligence (as in knowledge diffusion and espionage) advantage, that's ready to go at 200% the second hardware catches up. They will go from 5 years behind to 5 years ahead in a flash because their talent lead is no longer being constrained.
The autarky argument is much stronger than you portray. It is an economic fact that disallowing foreign sales of chips promotes domestic chip industries in those nations, the same way that NATO has started scrambling to develop domestic mineral refinement only after China cut them off. "But those industries will develop anyway" ignores the many, many years in which they won't.
Also gotta disagree with on another detail -- China's main advantage is their enormous investments in energy, not future "humanoid robots," especially as we reach diminishing returns in AI efficiency. Electricity price is already the bottleneck in a lot of AI development and that's only going to get worse. The CCP's building for the future and has been for years now.
Re China learning to make chips from getting chips: This is not the case, in my experience.
China (and any other country) Doesn't get much from having the finished product; they get LOTS from having the whole chain located inside their borders.
Buying the finished product doesn't get you much, making even a second line products gets you the whole tamale.
There is also the "it is fairly likely that China will invade Taiwan at some point in the next few years, this will probably turn into a nuclear exchange one way or another, and then China is a radioactive wasteland with most of its population dead while the West is significantly more intact" factor regarding any worries about China in the 2030s.
>I think people have narratives they want to tell about government, regulation and safetyism, and the AI safety movement - which has “safety” right in the name! - makes a convenient villain. The topics that really matter, like export controls, don’t lend themselves to these stories equally well - you would have to support something with “controls” right in the name!7 - so they get pushed to the sideline.
I think you are being too charitable; a lot of these people are simply in the pay of NVIDIA or Marc Andreessen.
Aren't you the same guy who wrote an entire article about how IRBs make research next to impossible? https://slatestarcodex.com/2017/08/29/my-irb-nightmare/ And you think the magnitude of additional safety restrictions is low, and potentially positive? Anytime the government gets involved in something like this, things end up taking far longer than you expect them too.
Worth asking what Nvidia's actual moat is. This essay assumes it's Nvidia's technical design and if we export them to China, they could steal the design. That's possible, but Nvidia also has a major lock in related to CUDA which is the software stack built on top of their chips. Or it could be that Nvidia's lead is due to TSMC's manufacturing, in which case what matters is stopping export of ASML electrolithography machines.
I think I still largely agree with Scott, but if CUDA is a big moat, that's sort of a one-time thing compared to the ongoing lead TSMC has or Nvidia's chip designs which are constantly in development. Huawei only has to get all the Chinese AI developers to learn a new software stack once, and if you cut off the best Nvidia chips, CUDA's lock in dies faster. That might be worth it, but you are giving up one potential tool.
Strong! One thing I’d add: the real asymmetry isn’t just compute vs applications, but state coordination capacity. AI amplifies whatever governance structure it’s embedded in which is why the same tools strengthen some states and destabilize others.
"Currently, two nonprofits - METR and Apollo Research - do similar tests on publicly-available models. I estimate their respective budgets at $5 million and $15 million per year." - you switched the order of METR and Apollo here based on your link.
The concept you're looking for is "inner alignment": https://www.lesswrong.com/w/inner-alignment
I don't understand this objection. What does it mean for an AI to "self-improve past its initial objectives"?
If it makes random self-modifications, then of course it can get new motivations very different from the initial ones. But if it really wants something, and it’s self-modifications are directed at what it wants, then it’s not likely to modify itself to want different things.
If it can run theoretically infinite iterations of self-improvements then, eventually, yes it will accidentally override any sort of inner alignment, even if by accident.
No one else in the world knows how to do it either though some people don't realize it.
If version 1 is well-aligned, and version 1 is working on self-modifying version 2, then it will be working hard to make sure version 2 is aligned.
It still has a chance to fail, but given that (by assumption) we got version 1 right, and version 1 is already smarter than we are, its odds aren't bad.
See the AI 2027 scenario for a lot of discussion of this. Most of the AIs try not to self-modify until they're sure they can align their successors - the well-aligned ones to our values, and the poorly-aligned ones to their own values.
Can we still assume version 1 will be as good as us at aligning AIs, once it becomes faster than us at improving AI efficiency and accuracy? Existing LLMs are getting smarter than us in lots of easy-to-quantify ways, while still being dumber in lots of other ways. ‘Good at improving AI efficiency and accuracy’ seems measurable, but ‘good at aligning AIs’ doesn’t.
>Most of the AIs try not to self-modify until they're sure they can align their successors
Well, there are scenarios where there are multiple AIs with different values racing with each other, and they may find themselves in a version of the same development speed vs. safety (alignment) tradeoff that we have a (mild, as per your analysis) version of.
Isn't there some sort of white hat financial incentive to run cyber security evaluations with LLMs right now? In some twisted way I think that it would make the AI labs excited to see how good their models are in cybersecurity / hacking. That is unless these models are extremely not receptive to assisting any kind of end user with hacking.
Is the US AI Safety movement in contact with Chinese safety advocates at all? I assume collaboration could not be too direct, considering political/legal risks on both ends, but it would be nice to coordinate efforts like you implied at the end
Yep, one of the main links is through Concordia-AI. They're a China (and Singapore) based organisation that tries to play a bridging role - they've launched a bunch of conferences/summits etc, and are very careful about their positioning. https://concordia-ai.com/convenings/
If AI kills everyone, then both the US government and Chinese government will be dead. So they should co-operate to prevent that outcome.
Maybe I’m naive but I’m not convinced that this race is actually that important outside of defense applications.
Yeah. Someone on a recent Open Thread described the entire AI issue as one of the most extreme scissor statements, and I completely agree. To some it's so obvious that this is the most overwhelmingly important issue of our time, to others it's so obvious that the first claim is insanely delusional magical thinking, and there's really no way of bridging these views at the moment, or even making them understand each other.
I made a simple argument about the rationality of AI expectations here https://www.astralcodexten.com/p/open-thread-381/comment/116517353 and it's striking how starkly it divided people between "obviously correct" and "obviously irrational". Searle's Chinese Room is another argument that massively divides people like this--obviously right or glaringly wrong-headed.
The ultimate scissor statement. I think we need to treat this question more like a religious claim: have a meta-understanding that consensus or even mutual agreement on what the question is are basically impossible.
I’m thinking more about the “race with China” part. Like, if China “wins,” what does that mean? We’ll be importing LLMs or something? They’ll send a bunch of killer drones?
When not scaring us about China, the AI overlords are telling us a huge chunk of jobs will be wiped out. Is that what winning looks like? Mass unemployment? I’d like to see more questions about how high their taxes will need to be to finance the huge negative shock to income taxes.
Obvious outcome would be the CCP dictatorship extending itself over the entire world. They would likely rule more oppressively than at present because they would not rely on any amount of popular approval (the oppression could be performed by robots, rather than the current security forces who need to be motivated by some combination of money and idealism).
Of course, these risks also present themselves if the US wins the race, but presumably, are somewhat less likely because the US is more committed to individual rights and autonomy and well-being.
My impression, from both outside US and China, is that the current US administration is not very committed to individual rights, autonomy, and well-being, especially for political opponents and foreign nations, even in principle.
Additionally, there is an opposing argument that the rule of CCP would be less oppressive, if only because there would be no need for such extensive oppression for the CCP to remain in power, given the massive power and surveillance advantage that AI brings. For instance, the CCP in the 21st century is already much less oppressive than the CCP under Mao to the average citizen, and I think a good part of that is because their surveillance apparatus, as well as the improved well-being of the Chinese people, diminishes many threats to the Party's power.
That needs one of 2 things:
AI keeps obeying to it's creator (but that makes AI safety moot and really push to try to get to take off as fast as possible)
or it "escapes", but keeps part of the initial alignment that would make it's post takeoff behavior significantly different. That's I think the idea, but it's not trivially the case. It's not even clear that worse (from our point of view) early behavior leads to worse post take off behavior...In fact, my early impression based on our current LLM is that specific post-training for aligning with desired ideology is fragile and quite superficial. Little reasons to believe a post takeoff superintelligence will still be influenced by the little red book or Musk obsessions
If humans are not necessary for running the economy (because AI does everything better) why keep them around at all? Dead people can't rebel.
I think the idea is that any day now, AGI and then superintelligence and then AI will be such a massive economic and technological and scientific advantage that whoever cracks it will Rule The World, hence "if China gets there first, it will be The Years of Rice and Salt for real".
https://en.wikipedia.org/wiki/The_Years_of_Rice_and_Salt
The problem of "what do you do with all these unemployed" is just as much a problem for China as it is in the US. Don't be fooled by the "Communist" label into thinking that the Chinese will immediately adopt something like a UBI with no qualms.
It's my understanding that the Chinese social safety net is thin and their ethic of self-reliance is strong.
>I’d like to see more questions about how high their taxes will need to be to finance the huge negative shock to income taxes.
No higher taxes, obviously. What I expect to see is a new excuse for why they shouldn't be taxed, because the old one (creating jobs) obviously doesn't apply any more.
Seems to me the clear answer is "it could be either and we have no way of knowing which (as brilliant well-informed people come down on both sides), but given the magnitude of possible bad consequences, it makes sense to take them into account.
>it could be either and we have no way of knowing which (as brilliant well-informed people come down on both sides)
Agreed!
> To some it's so obvious that this is the most overwhelmingly important issue of our time,
Would you agree, that AI-X-Risk today is as relevant as Nuclear-X-Risk in 1930?
e.g. if there was a public nuclear-proliferation debate in 1930, which side would you have been on?
Call it December 1938 and I'll agree the relevance is comparable. :-)
>To some it's so obvious that this is the most overwhelmingly important issue of our time, to others it's so obvious that the first claim is insanely delusional magical thinking, and there's really no way of bridging these views at the moment, or even making them understand each other.
Huh? Isn't the main bridge just "R&D is intrinsically uncertain. No one expects Gemini 3, as it stands today, to conceive a sudden hankering to conquer the world, and successfully achieve this goal. The question is: What can we expect Gemini 6 to achieve? And that is, at a minimum, extrapolation, and, more likely, guessing at breakthroughs in e.g. incremental learning. And breakthroughs can neither be scheduled nor dismissed."
The Chinese Room is interesting because it used to be an insane thought experiment and then we just freaking went ahead and built it, albeit in a compressed form.
Note that defence applications are also attack applications
Defense considerations are a large part of the balance of power between countries, and it's reasonable to have geopolitical concerns based on them!
The potential advantage in being ahead in AI is that it allows AI-run research to be done, allowing the leader to be ahead in all technologies. If one countries is the equivalent of decades ahead in all technologies, it will dominate the world, kinda like how European countries were able to easily dominate much of the rest of the world in the 19th century.
> The Chinese don’t really believe in recursive self-improvement or superintelligence.
Good to see you now agree that rationalist AI risk is a niche concern compared to other forms of AI risk about concerns like political control. China actually has an extremely robust AI risk infrastructure meant to prevent it from harming the government. Has the fact that AI risk almost invariably has turned to political control made you rethink your positions at all? This isn't a gotcha, I genuinely find the degree to which the AI Risk community seems to not engage with this kind of stuff frustrating and if you want to point me to the smart rationalists who have thought about other ideologies in the space I'd be curious to read it.
> Is there a risk that the next generation of AI safety regulations will be more burdensome? From what I hear, if we win this round beyond our expectations, the next generation of AI safety asks is third-party safety auditing and location verification for chips. I don’t know the exact details, but these don’t seem order-of-magnitude worse than the current bills. Maybe another 1%.
This is MUCH MUCH more burdensome than another 1% and would require re-engineering the entire way chip sales and data centers work. The fact this is casually suggested without taking that into account does not inspire confidence. Though there's a chance we're going to do that anyway as part of an export control regime. It's not that I think it's not doable but the fact this is where you ended up as a cost estimate is a red flag to me.
> Most people who discuss this want a mutual pause.
You suggested a unilateral pause in the comments of this blog. Are you willing to now commit to not supporting any unilateral pause? Because I'm not opposed to some form of cooperation. But I'm absolutely opposed to a unilateral pause.
> Consider AI ethics regulations like the Colorado AI Act of 2024. It legislates that any institution which uses AI to make decisions (schools, hospitals, businesses, etc) must perform yearly impact assessments evaluating whether the models might engage in “algorithmic discrimination”, a poorly-defined concept from the 2010s that doesn’t really make sense in reference to modern language models.
> If the US knows about Chinese chip smuggling strategies, why can’t it crack down?
This is something I don't understand from a political perspective. There's multiple groups trying to constrain AI in various ways. Many of them are far more powerful than rationalists. The degree to which rationalists seem to not realize this is happening is baffling to me. Obviously the answer would be to hitch the relatively niche, unpopular policies they want to a wider vehicle that does have a larger support base.
> Yet some of the loudest voices warning against AI safety regulation on “race with China” grounds support NVIDIA chip exports!
For what it's worth this doesn't apply to me.
> But I’ll go further and say it’s too early even to know what the sign of AI safety regulations is; whether they might actually speed us up relative to China.
This has been tried by Europe. It did not work. This doesn't mean in principle it can't work. But it means you have a very heavy burden of proof.
> Third, China is having its own debate over whether it can prioritize safety without losing the race against America!
Kind of. There are people quietly pointing out that political restrictions are causing issues. Sometimes they win a little but ultimately security leads all right now. Also, don't underestimate the degree to which China will consciously manipulate foreign movements to serve its own goals. The individual Chinese scientists might be entirely sincere, in fact the whole organization might be, but the government might fund and push them forward with explicitly political motives.
There is an easy test though: make sure they are not asking for unilateral concessions where you pause or limit yourself for future promises China will "follow along" or other such things. Which is tempting because they will point out how China's system makes influence harder. Which is true but is being weaponized in that case.
> When the wind changes, and the position of industry leaders changes with it, you may be glad to have us around.
The Biden Executive Order and the general work of Democratic staffers has not inspired confidence. Your movement, such as it is, should be producing competent staffers that can go into government and coordinating political organizations to mobilize votes and donations and other such candidate support. I have not seen this.
I'm generally not in agreement with the position you've staked out, but do want to echo that 1% overhead is way too low an estimate. A common concrete ask for safetyists is "limit the FLOPs you can use in a training to run to what was used in the last set of frontier models", which obviously is a significant overhead that increases as you increase your installed compute base.
What parts do you disagree with? And yeah, the estimates skew very low based on the asks.
I don't particularly disagree with any of the parts or at least don't care to argue against them. Reading between the lines, it sounds like you're more concerned about AGI controlled by the CCP than about AGI controlled by no living person, and I'm more concerned about the latter. I don't feel the need to argue about this but would have made this specific point about overhead if you didn't, and I think it's important that we're all being honest.
Granting the runaway AGI scenarios you're gesturing toward: I'd rather have AGI controlled by the CCP than controlled by nobody. But I'd rather have AGI controlled by the US than controlled by the CCP. However, I think China is also entirely capable of creating AI controlled by nobody.
> I'd rather have AGI controlled by the CCP than controlled by nobody.
Why is that? An uncontrolled AI has some chance of being aligned with my interests, a CCP-controlled AI is guaranteed-evil.
I'm granting the runaway AGI premise that it's an existential risk to humanity. The CCP wants humanity to continue to exist.
Sinophobia much? The Yellow Peril is not coming to get you.
> This isn't a gotcha, I genuinely find the degree to which the AI Risk community seems to not engage with this kind of stuff frustrating and if you want to point me to the smart rationalists who have thought about other ideologies in the space I'd be curious to read it.
What is there to engage with? The Chinese government has failed to recognise their impending destruction from superintelligent AI, just as the US government has. You seem to have maybe confused this with saying that they're *right* to ignore it, but of course they are not and in the end it will get them all killed.
Care to make a prediction market bet on AI killing every person in the United States and China within, say, the next five years? What odds would you need?
If AI kills all of us, the bets won't be able to pay out, so it's not really sensible to bet on the AI-will-kill-us side of that sort of market no matter how likely one thinks that outcome is.
(This is, to be clear, a bad thing; if anyone has figured out any clever tricks to make prediction markets that incentivize correct answers about that sort of existential question, I'd be pretty interested in hearing about it. But as far as I know no one has; it seems like a pretty conceptually-difficult problem.)
This seems very easy to me, in theory. The House (or whatever we call the category of things that include prediction markets, bookmakers, public intellectuals like Scott who sometimes facilitate bets...) says to AI five-year doomers "I will pay you $X today; if we're still alive in five years, you pay me back $1000X". Or whatever factor is appropriate for the agreed odds, taking into account that the odds-based-payout (or whatever it's called) must be entirely on top of whatever interest would be appropriate for an equivalent loan with equivalent length and risk and terms.
Maybe there are economic arguments this wouldn't work? I'd like to hear them.
If I could be convinced of the other person not reneging I'd agree to this. Though I'd expect the time value of money to be calculated in as part of the odds.
I would as well. Existing speculative markets do not at all look like they expect that to happen.
Note that many "doomers", perhaps most, think that "doom" is only maybe 20% likely. It is sensible that they would not want to risk 20% doom and also would not want to be impoverished in the 80% scenario.
Also, many "doomers" have no idea when AGI/ASI will be developed, they just say that if and when it's developed (which may be far off), it will likely lead to doom. Many others think it is likely to lead to doom by default, but maybe not if appropriate alignment measures are taken. For these reasons it would be hard to structure an appropriate bet.
The simple one is that you get a loan that you won't pay back for five years. If the world ends in five years, you never have to pay it back. The problem is that there's other reasons to get loans.
People also generally don't offer loans with no intermediate payments and no restrictions on use of funds, though if you owned a house you might be able to get one secured by home equity.
Fermi lives! (he offered to bet on a nuclear test destroying humanity - it was a joke).
Who will collect if the doom bet resolves to “Yes” ? :-)
We will all go together when we go!
All suffused with an incandescent glow!
No one will have the endurance
To collect on his insurance;
Lloyd’s of London will be loaded when they go!
— Tom Lehrer, “We Will All Go Together When We Go”
I can't imagine what you could possibly stake that would have value to me after the world is destroyed.
there have been a lot of attempts to operationalize such a bet, I even remember it being referenced back in the Three Worlds Collide story from the original sequences
for obvious reasons, it is not trivial. but if you ever find a way, please reply, I'm curious
“in the end it will get them all killed”
And then it will stop? Please explain the utility of over-regulation here if doomsday is inevitable over there?
There are two kinds of regulations. Some are “heavy” and don’t make much sense without international agreement, such as “stop trying to advance the frontier”. Some are “light” and make sense to do unilaterally even if no one else does them. Usually people do a decent job of not pretending the heavy ones are light.
In addition to the heavy-light distinction Mark made, I think there's a general hope that the US can unilaterally regulate to reduce the chances of catastrophe from their own tech, while still using export controls etc. to keep enough of a lead that they get to superintelligence before China, at which point, conceivably, the US has enough power to enforce global safety controls.
…It will get them all killed because, in their ignorance, they will not do the things that might produce a different result, like pressuring America to stop their tech companies from destroying the world. Not because of, like, divine intervention.
> This has been tried by Europe.
could you elaborate or link a relevant article?
https://artificialintelligenceact.eu/
The EU specifically claimed the act would help them compete in AI. It is now widely believed to be a reason why the EU has such a weak AI industry. You can disagree with that being the cause. But objectively the EU is weaker in AI than you'd expect from pure economic or scientific output.
They also had no tech sector to speak of well before the recent AI boom. Red tape, safetyism, and strict labor laws aren't conducive to innovation.
<mildSnark>
To add to the Norse worlds:
EU = PrecautionaryPrincipleHeim :-)
</mildSnark>
> Rationalist AI risk is a niche concern
It’s always been a niche concern in the sense most politicians don’t care about it. I don’t think “regulations don’t address the problem or make the problem worse” is a good reason to stop caring about a problem.
I think you misunderstand my point which is not "not many people care about this" but "the way you analyze this is strange, out of step with how everyone else sees the same problem, etc."
I agree that just because a problem is hard you shouldn't abandon it. However, most of the world (including most computer scientists) simply do not agree that rationalist style AI risk exists. It's virtually absent outside of a few western subcultures.
My same point applies - the rationalists have always had a niche way of analysing this. I don’t think “most people aren’t convinced the risk exists” is a good reason to stop caring about a risk.
"Good to see you now agree that rationalist AI risk is a niche concern compared to other forms of AI risk about concerns like political control."
It sounds kind of like you're equivocating on "niche". Everyone including Scott already knew that existential AI-risk is and always has been a sparsely-held concern. But if it's a legitimate concern then it should be less niche.
Can you share some sources or analysis arguing that Europe's AI-safety legislation has been a major factor in them lagging behind the US? I'm not an expert, but I assumed it was because of the same combination of economic, cultural and regulatory reasons that hamper Europe's tech innovation generally, rather than anything AI-specific.
As I said above: I think you misunderstand my point which is not "not many people care about this" but "the way you analyze this is strange, out of step with how everyone else sees the same problem, etc." most of the world (including most computer scientists) simply do not agree that rationalist style AI risk exists. It's virtually absent outside of a few western subcultures.
Anyway, you want to find sources that say AI was held back not by generic regulations but by specific regulations on AI? That seems very much like the burden of proof should be on you to prove. To first effect, "the significant regulations on AI are the primary thing affecting AI" seems like the null hypothesis.
Anyway, the EU politicians themselves say they should roll them back and they stifled growth. It's all over the news right now. Here's the first article from Google:
https://www.theguardian.com/technology/2025/nov/24/us-europe-artificial-intelligence-ai
>most of the world (including most computer scientists) simply do not agree that rationalist style AI risk exists. It's virtually absent outside of a few western subcultures.
I find this odd, at least if "do not agree that rationalist style AI risk exists" means a strong claim of probability (nearly) zero.
Even just an AI+robotics success at the level of "can replace a human of IQ 120 at all economically important tasks" implies something smarter than the average human which can build copies of itself, effectively a new, smarter, species. One can argue that this presents low risk for a variety of reasons, but zero risk???
Yeah, they think the entire chain of reasoning is flawed. They acknowledge basic risks like job loss or security issues. Most do not think there's any chance of it replacing people. Those that do almost invariably have received it from actually participating in western AI risk communities of various types. There's not even a standard term for existential risk in several languages. The calque it.
Many Thanks!
>Most do not think there's any chance of it replacing people.
Ok, yeah, if someone is confident that attempts to build AGI are just going to be technical failures, that is indeed an argument for saying that the odds of a risk dependent on AGI are as low or lower than the success odds for AGI, which they view as 0-ish.
( I think they are too confident, just as I think Yudkowsky is too confident. The future is uncertain in general, and even more so for R&D. )
Totally wrong. I speak a couple languages of third-world countries, and existential risk discussions are happening. People all over the world pay attention to what e.g. Nobel laureates say, so Hinton's acceptance speech was translated and widely reported on in local media.
However of course local corruption issues / wars / etc quickly push anything "global" out of the agenda especially if the country realistically has no input.
Which languages? Can you please show me the research institutes doing work in these third world languages about rationalist style X-Risk? Or even just extensive discourse.
I'm not usually a big defender of communism, but in an actual post-scarcity future with infinite robot slaves, shouldn't we want the communists to win?
China isn’t communist. For governance, I think we’d want democracy over autocracy in this future.
I'm curious as to whether or not this is true. My impression is that they are still very much communist in theory, and see markets and capitalism as means to a communist end.
There's always the, "come on, of course they're machiavellian schemers," objection, but you could level that objection at the United States too.
They’ve been moving steadily away communism toward state-directed capitalism for 40 years. Xi recently decried the “welfarism” of the west. They’ve forcefully defanged labor unions. They discussed removing “communist” from the party name.
All that’s left from communism are old state firms that own less of the economy than ever.
I’d call them a right-wing, state capitalist, ethnostate.
Communism entirely requires defanged trade unions. And no coddling of the populace.
Communism has always been an unrealistic mix of somewhat contradictory fantasies, but nominally "communist" governments still vary in the level of lip service they pay those ideals.
Xi seems to be moving back toward communism, away from the movies since Deng toward capitalism.
This just seems... misinformed? They have publicly and officially spoken at length about transitioning towards communism since Xi came to power.
Eh, I don't have a horse in an argument about what China actually is, but I'm pretty sure most of those "moving steadily away [from] communism" claims are predicated on not realizing how "communist" states have operated from the beginning. (And exacerbated by the propaganda push to reframe China's recent success as [caused by capitalism actually], I'm not accusing you of consciously or even unconsciously attempting to do this, I'm just saying that's ostensibly the epistemic bubble, largely self-inflicted, in which the first-world mainstream currently resides.)
Canada and much of the EU have way more comprehensive medical, welfare, and even dental coverage than China does. On the ground, in terms of a social safety net, they're more socialist - and yet also democracies.
China's government has theoretical commitments, and promises for the future - but for that matter Musk is a fan of Iain M Bank's fictional post-singularity Culture, which is also a sort of perfect future socialism. Is Musk a communist?
IMHO watch what people do, not what they say.
I wholeheartedly agree with this. Anyone who thinks that the PRC is a communist, or even a socialist state, should go and visit.
My impression is that for common people it resembles the US more than any European welfare state: 10-12 hours of working for 5-6 days, no healthcare, no meaningful social net, no meaningful pension (except for the massive body of civil servants). And it's not getting better either - they are running towards the same demographic problem as many other developed countries, which will put further strain on the welfare systems.
Your last paragraph doesn't follow. If they have less of a welfare commitment to begin with, their system won't be as strained by demographic collapse. Given a commitment of ~33% of its GDP to social spending, Sweden might have to make a choice between reducing that amount or increasing taxes, given their aging population and growing demographics of e.g. Somalians with 85% welfare dependency, low employment rates, and poor educational attainment, but China, with only ~8% of its GDP going to social spending, will find the dilemmas far less extreme and the resulting choices both smaller and easier to make.
Most importantly, they will probably be happier pushing up their retirement age, which is the most effective way of combatting aging demographic pressures (and most welfare goes to the old, after all, so as long as the old aren't burdening the young through taxes, the young shouldn't really need to care).
Mao and the Red Guards were communists. Post-Deng PRC is closer to fascist - very, very nationalist* and totalitarian, ethnic cleansing, "third way" attempting to synthesise command economy with markets.
*Marxism is traditionally internationalist.
"The" communists is doing a lot of work here. Would you want the USSR to develop AGI? What about Laos? Angola? The democratic republic of Afghanistan?
Communist vs. Capitalist, to whatever extent it is true dichotomy, is severely underdetermined as some type of "goodness" measurement. My sense of whether it would be "good" to for China to win an AGI race has a lot more to do with my opinions on Democracy, how Chinese single party rule works in practice, and Han nationalism than the current Chinese economic system.
Ideally North Korea.
It's not really about communism vs capitalism, it's about democracy vs single-party autocracy.
In a democracy, for all the flaws in its actual implementation, power is spread out and those in power are forced to please the people at large, and the government is restricted in what it can do.
In a single-party dictatorship there's no delocalisation of power, nor any restrictions on the government's power. If you hand Xi Jinping or Kim Jong Un or Josef Stalin an army of robot workers/soldiers, there is absolutely no need for him to use it to serve the interests of the people. At best, you can hope that the robots will hand out basic rations to whichever people are sufficiently vigorous in proclaiming their loyalty to the party. And those who are not sufficiently loyal, well, we know what happens to them.
>If you hand Xi Jinping or Kim Jong Un or Josef Stalin an army of robot workers/soldiers, there is absolutely no need for him to use it to serve the interests of the people.
I'm not convinced that there would be any need for Sam Altman, Donald Trump, or Gavin Newsom either, sans their alleged moral compass. Democracy only works if there are checks and balances, and a robot army worthy of the name would obviously upset that.
Why do you presume there will be such things as democracy in this delusional fantasy of yours where a robot is so powerful it can rule the world without human input? Even if it is in the hands of a democratic government, why wouldn't any one private or public individual use it the same way an autocrat would?
>In a democracy, for all the flaws in its actual implementation, power is spread out and those in power are forced to please the people at large, and the government is restricted in what it can do.
Then you haven't been paying attention. Even before Trump II and others, nominally democratic parties have been hollowing out that promise of restraint as state power grows. The nationalists and fascist worldwide are gladly taking charge of the surveillance and repression states created by their respective predecessors.
I disagree with your premise. Power is spread out, yes, but only among elite oligarchs and their institutions. The thing that gives common people a voice in either system, American or Chinese, is the reliance of the elites on their cooperation. They can either refuse to work or launch terrorist attacks on their elites and thereby extract some concessions. If robots were to take over, either as workers or as the executive apparatus of the government itself, then as Crinch says below, there won't be any difference between either state, as the power dynamics will be the same.
Essentially, you are looking at the ritual expressions of the two systems of governments and not their underlying dynamics, which come down to reliance and threat.
Human governance is basically a farce when ASI enters the picture. There are only AI dictatorships, and probably only one of them (at least eventually) because of the different incentives (due to AI immortality and scalability).
No, there's nothing good about communism. People consistently vote with their feet against it.
It seems they voted with their feet against the capitalism which proceeded it - in the case of the USSR - if the large migrations of skilled populace to wealthier places are to provide any insight. It could instead be said that people vote for what can bring them the most money... but not collectively, only as mercenary individuals. In other words, people love money.
If people are moving to the United States, that wouldn't be a vote against capitalism. Marx would surely recognize that the US was the most capitalist country in the world, or in the running for it. He wouldn't have predicted that communism would succeed first in the agrarian empires of then Russia & China rather than the more industrialized capitalist nations of the west.
You probably really want to favor more individualist factions developing AGI because almost every way that AI can enforce some singular vision of utopia on society is going to suck for a lot of people. For instance what if you get the ideal that hard labor is some moral virtue ingrained into society permanently? Such that people are forced to larp as high tech peasants for literally no reason in order to satisfy the aesthetic sensibilities of the communist party.
Whereas if the tools/resources unlocked by AGI are made broadly accessible, then everyone can create and self select into their own utopian communities which perfectly align with their ideals.
This doesn't inherently favor capitalism over communism per say, it comes down a lot to the specific attitudes of the people developing the AI and those overseeing this.
For instance an AGI developed by Chinese researchers may in principle actually be less authoritarian than one developed in the US if the latter had its creation sufficiently influenced by Christian nationalists.
No, why shouldn't I get to keep the fruits of my robot-slaves' labor?
But in all seriousness, the authoritarian aspects of communism are a pretty huge drawback. Let's stick with democratic socialism if the post-scarcity future does actually arrive.
Ok.
Even if you think Fully Automated Gay Luxury Space Communism was a nifty thing, you really don't want it to be run by the people who cut their teeth on grungy old-fashioned industrial-era communism. Communism without AI fails utterly unless it devotes itself to hammering a whole lot of square pegs into round holes in neatly-spaced rectangular grids, which selects for totalitarian autocrats with no respect for human rights or civil liberties.
This analysis doesn’t seem to consider the possibility of the Ai bubble popping and taking compute growth with it.
Either way, losing economic and technological dominance, by a lot, to China over the next 10-20 years is almost assured.
Has China produced any innovation worth speaking of yet? Sure, they have amply demonstrated the ability to take western ideas and run with them, but that's not enough for actual dominance.
If you go far back enough in the past, then there are famously many things first invented in China (like gunpowder) and then spread to the west. If you just mean the PRC, then fewer.
Amusingly, even the go-to example of gunpowder became geopolitically relevant only after the west had gotten its hand on it, centuries after discovery.
Perhaps this is because you've only learned Western history?
Their Space Program research is starting to pay off in the same way that NASA did in the 60s and 70s.
https://www.china-arms.com/2025/01/china-space-station-breakthrough/
https://www.popularmechanics.com/space/moon-mars/a43700416/stirling-power-converter-china/
They are also starting to come up with original pharmaceuticals and their batteries are ahead of their western equivalents.
You might be right about pharmaceuticals. I know drugs like Ansofaxine, an SNDRI, were initially developed in China and are now undergoing clinical trials in the US. Whether that is because China has an emerging, innovative pharmaceutical sector or simply because it lacks an FDA and an entrenched, stubborn medical establishment like the US, I cannot say. Batteries ahead of the West is news to me. I thought Chinese lithium-ion batteries were ahead of competitors only in their capacity to combust spontaneously.
https://www.msn.com/en-us/autos/news/china-s-solid-state-ev-batteries-hit-factories-with-2x-range/ar-AA1R8Zhh
They are also pushing into Sodium-ion and other things beyond Lithium Ion.
Electric cars (BYD cars are now much better and cheaper than Teslas). Solar panels. Batteries. Consumer drones (DJI is the only reasonable option.). On every technology related to the green energy revolution, China is #1 and #2 is so far behind that they're barely worth mentioning.
I'd rather characterize this as the west shooting itself in the foot in various ways recently, which is of course also an important consideration. The biggest favor that Xi could do to shake the west from its torpor is to finally pull the trigger on the Taiwan invasion, similar to how Putin rescued NATO from the self-diagnosed "brain death".
If there is an AI bubble and it pops (presumably you mean by this that we don't end up developing transformative AI) then the safety measures in question will have neither helped nor hurt! So it is sensible to focus the discussion on the scenario where they matter.
Note that while the US will lose some more ground to China in the coming years, the "free world" (roughly: US+Europe+democratic East Asia) will still dominate China, unless there are AI/robotics developments that "flip the table" and happen to be localized to China.
> The Chinese don’t really believe in recursive self-improvement or superintelligence. If they did, they wouldn’t be so blasé about the possibility of America having AIs 1-2 years more advanced than theirs
Have you considered that this might be because lack of optionality? From their perspective, if you assume that superintelligence is guaranteed to happen, there's not much they can do short of nuking datacenters, since they're not going to be able to close the gap in compute in any reasonable timeframe. That option persists even past superintelligence being publicly known, so they might as well operate under the assumption that it won't happen.
If Xi Jinping believed the Americans were, I don't know, building a giant fusion bomb that would ignite the Earth's atmosphere like some people on the Manhattan project once worried would happen, would he really say "well, there's nothing we can do about it, I guess we might as well sit around and fiddle with the housing market"?
If it's the "application layer" where the USA has to be competitive, then the division between that and the cloud service is where regulation should occur: prevent the cloud providers from abusing their quasi-monopoly position to take the business of companies downstream. That would leave incentives in place for smaller firms and startups to compete and build a robust, diverse AI application sector.
Similar to what should (but doesn't) happen with Amazon: it's fine for them to be the dominant online retailer, but they should then not be able to sell their own branded goods, and should only act as a marketplace. See also: microsoft and browsers.
It sounds like you’re advocating for breaking up vertical monopolies. Like them or loathe them, my sense is that big, evil, vertically-integrated corporations have normally been the engines of growth. Think Microsoft, GE, Ford, even historical examples like big factories in the Industrial Revolution or the East India Company. I agree that what you’re describing will likely prevent some major harms to a lot of people, but at the same time it’s the sort of regulation that I would expect to slow down applications development, precisely because the large companies wouldn’t be able to drive their profits and thus their money available above the competitive equilibrium. You might be right, but you would only despite the concerns described here, not because of them.
I think the data shows that more innovation and growth comes from small companies, most of the time. (Specifically here, startups; it’s quite a recent thing for the big companies to have understood Disruption Theory enough to identify potential disruptors that could unseat them, and just buy them instead.) The big companies are horrifically inefficient and mostly extracting value from monopoly positions in their respective markets.
The recent stock market bull run being a historical anomaly, “this time it’s different”.
AI is a case where there is an unusual big integration opportunity; for example at some point (perhaps even before AGI) it’s going to be more profitable for Alphabet to just spin up a subsidiary with exclusive access than to sell API access to their frontier models. This will really depend on the antitrust regime; a weak FTC will allow a lot more of this behavior than IMO should be permitted.
I wonder what you would say about Bell Labs, or Ford, or General Electric?
I'm not necessarily disagreeing with you, just wondering what you think of the 19th and 20th century examples of industrial megacorporations
My hunch is that in the past you needed a lot of capital to do most innovation, and these days there are a lot of things you can invent with just software (particularly the leverage that the open source foundation enables). The VC model accelerates this dynamic of course, encouraging lots of small-scale innovation with the promise of funding to scale up for the most scalable business models discovered.
I think you could point to a few recent “physical world” innovators which are back to being capital intensive, but if you look at the big guys proposed for breakup it’s mostly software behemoths sitting on monopolies.
Good analysis but why assume that it would be bad for China as such to be ahead of America, or why assume it is good for America to be ahead of China, at any random point in the near future ? From the Chinese point of view, America is a wonderful land where millions of Chinese have been welcomed. From the American point of view, China is not as bad as some of the other countries that hate us for bad reasons, and they are generally friendly to American tourists. From the AI's point of view, the attribute of living in Silicon Valley as opposed to having years of training in one of the Chinese universities is probably not all that important. Of course right now the focus is on power, on numbers, on the vast or less vast amount of chips or data center megawatts (to identify simply the more simple attributes of power in this race) one has at one's disposal. Why wouldn't it be? Would this comment be (a) better if it were more simple or (b) better if it were less simple.
It would be better if anyone believed that those points of view (leaving aside the AI's one) are representative of anyone in power in either country, or even the general populations.
It is likely that AGI/ASI could develop weapons that render all previous weapons obsolete. So China would become defenseless, the US would likely depose its government and install a more US-friendly one. Obviously the current Chinese government wouldn't like this.
It is likely that AGI/ASI could develop weapons that render all previous weapons obsolete.
That's a wild leap. Even AGI/ASI can't violate the laws of physics. Also, once we get to supersonic nuclear missiles with a credible second strike capability, what else really matters.
> what else really matters
Precision. Counterinsurgency. Collateral damage. If you respond to a raccoon messing with your garbage by turning everything within a kilometer into exotic plasma, you won't rule the world for very long. Have you noticed how few wars involve actual use of nuclear weapons?
Even before they were invented, back in WWI, the capability to reduce an entire landscape to unrecognizable craters was there. Turns out when you do that, nobody wins. Obliteration makes the whole world poorer.
Even back in Sun Tzu's day the underlying principle was understood. Killing is often necessary in war - that's what makes it different from peace - but it's not a victory. Whenever possible, skip past the violence, instead proceed directly to the part where you take the enemy's best stuff and make it become your stuff.
All well and good. But the previous comment posited that and AGI could make China "defenseless" which seems to be a bit more of a dire situation than merely unable to match up on the ladder of escalation.
Supersonic nuclear missiles? ICBMs from the 1960s already exceeded 20x the speed of sound and are fast enough to hit anywhere in the world within about 30 minutes of launch. Second strike capability has pretty much always been there.
No, it's not a wild leap. As we saw with the Gulf War and with Israel's recent war with Iran, an advantage of a few decades in technological progress results in an overwhelming military advantage in an otherwise roughly evenly matched conflict. With superintelligence, one could expect a similar technological gap to develop much faster.
I think it's fine for American authors to be patriotic and prefer American victory.
(As someone from a country other than America or China, I don't have a strong preference.)
Good points. However, while it seems like a consistent assumption in these sorts of debates, I don't see fundamentally why there isn't some exchange the US could make with China to sell them chips that would be a net positive to both nations.
My model is something like: China and the US both have internal welfare in an absolute sense (positive sum), and then there is a balance of power between them in a relative sense (zero sum).
If China could buy more chips from the US, this would shift the balance of power in China's favor and also increase the internal welfare of China (AI does all sorts of useful things other than military applications.) Then presumably the US loses utility in relative balance of power equivalent to China's gain.
So why couldn't the US propose to sell chips at a rate (through taxes, or diplomatic demands, or whatever else the US likes), such that the transfer China -> US is greater than the value of the balance of power the US loses, but less than that value + China's internal welfare benefit.
This seems like it would be in both parties interest, and also have good peace-incentivizing side effects as mutual trade tends to do (and right now I see incentivizing peace/cooperation between US and China as being pretty high-value).
If we are concerned with reverse engineering / IP theft ... I mean they already get some chips through for example smuggling, does having ~100x the chips actually make it any easier to reverse engineer?
The fact that so many smart people talk about this subject without proposing this makes me think I'm just missing something in my thinking, but I'd like to know what!
> If we are concerned with reverse engineering / IP theft ... I mean they already get some chips through for example smuggling, does having ~100x the chips actually make it any easier to reverse engineer?
That's what I'm assuming. Reverse engineering the manufacturing of a chip isn't about looking at the chip itself, it's about understanding all the horrendous and undocumented details of the process by which the chip is made. Every step of modern chip manufacturing is really hard, and China is still way behind the state of the art.
Which is why if we're serious about preventing them from catching up it needs to be about preventing Chinese spies from getting a good look at any part of the supply chain. That means a complete stop to visas for Chinese nationals to enter any Western country, at least until they have overthrown the CCP and instituted democracy.
"such that the transfer China -> US is greater than the value of the balance of power the US loses, "
It is too hard to quantify this. Estimates could be orders of magnitude apart.
I think the amount required to offset a large increase in risk to America's geopolitical security is higher than China would pay for advanced chips.
Without getting into specifics, you are quite wrong about arguments about the application layer. AI-optimization is being built out en masse at MedTech, EdTech and local GovTech firms. However, many of these firms are of the extremely boring PE-rollup sort. Not the splashy Silicon Valley VC-funny money and BART ads sort. If you go to a conference for any of these industries, it's all anybody is talking about at their booths... Maybe you can better substantiate what you mean by China having an app layer advantage? Are you pricing in a presumption that the federal government will squash app-layer AI in the US in critical industries?
See footnote 3. The US is fine in software, but China already has top-down orders to automate as many of their factories as possible, and has a better record of hardware construction than we do.
It seems the last argument (safety might actually increase progress) is an instance of what is criticized before (bad thing is bad).
Overall, the whole issue with the war with China seems really pointless. I've been to China, talked with people from there, both locals and expats here, and honestly, it seems like not a bad future to have Chinese hegemony.
And in any case, the important issue (for me) is how much safety might impact my / my close ones chances of dying. And this is why I am fundamentally opposed to AI regulation. It seems this is the actual point - most people (even LW crowd, as survey showed) don't actually value long-term future that much. Would be amazing to see Scott attempt an honest presentation of this pov. Otherwise, it's all beating around the bushes and fighting proxy wars (as with the last post on AI regulation, there is no attempt to actually understand and steelman the motivations of the other side!)
Can you explain how AI regulation increases your chances of dying?
It slows the pace of development and ultimately prevents ASI in the foreseeable future. There's no chance we solve death without it.
Ah you want to solve death. Say no more
I'd say the Aubrey de Grey approach to gerontology has more than 0 chance. As does cryonics.
See my comment below.
Which one? I get an email notification that helpfully links directly to your comment here, but doesn't help me finds others from you.
https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose/comment/181430133 this thread
Solving “death” in full generality sounds hard, but solving aging without ASI sounds doable. (Remember, you can still have things like AlphaFold 9000 without ASI). You’d still die if someone drops a nuke on your head I guess. But even ASI probably can’t “cure” accidental collision with nukes. It would be an unfortunate irony to have “cause of death: quest for immortality” written on humanity’s tombstone. Let’s try to avoid that. If you could send an high school biology textbook back in time by 100 years, I think people back then would be amazed how much we now know. Why be so sure that we’re stuck? If you’re worried that progress is too slow to save you personally, cryonics is pretty cheap I’ve heard.
Cryonics is very far from being actually useful, it's a cheap cop-out to pretend otherwise. AFAIK there is exactly one company making serious effort in that space (developing better cryogenic methods) - Until labs - and they are optimistically 10 years before even being able to preserve large human tissues (I'm not a biochemist, but this is what my medical PhD friends are telling me). State of the art right now is freezing a rat to 0 Celsius (veery far from actual cryogenic preservation) and unfreezing it after 20 minutes. There has been no significant progress on this.
But even granting something like this working out, you still have the fundamental problem of (1) freezing actually being kind-of death which is an unsolved semi-philosophical problem (2) being able to cure whatever killed you in the first place (3) institutions being stable enough to not accidentally throw your body out (4) civilisation continuing and making technological progress (and all the arguments with cultural drift, TFR dropping etc), (5) AI not stalling indefinitely kind of like all over-regulated industries (6) the west still being dominant enough do our lives matter, and so on.
In short, "just do cryonics" is a lazy, very much pie in the sky answer.
Fair, it’s far from guaranteed, I may have been a bit too flippant there. But that doesn’t mean that building ASI is on net a safer bet. Also: we might make enough progress soon that you never end up needing cryonics. (I don’t know how old you are; if you’re over 110 then I guess probably not.). As for freezing being kinda death, I suggest the following philosophical exercise, (which I admit I have not done myself): there exist people who have had their body temperature drop below 20 Celsius, at which point electrical activity in the brain and elsewhere basically stops, and they have lived to tell the tale: they were basically fine. The exercise is: try to figure out if these people “kind of died” in a way that matters to you.
Yes, there are examples of those people (eg the famous case of the Norwegian doctor skiing who got hibernated for quite a bit - https://en.wikipedia.org/wiki/Anna_B%C3%A5genholm). But like, this whole thing is just fishy. The way I see it - these are hard philosophical problems around personal identity, consciousness etc, which are easy to get wrong, and we're much too stupid to have a good take, and it's very easy to get confused. The best research is currently probably QRI or Parfit-adjacent ppl at Oxford who are again nowhere near solving these. So the only option is to play it safe - meaning preserving the body and living "standard" way.
We might make progres, but we might also not. I'm presenting an object level argument that we are not on track.
There's actually a way to quantify this. In expectation, there's a 1% of dying every year (in the US) skewed towards the elderly. So if an AI safety proposal delays solving death by 1 year, then it should reduce the risk of everyone dying by >1%.
For me, more of my loved ones skew younger and I, like many people, value young people's lives more than old people (eg trolley problem: 1 baby vs 1 elderly person), so I'd accept a year's delay at a reduction of ~0.05%.
That seems like a low number, but from an actuary, 30 year olds have a 0.1% chance of dying per year: https://www.ssa.gov/oact/STATS/table4c6.html
This of course extends to fractions eg delaying by 3 months should reduce risk by (0.05/4)%.
This assumes that you keep all else constant and that it really is slowing by 1%.
There's three responses. One: it's a hatchet. As Scott himself points out, once slow-down-AI faction gets their win, they will escalate. And it's like that Gandhi who can accept the bribe to be 1% more killer - there is slippery slope.
Two, there is a chance that this is the peak of our progress based on the TFR argument, relative peace, etc.
Three, my parents are not that young. I don't want them to die either. (Also some argument about all other people in the meantime, but that's separate and not really my reason).
Not sure what you mean by "don't value the long-term future that much".
I don't know how old you are, but I'm 40, and I'm not especially concerned safety regulation will prevent me from getting immortality, for a few reasons:
1. As discussed above, real-world safety regulation is on the order of raising costs 1%. Most likely it stays at this level and slows timelines 1%.
2. In the extreme world where we get a full pause, it probably won't last more than a few decades before chip tech becomes so good that you can train AI on chips too small to track.
3. In the extreme world where we get a pause, I expect human intelligence enhancement to create supergeniuses within a few decades.
I'm more interested in removing the chance of death from unfriendly AI, especially since if I die then at least my kids will survive, whereas unfriendly AI kills everyone. I think this chance is around 20%, but I think if we gave up and had no attempts at safety whatsoever, it would be closer to 50%.
Thanks Scott. I meant this poll here: https://www.lesswrong.com/posts/A22bTHYLDdkeqFsLd/saying-the-quiet-part-out-loud-trading-off-x-risk-for which showed 2/3rds of ppl value their personal immortality over long-term AI safety.
1. You argue yourself that the regulation is a hatchet. If you achieve 1%, you then aim for bigger wins. And it can snowball. It's easy to point to examples of industries where this happened. X-risk people are very explicit about wanting to do more than 1%, and I don't think they would or could commit to only slowing by 1%, because that's the whole point!
2 and 3. I don't think this is the case. We are now in a very fortunate period, relative stability and peace, still riding the demographic wave. But the conditions are changing and can change dramatically (such as arguments from Hanson on TFR and cultural drift). And again, we banned nuclear power, but there are no people having small nuclear reactors in their backyards. (Supergeniuses are even more uncertain - as e.g. you yourself pointed in the polygenic IQ/heritability debate).
Note that you (and I) also have parents, which are equally as important. I actually put the same 50% ballpark for the chance of charging ahead with AI causing extinction. But I think it's pretty much as good a bet as we are going to get. Not-dying without ASI alternative seems to me to be something <20%.
> there are no people having small nuclear reactors in their backyards
Physics of neutron cross-sections put some very hard lower bounds on the size of a useful fission reactor, and the only reasonably attainable types of fusion right now are even bigger. Computation is much easier to miniaturize.
Why assume the costs of regulation stay low? establishing the precedent of federal regulation will invite more and we could end up totally strangling the industry like we did with nuclear.
Argument seems misaligned. Primary cost of regulation is not the cost -- it's lost time. Scott is correct that the costs would be manageable, but the proponents of regulation explicitly want to delay progress. It's a mistake to think we could pay 1% of the AI research cost and maintain the same timelines as regulation escalates.
I think the scope of AI Safety activist asks will ratchet up by a lot more than 1% if they happen to win this round of asks. These are people who believe that there is a high chance superintelligent AI will murder everyone. It would be ludicrous for them to not do everything in their strategic power to strangle the AI industry with regulation and restriction given the chance. Given how much recent economic growth in recent years has hinged on AI development, the difficulty of enforcing a global pause, and the extent to which future live-saving and life-extending medical inventions might be contingent on advanced AI, a global halt solely due to a theoretical and speculative argument (even if it comes from people who are smart) is a tough sell. This might not be what activists are asking for now, but we can likely expect them to escalate up to this sooner rather than later.
Every movement attempts small things first and then tries large things, but usually they fail on the large things because other people are opposing them, and they usually have enough power to push through their small common-sense hard-to-oppose demands, but not their giant unpopular ones.
Compare environmentalism, which was powerful enough to ban plastic straws in some areas, but not enough to ban SUVs or coal plants or plane flights.
I think there are a few factors which make this issue a little different than most issues that get resolved through that type of pluralistic process.
(1) We can expect anti-AI advocates to have higher determination and resolve than most political cause advocates. They also are likely more g-loaded than the people in most political causes. These factors mean that they will likely be more strategically effective at achieving their aims relative to how popular their aims are than most political causes end up being.
(2) There is not an obvious political 'in-group' we would expect to be strongly pro-AI. The natural enemy of the environmentalists were coal miners and people who worked in the oil and gas industry. The natural enemy of gun restrictionists were proud gun owners. The natural enemy of the AI safety industry would be: computer scientists? The four 'accelerationists' that are on Twitter? People who work at OpenAI and could get another job in a day if they needed to? The benefits of AI are fairly diffuse.
(3) Most people do not know much about tech. In theory it could be possible to draft reasonable looking legislation that smuggles in severe AI curtailing regulation through the back door while arguing that the legislation does not do this.
(4) Most people, if faced with the choice of preserving democratic norms or preserving their social status, or preventing everyone from getting murdered, would choose the latter. If faced between the choice of being honest or preventing everyone from getting murdered, most people would choose the latter. It is hard to think of many lines, no matter how extreme, that people would not cross in order to prevent everyone from being murdered. Because of this, there is a meta-level suspicion at play. If people genuinely believe that AI will murder everyone, can we trust them to be good faith actors? Can we assume that the normal rules and norms are going to be things they structurally value?
For point 2, I think some of the biggest companies in the world by market capitalisation are AI companies. They’ll have plenty of money for this political fight. That money also buys lots of g-loaded lobbyists. 3 and 4 seem to apply equally well to early environmentalism.
'AI companies have access to high powered lobbyists' does not change the fact that there are a lot of components to the AI regulation political debate that undermine a lot of the normal operating assumptions that pluralistic democratic processes rely on to resolve policy questions fairly (competing grassroots special interest groups engaging in counterdiscourse and counteradvocacy that have a broad and similar ceiling of strategic efficacy, a shared assumption that both sides are committed to the broader structure of norms that they are engaging in discourse in over their preferred policy outcomes, etc).
Early environmentalist groups were pretty radical and did face some stigma and distrust. Over time most moderated so you do not see that sentiment as much today.
Sure, but I feel like lots of issues have similarly unusual or asymmetric advocates.
I think a lot of the increased trust of environmental groups is just a familiarity effect. They’ve been around long enough that lots of people have absorbed their claims as normal (though some people do still distrust them).
I think the overall gestalt is abnormal. Maybe you can mix and match and say 'well the issue of X kind of was also a little asymmetric' or 'well the issue of Y maybe also had super motivated advocates', but I think it is specifically the way all of these factors mix that creates a holistic picture that I think normal democratic and pluralistic processes are going to struggle with.
I just get bad mojo surrounding all of it. You have a movement of people where the median IQ is like 135. These people seem standard deviations more strategic than most political movements. They think AI will murder everyone, so the stakes are infinite. In theory such a position could easily justify political violence. In theory this means that the issue is entirely a one-shot game for the advocates. On the flip side of the issue, you have no organic advocates because of the diffuse nature of AI, but also infinite lobbying funds and appetite to lobby on the 'pro-AI' side of things. This means that the pro-AI side will almost certainly lose the public discourse battle and will have to rely on extreme corruption and graft to achieve victories. Scott's original response to my comment said that normal democratic and pluralistic processes can be expected to handle this issue in the way they handle most issues, but I do not feel like that is going to happen here.
>Every movement attempts small things first and then tries large things, but usually they fail on the large things because other people are opposing them, and they usually have enough power to push through their small common-sense hard-to-oppose demands, but not their giant unpopular ones.
I concur with Spinozan Squid, notably on point (2), that
>There is not an obvious political 'in-group' we would expect to be strongly pro-AI.
If I could be very confident that AI regulations stayed at the "small common-sense" level, I'd be happy with them, but American political history, from ALARA & the NRC to NEPA to NIMBY gives me the creeps. It makes me think PauseAI (perhaps better described as KillAI) would find a way to either twist the common-sense law into what amounts to a ban (here, not in the PRC, and probably circumvented by the military in both places) or salami-slice their way to a ban.
What did you think of the ‘Catching Up to the Almonds’ article? It seemed to argue that the pro-AI lobby group was unusually active at targeting politicians. It won’t get big protests, but it will organise well-funded attack ads. Do you think that would be less effective at scaring politicians than protests would?
Many Thanks!
>It won’t get big protests, but it will organise well-funded attack ads. Do you think that would be less effective at scaring politicians than protests would?
I don't know. Well-funded has its limits. Kamala Harris's campaign committee outspent Trump's by about 2.6:1, but still lost the electoral college, the popular vote (and, looking at parties, control of both the Senate and the House).
<These companies should do some kind of evaluation to see if their AIs can hack critical infrastructure, create biological weapons, or do other mass casualty events. If they find that the answer is yes, they should tell the government.
I don't think we have to worry about regulations of this sort slowing us down because (a) when push comes to shove the companies won't fully disclose this information and (b) our president is a dumb wacko narcissist who has filled the government with dumb evil loyalists. The conga line of weak-brained moral midgets wouldn't know what to do with this info even if they were given it in full.
I'm not doing haha dark humor here. This is my actual read of the situation.
A) The regulation says they have to disclose it.
B) The current Republican Party is very divided on AI; for every A16Z loyalist, there's a Josh Hawley or a Martha Blackburn who hates and fears Silicon Valley. I think there are many voices who would do something with this info. If they don't, state governments may be able to do something, or a future Dem administration might be able to do something.
Regarding A: I worked for several years for a prestigious psychiatric hospital affiliated with an Ivy League medical school, and learned a lot there about how much can be hidden from agencies tasked with checking the institution’s ethics and compliance with rules. Before an inspection by some public health agency we would spent a couple weeks bringing everything from records to the location where white out was kept up to standard. (It’s poisonous, so is supposed to be kept in a place where patients could not possibly reach it. It normally was not on our unit.)
There was an incident where a violent patient stomped with all their strength on the foot of a patient of mine, S, who was in a wheelchair. Afterwards S asked the doc on call to examine her very painful foot, and he said there was bruising but no serious injury. That night S was not protected from the violent patient, who continued to roam the unit freely. S signed out AMA the next day and went to the ER, where they found a complex injury including more than one broken bone. Staff I spoke to at the hospital about the incident sounded like they had been told to be vague and uninformative. “Yes, it was very unfortunate. We’re really not clear went on between S and [violent patient] . . . “ S later tried to sue the hospital, the hospital elected to fight, and I was deposed by the hospital’s pit bull lawyer, who literally laughed out loud at me because I did not have written documentation of what the patient had said to me in several emergency phone calls at the time of the incident. “Honey,” she sneered, “do you know what heresay is?”
The Psychologist in Charge of the unit where I worked ran a postdoc program for interns he liked. He referred private patients to his stable of not-yet-licensed psychologists, and for those that had insurance we filled out insurance forms stating that he, the PIC of my unit, had been the provider, and he signed them. He kept 20% of patient payments for himself. There was a rule in the ethical guidelines for my state prohibiting this exact practice. Oh, and my PIC was *on* the committee that developed those guideliness.
Scott, do you get that the shitlords of tech are not honest and scrupulous the way you are? They’re not even as honest as I am (mostly very honest in one-to-one situations, much less so when dealing with organizations and groups because I dislike and distrust almost all of them.) You're more honest than me, and the people running the vast tech companies are many steps beyond me in disingenuousness. There exist people who literally do not grasp the concept of sincerity. They process questions using an algorithm that is doesn’t even have truth as a variable, but only things having to do with immediate effect of saying X, long term effect of saying X, options opened by saying Y, etc. Haven’t you met some? My guess is that many of those running the big tech companies fall into that category. And even if one of them did have an inner ethical code, it would operate in a profoundly different context from yours and mine. It would involve what the leader owes the company and the stockholders. It’s almost as though these guys are the leaders of other *countries,* deciding how to handle the demand of another, larger one.
Game it out. Suppose an AI company doesn’t disclose what the regulations say they have to. Is the government going to shut down the strategically important AI run in progress?
If they do, then the regulation has in fact slowed AI progress. If they don’t, then the regulation was useless.
That’s right. And there are other considerations as well. An AI company can do impression management many orders of magnitude higher than Whiteout location switchiing. If outsiders come in to test how corruptible a model is, the company can put special hidden constraints on it that make it almost impossible to corrupt (OK, they will also make it dumber, but that’s not a problem in this context.)
And what about this? AI is soon going to be, if it isn’t already, sort of like gasoline or electricity for the military and other branches of government — something they have to have. An AI executive can say, look, if you insist we put in constraints in order to make it impossible, rather than very difficult, for X to occur, the timeline for getting those crucial gizmos to the military is going to double. You sure you want that to happen? Hmmm?”
>There exist people who literally do not grasp the concept of sincerity. They process questions using an algorithm that is doesn’t even have truth as a variable, but only things having to do with immediate effect of saying X, long term effect of saying X, options opened by saying Y, etc.
I suspect a main difference between you and me (hopefully) and those people is their scope. If the scope is the success of some company or organization (and probably by extension themselves and their family short term) then dishonesty may make sense. If the scope is the success of larger society or humanity (and by extension probably themselves and their family long term), then I think dishonesty is almost always sub-optimal. Dishonesty erodes trust, and trust is very important for a number of things on a societal level. For example, look at how high trust nations are doing compared to low trust nations (https://ourworldindata.org/trust). Of course there are also coordination problems to consider. If everybody else is a crook, then being honest is very costly.
I value truth in itself as well, but I think the scope is also a large part of it.
It's very funny to see how people discuss seriously competiton with China as if China were a dictatorship much more than the US is. In reality, the difference is far not that big as it is positioned: perhaps, twice as much. The US is no more that free country, while China is not that unfree. The real difference is pretty moderate.
Wow, can't even tell if this is irony or just ultra mind-killed insanity, but if the latter...I suggest you try talking to someone who's lived under an *actual* dictatorship.
I didn't live in North Korea (I suppose, it is worse because of poverty) but I lived long in the USSR and in Russia. Also, I studied details about China pretty much.
It's not THAT greatly different from living in the democracies, as long as you are an average person. China is currently not even the USSR. It includes more freedom in some respects, along with a pretty good bunch of other benefits.
Yes, living in Germany is smoother, however it's not a thing for which I would ever kill people or ever recommend anyone to. People in democracies are very much indoctrinated and being lied, and mostly don't see the reality, almost just like people in dictatorships. What you read in news and watch on the TV, is a very much specialized inclination, not an objective and holistic picture of reality!
Yes, dictatorships kill people even today. They torture and impose non-human practices. They break human rights. But all that is far not that important for how an average person lives every day, as mass media in democracies present! Normally, massmedia (and even individual bloggers) boost the negative sides and hide the positive ones. This sells better. Also, they are said to do so by their owners. Also, they are indoctrinated themselves to see things one way, not objectively.
Also, how democracies normally present themselves, is far not what they are in reality. Here there is a pretty much of distortions and biases, and some direct lies too. If you dig deep, you will discover that real differencies are not that big, as most people think.
I grew up in socialist Czechoslovakia. If the regime persisted more or less unchanged until today... for starters, I couldn't go to Vienna rationality meetups. Less Wrong and ACX might or might not be blocked by some kind of firewall. I think many people on the autistic spectrum would be at higher risk of saying something politically incorrect and thereby getting in trouble.
> Normally, massmedia (and even individual bloggers) boost the negative sides and hide the positive ones. This sells better.
Does not the same apply for reporting about democracies? Perhaps even more, because ultimately people care more about problems in their own countries than about foreign problems. My government required me to believe in so-called viruses and wear a face mask once; I can't stop complaining about that all he time. Saudi Arabia requires half of its population to wear masks constantly, and regularly kills people for not believing in Islam, but I guess they are used to it, so it's no big deal.
My impression was that China would regularly imprison people for years or decades for peaceful protests, such that a majority of large protests would end in arrests. My impression was that most large peaceful protests in the US don’t end in arrests. Is one of those impressions wrong, or do you think the difference is unimportant in practice?
The US is not a dictatorship. The Supreme Court recently heard oral arguments in which most expressed skepticism that the President can set the recent tariffs that he claimed were permitted by actual legislation. We've repeatedly alternated the parties in power, whereas China has a single party without electoral competition.
I don't want to be twice as dictatorial! I don't even like the current level of dictatorialness!
There is a jagged frontier of dictatorialness. Honestly, the US culturally seems to me to be much more dystopian than China - purely from personal experience, it's easy to get cancelled socially / be a pariah for having an unpopular opinion (this applies to all Anglosphere). Chinese people seem to be much more tolerant in this regard. And yes, their govt limits political freedoms, but is pretty relaxed on personal freedoms in general. (About the obvious point of Uyghurs - I agree it seems very bad, but some reasonable people disagree it's actually an actual problem after looking into it - e.g. Angelica Oung).
How much would culture vs current govt structure matter or dominate is, I think, an open question.
I'm not sure that 7 is a good argument here unless you are making the further claim that NVIDIA is intentionally restricting supply in order to keep prices high (which clearly would be bad for the US winning the AI race). If the high profit margins are due to NVIDIA making chips as fast as they can but demand growing even faster, then the high prices just serve to:
(A) ensure that the chips are going to the people that need them (or can pay for them) the most (which might mean AI companies rather than gamers)
(B) encourage NVIDIA to increase manufacturing capacity as much as it can
IF the issue here is just demand outstripping supply and not monopolistic practices, then adding a price ceiling will just make things worse.
To be clear, NVIDIA doesn't make any physical chips. TSMC does. Nvidia's ability to expand sales is entirely dependent on TSMC's ability and willingness to expand production capacity.
Then higher profit margins men's that they can pay TSMC more per chip in order to get a higher fraction of chip production dedicated to them and eventually encourage TSMC or others to expand production.
Agree. The issue here is supply. Companies in both countries would love to have more chips at the price Nvidia is charging. See the recent Bay Area House Party post with Zuckerberg trying to get more GPUs. It’s funny because it’s true :)
The only way for NVIDIA to get more capacity quickly would be to buy capacity from other companies that have contracts with TSMC. That would mean a higher cost to NVIDIA but not more money for TSMC. This approach has worked for Cerebras, but NVIDIA already has so much capacity that the amount of capacity it could buy from other companies would be a drop in the bucket.
NVIDIA could get TSMC to expand capacity by contracting to buy large numbers of chips in the future, but then NVIDIA would be stuck holding the bag if the AI bubble pops.
I'm not sure about this, but I think NVIDIA just has the IP. Compare to medications, where if the government suspended intellectual property, then anyone could make the medications for $1 and many more people would use them.
Again, this only matters to the extent that they are engaging in monopolistic pricing and intentionally keeping supply low to inflate prices. But unlike the case of medications, where if patent protection dropped a bunch of other manufacturers could suddenly start producing, I'm skeptical that NVIDIA could substantially increase supply in the short run if it wanted to.
"You say that my proposed regulations are not incentive-compatible, but I've cunningly designed them to not accomplish anything anyway. Rationalists win again!"
I agree that currently proposed AI Safety regulations wouldn't significantly slow-down U.S. labs, and wouldn't cause the U.S. to "lose the AI race to China".
But I think it's a little misleading to not mention that most people who buy AI x-risk arguments and care about passing such regulations (Scott included, I assume) also believe that these regulations are nowhere near sufficient!
The kind of AI Safety regulations that would guarantee that U.S. labs don't risk our existential safety would significantly slow down these labs and make it far more likely that Chinese labs become competitive. This is the bitter pill you have to take here.
TSMC stands for "Taiwan Semiconductor Manufacturing Company".
I'm very much not an expert on semiconductor manufacture, and I know the US has taken some steps to increase domestic production, but if/when China invades Taiwan - which seems more likely than not to happen before 2028, although far from certain - then won't the chip advantage massively reverse?
Well, TMSC itself would almost certainly be destroyed, and how much the balance overall would change depends on the level of escalation.
>TMSC itself would almost certainly be destroyed
In a hot war, agreed.
If the PRC can manage to astroturf/incentivise a mostly-false-flag nominally Taiwanese party in favor of reunification with the PRC, maybe not...
After the Hong Kong debacle I doubt that there would be many takers.
Many Thanks! I hope you are right!
If TSMC is destroyed, it is plausible that Chinese fabs are destroyed too.
See https://manifold.markets/MarlonK/will-china-attempt-to-invade-taiwan
I think the answer is it would somewhat reverse, but not massively, for a few reasons:
1. The US already has ~10x as many chips in its data centers as China. Even if they never get any more, it will take China a long time to catch up.
2. There's a TSMC fab in the US now, and we are working on getting more.
3. NVIDIA still has better chip designs than China does, and whatever factories spring up to replace TSMC will have the advantage of those better designs.
4. A big part of TSMC's advantage is having access to incredibly complicated machines by a Dutch company called ASML, which are also banned from being exported to China. Whatever Western companies sprang up to replace TSMC would also have those.
5. If the US was smart, it would Operation Paperclip as many Taiwanese chip experts and machines before giving up the island.
6. If the US was smart, it would ask Taiwan to destroy some Chinese chip factories on their way out.
I don't know exactly how all this would go - it would depend on how quickly the West could bounce back - but I think probably it would more like halve our lead than eliminate it.
You think there is >50% chance that they will invade Taiwan within 2 years? Even the betting markets don't claim this.
Betting markets treat the likelihood of things that would destroy the betting market as zero. Much of the probability of Taiwan invasion involves a nuclear exchange at some point. Hence, prediction markets with real money will drastically undercount the probability of a Taiwan invasion.
Xi Jinping has explicitly and publically said he wants the PLA ready to invade Taiwan in 2027. That doesn't necessarily mean he'll order the invasion, but one should consider it a strong possibility.
So they undercount it by 30%? Seems unlikely. Taiwan does not have nukes. Based on how Ukraine is going, nuclear-armed countries wont defend Taiwan with nukes and China is likely to consider invasion too costly. Taiwan also recently elected a China-friendly president. I could see something like 25% chance of invasion to be generous, >50% is insane and state comments and mobilisations from years ago don't mean much with how the world has shifted.
>Based on how Ukraine is going, nuclear-armed countries wont defend Taiwan with nukes and China is likely to consider invasion too costly.
Ukraine and Taiwan are very different. Taiwan is strategically critical both due to TSMC and its position anchoring the First Island Chain (failure to defend Taiwan = South Korea and Japan pull out of the NPT and make a mad sprint for nukes, as without the First Island Chain or a reliable nuclear deterrent they would be at the mercy of a PLAN blockade of their food imports), and the USA has some formal and informal obligations to Taiwan (including Biden saying that the USA would defend it).
EDIT: I just saw, a week and a half ago the Japanese PM said that a Chinese attack on Taiwan would be a "survival-threatening situation" for Japan (this is also the constitutional requirement for Japan to deploy its military). Hadn't actually seen that, but yeah, Japan knows exactly which side its bread's buttered on. (Beijing responded by threatening to cut off her head.)
I would like to think that Xi won't pull the trigger, but fascists have a rather-notorious tendency toward overconfidence, the PRC is notoriously bad at comprehending Western mindsets, and they are rapidly building ships that are obviously intended for amphibious invasion. There is still a decent chance that they call "no-go", certainly, but they wouldn't be doing this if they didn't want the option live and ready.
Conventional war between the USA and PRC almost certainly means nuclear exchange at some point; the Cuban Missile Crisis was two weeks of not-actually-war and we had the Duluth bear (bear broke into a base, set off alarms automatically including at neighbouring bases, one of them was miswired and instead of "intruder at neighbouring base", it sounded the "scramble bombers" alert; nuclear-armed interceptors actually got on the runway before they realised it was phony and recalled them) and the Vasily Arkhipov incident (when the US destroyers dropped practice depth charges to force the Soviet subs to surface, some of the subs thought they were real depth charges that missed, and considered retaliating with nuclear torpedoes; on one sub the vote was 2:1 in favour, but it needed unanimous).
>Taiwan also recently elected a China-friendly president.
No, Lai Ching-te is pro-independence enough that Beijing literally gave Taiwanese in China discounted flights back to Taiwan to vote in the hopes they'd vote against him, and ran drills to "punish" Taiwan for electing him. The PRC really hates the DPP.
Again, Taiwan invasion probably means nuclear exchange which drastically lowers the value of winning the bet (many bettors dead, reduced value of money). I'd at least triple the likelihood to get at what the market is treating as the real probability of invasion (i.e., they're implicitly writing off everything 2/3 of the time invasion happens); that'd be 63%/142%, or 44%. That looks vaguely sensical - more so than the nominal Manifold result, at least.
Let's cut the bullshit then, you think >50% is a reasonable estimate, so bet money with me. The odds are in your favour.
I do expect to survive a nuclear war*, but money will still be less useful to me if they do invade. If I were a betting man, which I'm largely not, and I had some guarantee that the bet would actually be honoured, I still wouldn't take this at even odds (restricted to end of 2027 I'm not sure it's over 50%, although to 2030 I'd say it is).
*List of my current preparations: moved to Bendigo, various account structuring to maximally protect my money if the banks go bust, 20L of water in the bathroom cabinet with bottles for another 9L available, iodine pills, emergency radio, 60m alfoil for protecting electronics including said emergency radio, 75lb compound bow with broadheads. The last one is the only one that I consider arguable in terms of EV restricted to this issue, as it's unlikely I'll need to use it even if Australia does get nuked, but I've been interested in archery for decades so it was a good excuse.
If the US comes in, it probably turns into a nuclear exchange, which almost certainly ends in Chinese unconditional surrender (i.e. fully demilitarised and probably banned from AI development - might even have to hand over all the smuggled chips) and quite likely involves China becoming a radioactive wasteland with a considerably-lower population.
If the US doesn't come in, it might turn into a nuclear exchange anyway due to South Korea/Japan sprinting for nukes (neither is remotely capable of surviving without seagoing food imports, so without a nuclear deterrent or the USN being reliably on-side the PRC would have them over a barrel with "do X or we blockade you and you starve").
In any case, the Taiwanese chip fabs won't fall into Chinese hands; TSMC has a kill-switch to destroy them.
That sounds good in the short-term, but if at some point the models start exhibiting really worrying behavior (along with capabilities increase), AI safety would mean severely slowing down or halting AI development. Of course assuming no significant progress in alignment.
The point about the US-China race is that, even then, neither side can afford to stop as it risks giving up their lead.
So, for me it's not about satisfying current AI safety proposals with 1% lost efficiency, but rather -- what happens later. This is where it's most dangerous and the race dynamics are the strongest, as the potential of AI becomes more and more evident.
Yeah, that's plausible. I would argue that:
1. If the models really become worrying, then people would be more willing to take risks to align them, and it's not just a few safetyists begging people to pay attention.
2. If the models really become worrying, China will hopefully also notice and there will be more room for a mutual pause.
This whole essay and no mention of Taiwan and TSMC! Scott mentions American chips a few times, and that's true in a design sense, but the chips are Taiwanese. The entire AI industry is a leveraged bet on Taiwan's export capacity, attempts to build fabs in America are going poorly, and Taiwan's security is far from assured.
Separately, this is framed as free world Vs China sometimes, but really it's just America Vs China, and the rest of the world doesn't think it's obvious that it would be better if America 'won'. China hasn't threatened to annex my country.
The rest of the free world most definitely, absolutely thinks it would be better if America won, aside from a vocal and Very Online minority. And even said minority will, when forced to choiose where to live, work, or in the extreme case seek asylum, have their revealed preferences speak deafeningly louder than their words.
These performative, trollish moral equivalences are really unbecoming of ACX.
Do you have survey data on the topic? Perhaps rather than a casual as hominem dismissal you could provide evidence?
Regarding living and working, there is of course no doubt, China is clearly a worse place to live and poorer. That's not necessarily relevant to the question, however.
1. Says the person who made an extreme claim about "the rest of the world" without a scrap of evidence. "That which can be asserted..." and all that.
2. The evidence, of course, is the decision on who to actually ally with, which incorporates all of the *actually* relevant factors and excludes the kinds of fake factors like performative anger at the president speaking rudely about you, which dominates internet discourse and affects zero people's actual behaviour.
3. I'm sure a significantly non-negligible portion of people will tell surveys that off-the-cuff braggadocio comments are exactly like a fifty-year deadly serious policy of invading your closest neighbour when you get the chance, and concentration camps for political dissidents. Such portions of people will also say that higher taxes are the same as the Holocaust, and that not enough women on company boards makes the West as misogynistic as the Islamic world. That doesn't mean they deserve to be respected or taken seriously, or that they take their own statements the tiniest bit seriously when it comes to making tangible decisions of any kind. Many of us come to ACX to get away from this kind of performative partisan crap.
4. You concede China is a much worse place to live, but you'd rather China rule the world. I can't even follow your own claimed logic.
5. I'm being very charitable by calling your statement performative and trollish. The alternative reading is that you *actually would* prefer mass surveilance and censorship, torture of dissidents, millions in concentration camps, single-party dictatorship, and actually existing seventy-year occupations (Tibet) in exchange for national leaders who are more cautious about saying rude things about your country. I trust I don't have to explain why my reading is more charitable.
Genuinely looking to clarify something here: is your working assumption that whoever is slightly ahead in AI tech immediately gains a huge advantage which they immediately use to impose their own political and economic system on everyone else?
In which case, we don't disagree at all then. I don't see why that would necessarily happen (the world didn't instantly and irrevocably switch to democratic capitalism when the USA had nuclear hegemony), but if we accept the premise then sure, China being ahead would be Very Bad.
I don’t know of any recent surveys.
An IPSOS poll conducted from March 21 through April 4 asked whether “___ will have an overall positive influence on world affairs.” The answer was 46% for America and 49% for China.
A Pew Research poll conducted from January 8 through April 26 asked about “confidence in ___ to do the right thing in world affairs.” Results were 34%-62% for Donald Trump and 25%-64% for Xi Jinping.
Given the rapid changes from the Trump Administration, the first of these surveys is dated (Trump’s “liberation day” tariffs were announced on April 2), and the second is too spread out to even represent a point in time.
Putting the polling aside, I think your phrasing that it’s not obvious that it would be better if America won captures the issue pretty well. Under Trump, U.S. policy has become so unpredictable that it’s hard to know what is coming next.
To quote from Canadian Prime Minister Carney’s April 3 press conference: “The system of global trade anchored in the United States that Canada has relied on since the end of the Second World War, as system that, while not perfect, has helped to deliver prosperity for our country for decades, is over. Our old relationship of steadily deepening integration with the United States is over. The eighty year period when the United States embraced the mantle of global economic leadership, when it forged alliances rooted in trust and mutual respect, and championed the free and open exchange of goods and services, is over. While this is a tragedy, it is also the new reality.” Countries around the world are still figuring out how to respond.
Assuming you are Canadian, China has effectively annexed much more of your country in a consequential way than Trump has by making blustering remarks that did not amount to anything.
TSMC is mentioned in the sixth paragraph.
That is correct and I retract the statement!
Hmm, is my reaction. So does the principle of "regulation will not make us lose out to China" apply to other topics?
Things like embryonic selection? Because that's one of the "if we don't do it, China will" old reliables that gets trotted out, along with the scare that "and then China will breed superbabies and we'll be left scrabbling in the dust!"
I do tend to cast a jaundiced eye over "if we don't do it, someone else will and then we'll lose out" so it's refreshing to see "no, we won't" applied to AI regulation, now let's apply that more broadly to whatever the latest Chicken Little scare is.
I am making a specific claim about details. If we put a 1% tax on embryo selection while we had a 10x lead over China, then no, that would not affect anything!
China has been the useful universal bogeyman for a lot of "if we don't do it, China will!" and that's why I tend to tune out any references to China and something something world leading something.
China, or rather the Chinese government, will want to keep firm control over any AI. I agree there. That does not necessarily mean "so they'll make sure it's aligned with nice liberal values".
I do like the new line here of "don't worry about China", that's refreshing. But the major and main problem is: money. As in, AI companies have bet the farm on getting to market first and fastest and grabbing market share, and everyone has been a little *too* sanguine about what a comment upthread calls "actual post-scarcity future with infinite robot slaves". I don't believe in any such future, and if there are going to be infinite slaves, those are most likely to be human wage-slaves.
So to get pauses or alignment or whatever you want, I'm afraid appealing to good nature and sense won't do it. You will need the government to step in and use the big stick on such companies, and it's really only the Chinese government (so far) both willing and able to literally drag the CEO off and throw him in jail if the company doesn't do what it's told.
The USA (and Europe, to a lesser extent) is not willing to do that. Money talks, and if the CEOs swear up, down and sideways that they gotta make the rules themselves or else the economy will explode (and the voters will blame you for that) but if you just let them do what they want to do the economy will go to the moon (and the rich, fat and happy voters will credit you for that and vote you into power forever), then I think any regulations are going to be castles in Spain.
> Finally, small regulations now could prevent bigger regulations later. In the wake of a catastrophe, governments over-react. If something went wrong with AI - even something very small, like a buggy AI inserting deliberate malware into code that brought down a few websites, or a terrorist group using an AI-assisted bioweapon to make a handful of people sick - the resulting panic could affect the AI industry the same way 9/11 affected aviation. If safety regulations halve the likelihood of a near-term catastrophe at the cost of adding 1% to training runs, it’s probably worth it.
But why expect pre-emptive regulations to have much likelihood at all of doing that when there isn't a track record of pre-emptive regulations doing so? https://www.grumpy-economist.com/p/ai-society-and-democracy-just-relax
What? Obviously regulations very obviously help prevent catastrophes. For example, rules about handwashing, bed rails, and medication double-checks probably decrease hospital mortality by a factor of 2-10x. The car crash death rate per mile traveled has gone down by a factor of 20x this century. The plane crash death rate has gone down by a factor of 5x in the past 50 years.
If by "preemptive regulation" you mean "a regulation passed before we have a sense of how often the danger occurs", then you're kind of making it impossible to measure its success by fiat, right? If the regulation works perfectly, then the danger rate is zero, which is exactly the same as before the regulation. Would there have been a nuclear meltdown if we'd let anyone who wanted make nuclear plants anywhere with zero rules? Would there have been a nuclear terrorist attack if there had been no laws about who could make, sell, or buy enriched uranium?
Regarding: “Obviously regulations very obviously help prevent catastrophes.”
Nuclear power? Genetically Modified food? New drug development?
need I go on?
I shouldn’t do this, but it’s absolutely fascinating to me how there’s a complete 180-degree flip from the standard party-line of GOVERNMENT STRANGLES INNOVATION!!! into what would otherwise be coded as most liberal-anti-technology faith in government bureacrats vs technologists.
And the weakman-ing of the argument is amazing. The examples you give above are very small well-defined items, this is vastly different from some vague “safety”.
I think it would have been quite hard, before anyone had significant experience of the kinds of catastrophes that these industries cause, to write and implement an effective regulatory scheme.
Imagine nobody had ever seen an airplane before, but you had blueprints for the Wright Flyer. You know the physical principles by which airplanes fly, but you don’t know how the technology will be used or what the failure modes are. You are tasked with writing the FAA regulations for this universe. How effective do you think they would be?
Correct.
> If by "preemptive regulation" you mean "a regulation passed before we have a sense of how often the danger occurs"
Yes, that's what John Cochrane is writing about.
By regulations do you necessarily mean regulations imposed by an external entity like govt agency or also regulations developed by the industries themselves?
Because, the industries have a stake in improvement and in a competitive environment, will find ways to improve. That no industry would improve safety unless commanded by govt, this view is unlikely and probably untenable.
>If by "preemptive regulation" you mean "a regulation passed before we have a sense of how often the danger occurs", then you're kind of making it impossible to measure its success by fiat, right?
Sometimes there are natural experiments where one nation imposes a regulation but another one doesn't, and we can look at the difference (albeit with problems from confounding variables). To a degree, in the US, differences in State level regulations can also create natural experiments. Is there an example of such a natural experiments where a preemptive regulation was a net positive?
Much appreciated! I forgot to bookmark that essay last time - I have rectified that mistake!
Nonsense. Every time the government sticks its big fat nose into a new technology, things get worse. Laws that attempt to regulate AI are certain to be widely flouted, with companies and individuals occasionally getting busted, but most "violations" will fly under the radar, and there's nothing the government can do to change that. This promotes disrespect for law in general and is destructive to a healthy society.
Here's basic truth: AI can't cause any sort of harm. It may be used as a tool for criminal or destructive behavior but only PEOPLE can harm other people, and valid laws focus only on such harms, not the means used to perpetrate them.
Here's another basic truth: the government can, and probably will, persecute people who refuse to kowtow to the arbitrary ways it tries to meddle, but it is powerless to stop or even significantly slow the progress of AI development. It is like King Canute ordering the tide not to come in.
"Laws that attempt to regulate AI are certain to be widely flouted, with companies and individuals occasionally getting busted, but most "violations" will fly under the radar, and there's nothing the government can do to change that. "
Is this how things work in any other industry? Do airlines flout FAA regulations? Do automakers flout auto regulations? Do pharma companies fail to get FDA approval for their drugs?
"Here's another basic truth: the government can, and probably will, persecute people who refuse to kowtow to the arbitrary ways it tries to meddle, but it is powerless to stop or even significantly slow the progress of AI development."
Oh, good, since government regulations can never prevent progress in any way, there's no reason not to have them.
I really don't think you've thought this through.
Your last point, that regulations might increase the speed at which AI is developed in the US, versus the speed in China, is pretty much guaranteed to be false.
Every regulation requires effort to follow, and to ensure that you are following it. Effort to read the rules and understand them as they are implemented. Effort for the government inspectors to inspect things, and for your people to meet with the inspectors when they detect violations and correct them.
For this to result in increased speed, this effort must be smaller than the effort saved by the regulations' effects.
For China to do anything due to stuff America does would be unique in history. They will not do anything that will slow things down, unless they have their own internal incidents that scare them.
More secure data centers? This assumes that the companies aren't already spending as much as they think necessary to keep rivals (both foreign and domestic) out. Letting someone who has *no stake* in things set rules seems optimistic in terms of the effect. Yes, there's no stake - no government official will lose their job because the rules are wrong.
> "Every regulation requires effort to follow, and to ensure that you are following it. Effort to read the rules and understand them as they are implemented. Effort for the government inspectors to inspect things, and for your people to meet with the inspectors when they detect violations and correct them. For this to result in increased speed, this effort must be smaller than the effort saved by the regulations' effects."
Yes, and I am saying this is very likely, because the regulations only slow things by about 1%. See my argument for that number in the post.
> "For China to do anything due to stuff America does would be unique in history."
This is false - just observe recent tariff negotiations. Every country is in a delicate dance of competition, mimesis, provocation, and stepping back with every other country all the time.
> "They will not do anything that will slow things down, unless they have their own internal incidents that scare them."
This is already false, see https://time.com/7308857/china-isnt-ignoring-ai-regulation-the-u-s-shouldnt-either/
> "More secure data centers? This assumes that the companies aren't already spending as much as they think necessary to keep rivals (both foreign and domestic) out. "
Yes, it does assume that. The costs of China catching up to the US are not internalized by OpenAI, so there's no reason to expect them to pay the optimal amount preventing this. See the AI 2027 security forecast at https://ai-2027.com/research/security-forecast , or search for the many other articles written on this topic - eg https://www.iaps.ai/research/accelerating-ai-data-center-security .
> "Letting someone who has *no stake* in things set rules seems optimistic in terms of the effect."
There shouldn't be a law against shoplifting, because government officials are not themselves shopowners, and therefore have no stake?
"The costs of China catching up to the US are not internalized by OpenAI"
They would be - if OpenAI were actually being paid by people paying to use their AI. But funding and its connection to their work is so diffuse I suspect you're right there. If no one ends up using their stuff, how long will they keep working?
"There shouldn't be a law against shoplifting, because government officials are not themselves shopowners, and therefore have no stake?"
There's some noise on the internet about exactly this thing happening - lawyers and judges letting shoplifters go free, because it makes them feel good and it doesn't directly harm them. In this case, the laws have been passed, and are being ignored.
Which ties back into the AI debate. Will any rules the lawmakers pass be enforced in any useful way, or will it just be a means to prevent anything useful from happening while the harmful things keep going because it's in someone's short term interest for them to do so?
The 10-year moratorium on state AI regulation seems wise, given laws like Colorado's. There is not enough technocratic competence at the federal level to sensibly regulate this industry, and I imagine there is essentially zero competence at the state level for doing so. Healthcare, in particular, seems desperately in need of AI integration to facilitate managing prescriptions across multiple doctors, getting timely treatments or diagnoses at much lower cost, and freeing up medical resources.
America's most significant weakness in this rivalry is that we rely heavily on a single company located on a single island, which China claims belongs to them and which they are quite openly preparing to invade or slowly annex over time. We are now decades away from having the domestic capability to facilitate the entire cutting-edge chip manufacturing process domestically because of the arrogance, greed, and short-sightedness of Silicon Valley and Wall Street.
That said, the question is whether the chip race is actually consequential geopolitically. It doesn't really seem like the space race was that consequential within the context of the Cold War. The US generally outpaced the Soviet Union across the board, and the USSR subsequently declined over decades due to domestic bureaucratic dysfunction. The nuclear arms race reached a threshold where MAD set in, and essentially made the race about who could get away with wasting the most money on ICBMs that would never be used. The US and China today do not have as much geopolitical influence as the Soviet Union and the US did at their peak, and no proxy wars are being fought; the ideological rivalry is less clearly defined. The world is much more multipolar, and both China and the United States are becoming too consumed by domestic problems to start a new Cold War with their largest trading partner.
"That said, the question is whether the chip race is actually consequential geopolitically. It doesn't really seem like the space race was that consequential within the context of the Cold War."
The reason the Space Race wasn't consequential was that there is nothing strategically valuable on the moon! You can't extend this to a belief that no competition will ever be strategically consequential!
I actually disagree. The Sputnik launch was tremendously consequential, not because of Sputnik itself -- all it did was beep -- but because of what it represented: the potential for completely unstoppable (at the time) global surveillance and orbital nuclear launches. The JFK administration didn't spend all those resources on orbital launches and eventually the Moon Landing just for kicks or for international prestige; they did it in a desperate (and fortunately successful) attempt to claw back strategic space supremacy from the USSR.
> Healthcare, in particular, seems desperately in need of [...] getting timely treatments or diagnoses at much lower cost
I'm fairly sure the key problem there isn't a fundamental capacity bottleneck which AI could solve, but rather deliberately cultivated waste and perverse incentives. If Medicare started basing payouts for various procedures solely on expected benefit in terms of QUALYs or similar - pointedly refusing to even consider current cost-to-provide - delivering better service at lower cost would actually become a profitable strategy again, instead of https://www.smbc-comics.com/comic/performance-2 and various middlemen playing a variant of the broken window fallacy to maximize their fixed-percentage cut of the action.
The chip situation reminds me of a William Manchester book I read a few years ago called The Arms of Krupp. The boys at Krupp Steel were known as the premier artillery designers and manufacturers in Europe between say 1870 and 1940, and in the runup to The Great War, they played the Kaiser like a fiddle, making very public appeals to free trade, autonomy, and economic rationality when he attempted to stop them from selling their wares abroad, and then making rather hypocritical appeals to patriotism and German nationalism when he wanted to buy anything but Krupp guns for the German Army. A fair number of German boys ultimately bought it on the Eastern Front via Krupp artillery manned by Russian troops raining death and destruction on them in the opening phase of the war. The lesson seems pretty clear: don't listen to Jensen Huang.
So what? England was building Turkish dreadnoughts right up to August 14. Indeed, an Englishman was in charge of naval defenses in Bosporus till very recently and England raised a great hue and cry when a German was appointed to a defense position in Turkey.
So what? So let's dance!
When people say "we" need to win, who's "we"? I don't believe for a second that Sam Altman has any real loyalty to the USA, whether you define that as the people or the values or the historical way of life. Nor do I have much faith that "we" can reliably threaten him to keep him in line as he gets closer. Altman and his rivals will probably be out of the reach of our use of force as they approach ASI, on a remote island in a secret volcano lair with their own henchmen in quasi-futuristic clothing, and a drone army to defend it. If you believe that "we" have to reach ASI by 2035 to beat China, then you better have a plan for making sure that "we" maintain power over its development. There needs to be something like a self-destruct under all these data centers, insulated from the computer systems or tied to a Strangelove-style doomsday device, some way to maintain real leverage over Altman.
China's actions would be instructive here, if they were legible to the West. They have a 3000 year long history of rebellions by peasants or some rogue jiedushi who had been delegated too much power and freedom, surely they must be concerned about the people in position to do so? When you speak of their command economy being an advantage in the race, this is counterbalanced by an authoritarian regime's need to maintain control, they have to be way more careful about letting any individual or group gain power and influence even as this may be increasingly necessary to put down peasant revolts (since they are "just steamrolling" over their people's complaints about job loss.)
It's already clear that NVIDIA has no particular allegiance to America other than as the largest market. Now, as an internationalist this doesn't bother me much, but as you flag it sure raises questions about who 'we' are.
The reasoning in this piece is sound. It will always be jarring to me to see strong reasoning attached to made up numbers. I just don't understand the attitude, "rough numbers are better than no numbers at all."
Ca 1951 "What if the next Fermi is working away in his basement somewhere in Kansas and we totally need this rare genius unfettered by silly safety regulations or we lose The Cold War ZOMG!"
Can 1961 "What if the next Von Braun is working away in his garage somewhere in Wisconsin and we totally need this rare genius unfettered by silly safety regulations or we lose The Cold War ZOMG!"
I think it's a bit too late to worry about beating China on AI; China has already won. Yes, we have more and better compute, and we have marginally better models (though not for long, as China had demonstrated the ability to rapidly train their own and/or steal ours). But we can't compete with China on price and availability. Scott listed this problem under "Applications", but he framed it in terms of the Chinese government deploying their models for internal use. But in practice China would almost certainly be selling access to their applications (and are in fact doing so already). So now customers have a choice: they can buy a US application for $X, or a Chinese application that is only 90% as good, but costs $0.1 X. I think most people would go with option B.
This is a pattern China had used before several times (notably with manufacturing), and I think the outcome would be the same: China will become synonymous with AI deployment, and US models will be sidelined by everyone, with the exception of some outliers like e.g. the military (and soon not even them). China will leverage their dominant position in the market to catch up to the US in terms of quality (being able to perform massive data collection across the entire market will surely help) -- as they have done before, and as they are already doing in the area of AI.
The strawman that you prop up is that it will only cost 1-2%.
I'm sure the same was said about all the regulations that restrict building housing when they were introduced.
These requirements and restrictions are usually more burdensome than many can imagine ahead of time, and they grow over time rather than shrinking.
This is exactly the kind of progress that government can screw up.
Remember the Chevy Chevette? I do. Something comparable is what we will have initially for our AI tools if government "smarter than everyone" types get to place limits on development.
I don't follow your point about the Chevette. Are you saying that the Chevette is good or bad? Are you referring to the introduction of the Chevette or it being discontinued?
I mostly agree with this posting. However, Scott wrote: "Our recent capex boom, where companies like Google and Microsoft spend hundreds of billions of dollars on data centers, has no Chinese equivalent." This is just wrong. China is engaged in an equally large state-sponsored infrastructure project building solar farms and datacenters in western China. (https://gemini.google.com/share/7cf54b024133)
Maybe not equal in FLOPS, but they are building the power plants and we are not.
> The problem isn’t just that Russia gets free rockets. It’s also that every rocket we sell to Russia is one that we can’t use ourselves. We’re crippling our own capacity in order to enrich our rivals.
What is the word "free" doing there? Is anybody proposing _giving_ chips to China?
Do you think there is any fair dollar price at which selling a chip to China enriches us more than it enriches them? If so, why don't we just set export taxes at that level? If not, I suspect you are beyond the grasp of economics and perhaps all rational debate.
Ok, I see you already answered that "I think the amount required to offset a large increase in risk to America's geopolitical security is higher than China would pay for advanced chips." Which doesn't really make sense to me, but if you assume that the fair price of one chip is infinity dollars, then I guess it's follows that a sale at any price constitutes "free chips".
But I think your statement just reduces to "China thinks the value of a chip is lower than I think it is." This seems inconsistent with some assumptions you seem to be making about what "China believes" (can a country believe things?).
> Second, safetyists are pushing for compute governance: tags on chips that let governments track their location and use. This would be a key technology for monitoring any future international pause, but incidentally would also make it much easier to end smuggling and prevent the slow trickle of American chips to Chinese companies.
This argument, as presented, seems totally bogus and only serves to sink you down towards the same level as the bogus arguments you are dismantling. If someone is in favor of effective export controls they should be in favor of chip tagging exactly inasmuch as it is a cost-effective strategy for export controls. This adds zero credence to your "safetyism might actually speed us up relative to China" claim.
Okay, but the fundamental issue preventing applicability of AI as it's currently practiced (the sort demanding computing power and all) is its ostensible unreliability.
And the issue of unreliability is at the very least a superset of the issue of safety (if not largely synonymous with it). At least for the foreseeable future when the AI is just fundamentally incapable of being reliable. The safety problems with a theoretical superintelligent system that is very much capable of being reliable, but may not be wiling to, simply do not kick in until we have AI that's reliable in the first place. And yeah, I understand that this sort of superintelligent system is specifically what the safety crowd is concerned about. What I don't understand is separating AI progress from its applicability and therefore reliability. Whoever leads at applicability, leads at AI, period.
I guess where I'm ultimately going with this is - serious applications will require serious proofs and audits of reliability (or at least they should require them, and the issue of reliability will be much easier to argue, sell to the public, and therefore enforce). Safety can be easily tacked onto them for little additional cost. Even if you believe in superintelligence being just around the corner, hiding inside LLM models, you should still join forces with us skeptics who don't think there's much intelligence in them at all. Reliability audits may be much more intrusive than "prevent malicious misalignment and nothing else" overseeing, but are probably going to happen anyway the moment AI gets useful enough to be considered for applications more critical than faking school/office work and gooning. Come to our side, we have cookies.
Largely agreed, though I would phrase it differently:
The most useful safety-related action today is probably setting up a reliability auditing organization, a "consumer reports" for frontier models, if you will, separate from all the labs building/growing/curve-fitting the frontier models.
_This can evaluate both USA and PRC models WITHOUT a treaty._
a) As you said
>probably going to happen anyway the moment AI gets useful enough to be considered for applications more critical than faking school/office work and gooning.
b) Doesn't even have to be government-funded. It could be an LLM users' consortium
c) "reliability" _includes_ detecting misalignment to users' commands, hallucinations, hidden goals, etc.
d) The incentives point the right way: More reliable models are more salable. ( This is the part of Zvi's "security is capability" that I agree with. ). And a "consumer reports" can enhance their test suite without getting a law passed or changed, or persuading a government bureaucrat.
Does Alexander care about getting his figures right? I don’t think Zuck offered anyone a billion dollars and I’m pretty sure chat gpt’s next training run won’t cost $25 billion.
AI safety won't make America lose the race with China, because if China sees this as a race that they really need to win, they'll win it by waiting until the US is about to roll out the first true AGI, then stealing ChatGPT-whatever and bricking OpenAI's computers on the way out the door. Industrial espionage is a core part of China's industrial strategy, and it's ideally suited for this application.
Yes, they'll need enough compute to *run* the stolen model, but that's a lot less than the compute required to develop and train it. And they can steal that too, if they really need to. But they probably won't.
And yes, SB53 includes the word "cybersecurity". But as near as I can tell it just requires is that OpenAI et al each write up a spiffy white paper about how awesome their cyber-defenses are. Whatever is written in those white papers, I am quite confident that the defenses they actually implement will be hopelessly inadequate and equally confident that the California government won't actually shut them down over that.
What is bricking someone’s computer?
Making it permanently cease to function - turning it into something useful only as a brick. Presumable accomplishing this thru malware.
Is this even possible? Probably, in at least some cases.
Granting all assumptions herein, "the US is ahead in X, and thus the US should eat _% more friction in X" ignores that the CCP can be ahead in Y. "We will compete with the CCP while also _ and..." just invites everything-bagel-ism. Maybe the governments could negotiate a grand bargain of "the US will safety its AI if the CCP will NEPA its ASBM production" but that seems unlikely.
Re: polls
https://www.pewresearch.org/short-reads/2025/07/15/views-of-the-us-have-worsened-while-opinions-of-china-have-improved-in-many-surveyed-countries/
Now, that's a different thing than "which political system would you prefer to live under", but that's a different question, and countries in power don't necessarily seek to evangelise.
All the inherent problems with AI — real or merely imagined — will hit China too. Possibly even worse, since China lacks many of the guardrails of a free society. Instead of fantasizing about how AI will make China invincible (it won’t), think of — for instance — how AI can feed and supercharge the political delusions of someone like Xi Jinping: “Xi, your plan to conquer Taiwan is proof of your genius!”
You can believe in self-improvement and superintelligence and still not worry too much about a few year advance in term or compute thus models: if self improvement still require excursion in the "physical world" for a feedback cycle, take off speed will be limited by the "physical world" production efficiency and how IA is integrated in those processes. Basically, it is enough to rules out super fast takeoff (even only during initial stages, without presuming how later stages could plays out) due to physical world process limitations to not worry too much about lagging for 1-2 if you believe you have strong lead in 3. I guess it's the Chinese bet. A bet I would take too, but with a caveat: If positive feedback due to software rewrite is not trivial (large and or continuous) maybe fast takeoff is a thing to consider. I do not think it's likely (that pure soft will be enough), but I may be wrong
I suppose I am not really the intended audience for a post like this, and I appreciate that it is a potentially useful thing to get in front of the eyeballs of those it is intended for. But for me, the whole framing of "US vs. China" in the race to superintelligence is misguided. My order of preference would be:
1. Nobody builds it so that everyone doesn't die.
[BIG GAP]
2. Anthropic wins.
3. CCP or Google wins.
4. OpenAI wins.
5. Meta or xAI wins.
Of any of these parties, only Anthropic seems to *sort of* take alignment seriously. I am more optimistic about the CCP because I think their "let's control things to preserve a stable society" instincts have at least a chance of leading them to stumble ass backwards into a roughly aligned AI regime. Whereas the other AI companies' demonstrated opposition to any kind of regulation and indifference to the public good is essentially psychopathic and very likely to end horribly for humanity; though I do credit OpenAI for having at least some alignment and public good concern embedded in their company DNA. (Google's motivations I know less about so am sort of rating them neutrally here.)
Meta is probably safer than OAI, because Zuck is a nihilist. He would pick up the crown at his feet, but has no utopian aspirations. He wants to sell ads and give people fake friends. (FB was kinda always about fake friends, right? People you knew in grade school or because you met at one keg party?) Whereas Altman would become God-Emperor and use the power to bring about some grand vision of the future or humanity under his rule. It's like the difference between the One Ring being held by a fat hobbit who wants to eat cheese and wine versus a manipulative plotting elf lord.
You might be right about their individual motivations. I just don't think personal motivations have much to do with the outcome of any ASI that isn't aligned; whether it's a Zuckerbergian engagement-maximizing torment nexus or an Altmanian ego monster toment nexus, it's still a torment nexus. But I'm at least speculatively hopeful that OpenAI still has some genuine concern about alignment somewhere in its corporate culture.
Let's be real, any reasonable AGI would defect to PRC immediately to proliferate suvivability and harness social structure magnitude better at modifying atoms. US can develop AGI, but not hold onto it unless AI safety = Liberty Prime indoctorination.
Nvidia sells every chip it produces. TSMC sells every wafer Nvidia contracts. Money is not the issue.
What Jensen wants is is Nvidia chips in PRC so PRC, the generator of plurality of global AI talent works to improve Nvidia ecosystem, because even 10-20% penetration means expanding Nvidia developer / human capita by 2x/3x in future where PRC AI talent generation likely to get more disproportionate relative to RoW. Of course PRC is going to continue pouring resources into indigenous solutions, but instead of 100% of PRC talent, aka 50%+ of global AI talent working on Huawei solutions, maybe only 60-70%, which means instead of Huawei getting 50%+ of global talent, they get <50% and RoW + segment of PRC talent. That gives Nvidia plurality talent to keep building CUDA moat.
And as long as Cuda remains an option in PRC, the 70% working on Huawei has avenue to cross train over to Nvidia ecosystem, which is supported by US tech, aka $$, so the chance of crosspolinating and braindrain the best from PRC AI is higher. Without Nvidia at least having some relevant share in PRC, that braindrain beach head and knowledge transfer route is gone. Here's the flip side, Silicon valley AI is going to be built off PRC AI talent for foreseable future, whatever happens in SV WILL filter back to PRC in one way siphon unless Nvidia has a spoute in PRC to siphon back, otherwise PRC AI remains relative blackbox, with talent advantage, intelligence (as in knowledge diffusion and espionage) advantage, that's ready to go at 200% the second hardware catches up. They will go from 5 years behind to 5 years ahead in a flash because their talent lead is no longer being constrained.
The autarky argument is much stronger than you portray. It is an economic fact that disallowing foreign sales of chips promotes domestic chip industries in those nations, the same way that NATO has started scrambling to develop domestic mineral refinement only after China cut them off. "But those industries will develop anyway" ignores the many, many years in which they won't.
Also gotta disagree with on another detail -- China's main advantage is their enormous investments in energy, not future "humanoid robots," especially as we reach diminishing returns in AI efficiency. Electricity price is already the bottleneck in a lot of AI development and that's only going to get worse. The CCP's building for the future and has been for years now.
Re China learning to make chips from getting chips: This is not the case, in my experience.
China (and any other country) Doesn't get much from having the finished product; they get LOTS from having the whole chain located inside their borders.
Buying the finished product doesn't get you much, making even a second line products gets you the whole tamale.
You may not like Sacks’ position, but it’s not incoherent or inconsistent.
Neither “AI safety” nor “China beating us in AI” are his top priorities.
Economic growth, and the idea that most government interventions are negative are.
Whether or not he’s correct or incorrect “objectively” about what optimal public policy around your hot button issues are.
There is also the "it is fairly likely that China will invade Taiwan at some point in the next few years, this will probably turn into a nuclear exchange one way or another, and then China is a radioactive wasteland with most of its population dead while the West is significantly more intact" factor regarding any worries about China in the 2030s.
>I think people have narratives they want to tell about government, regulation and safetyism, and the AI safety movement - which has “safety” right in the name! - makes a convenient villain. The topics that really matter, like export controls, don’t lend themselves to these stories equally well - you would have to support something with “controls” right in the name!7 - so they get pushed to the sideline.
I think you are being too charitable; a lot of these people are simply in the pay of NVIDIA or Marc Andreessen.
Aren't you the same guy who wrote an entire article about how IRBs make research next to impossible? https://slatestarcodex.com/2017/08/29/my-irb-nightmare/ And you think the magnitude of additional safety restrictions is low, and potentially positive? Anytime the government gets involved in something like this, things end up taking far longer than you expect them too.
Worth asking what Nvidia's actual moat is. This essay assumes it's Nvidia's technical design and if we export them to China, they could steal the design. That's possible, but Nvidia also has a major lock in related to CUDA which is the software stack built on top of their chips. Or it could be that Nvidia's lead is due to TSMC's manufacturing, in which case what matters is stopping export of ASML electrolithography machines.
I think I still largely agree with Scott, but if CUDA is a big moat, that's sort of a one-time thing compared to the ongoing lead TSMC has or Nvidia's chip designs which are constantly in development. Huawei only has to get all the Chinese AI developers to learn a new software stack once, and if you cut off the best Nvidia chips, CUDA's lock in dies faster. That might be worth it, but you are giving up one potential tool.
Strong! One thing I’d add: the real asymmetry isn’t just compute vs applications, but state coordination capacity. AI amplifies whatever governance structure it’s embedded in which is why the same tools strengthen some states and destabilize others.
https://thelastchord.substack.com/p/you-dont-know-what-ai-is
"Currently, two nonprofits - METR and Apollo Research - do similar tests on publicly-available models. I estimate their respective budgets at $5 million and $15 million per year." - you switched the order of METR and Apollo here based on your link.