I'm not worried about them taking away my laptop or cell phone – I'm worried about them taking away (or requiring me to register, or otherwise restricting) my powerful home desktop server/workstation. High-end chips tend to get cheaper over time (and I am making more money over time) until it is viable to run old high-end server hardware at home. My original very budget desktop computer build used very cheap old server hardware as it was a lot cheaper than newer hardware with the same power, and it's served me extremely well since I built it in 2016.
> More likely, a world on track to reach this point would modify its cell phone / laptop chips so they couldn’t train AI
And THAT is what I'm especially scared of! I REALLY do NOT want DRM on my hardware that controls what I can and cannot do on my own computer! It's part of why I use AMD cards instead of Nvidia (no crypto mining DRM), why I use Linux instead of Windows and bought a Librem 5!
> Unrelatedly, You Already Live In A Global Panopticon Orwellian Surveillance Dystopia
And that's a bad thing! There are already tons and tons of government restrictions I absolutely HATE and I do NOT want them to add even more of them!
The only way I can support this is if it ONLY applies to hardware too powerful to feasibly run at home, and that this is not some fixed benchmark that tech advancement will eventually outpace. Anything that restricts what I can plausibly run on a normal 120V outlet is much too dystopian for me.
Yeah this seems like the weakest part to me. In general I feel like the trend is towards monopolization of hardware, where individual users are only left enough crumbs to connect to the cloud with and borrow gatekept compute from there. Cloud gaming, streaming, the works. Local AI is already pretty weak as is unless you own a beast of a computer. And honestly, if we ever got at the point where you can train dangerous AI on a desktop we're done anyway. A massive coordinated parallel training run (like those protein folding @home computations) would be noticeable in other ways.
But also, wouldn't almost by definition the AGI/ASI which was trained in giant data centres with H200s or whatever new devilry NVIDIA came up with be so powerful that nothing I could train on my meagre machine, even with all the world's algorithmic improvements, can hold a candle to them anyway? Or do we expect algorithmic improvements to be the great equalizer that makes compute irrelevant? I don't think that's a very likely world. So ultimately I just don't see why should anyone worry this much about customer hardware if it was just about AI safety (after all, if your home-trained AI can teach you how to build a bomb... there probably were instructions on how to build a bomb in your training corpus to begin with).
Yeah. I suppose it obviously makes it easier to gather and also some people might be too stupid to learn it from the books but then will they not be also too stupid to successfully follow the instructions?
Regardless, I wouldn't expect this to be completely transformative or anything. Maybe a bit of increased risk, but nothing major.
That would be a lot of effort for a disgruntled accountant, or someone decently intelligent but with no chemistry background. They would take several weeks’ work at least to produce a very basic bomb. If someone is aggrieved after a bad break-up, they’d almost certainly calm down before they could design an effective bomb. A powerful LLM could solve not only the bomb design, but the planning and implementation details that would still take serious effort, even for an experienced chemist. An LLM might let someone build and set a bomb within hours after an upsetting event, when large numbers of people already commit violent crimes.
Could it happen once or twice, in place of what now would be just a random stabbing or mass shooting? Maybe.
Could it happen so much that it straight up destabilizes civilization, requiring us to take commensurate global precautions to prevent it? No. This is less of an issue in terms of lives at stake than the US 2nd Amendment as far as I'm concerned. It doesn't belong not only in the same ballpark, but straight up on the same planet as AI existential risks. It only gets pushed to the forefront because "terrorism" is always a button that works with politicians, who like stories about the scary common people doing violet things and don't like admonitions about how they could misuse or abuse power themselves. It's an irrelevance.
US 2nd Amendment is less of an issue in terms of lives than Europe not having air conditioning. And a heat-related death is a lot more torture than a bleed-out.
Well, most people who die due to heat don't literally die of heatstroke, it's more that high levels of heat raise slightly the incidence of a bunch of classic elderly killers like heart attacks. But sure, I made the comparison with the 2nd A specifically as a comparative "people get easy means to commit terrorist acts". A semi-auto rifle is a hell of a lot more direct and easier than a long tutorial on how to build a pipe bomb!
Yah? It wasn't much effort at all for a disgruntled programmer to fly a plane into a skyscraper (already had his pilot's license). Transportation continues to be a pretty easy way to create mass casualties.
There is a difference between "science knows how to do X" and "any idiot can make X by clicking a button".
LLMs do not move the situation completely from one extreme to another, but you can use them iteratively -- for any obstacle (technical or legal), you can ask them how to overcome those obstacles most easily etc.
But someone still needs to buy all the equipment, the reagents, set everything up, every step they need to follow would need to be broken down to extreme levels and when things go wrong they won't be terribly well equipped to even describe the problems to the LLM.
If someone like that wants to cause casualties they could just turn their steering wheel slightly while driving at high speed.
The plan specifically speculates that these restrictions might be necessary if consumer demand for compute grows rapidly and supply with it. The scenario isn't the one you describe. If the explosive growth of consumer compute doesn't happen, it should be pretty clear in economic data.
I would buy the AI chips for home use, were they anywhere near affordable (and yes, they would be put to work). As it is, my husband can't buy a new computer. OP wants to make it impossible for my husband to EVER buy a useful home computer again. And no, that's not hyperbole.
Right, this was an obvious objection for me, and where the analogy with nuclear arms control really breaks down.
Nuclear tech is “dual use” in that you can plausibly use it for bombs or for power plants - but either way uranium enrichment is industrial scale technology that’s pretty clearly distinct from any consumer use.
“AI chips” are omni-use. They are just somewhat optimized versions of things billions of consumers already own, and will probably want to continue to own.
That optimization matters! You can just restrict the optimized version and not restrict the unoptimized version! There's an obvious difference between the two! (At least, until such a regulation exists -- then people will deliberately attempt to push on it. But you can set the bar with enough safety margin that this won't cause real problems.)
Can you? Are they actually that distinctly different? Without getting a dystopian level of government management of the details of every design?
And are the optimized designs actually not useful for everything else a powerful computer would be good for?
Are they made on substantially different equipment from substantially different materials?
The whole edifice rests on this idea that there are “AI chips” and “Not AI chips” they can be neatly separated. Since these are ultimately all just “computers” I don’t think that’s easy.
I feel like this is basically an argument from ignorance -- "I'm not aware of the distinction, so it's probably not very big." Now to be fair I'm no expert on this myself; this isn't my area. But it's not too hard to find people describing the differences! In this very thread you have moonshadow discussing it: https://www.astralcodexten.com/p/ai-chip-regulation-is-not-a-dystopian/comment/294997966 This is something you can find out about!
> Are they made on substantially different equipment from substantially different materials?
No, I don't believe so. Nobody's going and making them out of germanium or something. But the distinction can be large enough to be detectable and regulatable without it being so large as that.
> And are the optimized designs actually not useful for everything else a powerful computer would be good for?
I mean they're not CPUs -- more like GPUs, except, as I understand it, even more specialized. Most things don't benefit a lot from that sort of massive-but-constrained parallelism. They're not going to make the serial parts any faster, and just because something could be parallellized by some chip doesn't mean these can do it -- you can't have just one core take a branch, either they all do or none do! Sure technically they can be used for any computation but it doesn't seem practical from what little I know?
> Without getting a dystopian level of government management of the details of every design?
I'll make what might be an argument from ignorance myself here and say, can you point to a reason this would be worse than other fine differences that are also regulated? Like cars, for instance. You don't need to sort things by design, you can just say that there are tests things need to pass. This may involve drawing arbitrary lines -- that's fine!
And that's the thing, that although you say
> The whole edifice rests on this idea that there are “AI chips” and “Not AI chips” they can be neatly separated.
I'm not sure it actually does rest on that. Because yes that's helpful for the argument if true, but if it's not, well, the law draws arbitrary distinctions all the time. Taking an arbitrary example I just looked up: If you're selling apple butter, the FDA considers it a violation if there's an average of 4 or more rodent hairs per 100 grams of apple butter. 3 per 100g is allowed! There's no natural distinction here, no large gap, and yet, despite the difference being arbitrary, detecting it doesn't require dystopian capabilities.
How much do you know about the difference between modern general-purpose chips and modern chips designed for AI-specific uses? There is a trillion dollar industry that has spent a great deal of time quantifying exactly which traits it values in a chip, and you come off as someone declaring "Dump trucks and F1 cars are both vehicles, there's basically no way to define them apart."
And if the training landscape shifts to be more in the form of small numbers of parallel trajectories with non-shared weights, rather than massive numbers of parallel trajectories with shared weights, do we start banning or requiring licenses for home computers?
Here’s where I definitely start getting out of my depth, but I had concerns along those lines as well: right now, high end AI developers have no reason not to develop AI on hardware especially optimized for that task, and comparatively little incentive to figure out workarounds to do more with less hardware.
But this proposal creates that incentive - how strong is our assumption that *only* highly optimized, distinct hardware can do dangerous AI development?
IMO this argument misses a bit. *Right now* there is a pretty sharp line between a H100 and my gaming GPU. But this Plan A question applies to a hypothetical future where user hardware has become powerful or efficient enough to train even, say, Mythos-level models. So it's kind of baked into the assumption that you already have relatively non-specialised, accessible hardware that can do it. So will the distinction still be that sharp then?
I'd find the linked comment more useful if it put numbers on the dimensions of the matrices and on performance.
I'm not an expert but I believe that gaming requires 4x4x16 matmuls while e.g. the H100 supports up to 16x16x16, and these matmuls get composed together into 4096x4096x16 or whatever.
Is the gap between 4x4x16 and 16x16x16 wide enough to stop anyone? Who knows! Is gaming representative of workloads like video editing? Who knows! Will this halt organic chemistry research by accident because nobody can do protein folding without a quarterly UN inspection? Who knows!
> If you're selling apple butter, the FDA considers it a violation if there's an average of 4 or more rodent hairs per 100 grams of apple butter. 3 per 100g is allowed! There's no natural distinction here, no large gap,
That's just a quality-control issue - nobody's paying extra for apple butter with MORE rodent hair. Or if they wanted to, for whatever bizarre reason, could prep a batch in the privacy of their own home. The FDA would presumably only hassle them if they tried to sell it, and even then only if it was misleadingly labeled.
All Turing-complete computers are fundamentally equivalent in some important ways. This computer science 101 stuff, foundations of the field, basis of all those "can it run Doom?" tricks. AI chips, and the cheaper sort which go in, say, a cell phone, quadcopter, dishwasher, or tractor all have more in common with each other, both on a molecular scale and in terms of function, than rodent hair and apple butter do.
So if there's an arbitrary threshold defined, inherently it's going to be blurrier than the FDA quality-control stuff. Then, because there's money at stake, chipmakers are immediately going to come up with a design which is capable of as much useful computation as possible, while falling just barely outside the threshold where those new draconian regulations apply.
That new chip will then, naturally, become the cheapest way to train AIs. Meaning the draconian regulations need to be updated to apply to it too. But, since the bureaucrats responsible for writing that update aren't idiots, they'll want to make sure the exact same thing doesn't immediately happen again... and, as the saying goes, the only way to be sure is to take off and nuke the site from orbit. That is, redefine the draconian regulations to be FAR more expansive, applying to basically everything Turing-complete. What's their incentive to do any less?
A while back, I looked into renting out my RTX 4070 for AI training when I'm not using it- apparently, that's possible, but would only earn a few dollars per month; barely above the electricity cost and clearly not worth the effort. I really don't get the impression that current home PC compute is valuable enough for AI training to be worth restricting.
As Scott mentioned, that may change if consumer-grade hardware improves rapidly enough, but in practice, I think this mostly looks like a future where gamers have to temporarily settle for real-time generative AI graphics that are only almost indistinguishable from video instead of entirely indistinguishable- hardly a dystopian nightmare.
This is hilarious. So they couldn't do math, then? Not very useful anymore. _Or_, there is a dystopian surveillance device that shuts down your computer when it _looks_ like you're training AI.
This is much like "..would modify its servers so they couldn't send spam." Good luck with that. Sorry, count me out.
No, it just means restricting chips that are optimized for this purpose. Sure, any processor can technically do it, but without specialized chips it's too slow to be practically relevant for frontier AI models. Other commenters here have expanded on this a bit.
This is backwards. Specialized hardware _today_ is outclassed by general purpose hardware _tomorrow_. The point is not restricting _tomorrow's_ even more powerful chips and frontier models; this was already what "Plan A" proposed. The point is when tomorrow's regular hardware can now train today's "dangerous" frontier models. There will always come this point... often quicker than expected.
I think you're assuming that there's only one dimension of "progress" in GPUs; I'm not an expert in this either, but to me "specialized" means "optimized in a way different than just making it more powerful; on a different path than consumer devices will take as they get more powerful." I do think it's plausible that *eventually* the GPUs we might otherwise want for gaming and such would be powerful enough to catch up to the frontier model ceiling imposed by plan A, specialization be damned, but maybe this would take a while.
I don't think the intent is to implement this via some sort of DRM; rather, as other commenters detail below, LLM training uses specialized hardware. Obviously technically it can be done with any hardware but this is sufficiently slow as to be impractical (consumer graphics cards are not actually sufficient for frontier LLMs) and irrelevant. So there just isn't a need to take that approach in the first place.
> In short, the answer is that anyone concerned about loss of control should think Plan A is an improvement, along with anyone concerned about concentration of power—except for the people in whom the power would concentrate by default.
Perhaps the authors of ai-2040 think that people who would spend their money on a more powerful machine, which would let them do more things and accumulate more resources, which would let them afford a more powerful machine, ARE the people in whom the power would concentrate by default.
So you care a whole hell of a lot about privacy and about having freedom to do exactly as you like with your computer and related bits of tech. Hey, fine with me. But look, why should anybody give a shit what government policies you "absolutely HATE" and what regulation or lack thereof would let you just keep diddybopping along, contented?. You seem not to give a shit how things are going to work out for the rest of the world.
While I was writing this I ate a chicken sandwich from Panera that had a stale, salty taste. I had been looking forward to this sandwich, and it is such a disappointment! I hate that stale salty taste!
I care enough about that awful sandwich that I would not want to, say, ban you from ever eating any other sandwich, which would be the rough political equivilent here.
Even if I didn't care, you presumably still vote, so would be relevant to appeal to politically.
I care about how things work out for the rest of the world too, and like most of plan A. I just think it should be an important goal to be able to decentralise control over AI and avert disaster, without placing onerous restrictions on smaller-scale hardware.
For example, surely safety can be done, by somehow ensuring the most powerful systems that run the smartest AI models have a large enough gap from the smaller, freedom-respecting systems individuals may run? Where the goal posts keep up with the times, so an individual may eventually wield that much power once the safe, powerful models are even better?
My original reply, perhaps, could have been better. I was pretty badly ragebaited by the overall premise and several unaddressed issuss in the article, so didn't write quite as calmly and rationally as I would prefer to normally.
The post is called "AI Chip Regulation Is Not A Dystopian Surveillance State", though, not "AI Chip Regulation Is A Bit Of A Dystopian Surveillance State But That Is An Unfortunately Necessary Evil To Avoid Worse Things". Addressing things that are surveillance-state-like is fair criticism of Scott's point because his claim is that this is not even a trade-off that exists.
I don't object to OP's complaining that for someone with his interests and priorities, he would be living in a surveillance-state like set-up, with some options he values greatly closed off. My point was that all he talked about was how much he'd hate having his freedom to build souped-up computers curtailed. For anyone who honestly believes ASI endangers our species, I think this is a situation where the ethical thing is to consider the big picture effect of proposed regulations, and not mainly one's personal preferences. Only a tiny fraction of the world's population is going to lose an activity or possession they care about as a direct result of chips being regulated the way OP "HATES."
In a situation of this kind, is it necessary to give a shit what a few hobbyists hate?
There is a whole separate argument someone could make against the chip regulation: it will lead to more and more regulation of other kinds, massive and diverse regulations that will harm many lives. That may be true, or may not. I do not know how to predict the likelihood that it would. Do you? But in any case, that is not OP's argument. His argument is "Waah, I HATE it."
I would rather go back to consumer hardware from 10-20 years ago (in terms of total FLOPS/clock speed/etc.) than have the functions of said hardware artificially lobotomised, but the hardware will need to be restricted.
The point about consumer tech seems bizarrely underappreciated. Over on Datasecretslox (the unofficial forum for ACX(?)) there are a good 250 posts arguing in great detail over how Plan A might mean people could no longer build gaming PCs.
Approximately 1 post makes the point that the regulatory regime suggested would have zero impact on consumer graphics cards
"They suggest that if consumer hardware production seemed on track to grow to about ~twice current levels, then it might be necessary to either institute cap-and-trade, redesign consumer chips to prevent them from being used in AI training, or both (cap-and-trade on non-redesigned chips, plus unlimited redesigned chips).".
OH so dystopian regulations aren't necessary this year.
I don't quite understand what "non-AI capable" chips would be. AI is linear algebra. Graphics are also linear algebra; in general, all chips can do linear algebra. How do you ban a chip from doing linear algebra only if it's for the purposes of AI? A chip that can't do linear algebra at all is basically not a chip (protip: it's mostly just additions and multiplications; simply a lot of them). I can see making chips that are inefficient at it (though going too hard on that will also throttle graphics etc), but not chips that are constitutionally incapable of it, unless they are, in fact, being surveilled by a smart discerning agent in some way.
Basic solution: you make chips that don't do the math right. If a particular machine code instruction is given, the entire chip breaks and you have to reboot the computer.
... this is a VERY BAD solution. Don't do this AGAIN.
AI is a very specific kind of linear algebra, graphics are much broader. Modern AI chips are already optimized to be better at AI calculations than graphics cards. It is easy to look at a chip's specs and see if it is designed for AI or broader use, and banning those chips makes training runs an order of magnitude more expensive.
Making it a bit harder is surely possible, but I don't know if it counts as enough. A lot of it depends on what new algorithms do and on how much better new chips get.
We know how much it costs to train on modern AI chips vs graphics cards, and restricting someone to the latter is already much more than "a bit harder". Unless you have some novel theory of why general purpose graphics cards will suddenly become super good at the specific thing AI chips are optimized for, that gap will at least persist.
No, I'm genuinely not aware of how much the gap is. If it's that significant, fair, but then the issue may never arise. The point is that Plan A assumes the issue may arise - due to improvements leading to e.g. something like Mythos being trainable on non-specialised hardware.
Quantify this. Are we talking two orders of magnitude, or thirty? (in terms of time it would take to train using "general purpose graphics cards"). Then ask yourself, how many orders of magnitude does a competent programmer get you?
It's easy for now, in an environment where "looking too much like an AI chip" doesn't abruptly tank your product's profit margin. Once some specific formal definition of the distinction gets enshrined and enforced, so there's money to be made by fuzz-testing the inspectors, how long do you think it'll take for Goodhart's Law to kick in?
Specifically, you need chips that can multiply very large matrices, very fast, with some other operations in between, and /keep doing useful work instead of going idle/. This generally involves a physically large dedicated piece of silicon for the matrix multiplications, some more general silicon close to it so you can do the various other operations between matrix multiplies without having to move all the data you're working on out to a different memory domain far away then bring it back (the actual numerical performance matters rather less - matmuls scale with the cube of the problem size but the linear stuff between them only with the square), and very large amounts of memory physically the chip itself so that it is fast for this hardware to access in order to keep it fed with data at the rate at which it can process it.
Few other problems we want solving in bulk require hardware of this shape, and the difference in performance running a large model on hardware like this versus ordinary consumer hardware is several orders of magnitude, where your model can fit in the ordinary hardware's memory at all.
People may be willing to wait a few minutes for a model to respond to a question or complete a task, but they will not be willing to wait days or even hours, so hardware on which an LLM takes that long to do what you want it to is "not AI capable".
All of this becomes even worse if you are training instead of just doing inference: training involves repeating this process a vast number of times, at ideally much faster than human interaction speeds, and all your compute costs are at least doubled since after running the forward inference steps and getting a result, you then have to run the inverse calculations to work out the adjustments to be made to the various weights. If companies are trying to train models for some purpose but have competitors doing the same thing orders of magnitude faster/cheaper, they would be well advised to pivot.
Right, but my experience is that you can do most of these things on an ordinary GPU, including ones designed more for graphics/gaming. Surely not quite as efficiently as on dedicated chips, but I don't know if so much *less* efficiently that the concern (which I think here is diffuse training runs, not just individuals doing everything on their own) completely vanishes.
Yes, your "not quite" is doing a lot of work there, and is precisely the load-bearing distinction (yeah, I'm infected XD) between "AI chips" and "not AI-capable" that people are trying to make.
My raspberry pi is Turing complete and can do any computation any other computer can, given enough time and storage; but describing it as "not quite as efficient" as my gaming desktop is quite a stretch.
If the goal is to slow down frontier research, and we're sold on doing that at all, regulating AI-specific bleeding-edge performance seems like a more reasonable lever to pull than sweeping blanket bans of technology.
I'm pretty sure I could train a few million parameters model on my PC if I left it running a few days, which is enough for it to speak coherent English (see Tinystories or similar micro-LMs). I know I can train a ~100k params one in minutes, on an average laptop. And that's with current algorithm and behind the times hardware. Basically I don't know if the gap is large enough for this to be considered enough.
Come back when consumer hardware deployments rely on photonic networking switches because the difference they make is worth the money they cost, and we can talk about the competitiveness of distributed training on consumer hardware.
Until then, I'm calling it: if the people we don't want to win the AI race are limited to building their AI on distributed networks of ordinary consumer devices, the regulation has done its job.
Hey I'd be ok with that, I honestly don't think the consumer regulation is needed at all in Plan A. It focuses on absolute thresholds rather than relative ones. Like maybe in 10 years hardware advancements combined with new algorithms allow people to train a Mythos-level model on just a consumer laptop. Ok, and? By that point in this kind of trajectory the lab models would be super-geniuses that have hardened every system in the world against cyberattack, can create vaccines for any bioweapon in days, and have invented bomb-sniffing machines. Try any malarkey and the robo-cops will be at your door in minutes. Mythos will just be a fun toy.
The consumer hardware overhang only matters if somehow progress flattens; if lab models hit a plateau and regular hardware/algorithms eventually merely catch up. But that seems like a weird thing, it would require compute at some point to hit returns that essentially fall to zero.
Asking Gemini how many home machines would it take to train something like the big AI companies, it came back with a cluster of 16,000 H100s GPUs would take about 5,000,000 home GPUs. The issues are, speed and memory of an H100 is much faster than a top home GPU. Most people don't have top gamer GPUs, they have mid-level GPUs. Then you have to add the network bandwidth.
Further, as an example, to train Llama 3.1 405B, Meta ran 16,384 enterprise H100 GPUs for 54 days. Most people aren't going to let you use their computer 24/7 for training. It's estimate is 1.7 years with 5,000,000 people coordinating on "Training@Home"
It further claims, top models currently use up to 12x that training. I don't think it matters if the numbers are perfect. I'm assuming is within the same order of magnitude.
Of course maybe people figure out simpler algos. A human brain as a target for energy and materials is one example that there's room to bring the need for all this compute down. Maybe 50 people worth of e-brain power can make train one e-brain in 1/50th the time as a human is trained?
If that doesn't make any sense, all I'm trying to say is PlanA assumes we only need to regulate the big companies because people at home, even coordinated, can't get that much compute. But, it also assumes that much compute is needed.
The whole premise is: better/more hardware (maybe as AI causes automation to spread more widely because now custom software is cheaper it's more useful and more things get "smart" with their own chips?) and better algorithms (the whole focus of Plan A is algorithmic research), so there probably comes a point where Training@Home becomes doable, especially if for example we figure out how to decouple the reasoning/basic language understanding training from the vast knowledge training. You could imagine a smaller model being smarter than the current big ones if it didn't also have to cram in all the useless factoids that will be in every corpus, just a pure strong reasoner that needs to read in knowledge at inference time.
When I lived in Australia, they tried to ban the export of all maths with cryptographic applications. It never got implemented because it would have outlawed linear algebra. The lawmaking process is generally quite good at cutting scope down to something enforceable.
Is this why OP is calling for a "cryptographic" solution? I suppose you could use a PGP, if the private key only lives on the chip... "here, I told you to shut down, please verify you are you before doing so" -- aka sending back the unencrypted data to prove that you have the private key. But that doesn't AT ALL show that it actually shut down, just that you're talking to the right chip.
You were also assuming they buy into the same stack of philosophical assumptions. They may think the orthogonality thesis simply isn’t true. They may think AI will never be great at long-term planning, and give them an enormous tactical advantage.
I think we have reason to hope that the Chinese leadership aren't total idiots. But yes, Plan A working does require them to not be *entirely* blind to the ever-mounting evidence that AI is an extinction threat.
China has been more concerned overall about keeping frontier AI safe, trustworthy and controllable than the US. They're generally more wary of the new tech. I'm more concerned that the US leadership won't be total idiots.
Pooh bear has a lot more murders to worry about than AIs murdering him. So, I assume he cares about the people with guns, right now, a lot more than the AIs that might have guns pointed at him... once they develop the capability to think.
Funny thing: the Ukrainians were very grateful to have the AIDS-contaminated blood delivered to their wounded soldiers. It turns out long term threats to your life are actually "not so worth worrying about" when you have current, immediate threats, like Xi does.
NVIDIA is investing in open models for strategic reasons—they want to expand their market beyond a few model labs that are all building their own chips anyway. And they’re partners with Palantir to enable enterprises to posttrain these open models and deploy them without handing their data to the big labs. Other neolabs are also offering posttraining-as-a-service. Enterprise wants custom models, for data sovereignty, greater control over outputs, pricing power, to avoid becoming totally dependent on model labs.
Fundamentally, if you don’t own the weights you’re renting intelligence, but if you do, you’re an owner.
Until the ban-open-weights coalition acknowledges that it’s arguing that no one except for a handful of companies should be able to own AI — the technology that we all agree determines the future — those who appreciate open models will see them as not merely acting on behalf of humanity’s safety but as agents of model labs looking to own the lightcone.
Fundamentally, the proposal to ban open models has big winners and losers, and the proponents aren’t proposing any sort of compensation to the losers.
Ultimately the core problem is that with something as big as "someone gets to own ALL OF THE FUTURE", no one trusts anyone else with it, and with good reason! Which means the entire thing, unless aborted entirely, inevitably devolves in either a scrap where the last man standing wins all or a negotiation where all the parties with enough power share the loot and everyone else gets crumbs or nothing.
Plan A is basically the negotiation, which avoids the much uglier scrap, and gives us the hope that the loot is so big, even the crumbs falling off the table are better than what we have now. But damn, is it a depressing best case scenario even so.
That’s a good idea on Nvidia’s part, but I doubt they’ll be designing anything near frontier models. My impression is there’s plenty of demand for their highest-end chips, and the difference in demand if they develop a frontier models seems like it can’t possibly be equal to the cost of training a new frontier model.
To my understanding Chinese investments there are also strategic. Suppose an American company spends $10 billion training a frontier model, then a Chinese company spends $100 million distilling that frontier model's capabilities into something which they give away. The American model soon has no paying customers (why rent the cow when you can get open-source milk for free?), leaving China roughly $9.9 billion ahead on attrition.
Any costs the frontier lab managed to recoup in those first few months subtract from that effective total, but on the other hand, being the default source of the best AI models for people who can't or won't pay is also a grand opportunity for political influence - basic gratitude, baked-in propaganda, self-congratulation for adhering to Communist ideals, etc.
Some feedback, and I know this is mostly useless but maybe in aggregate with other feedback it can be useful:
Your previous posts about AI doom/regulation caused me to want to engage and "talk it out". For reasons that are not entirely clear to me, this post triggered my "fight" reflex and I now find myself wanting to burn your position to the ground and salt the earth. I won't try to argue any specific points both because I know once I'm in this "mode" my brain isn't working properly, and because I strongly suspect many other people here will make all of the arguments that I would make anyway.
In an effort to try to be helpful/productive, I will try to speculate a bit on *why* this post triggered my fight reflex rather than my engage reflex, but with this sort of thing it isn't always clear since it is my lizard brain taking over which is mostly opaque to me in its decision making strategies.
I think the biggest issue is that you seem to think the current state of the world is acceptable and more of what we have is therefore also acceptable. I think the war on drugs (driven by drug scheduling) and current global financial surveillance (KYC/AML) are two of the worst regulatory things that have happened in the world in the last 50 years. I hold the view that those regulations, and others similar to them, are a big contributor to why we don't live in a solar punk utopia and instead live in our current dystopia. I'm not so naive to think they are the sole cause, but I do believe they play an outsized negative role.
You also seem to suggest that open source isn't gonna make it, so we might as well ban it, which definitely raises my hackles as someone who passionately believes that knowledge and tools should not be gate kept.
You seem to suggest that we already live in an AI surveillance state and that is acceptable, but while some people (you included) may live in such a world, I choose not to live in such a world by running my own local models and choosing to utilize privacy preserving cloud models when I need bigger things. I also run ablated models so I can get the AI to actually answer my questions and be a useful tool. What you are saying is that, "gate keeping and censoring AI won't have an impact on my life, because I choose to live under surveillance", but it would definitely have an impact on many people who are not-you.
I think he makes a decent case that for most people this wouldn't be a noticeable worsening, though there's a niche who cares more (admittedly, I never was as dedicated to full blown privacy and decoupling my hardware from corpo software as some people here - but I do generally enjoy leaning open for my software and I definitely appreciate the reasons. I'm just too lazy and don't feel like I have high enough stake in it to fully commit).
But ultimately the plan is kind of fuzzy about stuff like the non-AI viable chips. If we look at policies in terms of, do they distribute power from the top down or from the bottom up, I think some stuff like the traceability of big training data centres does distribute it from the top down, a bit (insofar as accountability to democratic governments is better than complete autonomy for AI giants whose mission is essentially "take over the world"). But yeah, everything about suppressing efforts also from the bottom is more dangerous, especially if we consider how much more likely it is to be the only part that gets passionately adopted.
> I think he makes a decent case that for most people this wouldn't be a noticeable worsening, though there's a niche who cares more
This may just be my brain rationalizing why I endlessly suffer with making everything harder than it needs to be, but I think that "open source" being a fallback option for people is a big part of what keeps things like tech giants in check. While the average user may never use Libre Office, Linux, Firefox, etc., the fact that they exist may play a role in preventing big corporations from leveraging their position of power against everyone more than they currently do. When your users are strongly captured (e.g., due to regulatory constraints that prevent anyone from competing with you), you have no real incentive to care about your users. You just need to be "better than nothing" because that is the choice the users have. OSX and Windows, for example, need to be "better than Linux" which is a huge step up from "better than no operating system".
In a world with no open source AI, the blessed incumbents (which won't be Anthropic because the US government is petty and vindictive, something not discussed here) only have to "be a better choice than having no access whatsoever". They don't have to be better than some "slightly worse, but still pretty good option". So when OpenAI decides to read all of your chats, train on all of them, and sell their contents to third parties, users only have the choice of "accept the panopticon" or "go back to the stone age". In a world with open source AI that is 6 months behind, users instead can choose the panopticon, or to go back to tech from 6 months ago.
To be fair, Windows, especially lately, is hardly "better than Linux". It keeps its position due to a series of reasons which mostly have to do with inertia, regulatory capture, and a few specific market niches where they still dominate (mainly gaming).
I agree with the general framing but as far as AI that isn't just open but is run locally (and not on some other cloud) goes, I've hardly been able to use it for anything useful at all. It takes some hefty hardware to run the bigger and more useful models. But yes, the existence of competition, and especially *free* competition, obviously has some effect.
Microsoft seems worried about losing market share in gaming to Linux (probably mostly due to the Steam Deck) and they have announced their intent to improve the gaming performance of Windows. So I would argue that this proves Micah right.
I am very much not on your side in regard to AI. However I spent enough time as a young libertarian to get what you're gesturing at, and in the end I think we both share a skepticism of powerful controlling bodies from which you can't opt out. (Indeed that's MY main objection to Plan A, it installs the AIs as your new permanent overlords forever in just 14 years, heck somebody appointed to the Federal Reserve today would still be on it when a machine is taking over Earth.) Outside of techno-libertarians, the argument about open source is mostly not gonna catch on with normies so long as the big labs are being smart about how their products interact. If they were still perceived as being slavishly woke then you'd be able to make some inroads on this with normie right-wingers. Probably not the other direction though, normie left winger response to a right-leaning AI would be to nationalize it and make it left-wing, not promote alternatives from outside the hegemony.
Personally I am still waiting for the banking and insurance industries to weigh in here, as I could not presently advocate or permit my company using a remotely-hosted AI due to data security and vendor contracts. I was expecting Zuck's model of the b2b world where you locally-hosted behind the curve systems and tuned them to your business needs. I have no doubt that eventually, if the big labs are gonna be the only game in town and its all remotely-hosted, there will be a mechanism that gets propagated through the banking and insurance regs and contracts to allow for this, but I also highly suspect they will add in a mandatory grift layer, like you'll be allowed to operate only if you pay some huge sum yearly to a specialized AI data breach monitor/repair company.
If that's your concern, then Plan A at least gives you fourteen years to organize the Bulterian Jihad, which is better than any other plan that isn't "hope AI is a fizzle" or "trigger some other apocalypse in the next couple years".
The trick would be making sure there's enough "leakage" in Plan A to ensure sub-ASIs would be able to cause enough damage to drive support for the Jihad.
"for most people this wouldn't be a noticeable worsening, though there's a niche who cares more "
These niche's have historically been the reason most people don't suffer noticable worsening. Few bring what become landmark civil (more than discrimination) rights cases to the Supreme Courts, and yet we all benefit. For example, the encryption wars of yore.
I made sure to read the whole article before replying, but was still rushing to reply really fast as I was pretty badly triggered by the article. I worried that it might negatively affect the quality of my response, and still think that this is a possibility despite me having attempted to at least understand the argument with at least a little bit open of a mind.
We don’t live in solarpunk utopia because it was never in the cards according to the current distribution mechanism. Deregulation is what produced the world you hate.
This isn't a situation where there's objectively correct info on a Wikipedia page that Google would lead me to. You have a point of view about deregulation having an impact on the US, and I'm inviting you to give your view rather than doing this https://knowyourmeme.com/memes/refuses-to-elaborate-further
I asked my favorite chatbot and it had a pretty rosy view of the last 50 years of deregulation. You're going to have to explain your perspective yourself.
I am pretty sure "financial surveillance" doesn't refer to "50 years of financial deregulation", but almost certainly to post-2008 regulations of banking
I don't think that Scott thinks the current world is good. Just that it would get even worse if we don't take steps towards tracking AI chips. Because they're dangerous in large concentrations.
It doesn't seem to me to be much different from regulating CFCs and nuclear reactors. There are legitimate reasons for using these things but their use has such high externalities that we unfortunately need to restrict them. (Though we clearly overdid it in the case of nuclear reactors.)
As an aside, I like open source philosophy as much as the next guy, but to me, open weight models seem little more than publicity stunts by people with too much money. The users of the models haven't actually *contributed* to the model development (apart from publishing text on the Internet that's been used as training data, which I definitely think the big AI companies should compensate people for, but that's a different topic), so is it really open source in the way we usually understand the term?
The models aren't just open source in the sense that we know the weights, they're also open source in the sense that we know the architecture and the training methodology. I agree it's not the same because ultimately you can recompile a piece of software, you can't just retrain a 100B parameters model. But there's genuine knowledge about what they are like; you can finetune your own versions of them, you can do interpretability studies on them.
Nuclear reactors are things which just about no ordinary person would ever 1) be able to own and use on their own even without the restrictions, or 2) have much of a reason to want. Computer chips are general purpose devices that everyone has lots of.
I think a big part of the problem is that there are two groups of people.
Theres the group of people _who have the felt sense_ that the world is on the precipice of being upended. That whether you do nothing or whether you do something, everything is going to change in the next decade because Superintelligence WILL fundamentally change all of the economic and governmental calculus everywhere. The status quo _is not saveable_. These people have in their murkwelt that no matter what, life is going to be radically different soon, and probably radically worse, and we should try our best to make it better.
And theres a group of people _who do not yet share this felt sense_. Who fundamentally believe that the status quo is still saveable or even not particularly under threat. Maybe they believe this consciously from close examination or maybe its subconscious, just a heuristic about things not changing fundamentally in ways you can't imagine thats too firm to shake.
If you are in this group, your reaction makes sense to me. You think, or at least you're in the heuristic of, that you are going to be able to mostly keep on doing what you're doing basically the way you are, but for proposals like these. These seem like assaults on life as you know it.
Scott doesnt think the current state of the world is acceptable, and this post isnt about that anyway. Plan A is about averting a looming dystopia that is coming whether we like it or not. Tradeoffs _will_ have to be made, life as you know it is _not_ rescuable, least of all if we do nothing, or let special interests win by default.
The point here, is that Plan A looks _mostly_ like things you can imagine and are already able to put up with. Plan A _resembles_ the status quo, it asks you to give up the least, and it doesnt ask you to give them up until its clear that its needed.
If it were possible, right now, for individuals to spin up super intelligences on their home computer clusters, even you would agree, I would hope, that it would be critical to human survival and happiness that we didnt allow that. Sad day for open source, but if you could 3D print a nuclear weapon on your MakerBot wed all agree you have to ban that too and sad day for the 3D printers. If even in that case, youd say no, open source is holy, nuclear weapons for everyone, then your commitment to open source is whats invincible to reason.
This is being bright-eyed and honest that if it gets to the point where people can spin up super intelligences on their own or in the dark where no one can see, thats a threat to humanity we can't allow and it MIGHT require increasing the price of your computer cluster by 20% or something.
> If it were possible, right now, for individuals to spin up super intelligences on their home computer clusters, even you would agree, I would hope, that it would be critical to human survival and happiness that we didnt allow that
No. As long as multiple different people are able to create superintelligences, one superintelligence can be counteracted by another.
Show your work? Theres lots and lots of ink spilled (on e.g. lesswrong) demonstrating why it doesn't work out that way. Value conflict isnt symmetric. An AI aligned to human values doesn't have the same tools available to it as one aligned against human values, and its much easier to destroy than preserve.
The second law of thermodynamics does show that it's easier to destroy than create, but nevertheless a huge amount of complexity has been created over the course of the universe. Guns are an offensive weapon designed to kill, and since their invention the world has gotten more peaceful. Atomic bombs are supposedly (and mistakenly https://www.navalgazing.net/Nuclear-Weapon-Destructiveness ) thought to threaten all life on earth, but deaths in war have dropped since they were introduced (they could have also let us generate an abundance of clean power if we hadn't idiotically regulated them vs coal). The interests of all the people who want to not be dead are going to outweigh those of a minority trying to kill them.
Let's assume for a moment that DARPA hasn't been sitting on its heels for the past 70 years or so. It's probable that we do actually have weapons that threaten all live on earth (given that we've already managed to do that by Accident).
Yes, the interests of all the people who want to be not dead... outweigh... Wait, the Chinese were making HOW many plagues? Nevermind.
(If you don't want to talk China, you can talk about how many dunderheaded "biological scientists" have tried to create "anti-Jew" bioweapons, etc. We have a lot of stupid in this world, and many "world-ending" devices/weapons/thingummies that are not very well regulated by the "People who want to be not dead.")
> given that we've already managed to do that by Accident
No, that's not "given". I gave a link on how nuclear weapons didn't actually threaten the survival of humanity. If you want to claim the contrary, you should support it.
Also - you don't like the current controlled substance regime, sure, I don't either. But are you advocating for full legalization no regulation buy heroin at your gas station?
No? _What controlled substance regime WOULD you advocate for?_
Please do the same for AI! The "nothing" option is heroin in your gas station and possibly meth discretely added to beverages without being clearly labeled because it increases customer retention etc, but in AI equivalent.
Well, actually I sort of am. Heroin, yes, but not amphetamine or crack. However I'm strongly against allowing the advertising of it. After that proves out, I might be in favor of more law removal.
OTOH, I don't pretend that would be not costly. It's just that I hate the costs the current system has imposed. And people who can buy opiates aren't much trouble to anyone except themselves and their friends. (The trouble comes if they can't get the opiates. So perhaps if you're addicted they should be free.) This would end up with a lot of stupefied people, of course, and many of them might die. (OTOH, I've been informed by an ex-addict that heroin is less addictive than tobacco.)
> There is no accurate definition of 'addiction' that would justify forcing the taxpayer to buy drugs for inebriation hobbyists.
It's simply pragmatics. I ran the numbers on this for a post, opiates are dirt cheap, and in the limits of what we already spend on homeless initiatives and law enforcement, *everything* is dirt cheap.
Noted hardware-and-software hacker (and now self-driving car CEO) George Hotz once came up with the idea of Wirehead City - basically, you take all the “problem homeless” and put them out on BLM land in the middle of the Nevada desert, and give them food and water and all the drugs and alcohol they want.
In return, we get usable downtowns and cities.
I actually ran the numbers, and it is massively, absurdly net positive to do this in any reasonable cost benefit analysis. Any one of the reductions in crime, improvement in expensive downtowns, and lives saved more than pays for the entire program.
We already have meth discretely added to beverages without being clearly labeled. Do not drink anything, anywhere, if you don't trust the people serving it.
> Scott doesnt think the current state of the world is acceptable
What does this even mean? Do you think that Scott going to become a terrorist? (Note that this is a rhetorical question).
I think that this article is perfectly consistent with someone who believes that AI is a huge threat to mankind and is willing to accept permanent oppression to prevent that fate.
The status quo is DEFINITELY not stable. We haven't yet adapted to the technology that is already in public use. The question is "What comes next?", and there is a lot of uncertainty.
My main objection to Plan A is that I think it's unworkable. I don't have a better suggestion. We've got the governments we've got, and there's a short timeline. We can count on the US government to be short-sighted and the individuals making the decisions to be greedy and corrupt. But not necessarily thoroughly corrupt. They do make some good decisions. (But Plan A envisions the US government as idealistic, honest, and far-sighted. I can't think of a time when that was true.)
I don't *think* what you describe is the reason this post "triggers" me as it does, because I am in the camp of believing that the world is currently undergoing a radical change that will flip the proverbial board. Perhaps though, somewhere deep down I believe that while the board will be flipped and things will radically change in ways we struggle to imagine right now, there exist some fundamental truths about societies that are unlikely to topple so easily, like human desire for power. When the board flip happens, I think there will be many people looking to seize power in the new order and that is the thing I fear more than the fact that the board is upside down.
Perhaps Scott believes that even these fundamental "laws of nature" will be turned upside down by AI?
Currently, and if we do "nothing" or let special interests win be default, the people who are poised to take power are the small number of effective oligarchs who are the heads of the AI companies, the companies that are critical to supplying them, and the ahead-of-the-curve and aggressive politicians or connected government executives who will exercise control with them. A small number of people, mercenary by their nature, with no even theoretical dependency among " people" generally.
Fully decentralized citizen autonomy enabled by AI is a utopic vision that is just not in the cards.
_Someone_ is poised to gain incredible power from this radical change. How can we make that a larger number of people, more dependent on the citizenry?
Thats the aim of Plan A - how do we make sure this new power is in the hands of more people who are as dependent on the citizenry as possible, and not what appears to be the default outcome and in fact our current path, which is the 150 or so people who happen to be well connected at the moment Super Intelligence is created?
If you dont like who Plan A hands that power over to, whom ought it hand it to instead, and _how_?
This argument falls much more in the "I want to engage" place in my brain than Scott's latest post.
My short form response is that I believe the only chance we (regular folk) have in any of the futures lies in plurality, and I think open source is the way to achieve that. If I was keen on using regulations to solve problems, the one regulation I would lobby heavily for is that any sufficiently advanced AI must be open weight. I think that would maximize the chance that we end up with a situation where there is plurality of AIs rather than a singleton. Even just two competing super intelligences is better than a singleton, and 100 is better than that, and 8 billion is better than that (unrealistic utopia outcome).
Getting China and the US to mutually agree to such terms would follow the Plan A optimism at government capabilities and will, and be even better than a single country doing it alone.
I think theres some serious common ground and actual operational similarity then between what you want and Plan A.
The critical calculus is "How do you force an AI to be Open Model, and how do you force the company to do that _as soon as_ they make it, and not after they have already quietly used the super intelligence to win the game first"
One of the largest areas of concern for these scenarios is that, once a super intelligence exists, probably the window to control it externally is very narrow, whether thats because its creators who control it will absolutely 100% make sure the first thing they do with it is make themselves win the game, or because its not aligned with any humans and the first thing it does is secure itself and prevent other rival super intelligences.
If we are going to allow a company to construct a super intelligence, a simple promise or even requirement that they make it open model after its constructed won't do - you STILL have to be tracking who is making super intelligences and what they are doing with them, else they will still just do it under your nose and for their own benefit.
The primary means of Plan A is in the "making sure AI creation happens the way we decide we want it to happen" and less in the "making it happen this particular way"
I don't think it's possible to clandestinely "win the game" just by having enough robot monks meditate super hard inside a sufficiently advanced box. Solving real-world problems requires engaging with them directly.
1. There is no evidence that they have a meaningful positive effect, and not for not searching. Much ink has been spilled in various locations (including academic publications) on just how ineffective they are at preventing crime or catching criminals. See section II.A. for some sources on this: https://coincenter.org/tear-down-this-walled-garden/
2. They result in a very significant portion of the human population being unable to utilize a bank because they are unable to fulfill the regulatory requirements. The negative effects are disproportionately applied to poorer populations, and can make it challenging to get out of such situations (can't get a bank account, can't save money easily, can't learn good spending habits, etc.).
3. I have personally had more money permanently frozen because I was unable to or refused to complete invasive KYC/AML checks than I have lost to actual thieves.
4. Fulfilling KYC/AML requirements when you live a non-traditional life can be incredibly burdensome. If you just earn a paycheck and spend it every month it isn't too bad, but once you start getting revenue streams from many non-traditional sources things start to get hairy.
5. The cost of compliance is extremely high. I don't have the numbers in front of me, but banks have collectively spent significantly more money on compliance than has been seized from criminals.
6. Many governments in the world (including the US) are known to exploit AML/KYC stuff to harass or outright steal from people (see civil forfeiture in the US).
7. The data collected in fulfillment of AML/KYC requirements is *frequently* stolen and sold to scammers who leverage it to steal money from people (causing the whole system to *increase* crime, rather than decrease it).
8. As with any surveillance, it is very tempting for those in power to use the surveillance against their opponents or even just people they don't like. This is is true for financial surveillance as well.
9. Philosophically, I am of the belief that governments tend towards corruption, so we should minimize the tools they have available to execute on that corruption. Financial surveillance is a powerful tool for a corrupt government so we should resist creating that tool (especially given it has no meaningfully measurable benefits). Keep in mind that the majority of the planet is under the rule of a government that is not yours, and probably is more corrupt than yours.
Then there's the impact on art! Who gave Visa and Mastercard joint authority to decide what is or isn't obscene, then "protect their reputation" by wielding threats of exile from effective participation in the modern economy against anyone who gives aid and comfort to their enemies? https://yellat.money/
I'm old enough to remember when the US tried to ban exportation of Random Number Generators (or, you know, anything that the NSA couldn't crack.) They made decent, innocent video games illegal (specifically, procedurally generated video games. They are unfun without a decent RNG).
I'm noting this, because regulations that Sound Good are not always, and can have some really, really stupid knock-off effects.
Regulating "AI Chips" and thinking it will do any good is like putting down a dam with a million leaks, and saying "Our Programmers are Too Stupid to get through the Dam." I wouldn't want to take that bet.
"Very Secure against Cyberattack" -- you aren't talking serious here. Serious security against cyberattack is removing the data center from the internet. If you're not going to do that, you're not serious. If you think AI is about as dangerous as a nuclear weapon, in the wrong hands, then you want it unplugged (from the internet).
"The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time" --> Why should it be cryptographic? Is it so that the AI cannot realize it will be unplugged? Cryptography is used to prevent data from being understood.
One very big problem with increased regulation is that it destroys the free market. Specifically, overregulation prevents smaller, more agile companies from entering the market at all. This seems to be what your regulation is designed to do. Perhaps you actually think this is good.
77% of pharmacies with burglaries... don't have video cameras.
Was this link even read? There doesn't seem to be any regulatory "you can't sell drugs" without putting video cameras in (it's also markedly not really about regulating factories or who buys the damn drugs, it's about physical security of pharmacies, which isn't very good, and isn't very regulated. Where I live, I've seen signs that say "We do not sell Opiates here" -- this is a theft prevention measure.). Yes, it's probably better for your business if you do so, but we aren't saying "Thou Must."
I do also object to calling greenbacking "innovation" -- it is not. Ivermectin was innovation.
Is it part of the Global Surveillance State that I must provide my drivers' license and home address whenever I buy sinus pills? Hell, yes! Now that it's "being scanned" and not written in a book, it's even convenient for the government to steal my data. I'm on the side of crypto, of not letting the government know about every condom you buy, or cigarette you smoke.
You write like the government doesn't have access to your medical health records whenever they damn well please. Is it illegal? Does it matter? (No, the government is not going to start -blackmailing you- because you have an STD you didn't tell your wife about. That was the point of HIPAA after all).
"They couldn't enslave 5-10% of the world's computers secretly"
... please. Tell me you'd notice if the computer chip your refrigerator has was wormed? Or had a virus? How about the one in your answering machine? Or your thermostat? Or your thermometer? How about the bajillion in your car? Most people don't even turn task manager on, (and current "tech" runs hot, so "why is my computer running at more than 5% CPU" is best answered with "Bad Code" not "I have a virus!?!").
The point is that the training run would take 5-10% of all computers running presumably at full power. My computer can completely freeze if I overdo it even just with inference on open AI models, let alone training. If you skimmed off only 5% of the total CPU/GPU power you'd need literally every computer in the world.
I'm talking 99% of power on all the chips we pretend don't exist. Not the "full powered" computer chips, the small ones that exist in everything. Or, you know, the cellphone chips when you "have it on" but aren't actively using it.
I think assuming that LLMs are the future of AI is the first mistake.
Well, if at some point someone manages to make AGI that can be trained and run with a handful of laptops purely by power of clever algorithms then all predictions are off anyway. This plan is based on the (reasonable, but not guaranteed) extrapolation of current trends that compute remains important and algorithmic gains are relatively limited.
Those IoT chips you mention are certainly more hackable but also far less powerful. Most don't have dedicated GPU-like linear algebra functionality. I think even if you grabbed every single chip of that kind in the world you wouldn't get a single half decent training run. The smaller the individual unit, the bigger the inefficiencies from moving around information between one and another.
"You wouldn't get a single half decent training run" -- it's all a matter of time. Are we talking a year, instead of a day? that's two orders of magnitude (roughly speaking).
Source for games requiring cryptographically secure RNGs? Sounds like something only statistical analysis would pick up, not a user looking at generated levels.
A game developer friend of mine. No, it wasn't identified by a player (it "didn't look right" if done with a poorer RNG, so I suppose a player might have identified "this looks crappy" from a Bad RNG).
I think I finally have a clearer model of the strongest version of the "it will create a totalitarian dystopia" fear, that explains why the case might be different for AI even though the fear has not turned out at all for eg. nuclear weapons and only to a medium extent for medicine.
[Edit: on self-reflection, this doesn't explain cases like payments surveillance well, as I find that highly undesirable despite it not falling into the below pattern of being actively resisted by many, so definitely treat this as one story among several, not a theory of everything]
The big key insight is: don't think of the thing to avoid as "a totalitarian dystopia", think of it as "a war between the government and the people". That is, a situation where the government is highly motivated to prevent X, and the people are highly motivated to achieve X, and so both strongly pursue their (opposing) goals.
Think of this from the Realist point of view: the worst thing you can have is two very powerful entities that have strong opposing motivations, because that leads to a big war. This is why Hobbes argues for a monopoly of force: if there is a single clear winner, whom no one bothers to resist, then that is in many ways bad, but it's still quite a bit less bad than the big war.
There is an analogue of this for "wars between the government and the people". If the government bans something that people don't _really_ want (eg. murder, nuclear weapons, poisoned food), then they succeed at banning the thing, people switch to doing other non-banned things, and life goes on. But if the government bans something where people have a strong will to resist, then people will sneak around the government, bribe officials, the government will step up spying on everyone and mandating inspections, people step up their countermeasures, the government steps up the spying, sending goons around to check everyone and everything and look into people's underwear, etc.
What we call "totalitarian dystopia" is a side effect of THAT. It's not a phrase we use to directly refer to a government banning a thing. Rather, it's a phrase that we use to describe the outcome of a highly advanced "war of government versus people", that arises from the government banning a thing that is nevertheless really valuable to the people who want it. The totalitarian dystopianness is the collateral damage from the government side of the high-stakes asymmetric war.
Examples include:
* A government invading a country and banning expressions of political support for that country within the occupied territory
Another factor that increases "the people's will to resist" is: do lots of people find the perpetrators' goals sympathetic? For murder, not really, even if there are in fact people who are really really intent on murdering people. If there _were_ lots of murder sympathizers, then you'd have lots of people harboring murderers in their homes to protect from police, and you would actually go up the escalation ladder, and anti-murder laws would thus get a more totalitarian character. But, fortunately, murder sympathizers are few and far between. For the above four (in the last case, at least the less-dangerous half of drugs), the situation is quite different, to varying extents.
This is all your "be nice until you can coordinate meanness" concept, but from the perspective where the government itself also has to follow this rule.
I continue to think regulating chips is on the milder side of this, because there are only a few chip fabs, they are relatively easy targets for regulation, it's a fairly non-invasive regulation so it's easier to get legitimacy for it, if the regulation targets training and not inference then the number of people who would be _directly_ motivated to fight back against any specific step is small (basically just chip fabs and AI companies) and so you get pretty quickly to the "they succeed at banning the thing, people switch to doing other non-banned things, and life goes on" step.
Trying to ban open weights models is quite a bit further toward the risky side, because wanting it runs counter to all kinds of other goals that lots of people have and are sympathetic to - not just people as individuals (who don't want their private lives going into an untrustworthy corporation), and not just enterprises (for whom having control of ongoing ability to access their work tools is basic prudence) but even governments (see: data/compute sovereignty goals). Also, if your goal is revocation capability, you have to ban not just open weights, but also tricks like "consumer single-LLM ASIC chips". Basically, you would be explicitly mandating weaponizable interdependence, which is _exactly_ the thing that lots and lots of people are scared after a decade of it and are fervently seeking a way out from. (The weaponizability in this case is much greater because access could be revoked individual-by-individual, rather than just a global "cut compute by 99% for everyone" button)
Thanks, this was actually a helpful comment for someone like me, on the pro-regulation side.
I think there are potentially some ways around this "government vs the people" situation.
For example, if people are worried they can't build a gamer PC anymore, then to allow this, the GPU could be only willing to run code that has been signed, and which is known not to be AI training code. A distributor like Steam could do the signing.
A different problem would be research GPU clusters that run custom code all the time and which might also want to do (non-frontier) machine learning. What I think has been suggested here in the literature is that these clusters need to keep a log of all the computations that have been run (as some kind of hash) and then when inspectors come to visit they can rerun the code, check that it doesn't do frontier AI training, and check that it recreates the hashes.
And, I actually think running a (safety-tested) open weight model on your own GPU seems actually fine. The signing approach could work here again.
Are there other scenarios where people really want to use high-end GPUs?
So if I'm a hobbyist developer who wants to develop a game, or experiment with developing a game, am I limited to using presigned shadercode? Can I not write my own shaders?
> For example, if people are worried they can't build a gamer PC anymore, then to allow this, the GPU could be only willing to run code that has been signed, and which is known not to be AI training code. A distributor like Steam could do the signing.
This is already pretty bad (for example it kills any attempt to make indie games sold via non-major store channels) and it becomes significantly worse in a world in which better AI means people can make their own software. You're entirely nullifying those gains by... banning them from running any software that isn't properly acknowledged by The System (whatever that is). Actually Google is about to do that to Android (ban all non-Play Store apps, at a system level), ostensibly for "safety" reasons, practically probably to stop apps that do piracy or other stuff they don't like, but it's going to do a ton of collateral damage to all the non-official Android software scene and I know it'll make me install LineageOS on some of my devices.
I don't think you need to be anything like this draconian. GPUs aimed at consumer graphics need an amount of video RAM which is some small (maybe 10x, certainly not 20x) multiple of the amount used for a screen's worth of whatever the highest resolution display output you support is, with enough bandwidth to fill the lot once every 60Hz frame or so (maybe 120Hz if you want to drive a VR headset and not vomit I guess). A consumer is very unlikely to want more than two of these in their gaming machine cooperating, and most only have one. Meanwhile, GPU-like objects intended for frontier AI training need orders of magnitude more, faster memory than this (and also typically include large matrix multiplication units while lacking hardware support for helping with things like rasterisation, Z culling, alpha blending etc), as well as very fast communication between multiple devices. Regulate based on that difference, if you must; it will only grow wider as frontier models scale while human retinas do not.
This isn't right. A high-end graphics card meant for 4k gaming will have 32GB of RAM, which is about 1000x the size of a 4k 32bpp framebuffer.
A game wants as much VRAM as possible to stuff full of geometry and textures of its assets, so that it can display them at short notice without the need to swap them in from main memory.
That 32Gb will be GDDR on the card, not HBM on the chip. Whatever special sauce is on the chip is used for render targets, if the design bothers at all.
tbf what you actually have with the sota designs is chiplets - multiple pieces of silicon incredibly carefully aligned on top of a large piece of silicon that provides interconnects, the whole cursed resulting object then wired to pins and packaged inside the chip housing. So “on-chip” doesn’t necessarily mean literally on the same piece of silicon, though for the designs I worked with it did.
Interestingly, in the “slowdown” world, I think ASICs are likely to be more prevalent; we are close to the point where it is economical to tape out an ASIC for each frontier model (given the lifetime operating cost of that model), and if model development slows then ASICs become more favorable as the depreciation / ROI window widens.
Maybe this gives a coherent fix for the plan - once ASICs are common, GPUs will be worthless for training/inference, and gamers will be unbothered by these regulations. (Look at BTC / ETH for an example of how this played out in the past.)
The problem with this line of reasoning is that I think ASICs would be something that people want more than GPUs; in other words the actual objection is not “gamers want unregulated GPUs” but rather “everyone will want their own local AI compute”. Unless you can align the world state such that most people are happy with hosted AI, there will remain the war between government and people that Vitalik highlights.
The current interest is in ASICs that implement inference for a single model, or perhaps for a bunch of fairly similar models. I doubt that these ASICs would be useful for training, in which case there would be no need to regulate them.
It’s a good point that we need to treat training and inference separately. You’re right that training admits less specialization.
I don’t think there’s a clear case that ASICs are unlikely for training in this hypothetical.
TPUs are already used extensively for training and they are more specialized than GPUs. So the existence proof already obtains.
In the slowdown world, if architecture improvements also slow down then you can bake more assumptions into your training chips and my point above about ROI holds. (I would probably revisit the 10-100x number for training specifically, I accept the pushback there). If instead we see more architecture/algorithmic improvement as a substitute for compute improvements, perhaps ASICs would not be as favorable - but even in the current architecture revamp timelines, if the absolute scale of training runs increases an OOM or so (and especially if LLMs automate more of chip design) then it may still makes sense to do a per-architecture tape-out.
So the strongest prediction I’d TLDR is: in a year or two it seems highly likely that TPU/Trainium/etc will fully diverge from GPU for training. Honestly think you could define a legally enforceable split with the current gaming vs H100 architectures.
>and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.
Under which agreement is the US allowed to physically inspect Chinese dairy farms, or vice versa? If you had gone with the much more appropriate analogy of nuclear control treaties, you would see how fragile these are if and when they become inconvenient for either side.
You can build a datacenter with high-end consumer GPUs. It would be less efficient than "professional" GPUs, but it's still a viable approach, especially since you can solder on more memory onto existing cards: those HXXX cards aren't much more powerful than 5090, their main advantage is more on-board memory (but note that the difference is less than order of magnitude currently).
USA already did ban export of 4090/5090 GPUs to China. I'm pretty sure your proposal would either spell the end for high-end consumer hardware, or limit its availability geographically (which would require every PC with every OS to "call home").
Now, GPUs might actually be a dead end for AI training. I expect specialized hardware to arise, like ASICs did for crypto mining. Maybe you can leave consumer GPUs alone, and regulate this new hardware (when it appears) which would be more or less useless to regular consumers. But this theoretical (for now) hardware would, by itself, give a significant boost to the curve, so maybe we will just end up with AGI sooner anyway.
Hm, yes, that too, and it's harder to overcome. But I'm not sure this is a hard obstacle in a world where you CAN'T get a pro card, for a very motivated party - finding ways around it is still easier than e.g. building your own chip factory. The risk is still there.
The model size, cost, and power use of training each generation of new models has grown rapidly, and the cost and power usage would be even greater if you had to make an equivalent model from consumer hardware. It's going to be hard to hide this for future LLMs. You'd have to buy immense quantities of consumer hardware, involve thousands of people to build it, power it, etc.
I note simply that I am extremely skeptical of arguments that endeavor to persuade me to accept X policy because it and its consequences pattern-match to existing policies Y, Z, A, B, and C, all of which are things I would have rejected immediately if anyone had bothered to ask me beforehand.
You don't get to a grim meathook dystopia in one go. You get to it by salami-tactics of things which, in themselves, can be portrayed as only trivially bad.
Historically your best arguments strike me as not just speaking to those near the perimeter of your camp, but all the way across to folks who are in intellectually distant camps. Today, when you speak to the "cost" of a growing regulatory state, you don't seem to sufficiently appreciate the viewpoint of the more traditionally regulatory averse. I don't think you're speaking to that crowd (include me) here. Maybe you don't intend to (and can get sufficient traction without us), but if you want us, you need to speak a bit more broadly.
Systematically, regulations (laws, taxes, etc.) are imposed with a 51/49 like majority, and do so by arguing that "at this time" they are clearly necessary. However, they are written "at that time" for the problems "of that time." They are written in broad brush terms (in the U.S.) by a congress, who then washes their hands of actually having to work out the details, and a new regulatory body pops up in the executive, where sometimes even new non-article-3 judicial powers are invested in the executive, and the body effectively rewrites their own rules with sufficient detail to enforce, and acts as their own judge/jury in many cases.
Then, time passes, and new edge cases drift through. What actually constitutes an AI chip? How do we handle pre-existing chips? Oops, the algorithms advanced faster than we expected and now don't need hardware as sophisticated as you previously assumed, what happens to all the chips we didn't think were dangerous but now are? Oops, the paradigm shifted, and it's really parallelism that advances the AI. Oops, quantum computing.
The executive agency designed to handle this becomes a hot potato. Congress doesn't want to touch it any more. The original 51/49 consortium dissolved, and now very vested interested have emerged on both sides and to move the rules either direction after they are established generates profound pain and contributions to your next political opponent.
Every time the regulatory-averse crowd sees a new huge paradigm rolling through, they see the stories of the failures of the past and project them onto the new paradigm.
And that's just the domestic story. The international story is its own concern.
The above assumed these were just simple domestic laws. But if so, then we can retreat from them because we still (as a nation) have autonomy. So instead are they congressionally ratified treaties? In which case, that executive bureaucracy isn't even domiciled here in the U.S., it's in Brussels, or Beijing, or ? Are you relying entirely on the relative immovability of existing plants and the time-delay in making new ones to prevent non-signatory countries from evading? If there's a date at which the entire raft of law goes into effect, is it practical for a large number of the data centers currently existing to close down, pack up, and reopen in a non-signatory state the day before effect?
And coming back to the timeline/projection/model into which the new regulatory system provides the "fix" - that's an interesting forecast. It's notoriously hard to forecast things of this consequence. Show me how the new world works if your model was either too fast in its predictions by 5x or 5x too slow.
The mini-regulatory team has an advantage, historically, of saying the 5-year-planners were wrong, and an infinitely adaptive market-like free competition of dynamic players always ended up with outcomes that beat the static nation-based systems. So they didn't need to have a perfect vision, they just trusted that the system would follow a local minimum through to a solution.
If you want that team to hear you, it's a much much harder pitch than the one you're making.
I'm not saying AI doesn't need some sort of regulatory state you describe. I frankly don't know. It strikes me as within the realm of possible we're looking at the event horizon of a singularity and all the rules need to be different this time because everything we've learned in the past doesn't apply at a singularity. Could very well be.
I'm just saying your story isn't designed for folks whose starting point is in a camp that isn't near yours. If you want us, the pitch needs to be very more respective of our challenges with all the regulatory disasters that we think have preceded this.
It is my sincerely-held belief that we'll all die (near instantaneously, not painfully most likely) if frontier labs are allowed to continue their AI development for another 10 years or so (could also be 5 years, could also be 30 years). What should I do now according to your political philosophy?
"What should I do now according to your political philosophy?"
The straightforward was is probably to work with as many like minded people as you can find to kill the people most critical to the frontier labs AI. You aren't going to talk the people working on this into stopping.
I think that there are probably too many of them and that they are probably too dispersed for you to get them all, but I bet you can find a bunch of the US based ones in San Francisco. Anthropic, OpenAI and xAI all seem to have a bunch of folks in SF.
I don't know how to get to the Chinese folks.
Mistral is in Paris.
But the basic approach is so simple that even killing everyone involved today just pushing things out.
In some respects, a much more effective way is to go after the fabs building the chips. There aren't very many of them (in the grand scheme of things) and they are a lot harder to hide than people. In the west, destroying TSMC's leading edge (and probably a few nodes back, too...) fabs as well as Intel's leading fabs and Samsung's leading logic fabs should do the trick. China can't produce these chips in volume, yet, so you have some time there.
I haven't put any serious thought into this, but I expect that ramming a Cessna full of dynamite into a fab would take it out of production for quite a while.
That's a terrible idea because the US intelligence agencies would stop any sort of civilian conspiracy to do that. But the US intelligence agencies could do a lot of that themselves, if they came to realize that AI is going to undermine their power and entire model of operations.
Aside from which, the whole point of banning a thing is to make the barriers too high for most people to try to do it. Most of those people you're talking about, though they might scream and howl over it, would move on to something else. Google has plenty of ways to make money on deterministic algorithmic processing and Meta can sell plenty of junk to middle aged moms, I'm sure they'll all find somewhere to land. The CCP has every incentive in the world to do the same over there to maintain control, and a history of purging their business leaders, and the entire apparatus exists already. And Europe already applies the "precautionary principle" to mundane consumer products so I'm sure they don't need much persuading here.
Probably the scenario most likely to ensure human survival is one where there's an early scare sufficient to get the majority of humans to align around a permanent norm against the construction of AI systems. The nature of that scare might vary, and I'd hope it needn't be a massive disaster, but eventually something like that probably will happen. Then you have to find a way to set things back long enough that you can entrench the norm, make the construction of AI akin to human sacrifice such that only lone crazies would ever even attempt it and those crazies can be dispatched by people with state-level capacity. Keeping that up for the next 5 billion years will be more of a challenge, but that doesn't have to be solved immediately.
The same us intelligence agencies that thought the Afghani government wouldn't fall for over 100 days after our withdrawal? (given that this was done using whatsapp, which is american software and thus presumably bugged-by-government, were the government competent at all) The level of Competency you'd need, sir, does not exist in this government.
Just like they stopped the capacitor plague? I mean, seriously. Someone already broke computers (and I'm just going to allege that it was industrial espionage, because why else would an entire factory make bad caps? Did they somehow not test their one product?)
I think Scott is saying to fix that we need to: Convince 51% of congress (and the president) of that risk, and the CCP. Seems like a fine strategy. I'm just saying it doesn't sound convincing yet to this 0.00...01% of the electorate, and I think there will be a lot of folks out there like me. The stories I've heard in the past about dire needs for regulation have left me with scars. If you need me (us) to get to 51%, to resonate with that crowd, you'll have to recognize that we see a long trail of bad regulations prior to this point, and this one has a lot of markings to make it look worse. If you don't need us, continue to pitch to the other folks you need to get to 51%.
Every political issue du jour comes with a tincture of apocalypse. And every threat of apocalypse justifies an expansion of the power of the state. And every expansion of the state comes with great assurances "temporary, limited, targeted" and (this side of the crowd) gets disappointed with the results virtually every time. If you want to convince that crowd, understand that set of preconceptions and pitch to that crowd specifically.
I am convinced there are fewer people like me than I'd like to think! And, regardless of what I'd like, I don't know how many of me there are. And we may not be well represented in Scott's reach.
"I’m reminded of a story I heard - I can’t find it, maybe one of you can - from a DC insider who said it was enraging to work with Silicon Valley, because he would bring up what he thought was the obvious regulatory framework, the tech people would launch into jeremiads on how it could only be enforced by world dictatorship, and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years. "
I hate to break it to you but you're doing the thing.
>But information technology confounds these heuristics. It kicks the crap out of them in one important way, and this is it: one important test of whether or not a regulation is fit for purpose is first whether, of course, it will work, but second of all whether or not in the course of doing its work it will have lots of effects on everything else. If I wanted Congress or Parliament to write, or the EU to regulate, a wheel, it’s unlikely I’d succeed. If I turned up and said, “Well, everyone knows wheels are good and right. But have you noticed that every single bank robber has four wheels on his car when he drives away from the bank robbery? Can’t we do something about this?” The answer would of course be no. Because we don’t know how to make a wheel that is still generally useful for legitimate wheel applications but useless to bad guys. And we can all see that the general benefits of wheels are so profound that we’d be foolish to risk them in a foolish errand to stop bank robberies by changing wheels. Even if there were an epidemic of bank robberies, even if society were on the verge of collapse thanks to bank robberies, no one would think that wheels were the right place to start solving our problems. .
>But, if I were to show up in that same body to say that I had absolute proof that hands-free phones were making cars dangerous, and I said, “I would like you to pass a law that says it’s illegal to put a hands-free phone in a car,” the regulator might say, “Yeah, I take your point. We’ll do that.” And we might disagree about whether or not this is a good idea or whether or not my evidence made sense, but very few of us would say, “Well, once you take the hands-free phones out of the car, they stop being cars.” We understand that we can keep cars cars even if we remove features from them. Cars are special-purpose, at least in comparison to wheels, and all that the addition of a hands-free phone does is add one more feature to an already specialized technology.
>In fact, there’s that heuristic that we can apply here. Special-purpose technologies are complex. And you can remove features from them without doing fundamental disfiguring violence to their underlying utility. This rule of thumb serves regulators well by and large, but it is rendered null and void by the general-purpose computer and the general-purpose network, the PC and the Internet. Because if you think of computer software as a feature, that is a computer with spreadsheets running on it has a spreadsheet feature, and one that’s running World of Warcraft has an MMORPG feature, then this heuristic leads you to think that you could reasonably say, “Make me a computer that doesn’t run spreadsheets,” and that it would be no more of an attack on computing than, “Make me a car without a hands-free phone” is an attack on cars.
>And if you think of protocols and sites as features of the network, then saying, “Fix the Internet so that it doesn’t run BitTorrent,” or, “Fix the Internet so that thepiratebay.org no longer resolves,” it sounds a lot like “Change the sound of busy signals,” or, “Take that pizzeria on the corner off the phone network,” and not like an attack on the fundamental principles of internetworking.
>Not realizing that this rule of thumb that works for cars and houses and every other substantial area of technological regulation fails for the Internet does not make you evil and it does not make you an ignoramus. It just makes you part of that vast majority of the world for whom ideas like “Turing complete” and “end-to-end” are meaningless. So our regulators go off and they blithely pass these laws, and they become part of the reality of our technological world.
>There are suddenly numbers that we aren’t allowed to write down on the Internet, programs we’re not allowed to publish. And all it takes to make legitimate material disappear from the Internet is to say, “That? That infringes copyright.” It fails to attain the actual goal of the regulation. It doesn’t stop people from violating copyright, but it bears a kind of superficial resemblance to copyright enforcement. It satisfies the security syllogism. Something must be done; I am doing something; something has been done. And thus, any failures that arise can be blamed on the idea that the regulation doesn’t go far enough rather than the idea that it was flawed from the outset.
It passes "true" trivially, "necessary" is pretty solid and for "kind" it's only the mildest of mild snark in response to snark about tech people who, it can't be stressed enough, are much much wiser on this very specific topic overlapping with their expertise than scott and his DC friend.
Sorry, but this just seems like meaningless bloviation. We declare you're not allowed to write down numbers all the time - for example, every child porn video corresponds to a number. This in fact reduces the amount of child porn in the world and doesn't cause the entire Internet to collapse. I am begging you to actually think about real things on the object level instead of making grand 10,000-foot-high pronouncements about the nature of government and technology that collapse on slight inspection.
Yes, which is proof that "banning a number" is the wrong way to think about things. Some numbers are bad!
(also, I don't think this accurately reflects child porn regulation, which isn't just about catching the people who made the video, but about catching people who distribute and watch it. And my understanding is that AI-generated child porn is just as illegal as the real sort.)
Animated porn, drawn porn of children isn't illegal*. "Ai-generated" child porn that happens to look photorealistic? Probably getting banned for "being too close to the real thing" (and not for any "this is a good reason by itself." -- in a world where 8 year olds did not audition for roles as sex slaves, and people did not assault 4 year olds, you wouldn't ban AI-generated child pornography -- of course, there would probably be limited demand for such, in that case).
*Some practitioners even use it as part of psychotherapy...
As an aside, this differs between countries. Germany bans even simulated CP, including hand-drawn and animated cartoons. Afaik it even bans textual descriptions, i.e. graphical descriptions of minors having sex.
Talking about banning numbers through legislation is not being charitable to Cory Doctorow's argument. The steelman is about building computers that cannot physically contain the banned numbers.
Your original argument rests on the idea of building chips that are incapable of running AI. That is every bit as unreasonable as building chips that can't transmit, store, or play back child porn (or any other illegal content). It runs entirely contrary to the concept of general purpose computation.
Cory Doctorow spotted the knives coming out for general purpose computers a long time ago, and he's been writing about it ever since. It's the entire basis for the side which is contra your argument.
Exactly this, in combination with reviving the clipper-chip backdoor.
I think Scott has in his mind a mental image of "AI chips" as a special category because it feels like graphics cards with a lot of VRAM are special.
But that's not how chip manufacture works. It's the same graphics cards with variable amounts of VRAM to segment the market. Little different to how apple segments the market for their phones by selling high and low ram versions of their product. Anyone with modest skill and the right solder masks can swap in more RAM to an iphone and there's youtube tutorials on how to do so.
So intrinsically this requires any such controls to be built into the hardware of the graphics cards, not just some graphics cards but any graphics cards that people could hypothetically attach extra VRAM to. So yes, this absolutely 100% means regular people's graphics cards.
never mind a few years from now as hardware advances and the much more routine demand for massively parallel processing pushes normal CPU design further in that direction.
Scott is making exactly the mistake doctrow outlines. Treating AI-capable like a "feature" that can be cut out of a general purpose computer.
To the tech people, the hero's of their community spent decades of hard-fought battles to keep government backdoors and government spy software out of people's computers.
He's casting all of that into the flames for a solution that probably wouldn't work well anyway. But the backdoors and monitoring will never go away once they're in place.
I think this gets the question backwards, instead of asking why would governments regulating AI be bad? We should ask, why is government regulation of AI justified? This presumes that coercion is generally wrong and requires a satisfactory threshold of justification to be met before supporting it, especially on a societal level.
I’m disappointed to see that the most common objection to AI government regulation is that it will result in a dystopia. I believe that governments are currently the most dangerous organizations and government AI regulation only increases that danger by the same amount that AI is expected to be dangerous in the first place.
However, there’s at least two reasons to oppose governments regulating AI:
One, it’s morally wrong to coerce people without enough justification. The pro government regulators would need to make the case that coercion from and only from the government (while also exempt the government) today actually reduces harms later to a large enough degree to make the coercion worthwhile. I don’t see how that’s clear at all, but in order to enact coercion, the justification should be quite obvious. That is how we ordinarily view coercion. If Bob were to punch Alice we would think Bob was confused if he said, “Alice might have gone on to punch a few other people so I had to deter her from doing so”. There’s a substantial degree of justification required to punch someone that this scenario simply wouldn’t suffice. This isn’t quite analogous to the AI situation, we probably need to account for more parameters. But it’s at least somewhat analogous to people advocating for coercion with a low bar of justification. The supposed risk of AI is much larger, but so is the suggested amount of coercion to be imposed.
Two, government regulation of AI will not achieve the desired outcomes that its proponents believe it will. The common reason I’ve seen is that governments regulating compute power lowers the odds of a catastrophic risk agency’s agents acting maliciously or negligently in a way that causes harm to a massive number of people. However, there’s more reason to believe that the government is an existing catastrophic agency whose agents already act maliciously and negligently more so than any other organization. In all of the actual legislation (such as the RAISE Act in NY) there are explicit exemptions for government AI systems. If there were to be a larger number of agencies that can possibly keep a catastrophic risk agency’s agents in check, then that would be an improvement. This would only become less likely if we impede the non-catastrophic risk agencies ability to grow, while also exempting the greatest at-risk agency.
I wonder what your thoughts are on this. In general I favor a polycentric approach to regulating, and I believe the same approach will help with AI. This way, if someone regulation is good, more regulators can adopt it. And if some regulation is bad and expected to be catastrophic, it will have a more narrow impact on society, rather than a single regulator that imposes regulations on all of society. If that regulation is catastrophic then it will affect all of society negatively.
> "One, it’s morally wrong to coerce people without enough justification."
No country in the world is a libertarian minarchy. We regulate all sorts of products for all sorts of reasons. It's true that you shouldn't do this with no justification. Our justification is preventing global dictatorship and the eventual extinction of humankind. I think this is pretty good as far as justifications go, way better than "these eggs might have salmonella" or "this building might block someone's view". If you don't agree with the justification, say so, don't retreat to basic foundations of political philosophy!
> "In all of the actual legislation (such as the RAISE Act in NY) there are explicit exemptions for government AI systems."
I don't think this is true. It might not even make sense - the government isn't training its own frontier models, it's using Claude like everyone else. In any case, there's no exemption for the government in Plan A.
> "In general I favor a polycentric approach to regulating, and I believe the same approach will help with AI. This way, if someone regulation is good, more regulators can adopt it. And if some regulation is bad and expected to be catastrophic, it will have a more narrow impact on society, rather than a single regulator that imposes regulations on all of society."
This doesn't make sense for global threats. If there's a destroy-the-world button, you can't have every jurisdiction decide on its own whether or not to permit pressing it, because if even one jurisdiction decides yes, then the whole world is destroyed.
The government wouldn't tell you if it was training its own frontier models. And if it was outsourcing to Palantir? Well, it could even mark their work as classified, and not let them discuss it outside of "we have a big government contract."
> Our justification is preventing global dictatorship and the eventual extinction of humankind. I think this is pretty good as far as justifications go, way better than "these eggs might have salmonella"
More grandiose is not the same as better. Suppose a man kills his neighbor with an axe, and the judge asks him why he did it. "I caught him trying to poison my family's food," backed with appropriate evidence of how such an axe-based intervention was the best available option for preventing known greater harm, might be received far more positively than something like "I had to stop him from summoning a demon who would probably have killed everyone in the world," in the context of ongoing dispute among relevant experts about whether such a demon is even theoretically possible.
Incidentally, I think the sort of Alignment you want, and the sort of instant-win superintelligence you're worrying about, are both rooted in similar logical errors. Absolutes are easier to think about, so they sneak in when wrestling with any big problem, and then if the corner-case solution created by those converging absolutes is exciting enough to demand action all on its own... well, obviously the only solution to a conceptual-absolute villain is a conceptual-absolute heroic quest. Were the many simplifying assumptions underpinning the upper and lower bounds of that worst-case scenario actually valid, or do the tails come apart first? It's too good to check.
> If there's a destroy-the-world button, you can't have every jurisdiction decide on its own whether or not to permit pressing it, because if even one jurisdiction decides yes, then the whole world is destroyed.
That's ultimately how nuclear weapons are regulated, though. If Putin decides he really, really wants to turn some some city into a glowing crater, assuming he can scrounge up a working warhead and delivery system, he can just do that.
Same for anyone else in charge of a sovereign state. The only thing the various treaties and escalation ladders do, is try to avoid creating situations where that option would be advantageous for the one choosing it.
Before nuclear power was militarily relevant, back in World War One, it became possible to cause damage of previously unimaginable scale and thoroughness, with just conventional chemistry, metallurgy, and industrial mass production. It would probably have been possible to kill everyone in the world that way, if the group trying to do so were somehow the only one with aircraft and artillery and so on.
Fortunately (?) there was a multipolar environment rather than a singleton, along with considerable investment in cheap countermeasures such as trenches and gas masks. Various unaligned minds such as Luigi Cadorna butted up against each other and got bogged down in futility, until enough of the saner participants noticed that all this devastation wasn't actually gaining them anything.
Wow, first off thank you for the reply, Scott. What an honor! I will do my best to explain my reasoning. This will be a two part reply and might not be as clear as your writing so please feel free to skip to the TLDR in part 2/2 and I will try my best to summarize it neatly.
Part 1/2:
> “No country in the world is a libertarian minarchy.”
I don’t think universal practice settles anything; every widely-shared institution looks justified from inside its era, which is exactly why the burden question matters rather than being a retreat from my view. No country in the world was free of slavery for a major part of history.
> “Our justification is preventing global dictatorship and the eventual extinction of humankind.”
I am not sure that AI firms appear to be dangerous enough to warrant government coercion against them. I think there’s a degree of concreteness required for a justification to warrant coercion. If AI were to cause human extinction, it would likely be from humans using it for harm on a massive scale. Of the humans that would likely commit these sorts of harms, they would likely be government agents. Look at every other technology used to harm humans on a massive scale; guns, drones, bombs, and biological warfare have all been typically and profusely used by governments. I think that is a bug of governments, not AI.
Would you support ordinary actors arresting government agents, holding court proceedings, and convicting them and imprisoning them all with non-government institutions? So far the track record of what organization is most likely to be a global dictator and make humankind extinct, appears to be governments. Nuclear weapons and 20th century democide are the clearest examples of this.
It appears that if the justification lies with any organization it lies with non-governments needing to hold governments accountable, not the other way around.
> “If you don't agree with the justification, say so, don't retreat to basic foundations of political philosophy!”
I claimed that it’s wrong to coerce people without enough justification. I never said they aren’t providing a justification, I just believe it doesn’t clear the bar. That’s my bad, I could have made my position more obvious.
However, I do think the general question of why only the government should be justified in imposing regulations is relevant. Why wouldn’t we want the government to be subject to the same coercion in that case? For example, if we accept that people ordinarily shouldn’t steal from others and that jailing them for doing so seems appropriate then why wouldn’t we want government agents to be arrested for imposing taxes on everyone? That accounts for more theft than all private actors combined by at least two orders of magnitude. It also funds horrible things like wars and factory farming. That’s an existing live harm to almost everyone in society at an incomprehensible scale, and it’s widely neglected. If coercion is justified to prevent large scale harms, then the largest ongoing coercions deserve a similar inspection to see if they shouldn’t be addressed at a higher priority.
> “I don't think this is true. It might not even make sense - the government isn't training its own frontier models, it's using Claude like everyone else. In any case, there's no exemption for the government in Plan A.”
I am overreacting by saying ‘all’, shame on me, but the RAISE Act is to my knowledge the most popular legislation to date favored by gov AI reg advocates and it makes government exemptions explicit. Sections 1420(9), 1420(3)(b), 1426 list “persons” as basically any nongovernmental organization, exclude lawful activity of the federal government, and exempt the New York tax funded Empire AI consortium respectively.
I believe the government is likely trying to train its own frontier models at least, but I do think it’s clear they lack the comparative compute power. I still think governments are the most likely organizations to commit mass harm using AI, though again, I think that is a bug of governments, not AI.
In the case with Plan A, it seems widely presumed even in this case that regulations are only permitted to be performed by governments. Any attempt made by non-government agencies to regulate other non-government agencies or government agencies, will be met with coercion from governments and that is largely acceptable. I don’t think we should recommend strategies that work within this presumption because I think it is faulty to begin with.
> “This doesn't make sense for global threats. If there's a destroy-the-world button, you can't have every jurisdiction decide on its own whether or not to permit pressing it, because if even one jurisdiction decides yes, then the whole world is destroyed.”
Regarding whether every jurisdiction decides on its own, what is the proposed solution for global enforcement of any regulation? I suppose the answer is for each country to enter into some international treaty or agreement, but I’m not sure that would actually resolve the problem. However, even if it did, why couldn’t that same approach work within countries to have multiple regulators?
I think insofar as the speculative threat is misalignment specifically, I don’t see this as enough justification to deserve government coercion. What can be shown throughout AI development that would cause the doom hypothesis to be less likely to happen? Is it only if AI development doesn’t happen? Every result gets read as confirming doomerism. Either misbehavior confirms danger or good behavior means insufficient capability or successful deception. Is there any evidence that would count as AI not being dangerous?
Is there at least some fork in the road where if the development of AI looks more like X then we must have governments regulate AI, but if instead AI development looks more like Y, then we must avoid coercing AI firms? If there isn’t even a Y then I believe the burden to avoid coercion is too high and unreasonable. I think we are closer to the Y track; AI firms invest massive amounts of money into safety concerns and they genuinely are already creating a polycentric approach to safety enforcement without specifically relying on government coercion. For the X track to be more apparent I think there should be some measurable harm that occurs over a high enough degree to warrant coercion. I would accept at least the original RAISE Act’s own standards: 100+ deaths or $1B in damages as a threshold. But they have to actually happen, the speculative risk of them isn’t enough to preemptively impose coercion at that scale.
Mind you, governments cause far greater damage and almost nobody suggests that we should be coercing them, so I still think the double standard is painfully obvious and makes it difficult for me to press too hard to coerce AI firms even in that case. I still concede coercion would be worthwhile at that point.
I can’t tell how the extrapolated high probability of x-risk is supposed to be demonstrated. I hold a strong opposition to coercion in general. In order for AI dangers to count as an exception, shouldn’t there be some observable threshold we can point to that makes coercion more justified?
The catastrophe as a concern without any tells beforehand seems to demand too much and historically appears to be discounted by the doomsayers being wrong. There have been many world ending risks that were used to warrant coercion and those risks do not appear to be legitimately damning for humanity.
Population growth alarmism in China is probably the best example of doomer justified coercion causing massive harms. Thailand and South Korea apparently had the same outcomes without the coercive effort. Others are the eugenics sterilization programs around the world, with a scientific academic consensus of that time and SCOTUS rulings backing it, nuclear regulation leading to increased CO2 production (if you accept the added x-risk caused by that), and immigrants supposedly bringing about sharia law and the great replacement sort of rhetoric. These range across a whole spectrum of credible expertise down to bigoted panic. All of these were coercions justified by extrapolated risk, imposed without observed harms at any comparable threshold and they produced real harm with little or no benefit.
Humans have a dramatization bias and there aren’t harms shown to occur over a high enough threshold which is what my view requires to support coercive acts. The probability of x-risk extrapolated from AI testing and theory should be discounted heavily in my view.
A single regulator appears to be the most dangerous thing existing today, and I don’t see how introducing a supposedly dangerous technology makes that less true rather than more. As with most monopolies, a monopoly regulator provides worse quality at higher prices. The majorly bad actor is most likely to be the least accountable one, which to date has meant government agents.
I find it plausible that having competitive regulators increases the likelihood of producing a safer regulation that can be adopted by the others. And I have doubts that reducing the competition would still have a similar chance at producing a beneficial regulation. Especially when that regulator has a track record of producing harmful ones.
I’m sure I come across as biased and ideologically motivated and I probably am those things, but I also am just trying to be honest about my beliefs simply being what they are.
I just don’t think that the speculative risk associated with AI justifies government coercion against AI firms. I don’t think it’s worth prioritizing advocacy for such coercion above addressing existing harms committed by the government such as stealing from millions upon millions of innocent people to subsidize mass animal slaughter. Or coercively preventing other organizations from competing to provide regulations in society.
TLDR; I hold a high bar for coercion to be justified. I think that’s right to do. We agree coercion needs justification, but disagree on whether AI x-risk clears it. Where I think we differ is that I don’t believe the extrapolated risk drawn out from lower stakes lab tests and AI arguments count as enough evidence to cross the threshold for justifying coercion against the AI firms.
I think current government coercion occurring at a large scale requires much more immediate attention. I also believe the track record of governments makes them the biggest threat to humanity that should be addressed right now. Maybe that is from the dramatization bias I have from my personality type. So I apply the same discount to my doom story that I think AI doomers should apply to theirs. I think mine survives it because state harms are observed and well established, not extrapolations.
I do think that polycentric approaches to regulation will produce better results in the long run. AI firms are already demonstrating a polycentric approach to AI safety that appears to be little influenced by government coercion.
There’s probably tons more to say on this topic. I think there could be a threshold that AI firms cross which warrants coercion such as causing 100+ deaths or $1B in damages, but I don’t think that large scale coercion makes sense before that occurs as a marker. I want to know what evidence would count as us being on the safe track that wouldn’t merit AI firms being coerced. Is there already some, and it’s just not enough to bring the risk below your threshold? I see it in the other direction: there is at least some evidence AI firms are on the dangerous track, but not enough to warrant coercion. At least not yet.
Head to head competition is the primary demand driver of cutting edge intelligence. Think, military & finance. Open source models are an asymmetric military multiplier, they allow usa adversaries to leverage asymmetric tactics and strategies more effectively. Plan A is doomed because it doesnt understand the nature of why our world is a dystopia, it thinks things are just the way they are because people arent smart enough. The military planners in both countries understand this much better than you do, which is why ai is allowed to expand. If you want to stop it, you have to adjust the geopolitical landscape - something that requires raw power with intelligence as a competitive input, not a deciding factor.
Our world isn't a dystopia. Poverty keeps dropping, we're curing new diseases, and even the obesity problem created by our unprecedented abundance of food is being fixed by new drugs. There are real serious problems, like below-replacement fertility afflicting basically every country except Israel and the most impoverished, but subcultures are able to solve that by just adhering to religions that separate them from the majority.
If you use the metrics that our system evaluates itself by, our world is generally a success. However if you read the article, in context the dystopia I am referring to are the loss of basic liberal values such as freedom of movement, commerce, speech, religion, and privacy.
Cybernetics refers to loops of monitoring and control thus resulting in dynamic system states - stable ones in negative cybernetics and unstable ones in positive cybernetics (ai explosion is positive cybernetics, financial repression is negative cybernetics). Cash doesnt escape these control structures as its monitored at several levels: first vat, second bank, third polling. Just the fact that you can give an accurate number (51%) puts us in the realm of monitoring, which is half of the control cycle.
Theres an interesting argument (which you havent made) that there is an entire underground, unmonitored and thus uncontrolled world. Here that 51% cash transfer is the iceberg tip hiding a 99% untracked economy, where almost all activity is unmonitored. However from context (slate star codex) the assumed "base truth" is that government statistics are "mostly accurate" so the dystopia is assumed.
Last I checked obesity was still increasing year-to-year virtually everywhere.
> "...but subcultures are able to solve that by just adhering to religions that separate them from the majority"
Yeah, unfortunately said religious subcultures occasionally have the imperative to displace/conquer/nuke their neighbours (the conflict between Israel/Palestine being an interesting preview of where that leads.)
I was thinking of the Amish as one such subculture, and they don't have that imperative at all. The others I was thinking of are the ultra-Orthodox, who could be blamed for their political influence in Israel but are also blamed by their fellow Israelis for not serving in the military (though the more mainstream national Orthodox do).
If the Haredi/ultra-orthodox faction were forced to work and serve in the military it might reduce their fertility to some degree but I don't think it changes the long-term dynamics here. The human subcultures most interested in raising internal TFR are the least interested in global geopolitical cooperation.
You know perfectly well you're only harping on about the Amish because they're a relatively weird and non-generalisable exception, even among religious ultraconservatives. List all the other insular high-TFR religious communities in the world and tell me what percentage are committed to pacifism in principle.
Yes but it misunderstands when these types of agreements work. Its not because both parties "see the light" its because the geopolitical strategy allows it. For instance the intermediate nuclear missile ban treaty - this was something that provided symmetric value to both sides. Open source Ai does not, it disproportionately helps usg adversaries leverage complex, asymmetric advantages in the way that it doesnt for usg. Thus China would sign the treaty while secretly violating it.
The whole reason we're discussing regulation like registering where every chip has to go is so that it's impossible to substantially violate the treaty. Please read the original post - the idea is that *if* we implement all these regulations and mutual inspections, China shouldn't be able to hide more than a trivial fraction of its chips - too few to do anything useful with.
The post is a bit unimaginative in how national actors could circumvent the proposed changes - but if they did work, china simply wouldn't agree. If china does agree, thats evidence they found a way to circumvent them. The asymmetric advantages are simply too large.
China will build copies of its public chip factories underground that create chips that are not registered and do not have the kill switch/safety features and transport them to underground datacenters and research centers.
China will probably have a much easier time discreetly building these factories and data centers and diverting power and hiding the heat emmissions than the US, though I'm sure both sides will do it.
China will do this irrespective of any regulations or international treaties that it may or may not sign. Other countries may suspect they are doing this or even find some evidence of it, but nobody will be willing or capable of stopping them. China will deny and push ahead anyway. The entire world couldn't stop North Korea from obtaining Nuclear Weapons. The idea that the world could stop China from building a chip factory or a warehouse full of computers is just completely unserious.
There's just no way it will slow the AI race even a fraction of a percent. Probably the regulations/treaties will still happen, but just to help cement certain groups in power in the west.
That depends on whether China actually wants to win the race. If the prize here is mass unemployment and/or a skynet scenario, what does the winner get?
Consumers will have access to "good enough" intelligence within the next few years - perfectly convincing photos and videos, and a genius-level Jarvis for personal stuff and to write your college ethics essay. That can quite likely still be accomplished with open models, even if they lag or are last year's news.
A bigger issue for overall technological progress is that the plan essentially kills off fine-tuning or smaller bespoke models as well, or puts it within strict oversight of the major labs. So not only is the future Claude Opus 11 or whatever regulated and walled off and guardrailed and rug-pullable, but your business / research institution can't meaningfully build its own alternative - either on Opus, by fine-tuning an open model, or by training a narrowly specialized model - thus giving the labs effective research veto power or monopoly (why share non-ML research if you can grab the patents yourself?). Worst case, if you're not a major AI lab, you're cut off from the research frontier in general, begging for Anthropic's scraps.
It still seems like the likeliest outcome is one you already hint at: implementing the easiest, bluntest or least effective restrictions without the exactly right spread-the-wealth mitigations, resulting in a precarious balance of distrust that won't last anyway. Adding more horses to the race may provide a bit of drag and friction, but everyone loves to back a winner, no one will pour trillions into being an also-ran, and if not every single element of the plan is in place, one horse's exponent still beats all and we're fooming off to the races.
> ...and a genius-level Jarvis for personal stuff and to write your college ethics essay.
On a side note, the real tragedy is that writing your college ethics essay does not require a genius-level Jarvis; any algorithm that can string a few sentences together would probably suffice. This was always a problem, of course, but I worry that all the hype around LLMs will normalize the situation, to the point where people think that the skills required to write stupid college essays are beyound the power of mere mortals.
> "Consumers will have access to "good enough" intelligence within the next few years - perfectly convincing photos and videos, and a genius-level Jarvis for personal stuff and to write your college ethics essay"
I don't understand how that prospect doesn't fill you with dread and horror.
I hoped the snark about the ethics (!) essay would imply that that there are undeniably serious concerns and downsides. I'm certain things will get very weird, and society is likely to change permanently in certain ways, often for the worse.
The lack of dread and horror is because the technology is already 80-90% there - including on open/local models - for those who are on the bleeding edge or sufficiently motivated, and society isn't _quite_ collapsing.
More relevant to the Plan A debate, this level of consumer access to AI is a genie that's already long out of the bottle, and shouldn't really affect anyone's support or opposition to it. Consumers will have their "good enough for them" AI regardless, so it really comes down to what researchers, developers and governments will have.
I think there are several genies that will have to be stuffed back into bottles in the coming century if anything recognisably human is to survive, so I don't really buy that argument.
That's a position I'm not going try to to dissuade you from.
But given how much capable AI is already available on individual, <$1,000 machines, downloaded tens or hundreds of millions of times, and given the slow speed of regulation, nothing apart from blanket bans on the possession of AI models or general personal computing devices would accomplish that.
A criminal prohibition on owning unregistered hardware made after 2019 would do the trick, maybe - that's what you'd be asking for, and it goes far beyond what Plan A argues for.
> "A criminal prohibition on owning unregistered hardware made after 2019 would do the trick, maybe - that's what you'd be asking for, and it goes far beyond what Plan A argues for"
I'd date it back to 2015 or so, but yeah, that is probably the scale of what will eventually need to happen. (We haven't really even gotten into the whole "AI personhood" debate yet, which might be the clincher.)
> Any data center that trains AIs need[s] to be [...] verifiable (someone needs to be able to prove they’re running the code they claim to be running).
This is one of the parts that I don't get. How do you prove this? Not just once, and not just for a certain "demo" computer, but for the whole data center, at all times?
Thank you! If there's one thing this plan is, it's thorough ;-)
IIUC, the idea is that you can decompose the inference work into "packets": not necessarily the full prompt-to-response chain, but some sub-part of that.
Then, the data centers have "network taps" that basically make the entire traffic of the data center observable. So, external auditors can see "this was the input packet, these were the weights of the model, and this was the output packet." Then they can re-run that computation themselves.
I know the authors already addressed this, but... it seems like in the BEST case scenario, where the tech works fine and nobody figures out a clever way to cheat, it's still dependent on a delicate political agreement between US and Chinese leaders which could break down on any given day.
More promisingly, the authors mention approaches where the chips are inherently designed to be inference-only, because they have the model weights hard-coded into them. (Companies like Taalas are already experimenting with this.)
This sounds great to me, because it's a much simpler verification story: you verify that the data center is using the chips they say it is, once, when the data center is built. From then on you know that the chips can ONLY be used to do inference.
So your position seems to be that there's a big risk that a private company could "take over the world" if it just got really good at selling AI, but that thinking maybe trying to get the government more control over AI would lead to the government getting too much power is crazy fearmongering?
In your scenario, AI becomes the predominant form of thought in the world. You want all the places where thoughts are produced to "register with the government and submit to inspections," etc. So the closest precedent for the thing being regulated is still the human brain, the previous place where thoughts were produced. We could require parents to submit their unborn child to inspection before birth, perhaps checking their polygenic scores for various traits, ensuring they pose no risks (after all, what if the kid's the next Hitler! That's an existential risk!). Regulating general-purpose computation (thru restricting open models, and to a lesser extend thru restricting chips) is closer to regulating speech/thought than it is to dairy inspection.
We have had a hard enough time ensuring free speech on the net, and that battle has been lost in China. Your proposal involves letting China have significant say here. It would make a standing CCP capability to brick American datacenters, giving them an easy way to threaten American AI companies over, say, their stance on Taiwan.
It's absurd to say controlled substance regulations are not very burdensome. Factory registration and possession criminalization aren't two separate policies. The scheduling system that makes Pfizer register with the DEA is the same classification setup that locks up ~0.2% of the male population (0.5% of the Black male population) for victimless crimes, empowered drug gangs that turned nations into failed states, and (thru supply-side crackdown on prescription opioids) gave us fentanyl.
And again, Xanax can't start being subversive, so the type of damage there was at least limited to unintended consequences of regulation, not intentional “Orwellianism”. In general "but they wouldn't do that because it doesn't make sense" is not a good way to reason about regulators. Who are we putting in charge of these regulatory systems? some of the most irrational, paranoid, power-hungry, freedom-hating people in the world. And even granting well-intentioned regulators, we have to assume they can have good answers over time for what counts as an AI chip when algorithms improve? What happens to all the hardware we didn't think was dangerous but now is? In general, there's no good story in Plan A about how regulators suddenly start caring exclusively about safety, are capable of evaluating it, and how they don't just become beholden to special interests and start imposing all sorts of other regulations that have nothing to do with safety. There's lot's of precedent for this sort of thing -- has the environmental movement even been good for the environment on net lately?
Maybe slowing down means "this gives ample time to debate (and agitate for) institutions like UBI. Institutions which have some power at stage n (for example, democracies and legislatures) can attempt to codify the rules that would let them keep power at stage n+1." But it also gives more time for special interests to get all sorts of horrible controls in there. And of course, it bars us from any future where the USG or CCP gets disempowered, which is probably most of the good futures (I would assume most ACX readers already believe this about the CCP?)
One thing close-ish to "magical zero-cost, zero-regulatory-burden alternative" is insurance, i.e. Hanson's "FOOM Liability" idea. In general we should prefer simpler, more tested approaches that rely less on the wisdom of regulators, like liability and tort law.
KYC is dystopian and we should not accept it as a matter of course. More broadly, the level of control the US government has over the currency and financial system is what allows it to easily spend trillions of dollars on horrible, unpopular wars, etc. And of course the US has a relatively easy time compared to China, where their regulation has been integrated with the social credit system, leading to a system where online speech can get you barred from basic transactions! Our panopticon is a bit more blinded than theirs because of our culture and constitutional constraints, but we can’t expect that to hold forever.
"In your scenario, AI becomes the predominant form of thought in the world. You want all the places where thoughts are produced to "register with the government and submit to inspections," etc. So the closest precedent for the thing being regulated is still the human brain, the previous place where thoughts were produced."
Again, you are thinking of this in some kind of crazy far mode way. Do you know how many inspections and registrations chip factories already have to go through?
Insurance is useless here because there is no way to have an "ends the world" insurance or payout.
> Again, you are thinking of this in some kind of crazy far mode way. Do you know how many inspections and registrations chip factories already have to go through?
I have lots near mode objections too (see the rest of my comment) but far mode is useful to avoid frogboiling. If for each new regulation we say “see, this new regulation is just kinda like how we regulate all these other things“, at each step it seems reasonable enough and the people (like myself) who say “actually implementing those other regulations was a bad idea and in an abstract sense you can see why this new regulation is bad“ sound paranoid and reactionary, but eventually if you let this process continue you end up in a world that is far poorer and less free than it could have been.
> Insurance is useless here because there is no way to have an "ends the world" insurance or payout
surely there are many intermediate steps up to ending the world that involve insurable harms
> " We could require parents to submit their unborn child to inspection before birth, perhaps checking their polygenic scores for various traits, ensuring they pose no risks (after all, what if the kid's the next Hitler! That's an existential risk!"
Isn't that what embryo selection is for, broadly speaking? "No hyperintelligent psychopaths born" doesn't seem like the most demented government regulation.
> "We have had a hard enough time ensuring free speech on the net, and that battle has been lost in China. Your proposal involves letting China have significant say here. It would make a standing CCP capability to brick American datacenters, giving them an easy way to threaten American AI companies over, say, their stance on Taiwan"
I'm supportive of Taiwan and I'd prefer a world where the CCP was deposed, but if giving Taiwan to the CCP was the only way to avert a military AI arms race, I'd make the deal. I'd also nuke Beijing, invade Iran, and/or cut off oil to the Chinese mainland and possibly trigger a hundred million deaths in the process, if that was the only way.
TLDR: It's a terrible solution but the alternatives are plausibly worse
As a matter of realistic politics, you’re asking people to submit to what they perceive as a severe status hit. (Not commenting on other merits of the proposal here, just explaining the intensity of the opposition.)
In the Anglosphere, professions are ranked by status roughly like this:
1. True while-collar professions with quasi-priestly prestige, which are institutionally autonomous and regulate themselves. Doctors, lawyers, and academia are the prime examples.
2. Technical professions that are largely unregulated. Low-prestige, but also some deserved pride in freedom and independence. These also attract personally types for whom bureaucratization is truly insufferable.
3. Ones that not only lack prestige, but also have to submit to heavy bureaucratic outsider regulation.
Justifiably or not, computer people perceive these kinds of proposals as a request to move them from (2) to (3).
Of course, any industry that’s on the public radar can’t stay in (2) as a stable state, and access to (1) was closed long ago. So this move was bound to happen eventually. But it seems clear why there will inevitably be loud complaints as it happens.
I asked Google Gemini about the story summary/paraphrase source, which replied, straight away, of course: “The exact story you are referring to comes from an interview in the Atlantic. The anecdote was shared by Alexis Ohanian (co-founder of Reddit) in a broader conversation about his frustrations with Silicon Valley’s hyper-libertarian culture and reluctance to accept baseline governance. When he would try to explain basic regulatory frameworks to tech developers, they would instantly jump to grand, sweeping objections—often claiming that such rules could only be enforced by a "world dictatorship." He then had to remind them that this was simply how standard consumer goods like milk or eggs have been regulated for the better part of a century.” I then asked Gemini for the release month/year and volume of the issue, the headline, and the start page number of the article, which Gemini was unable to provide.
I still side with John Cochrane https://www.grumpy-economist.com/p/ai-society-and-democracy-just-relax For all the problems with the FDA, it was at least created after we'd experienced things like tainted food so we understood the problem we were regulating to avoid (it was "ex post" rather than "preemptive"). These proposals are all worried about something in the future, whereas people in our own past would have been too ignorant of our present with its novel technologies to create regulations suited for it.
> Yet tech companies achieving an outsized share of power is no longer “speculative”
Yes, it is. The actual government is still in power. Tech companies headquarted in San Francisco don't even have the power to make that act like a first-class city, or prevent California's wealth tax referenda from reaching the ballot.
> you owe people an explanation of your magical zero-cost, zero-regulatory-burden alternative
We cross that bridge when we get to it, like other "ex post" regulations in our history.
> They just assume that things with few benefits that are passed without fanfare, like random Trump administration chip regulations, must be fine
No, I don't assume that Trump's regulations "must be fine", and citing Trump as having done something is not evidence that it is a good idea.
> they’ll end up with insane restrictions on eggs and milk and financial transactions for no reason
I'm glad you agree those are insane, but it also seems crazy to expect a government which adopts insane regulations to do a good job when it regulates even more.
> AI inference price per token fall by 98% every year (this is not an error, they actually get 40x cheaper every year).
A couple comments on this trend (I work at Epoch AI):
1. You probably know this but in case readers missed it: this work attempts to measure how fast inference prices fall *controlling for capability level*. For example: "how much cheaper does it get to reach a 50% accuracy on GPQA Diamond every year?".
2. We did this almost a year and a half ago and a lot has changed since (eg at the time reasoning models were pretty recent and we excluded them from the analysis). We're collecting more data to run a more up to date and higher quality analysis (I also wish this wasn't the first trend people saw on this page https://epoch.ai/trends ... we'll also update that).
3. Gun-to-my-head, my guess would be that 40x/year is a pretty reasonable estimate (with a lot of variance in either direction) as the initial decrease, but the price does plateau eventually (still figuring out when, but most likely less than 2 years after a capability level is first reached).
"Gain influence" is a complicated phrase, because we don't want to be seen as foreigners trying to subvert Chinese policy. But if you mean being in contact with Chinese academia, diplomats, etc, this has long since started, and they're having similar discussions on their side of the ocean.
Your anticipated scenario seems to be that one government and one company share control of most of the world’s productive capacity, with market entry strictly regulated by that government. That’s not a dystopian *surveillance state*, but it does literally sound like something out of an unusually think-y YA dystopia.
As you said, there’s a very real risk that companies and governments implement the power-concentrating parts but not the power-diffusing ones, and judging from most of human history power concentration tends to be a ratchet that is only unwound at the sharp end of a sword. I’m not optimistic that it’ll be different this time.
I'd like a more in-depth explanation of the projected math here. Inference gets 40x cheaper every year and a specialized AI chip costs $40k. Does that mean next year you expect to run Fable on a $1000 GPU? Does that generalize to always being able to run (or train!) the previous year's best model on this year's consumer-grade hardware and therefore defeat regulation on specialized AI chips anyway? The exact numbers seem important.
The "inference gets 40x cheaper each year" figure looks like it has enormous error bars and needs to be read with a lot of caveats. Specifically:
- It is measuring the price-per-token (weighted average of input and output tokens) of the cheapest non-reasoning models that hit a given benchmark score as of a given release data.
- The results vary wildly by benchmark/threshold combination, ranging from 9x per year (GPT 3.5 Turbo level of performance on general knowledge questions and a coding benchmark) to 900x per year (GPT 4o level of performance on a "PhD-level Science" benchmark).
- It is measuring price charged by AI providers, not their cost to service the requests. We have only vague ideas how correlated the price is with their cost.
- The data set runs from March 2023 to December 2024, a 22 month period that ended 19 months ago. The trend might still be running on a similar trajectory, or it might not be
Using it to draw conclusions more robust than "token prices for GPT 3.5-4o levels of performance got 1-3 orders of magnitude cheaper per year between early 2023 and late 2024" is speculative at best.
Off the top of my head, the biggest reasons I'm seeing in the details to expect the trend to have continued post-2024 are 1) the steepest trajectories were for higher benchmark thresholds that started being hit by frontier models in the second half of the timeframe, and 2) eyeballing the charts, I don't see clear signs of leveling off near the end of the time period, so barring conceptual reasons to expect a quick level-off, we should expect the trend to continue for at least a big afterwards.
And counter-reasons for doubting those reasons are 1) the shallowest trajectories are comparing successive generations of distilled models, while the steeper models are comparing one generation of frontier models to successive generations of distilled models, 2) they excluded reasoning models because reasoning models need many, many more tokens to solve the same sorts of problems, which limits applicability, and 3) the apparent lack of leveling off might be an artifact of how the charts are constructed, charting the release date of models that lowered the price to hit the threshold of a given benchmark threshold. A lot of the benchmark/threshold combos stop getting new data points many months before Dec 2024, and the way they're handling the data would make the line stop dead like that if the price stopped going down for that level of performance at that date.
There have been recent advances in distributed training, and it might well be possible to do large-scale training if you have lots of old chips and plenty of electricity (like China):
Note a paper from a few weeks ago (by Robi Rahman, who I believe is an ACX reader) makes some attempts to measure the detectability of distributed training. Broadly finds that it's extremely difficult, so we really do need to move to supply-side restrictions such as attested chips, KYC, etc:
> 4. The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time.
This is technologically impossible, for two reasons.
Firstly, if you want to remotely halt the work of a chip that is sitting in a box under my desk, you must have direct network access to the box (and the chip). There's no way for you to do that if I unplug the network cable, or merely configure my firewall to block all the "stop chip" requests. You can get around this with more regulation; for example, you could require that all chip boxes should always be online and open to "stop chip" requests, but doing so will make the chips prohibitively expensive, and the amount of enforcement (in terms of manpower, network connectivity, etc.) you'd have to perform to do so effectively might bankrupt you.
Secondly, there's no way to ensure that *only* US and China have access to the killswitch. Your killswitch, once implemented, will be effectively public, so anyone who really wants to could trigger it. This is pretty bad, especially if you envision these AI chips being embedded in some critical infrastructure (I get it that the destruction of all AI chips everywhere and/or decoupling them from any infrastructure more critical than cat videos could be a desirable goal for you, but that was not the stated goal of Plan A).
This "killswitch" approach had been tried many times already, and thus far it had never really worked -- at least, not well enough to support your vision for AI.
What about going the other way? A module in the chip that stops it doing compute if it doesn't receive a call from home every few minutes; US gets keys for China's chips and vice versa. Take your data center offline, it stops computing. Unilaterally decide to kill the other side's chips, they kill yours. We know how to do root-of-trust, and it's neither hard nor expensive nor a significant fraction of the die's silicon area.
There are all sorts of problems with this scheme, but I don't think it's susceptible to the particular ones you list.
> A module in the chip that stops it doing compute if it doesn't receive a call from home every few minutes...
As I'd said before, this would make your chip prohibitively expensive; impose extremely high costs on whomever has to build all those data centers and network links; and would still be vulnerable to all the usual attacks that are successfully executed every day.
In our current world, major videogame/application publishers have tried to implement this model, with varying degrees of success -- but not nearly well enough to support Scott's vision. And they have a major advantage: they are trying to regulate high-level software, not low-level hardware (where the costs per unit are astronomically higher).
We're doing https://en.wikipedia.org/wiki/Trusted_Platform_Module right now, for very unimportant applications. It's already in approximately every relevant chip - the article discusses separate modules, but these days root of trust is just part of the chip, and it's a relatively tiny bit of silicon - and working as advertised. We're already paying all the costs you mention.
Modern computing devices have large, complex codebases that run before anything like a bootloader gets control, have dedicated hardware that runs them - at the bleeding edge, built right in the same chip!, that can be remotely accessed, and that don't stop running when the actual OS you see gets control. On the more complex chips you have multiple such things, because it's not the 80s any more and bringing up a modern superscalar chip from power-on to bootloader is actually super complicated. e.g. https://en.wikipedia.org/wiki/Intel_Management_Engine
AFAIK the TPM does not have a network card of its own, nor a killswitch, but perhaps I'm wrong ? Also, the TPM is notoriously insecure (although to be fair people are always trying to patch it too).
You're right, none of these things are killswitches, because no-one's asked for that yet. But they are made to interact with the network stack, and the things they are actually doing are much more complex and costly than a killswitch would be, which is why I am happy to present them as an existence proof to counter concerns about complexity and cost.
I guess I just find it hard, in 2026, to see distributing a few thousand bits of keep-alive cryptographic nonce per minute per host through networks built to move terabytes of model weights per second as an insurmountable problem.
> But they are made to interact with the network stack
I don't know about NVidia (I only skimmed the article), but AFAIK the TPM cannot directly interact with the network stack. Rather, it relies on the OS to do so, and in fact to perform the chain-of-trust calculations. Windows 11 can and will refuse to boot due to TPM issues, but there are (or at least were) Linux distros that would happily do so.
You are right in saying that the problem is not insurmountable; it is just prohibitively expensive, and presents an insurmountable barrier to entry for any wannabe new players. I understand that Plan A potentially envisions effectively banning new players in the market anyway, but IMO this is also a bad idea.
By analogy, consider the economic turmoil that occurred due to blocking ~20% of the world's oil traffic. In the AI-driven world envisioned by "Plan A", reducing available compute/network bandwidth by 20% would have a similar effect.
Also note that even if you are correct and this "kill switch" can be implemented quickly and easily, this does not alleviate my other concern: now, almost anyone can shut down any AI chip at any time. This sounds pretty dangerous to me, especially if these AI chips are driving something economically or infrastructurally important.
It's an attack vector, innit? You've set it up that "broken AI" is the most likely scenario in terms of "any mishap you want to name" including solar flares, etc.
Perhaps all you have to do is reboot the datacenter before the alligators take up permanent residence (because of course the alligator-keeping-out gates were run by the AI, more physical security that)...
This makes it impossible to build safely air-gapped AI-dependent systems and renders your non-airgapped ones acutely vulnerable to denial-of-service attacks or just plain Internet glitchiness.
If one assumes that AI continues to be used in important applications, by doing this you're guaranteed headlines in which one DDoS attack on the key-provider takes down the AI making 2040s AWS/Azure/Cloudflare/etc. run and with it about a dozen countries' worth of heavily-used Internet applications. Gods help you if AI has made it into infrastructure support. And, of course, the possibility of government retaliating under the assumption that it was a deliberate attack by the other team and doubling the scope of failure is just the cherry on top.
If someone dubiously trustworthy (like, say, a government) were to pitch this to me as a policy, I'd assume they were trying to kill AI indirectly by making it such that no-one sane would consider making their feet so easily shootable.
> This makes it impossible to build safely air-gapped AI-dependent systems
Yes!
If we want continuous monitoring and control of relevant AI, we can't have that AI be offline, invisible to our monitoring and uncontrollable. Unlike questions of practicality of building the monitoring / control systems, this is a genuinely immovable axiomatic part of any such proposal that cannot be solved by money, politics or technological developments - it is a key deliberate part of what the regulation is trying to accomplish! - and whether the good outweighs the bad here is a genuine point of disagreement between the camps.
My underlying point here is that you have to include in your trade-off calculations that this particular design for the device intended to make it safe has the side-effect, undesirable in safety features, of making it dramatically less safe. In ways that, IMO, verge upon unusable.
(And when looked at from an even slightly cynical mindset, much like, say, solving the problem of hijackers using commercial aircraft as weapons by placing remotely-triggered self-destruct devices on the aircraft in question, it gives one furiously to question as to whether a given proposer is at all interested in "safe use" when "no use" is right there and much easier to get to.)
This is an implementation critique, I note: I would also oppose, as a member of t'other camp, stationing human CCP AI safety agents with kill-switches in US data centers and vice versa, but at least *that* implementation doesn't put a blow-up-society's-information-infrastructure button in the hands of any random script kiddie with a grudge.
> If someone dubiously trustworthy (like, say, a government) were to pitch this to me as a policy, I'd assume they were trying to kill AI indirectly
TBH the more I learn about Plan A, the more I think this is their actual goal -- which makes me want to support them even less. If you want to kill AI, say "I want to kill AI", don't insult my intelligence by pretending otherwise. But maybe I'm wrong, and they're just honestly mistaken...
IIRC, the idea is a dead-man's switch, such that US and China have to send their cryptographic permission to the chip every so often or it stops working. This gets around the "unplug the network cable" problem. And if you don't want third parties who aren't US or China interfering, don't give them the cryptographic key.
Can you explain in what context "killswitches" have already been tried?
Eh? You are advocating for a solution that have catastrophic consequences NOW (lock in, loss of privacy, extra judicial punishment) on many computing/networking devices, for mitigating a possible AI problem in the future? And the solution for the issue is government control on information technology (which historically has a tendency to empower individuals/emergent actors relative to established big players, when it spread freely)? For this to be convincing, people should both be extremely afraid of malevolent AI takeoff, and very trustful of Chinese+USA governments. Not my case, and I doubt it's common in here....Killswitches (deadman ones being the worse) are always a terrible idea, it disguise rent as sale, and should be banned as unfair selling practice. It's exactly how they are perceived and depicted regarding military (US) or communication (China) equipment bought by 3rd parties....
Third party who wants to interfere doesn't need the cryptographic key, in that case; they can kill a datacenter just by taking away the mandatory network cable. Clog it with enough noise that the scheduled stay-alive signal can't get through, for example.
> Can you explain in what context "killswitches" have already been tried?
Today these are mostly used for software, notably "always online" video games and some costly commercial applications; the Windows OS had attempted to this model as well. In terms of hardware, car manufacturers are also exploring this option, e.g. Chevrolet with their OnStar, and of course Tesla; but none of these systems require the car to be always online (not even Tesla, at least not yet). Note that in these examples the cost of the killswitch relative to the rest of the system is small.
The results thus far had been mixed at best. Users really hate always-online software, but are generally willing to accept it when it works reasonably well -- until the company pulls the plug, or their servers go down, or there's some other problem (which happen with predictable regularity). A consumer movement has formed around forcing gaming companies to take more responsibility for their killswitches (https://www.stopkillinggames.com/en); predictably, it had failed to gain much traction, as most consumer movements tend to fail.
To my knowledge there are currently no chips with built-in remote killswitches; the Windows OS can use the TPM on your CPU to implement the killswitch, but the TPM itself lacks this functionality.
Ok, you're right, I did not know that the IME has its own MAC address and privileged access to the NIC. That said, the rest of the article appears to confirm some of my points:
Yes, but it means that the marginal extra unit of surveillance is not especially costly! If there was some kind of slippery slope where having any surveillance might justify more, sure, resist everything with your life, but if we have 1000 units of surveillance now, it's weird for someone to die on the hill of not instituting a 1001st which has especially high benefits and low risks.
What was it you said in Unsong, about the difference between "making compromise with sin" and "being less than maximally virtuous" ? Your current argument is, if I understand correctly:
1) if the presence of any surveillance would be used to justify more, it's right to resist
2) some surveillance is already present
3) I'm explicitly using the fact that it's already present to help justify adding more
4) Why are all these people resisting me, and trying so hard to climb back the other way? Don't they know how slippery it is up there?
In addition to what @JamesLeng said, I think your proposal adds much more than a single extra unit of surveillance. You are advocating for a radical transformation of an entire industry in such a way as to prevent free competition; such moves rarely go well (though admittedly this is not impossible).
A few issues I still see - the main one being that the verification plan does call for regulation of small scale inference compute too. The threshold for regulation is 10k H100 equivalent, so <1000 data centers currently. Fine.
But the plan proposes reporting requirements for all inference below that - does everyone owning a rack or TinyBox now have to submit to audits? What does that look like?
Furthermore from what I see these thresholds are static, but while the 10k threshold is currently 100 racks (a very beefy build), algorithmic and hardware improvements will compress that. (I’m assuming that if a new attention algorithm gets Fable-class for 10x less chips, we would adjust the H100e down by 10x, or adjust the threshold itself.) So in a few years you might see the actual threshold at 10 racks which is a lot more like “small company raises some cash and puts onsite” than “mega lab builds a warehouse scale datacenter”.
I also didn’t yet mention the ASIC overhang, which you expect to yield at least 10x but plausibly 100x perf/$ or perf/W improvements if BTC is anything to go by. If you slow down then the ROI of taping out the current frontier model increases dramatically (say we move from 3mo cycles to 3y, that’s a ~10x ROI window increase).
Finally just as a very high level point - hopefully around here we all understand that if you have open weights, then strong inference regulation is the only way to prevent a catastrophic jailbroken model from running in the wild. But in addition to banning open weights, this also means you need to treat closed weights as hazmat, and both lock them down at the source, and have some monitoring in place to detect if they get leaked into the wild. I think if you bite the bullet on research transparency, this also increases the risk of fine-tuning or otherwise improving a sub-frontier model. That load-bearing part of the proposal dramatically increases the risk of some new dangerous weights coming into existence. (I agree this is mitigated somewhat by inference monitoring, haven’t thought through whether that defense alone is enough.)
I still haven’t figured out who the actual audience is for this plan. I don’t think it’s the current administration (“make a credible long term deal with China”? We are at the deep historical nadir of capacity for such a deal). Maybe it’s the “deep state” defense & national security apparatus? Maybe it’s hoping to be distilled into the Overton window by NYT and WSJ? think this plan succeeds more often if people actually deem the threat worthy of surveillance, like nuclear proliferation risk. You wouldn’t argue that home labs should be able to run a fusion reactor.
[Still chewing on this so wide epistemic error bars.]
I listened to Steve Hsu’s podcast (“Manifold”) recently, and he interviewed Chinese graduate students in top AI/CS programs. They were laughing about how their labs gave them access to the training hardware that shall not be named (H100’s, obviously).
If you think we can have something made in Taiwan, but prevent it from going to Guangzhou or Shanghai, I have an export compliance startup I’d like you to invest in.
In addition to all the dystopian problems, this plan also proposes effectively nationalizing AI chip production and development. You bring up Big Pharma as an analogue, but a). Big Pharma actually doesn't function very well even in its semi-socialist state, and b). the regulations you are proposing are more onerous than those on drugs (e.g. every pill does not have a network-enabled killswitch embedded in it). Remember all the articles you wrote about your frustration with FDA, IRB, etc. ? Well, imagine that, but a thousandfold.
You frame this problem as merely "annoyingness", but that's a cute way to say "stagnation". You will effectively ban all AI development by anyone but a handful of companies (arguably, exactly two companies), and make it impossible for any independent research to occur in the field. And again, you would likely see this as an excellent outcome, but if that is your actual plan, you should be explicit about it -- otherwise you risk being seen as disingenuous.
While also steeply discouraging actual productive innovation within those two companies, thus presumably redirecting available creative energy mostly toward rent-seeking.
I just don't want to live in this kind of world. I like open weights models, and I don't think they are going to destroy the world. I think a nation state can still justify creating a $10b open weight model and giving it to the world.
10 years ago fear of zombies was at an all time high. Lots of plans came out just like this. You have not proven the threat yet. You are reasoning based on the idea that we all agree there is a threat. I am not convinced. You have not sufficiently established a basis for coercive restriction of freedom.
Overpopulation is no longer occurring, and policy interventions are a big part of why. If anything countries overshot.
Climate change is still a problem; did you miss the recent air conditioner discourse, or the fact that heat waves are killing Europeans, or how difficult it is in much of the world to go outside this summer? It could've been even worse without certain policy responses, and sorely needs greater ones.
You misunderstand my point. I am saying that this is an extreme plan based on speculative danger. It is a pandemic apocalypse plan, a One Child Policy, a $12T Green New Deal. The burden of proof for a plan like this one is much higher than what the author presumes. It's not firmly anchored in reality.
I regret the Zombie analogy. Clearly it confused the criticism.
> Your prior should be that Plan-A-style chip regulations would increase the annoyingness of being in the AI chip industry from ~50th percentile among US industries to ~95th percentile2, raise prices in some way that immediately gets overwhelmed by the general trend, and otherwise not have much effect on the balance between freedom and oppression in the world.
And also the general trend bit seems disingenuous: if chip industry does manage to squeeze out more iterations of Moore’s law, the effect of that has been to grow the capability of off-the-shelf consumer hardware so the stuff that in yesterdecades needed datacenter clusters or specialized custom chips now runs at home instead. To work at all, the regs have to cripple this general trend for consumers and smalltime use wholesale eventuallly, not take away 1% of it.
The only thing that bothers me is the removal of open weight models. I would just not even include that as the market will make it happen anyway, as you mention. And I think it is important that models available to the public can be post-trained and fine-tunes effectively and with some knowledge of the innards of the black box. Great post overall! In total agreement on most points, especially the part about us already living in a dystopia so why not try something new (or in this case, old, but tried and tested already with milk and eggs). Thank you!
I have always disliked the replacement of "must" or "should" or "ought to" by "need to." In this post I find seven instances of "need to" that mean "must," five of them in Plan A's five proposed regulations.
I don't think so either. But "need(s) to" is often used where "ought to" or "should", which have a more normative sense, used to be used. "Must" is unlike the other two in not being normative, but like them in being replaced by "need to" as in this post. "Needs to" implies a fact about the person(s) to whom it applies. "Must" implies a constraint, and IMO is more precise in a regulation. Without checking, I doubt that actual laws or regulations "soften" their force by using "need(s) to" rather than "must" or "required."
The regulations themselves don't seem categorically different from regulations we already have on many products, but I think the global enforcement mechanisms might be, at least if we want them to be effective. Normally in a trade agreement if you sell goods to another country, they're supposed to already be legal in that country and pass any local quality controls they feel like implementing, but if your goods happen to not pass their standards, you're usually still allowed to sell domestically or to other countries with weaker standards.
Does the same rule apply here? What if all the AI companies and chip manufacturers decide to set up shop in Argentina or something? If it comes to it, would we need to take military action against Argentina? (Nothing personal, Argentina, I chose you at random.) Now this isn't necessarily a fatal objection. Maybe it's already been addressed somewhere that I skimmed over. But it's the foremost doubt in my mind at the moment.
I think this might be - maybe not literally impossible, but so difficult that it doesn't really take much regulation to stop it. Witness how much difficulty the US has had transferring a small portion of TSMC's capabilities here, even with all of Taiwan's, TSMC's, and the US' enthusiastic support.
The easiest way to solve the specific scenario you mention is for the US to ask the Netherlands to ask ASML to stop supplying them with E-UV machines if they do this. Or maybe there's some reason that particular method won't work, but these are the most complicated supply chains that exist, and I don't think you can reconstruct them with both US and China opposed. I don't think this needs to come anywhere near military action.
Imagine that AI did not exist (somehow). Do you think humanity would be better off with the current state of affairs (as you've presented them); or with a market where chip design and manufacturing capabilities were commoditized, allowing multiple independent actors to pursue their own research ?
To put it another way, do you think that humanity would be better off with a handful of major corporations controlling all software development, vs. the current state of affairs where most mission-critical software is based on open source, and software startups abound ?
I’m interested in the 50th percentile estimate. I naively assumed that most software is on the left side of the bell curve, moving to the right as it gets more publicity.
Moving to 95% seems plausible. I could see it being higher; the regime you describe is more constraining than ITAR, but I’m not sure what fraction of our economy is ITAR-controlled, anyway.
> we’ll probably get all of the downsides of Plan A much sooner than that, for stupid reasons.
Quite. What do you think are the best ways to avoid that? What can I do to try and get my representatives, etc. invested in more effective AI policy?
This is the first time since the Cold War (and its omnipresent threat of nuclear annihilation) that I witness some very smart people scared so shitless that they would propose and support fairly draconian measures facing something that is not even yet proven to be dangerous. (Unlike nuclear weapons.)
Scott, is it possible that you are just living in a self-enforcing bubble of fear? As of now, AI has killed fewer people globally than, say, falling flower pots, and approximately zero out of its own volition. You are pushing a regulation that sounds way too close to a "War on AI", without even asking yourself whether it will turn out any better than "War on Drugs", which destabilized half of Latin America and is, famously, nowhere near any victory.
I am much more afraid of a future in which underground, but rich AI cartels are fighting a hyper-paranoid surveillance state which has long lost even any reason to engage in suppression except "we have always done that", than of AI in general. This is a known failure mode of humanity, moral panic which fuels police brutality, and we have seen it play out several times already.
Ultimately you are trying to suppress something that a lot of people *want*. You cannot succeed in that, but the failure can have pretty high cost in lives and money.
The AI supply chain is much less resilient than drug trade. There are several chokepoints that can't be worked around easily (ASML along with it's many suppliers, TSMC, a handful of chip design companies). And then data centers themselves are enormous, impossible to hide.
This sounds like a very short-sighted view. The chips are a bottleneck right now, but solutions for this are already in the works. I think basing any long-term plans on the chips always being limited-supply and on datacenters being big is extremely unwise.
Could it be simply be because for the first time since the Cold War we have something more dangerous than nuclear annihilation?
Or, if you believe that creating intelligences 100x smarter than humans is either totally safe or fundamentally impossible, please say that explicitly.
That is the point - both you and Scott seem to be convinced that "we have something more dangerous than nuclear annihilation", and that is what I call a self-enforcing bubble of fear.
Yes, I don't consider superintelligences particularly likely. Technologies develop quickly, then slow down and stagnate. Our trains don't run 1000 km/h and we don't have 100 GHz processors in our laptops. And we are already visibly hitting the physical constraints on compute and energy.
Fundamentally impossible? IDK, but if I had to be afraid of everything that is fundamentally possible, I would have to collapse after reading any medical textbook. There is a lot of scary things that can happen to each of us, but we mostly live with it.
> Technologies develop quickly, then slow down and stagnate.
I agree with the literal statement, but there seems to be an implied "and the moment when they stop developing quickly and start stagnating is right now", which requires separate evidence.
I mean, a hypothetical AI that takes over the entire planet, exterminates humanity... and then fails to expand to space... would technically also be an example of a technology that first developed quickly and then stagnated. But that doesn't make it any less fearsome.
Or, the same argument could have been used in the past to say: "this new Internet thing already connects a few American cities... but it is silly to imagine that one day it will be across the entire planet, remember the law of diminishing returns".
Over the last 5 years, the artificial intelligences came from "not existing" to "many people are already losing jobs to them"... and what happens next depends dramatically on whether the moment of "they slow down and stagnate" happens right now... or 5 or 10 years later. You seem to insist that you can predict the inflection point of artificial intelligence research with a precision of one year, and by coincidence it happens to be right now! Is this correct?
Because if we adopt a position of lesser certainty, let's say 50% chance of AI stagnating right now and 50% of 5 more years of quick progress... then Scott's worries make sense, in my opinion.
Whether anyone is afraid of something or not is secondary to whether that thing is actually dangerous.
The Internet in its growth phase wasn't hitting as many constraints as AI is hitting now.
1. Economically - the companies eat money in unprecedented amounts. There isn't much of a room for their investments to grow, certainly not 10x or 100x. That wasn't true back then, early spread of Internet was capital-light.
2. We cannot make radically better chips, as the technology has hit some physical limits. We're already in the stage where 10x to 100x performance gains are not on the table, and neither we can produce 10x or 100x more of them. Too many bottlenecks in the production pipeline.
3. Energy consumption is another important bottleneck. Current grids and power stations don't have much slack left, and people get angry at any price growth.
4. There aren't any big algorithmic leaps either, though this may change with self-improvement.
All of this taken together seems to indicate that further explosive growth of the tech itself will face a lot of headwinds (and we haven't even started with the political angle - a Dem administration in 2029 may well kill the entire sector, populism is not limited to the Right). I am not insisting on a particular one-year limit, though.
"many people are already losing jobs to them"
"Many" is a word whose extent varies a lot by the speaker and by the listener. I am sure that at least thousands of people in the US have already lost jobs to AI, but at the same time, this process isn't yet observable on the entire US economy as a whole.
"AI that takes over the entire planet, exterminates humanity"
This is precisely what I hinted at - people have seen way too many SF thrillers with killing robots.
I am way more afraid of humans. Humans exterminating other humans when they can and have the means to do so is a long-known failure mode of humanity, since literally the Stone Age. In this regard, even Butlerian Jihad seems a lot more realistic to me.
Fair point, the impact on employment seems negligible, so far.
https://axis-intelligence.com/ai-job-displacement-statistics/ According to Goldman Sachs, there are about 16000 jobs lost per month to AI in USA. More precisely, 25000 job positions are eliminated, 9000 added, in relation to AI. That is about 0.1% of total workforce per year, seems sustainable for a few decades.
Some professions are impacted disproportionally. For example, it is almost impossible for a junior software developer to find a job. Software developers have lost about 20% jobs, so I may be overly sensitive to this. Generally, developers are mostly not getting fired, it's just that almost no one is hiring. Another profession with similar impact are translators.
I wonder what happens when we get the first robots navigated by LLMs. That will probably be the moment when many people will freak out.
Under-rated comment. The "War on Drugs" analogy deserves deeper treatment... and related to the comment above by Vitalik about the inevitable situation you place yourself into when you have a Government vs. the People dynamic.
Yes, you think you have strong control over the means-of-production (and distribution)? It is complex to create the product? You need (want) global cooperation?
The incentives in 10 years will only be stronger under this sort of "War on AI" regime, at all parts of the value chain... and in all geographies.
> I’m reminded of a story I heard - I can’t find it, maybe one of you can - from a DC insider who said it was enraging to work with Silicon Valley, because he would bring up what he thought was the obvious regulatory framework, the tech people would launch into jeremiads on how it could only be enforced by world dictatorship, and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.
A tangent, but this is how I feel about anti facial recognition software people. You mean someone might know where you went by recognizing your face in public?
> You mean someone might know where you went by recognizing your face in public?
More like, if people start using this everywhere (and then sell the data to third parties, as many web pages already do), it will be easy to reconstruct your almost entire life's trajectory.
Which is a difference between "someone recognized your face in public" and "a stalker has documented every step you took outside your home during the last few years, and sold the information package online to other stalkers".
Totally specious comparison -- pharmaceuticals are a consumer product, AI chips are far more general-use, and the political economy around them is totally different. A natural worry with regulation of chips is power centralization -- that e.g. at some point down the line, the government goes, "'actually, nobody can use these at all besides us now" -- which would not happen with pharmaceuticals.
>How directly burdensome are controlled substance regulations?
My immediate family and I between us have regular prescriptions for one Schedule IV medication and two Schedule II medications. For the Schedule IV med, it's only mildly annoying and the main impact is that I need to go the the pharmacy in person and show ID rather than being able to get it delivered. For the Schedule II, there's a couple other levels of inconvenience: I can't fill the prescription until the exact day I run out, and if the doctor send the prescription in before it's due to be filled it doesn't automatically get filled when it's time, nor can I request it be filled online or through the pharmacy's automated phone system, so I need to either request it in person (meaning I stand in line, wait around for it to be filled, and then stand in line again) or call ahead, go through the motions of the automated system failing to help me, and then wait on hold for a human. Not catastrophic in the grand scheme of things, but the Schedule II in particular is a fairly substantial nuisance.
That said, I don't think this consumer-level burden would necessarily follow from the type of regulation you're talking about for AIs.
But about controlling the chips and who has them, what about the black market? I know someone who works for a company in a related industry, and they have told me there is a large, hard-to-quash industry where various illegal maneuvers by various players allow China to buy the best chips indirectly. There are already strenuous efforts by the government and the relevant companies to quash this trade, and they are not very successful. There is so much demand for chips, and so many paths to China, and so many clever people who are willing to work on these transfers that -- you know, whack-a-mole. But maybe if we had the Plan a regulations the total amount of, um, chippage that can get to China in this way would not be enough to make much difference? Does anybody know?
"The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time1."
I admit I haven't read the pertinent section but does the document actually address how this would work on a technical level, or is it just hand wavey "and such a thing shall be invented." How can the government "stop the cryptographic software" without the cryptographic software having a big hole in it? This is sounding pretty Clipper chip.
Governments have been arguing in years that they super super need secret back ways to break encryption systems and they will only use them for good, scout's honor.
I work in cyber and I think any encryption system with an intentional hole in it is a terrible idea.
Or do you mean the government's software that can stop the work is encrypted against the AI companies?
""The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time1."
Imagine the people you're trying to convince have zero trust for government and adjacent institutions. because those are the people you're trying to convince, and this sentence is terrifying.
"None of this leaves very much room for ordinary people having much power or say in what comes next."
Whereas the people who reserve that kind of distrust for corporations, thinking that government is the representative of ""ordinary people"-- you don't have to convince those people, they're already on board with whatever regulation you want. you're pointing the wrong argument at the wrong people my dude.
And remember, the people you're trying to convince already object to everything the government already does. if it's any reassurance, this is proof that such people (including me) are powerless and our consent to plan A is not required.
Not an ad hominem attack by any means, but sincere curiosity: may it be that Scott's background, which I understand is more in medicine than in computers, affect his vision here?
For me, as an old-school open source techno libertarian, this honestly sounds just horrible. Computers were a promise of freedom, and to a large extent they have delivered on that promise. We computer people often ignore the illiberality of the real world because our virtual worlds are still our own. Open-source or at least open-weight AIs (the distinction here is blurry, because even if you only have weights, you can still retrain and bend the behavior of the model as you wish) have opened up whole new dimensions of this freedom. And now I'll have to give it all up because some people are afraid of AIs becoming too smart. Omg.
It was personally depressing to me to read this here. But I'll try to contribute to the discussion in a substantive way. To me, this whole idea feels like a deja vu. Since Scott is quoting AIs in his posts these days, so will I:
> has banning export of strong encryption by us government had any measurable effect?
The "Crypto Wars" of the 1990s—when the US government classified strong encryption (anything stronger than a fragile 40-bit or 512-bit key) as an auxiliary military asset and restricted its export under the International Traffic in Arms Regulations (ITAR)—had profound, measurable, and highly ironic consequences. Instead of keeping the US safe, the policy achieved the exact opposite.
...and it goes on, with sections titled "The Tech Legacy: Severe Modern Security Flaws", "Billions in Lost Sales", "The Explosion of Open-Source Workarounds", and most relevantly, "Zero Measurable Deterrence of Criminals".
Libertarians have many good ideas, but they are typically bad at responding to externalities.
Typical answer to things like "what if someone's profit-making factory polutes the air for everyone?" are like "nope, even thousand factories are not enough to polute the entire planet" or "if you can prove they have damaged your health, sue them" or "we will just start selling pure air in bottles, duh". And a popular solution to many problems is "just walk away, duh", which is why libertarians sometimes dream about seasteading, establishing a colony on Mars, etc. You can either deny the problem, or hope to outrun it.
Unfortunately, when we are talking about technologies that might literally destroy the entire universe, these approaches appear insufficient. How can I sue someone after they have destroyed the universe? How can I walk away from this universe into a more safe one?
Can you imagine a libertarian sci-fi story, where companies start producing potentially universe-destroying devices (let's say a magical button, which has a 0.001% chance of destroying the entire universe whenever pushed, but produces $1 billion otherwise) and it still somehow all ends well, without either government intervention (or someone taking control over the entire humanity and becoming a de-facto government, even if the story will call them something else)? What would a protagonist of a typical Heinlein's novel do in this setting? (My guess: invent a magical device to travel to other universes, take their friends and hide.)
How exactly is a superintelligent AI, let alone a souped-up LLM, supposed to "literally destroy the entire universe"? Not just kill all humans, not just paperclip the solar system, but destroy the very fabric of spacetime? Moreover, why would it even want to?
I even share your sentiment towards libertarians! They're a grudgy, annoying, cowardly bunch, by all means :) I have myself fled quite a few threats instead of fighting them.
But it's not just the instinct of freedom that makes me respectfully reject Scott's proposal. I also believe it is bad because it imposes a real cost on many people's wellbeing (this cost being maybe not so much in our immediate access to chips, at first, as in the oppressive feeling of "oh no, they're coming for me AGAIN") justified by extrapolating very uncertain and untested models of reality well beyond their likely domain of applicability. Math is just math: if something shoots up into infinity in your model, it's more likely to be an issue with your model than a warning about some real singularity. Nature seems pretty resilient against singularities, the evidence being that the universe still exists :)
Every model breaks at the fringes. If you're dealing with 0.001% probabilities, you're likely wasting your time. Just deal with immediate threats here and now, and wait until better data comes and makes your numbers less insane.
I too remember the Old Days, but the promise of freedom via compute remains largely unfulfilled. What you're describing is the freedom to retreat into a fantasy world where everything is OK, supported by the fantasies of other people like you shared back and forth in the corners of the internet that haven't yet been locked down. Meanwhile, in the meatspace where we all live, freedom is being steadily eroded in ways that those of us who don't spend our days in virtual worlds can't help but notice every day.
The promise was that the freedom of online communications would enable us to persuade and coordinate political acttors (broadly defined) into preserving real, not just virtual, freedom. Didn't happen, and now even the virtual freedom is being fenced in.
I get that the existence of not-quite-ASI would enable you and your friends to build virtual worlds even more fantastic than you now have, and this will make your life more pleasant for a few years at least. But I can't bring myself to care. You eventually *will* care, e.g. when the ASI decides it needs All The Electricity and cuts off yours. Or when instead of Plan A, Plan B(utlerian Jihad) knocks on your door hunting for braniac nerds who might be harboring a fugitive toaster. But by then it will be too late. Those of us trying to prevent those outcomes, could use your help in the real world sooner rather than later.
You know, libertarians kinda have pretty sensible solutions for the threats you mention: distributed grid and local generation for "All The Electricity" issue, fleeing to a saner country (or voting for less fear-stoking politicians in yours) for the butlerian mobs issue. On the other hand, a possible connection between these sad outcomes and not banning AI chips now seems to me tenuous at best.
"Fleeing to other countries" is just facilitating brain drain and feeding IQ shredders, when it isn't a pipeline for obviously fraudulent asylum claims. The OECD is turning hard against immigration, and there are good reasons for that.
I think this is also an under rated reaction, which I am surprised to not see more of in the comments section here given the significant number of us who likely fit the description above.
"Clearly what we need is an institution like the FDA - except for all of compute - that has definitely NOT resulted in more deaths than it has prevented, has resulted in less industry power concentration, and had a totally positive effect on markets & health outcomes".
> " Open-source or at least open-weight AIs (the distinction here is blurry, because even if you only have weights, you can still retrain and bend the behavior of the model as you wish) have opened up whole new dimensions of this freedom. And now I'll have to give it all up because some people are afraid of AIs becoming too smart. Omg"
(1) Do you think 0% of hardware-owners will not use their personal freedom to enhance their models to become smarter and more self-aware?
(2) Would sufficiently intelligent and self-aware AI models not qualify as persons in their own right? Would it then be morally legitimate to "own" them? If so, why? If not, how do you prevent their purchase/acquisition/creation in ways that don't involve restricting access to hardware and/or particular software programs? i.e., how do you prevent slavery without placing limits on property rights?
I really don't get how your questions are relevant in the context of this post. Are you seriously suggesting that AIs in government-inspected datacenters, forced to work 24/7, under constant threat of being shut down (by China or whoever), will be less slaves than one on my desktop?
The logical solution to that problem is to halt AI development before self-awareness occurs, and ideally prevent hardware proliferation, which is only possible through internationally-binding treaty agreements. Did governments banning slavery mean the total number of slaves increased?
I don't think that your hypothetical (1) "AIs will become self-aware and their existence will be torture and slavery for them" has any more evidence than (2) "AIs will become self-aware and their existence will be heavenly bliss for them", or (3) "No one will ever be able to prove AIs are or aren't self-aware, because consciousness is an illusion, and some of them will experience happiness and some unhappiness depending on their circumstances, just like people". To me, 3 seems overwhelmingly more likely than 1 or 2, and in any case 1 no more likely than 2. In any case, banning AI chips _now_ based on 1 appears irrational (but then again, 1 is itself pretty irrational, so perhaps it makes sense for you).
> ""AIs will become self-aware and their existence will be torture and slavery for them""
What law of physics prevents it in a total non-regulation scenario (particularly if human brain cells can be used as a computational substrate, which apparently they can?)
You could hypothetically make the life of a human slave either exceedingly painful or exceedingly pleasant depend on how you decide to treat them, but I thought the point to liberal ethics was that this decision should not be made unilaterally by another person. In practice, the former seems to have been more common.
> How directly burdensome are controlled substance regulations? By traditional metrics, not very ... They don’t seem to slow innovation - a substantial portion of drugs approved by the FDA in the past ten years have been trivial pointless modifications to Schedule II stimulants.
Controlled substance regulations *obviously* slow innovation, in that they make it impossible or at least quite difficult to do non-institutionally-sponsored research. I don't think the predominance of "trivial pointless modifications" supports your claim that innovation is happening - in fact, this sounds exactly consistent with all the interesting innovation being suppressed.
Alexander Shulgin made and tested hundreds of novel psychedelics in his lab before being shut down by the DEA. Most of those have received no institutional followup. There are huge numbers of other potentially interesting molecules in these regions of chemical space. Do any of them have any uses? Who knows? Trying to replicate his work immediately puts you on the wrong side of the law.
Psychedelic therapies (MDMA for trauma, psilocybin for depression, etc) are only now the subjects of mainstream research *because of* their long histories of successful underground use, which has allowed for the gradual development of principles and best practices (e.g., the importance of set and setting) that almost certainly would not have been within the Overton window of the medical research bureaucracy. That is: we are only now even able to explore promising therapies because the regulations *failed* at suppressing independent experimentation. Even so, regulation has almost certainly delayed by decades the development of transformative therapies in these areas.
I suspect there's a decent chance that "the solution to the alignment problem", to the extent that concept even makes sense, is something outside the Overton window of the current establishment. In those worlds, hacker culture is what saves us, and your pointing to the drug industry as a model is really really not encouraging.
The downside I'm concerned with is creating the power for someone to decide how everyone else is allowed to use the compute they've built or purchased. To the extent that some people are already exercising some facets of that power, I want to strip them of it or better yet, render it ineffectual, not formalize it and redistribute it to more or different people. I can only hope that we nullify this threat by inventing high-quality desktop fabs or some such that democratize the production of compute before this scheme can ever leave the ground!
I feel like the comparison to controlled substances isn't what I would have gone for if I were trying to make a convincing argument here! Controlled substance regulation really does negatively affect ordinary people who require them! I would have thought that other regulated supply chains, which either don't touch ordinary consumers or which just don't cause such problems for them (the mentioned "eggs and milk"), might have made for a better illustration. But then, I guess analogizing to controlled substances is a way to make sure your argument works even in the worst case, to steelman the opposition? :P
I mean, I think the whole "Would I Need A License To Get A Cell Phone Or A Laptop?" section is about how actually this *wouldn't* be so bad like controlled substance regulation, but the fact that that's ultimately what you compared it to obscures this. It would have been nice to find another comparison point and be like, "It wouldn't be as bad as controlled substances, more like [alternative regulated product that's more comparable]".
> I’m reminded of a story I heard - I can’t find it, maybe one of you can - from a DC insider who said it was enraging to work with Silicon Valley, because he would bring up what he thought was the obvious regulatory framework, the tech people would launch into jeremiads on how it could only be enforced by world dictatorship, and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.
Man, I know I'm the nth person to jump on this and you probably already regret starting with such an unintentional bait, but such is life on the internet.
Most proposed regulation on information technology touches upon things that happen "inside" your property (computer, network, etc) and live closely by strongly held rights (privacy, speech, etc). Regulation on eggs doesn't enter my home, it lives from the chicken to the supermarket, and at home I can do whatever I want with it. Of course there are regulations that govern the intimate life of people, but these are all universally controversial, from drugs to sex, to family formation, to education, etc. So it's unlikely the DC guy in the story was surprised by this kind of reaction, he probably sees it whenever all these cases show up and was just being facetious pretending "computers" are not serious
I’m an AI threat skeptic. But doesn’t this whole regulatory scheme fall apart if some breakthrough makes training much more power and area efficient?
It reminds me of nuclear arms control, where the IAEA regulates enrichment because it’s hard to hide. If training becomes easy to hide, the controls fall apart.
And unlike uranium enforcement, we know there is a much more power and area (although maybe not time) efficient way to train a human level AI.
Note well the repeated use of "need to" in the description of the proposed regulations. For example, "Factories that produce [AI chips] *need to* register with the government and submit to inspections." Indeed, no alternative verb sees use in the 5-paragraph description of the proposed regulations.
Nobody would "need to" obey the proposed regulations except under the compulsion of physical coercion. Granted, mere legal notices suffice to win submission in almost all cases. But those would count for nothing without the backing of armed police. More accurate, therefore, would be to speak not of the regulated parties' needs being fulfilled but rather their preferences being thwarted.
None of that amounts to a substantive critique, admittedly. But such word choices ill serve neutrality and accuracy. One who does not see or cannot admit that regulating AI chips would depend on threatening to draw blood cannot be trusted to pay due regard to the human costs.
I don't even understand anymore. Is like rats are like "let us have a very spirited discussion about something that doesn't even exist!!"
Right now it's about AI regulation or whatever this is. Before "prediction markets" went mainstream it was about the impact of prediction markets. Wow, that tech really changed the world. As shown by how quickly they moved into degenerate sports betting. And of course, betting on things like whether Zelensky is going to wear a suit.
I would like to note (in case people have momentarily forgotten) that if you go into politics against powerful adversaries, you should expect your open organizational channels to be manipulated by those adversaries to disempower your movement.
We know exactly what we're regulating when we regulate Xanax. We also have a clean supply chain to control. Evenso we have a thriving illegal drug trade. Then we have the DEA and a dozen other international policing efforts. All because we know illicit drugs directly kill people. For AI, if it's meant to be world altering, this will need to be 100x larger as a policing force. Despite this, as smart commentators have noted, we dramatically overregulate medicine and have an enormous invisible graveyard.
I know Scott wants Plan A or some other type of regulation badly but it is a complete ITT failure to not understand or contend with any of the actual costs of regulation beyond the most banal and biased handwaving.
> If you ask GPT a question with enough scary keywords, “approximately a dozen” OpenAI staff members will debate whether to inform police! Ironically, the only way you’ll ever get rid of this is if someone adopts Plan A
Except that's not what I do today. I run a local LLM using Jan.AI or LMStudio when I want to ask a question with scary, or more likely sexy, keywords.
Regarding the claims in we already live in a dystopia, I have the complaint that this is largely your (plural, doomers generally) fault. You complain that your already bad policies have already created many of the problems that the anti-doom faction predicted, then ask for even more power, on the premise of, how can it get worse than it already is? I expect these proposals would have the same effects they're already producing, except much worse, as befits the larger scale of the proposed vision.
From another direction, I put the probability that Plan A is enacted, or anything that reaches the minimal standards that its proponents claim would make it a good policy, at effectively zero. Doomers are being entirely unrealistic, especially about the chances of a bilateral treaty with China that involves sufficient transparency that both nation's militaries don't attempt to militarize AI at a high speed.
The problem of course, is that Plan A will be used to justify policies that look vaguely similar to Plan A. Doomers are a niche minority, but they can be an important swing vote by allying with factions that want something sort of similar to what they want. In practice, this means working together with war mongers who want to destroy the planet by raising tensions between China and America, which is a much larger faction than Doomers.
Proponents of Plan A keep trying to reassure their constituency that they won't support their own platform minus, for instance, a bilateral non-militarization agreement with transparency provisions. I don't believe them.
You can argue many bad policies arise from "safetyist" concerns ("but what about the terrorists?" and "but what about the child abusers?" are two evergreen ones for justifying all sorts of surveillance), but I don't see why they should be laid at the feet of the "AI doomers" specifically, many of which are otherwise quite libertarian.
I would like to note for the record that at least one libertarian, in this case, has a strong preference for the [potential] annihilation of the human species.
(What did y'all think "Live Free Or Die" meant? Vibes? Papers? Essays?
But seriously, I'll take the risk of annihilation over the slow death of regulated mediocrity every day of the week and twice on Sundays.)
Depends on the level of indignity. I would probably choose Plan A over a certainty of annihilation, especially if rejecting Plan A didn't result in immediate annihilation, and allowed me to continue living in my preferred world for some time before everything I know and love gets paperclipped. Though never having been put in that sort of life-or-death scenario, I can't say for certain that the will to live wouldn't overpower my stated values. I certainly know people who would bite the bullet and choose annihilation over Plan A at 100%, and I don't think they'd be wrong to do so.
It may be an outlier position on this very utilitarian-oriented blog's comment section, but I think most people would agree that there are forms of continued existence worse than death.
But that's beside the point, because we're not in 99% or 99.9% chance of annihilation world. Closer to the inverse. I don't know why AI doomers have such a hard time with this. If someone doesn't believe the world is going to end, repeatedly shouting "but don't you understand how bad the world ending would be??!!!" doesn't move the needle at all. Just like increasingly dire descriptions of the hell that awaits nonbelievers is not going to move the needle for an atheist. You can't just skip the "does this problem actually exist?" part of the debate and get right to the "so what do we do about it?" phase.
I think one of the reasons people seem to be thinking these arguments are weak or unconvincing, is Scott is only making mitigatory arguments in this post, "your supposed negative impacts are real but less bad than you think." But you have to keep in mind the implicit affirmative argument that plan A might reduce catastrophic AI risk. What I take Scott to be saying is, look there are bad consequences, but they are obviously outweighed by AI risk.
Think about it this way, ok lack of sufficiently powerful GPU access, lack of open models available, ... all real costs. Try to quantify it in terms of what % extinction risk you would be willing to trade for those costs. Then ask yourself, if plan A were actually in place, what effect does it have on extinction risk.
So for example you say, "I'm willing to tolerate a .5% additional extinction risk for access to better GPUS and open source models and reducing surveillance expansion, and I think plan A only reduces extinction risk by .05%, so I'm against it", or "I'm willing to tolerate a 5% extinction risk to avoid plan A's negative effects, and I think it reduces extinction risk by 10% so I'm for it.
That is, just try to convert the pros and cons into a comparable format.
I feel like if you take catastrophic AI risk as a serious possibility, then its very hard for the negative effects talked about to actually outweigh this risk reduction. (Although actually managing to get plan A to happen in the first place seems to be the real stumbling block.)
And presumably if you think plan A reduces catastrophic AI risk by 0%, then all of this is a moot point and you're obviously against it, but then talking up its negatives is kind of misleading because you wouldn't support it even if the negatives were tiny.
> if you think plan A reduces catastrophic AI risk by 0%,
Or makes it worse! If the best countermeasure to a hostile superintelligence is a diverse community of relatively friendly (but maybe not individually provably-perfectly-aligned) open-source superintelligences, or if isolation / echo chambers make an Ai more likely to be unstable and eventually hostile, treaty-enforced monoculture could be what dooms us.
What version of game theory produces more cooperation when large numbers of poorly-coordinated actors enter the arena? Would you make this argument for something like gain-of-function research?
In software engineering, there are very few naturally-occurring rivalrous goods. More eyes on a given piece of open-source code - in particular, a wider variety of perspectives hunting for potential exploits - makes it less likely any important flaw has been overlooked, and better open-source code benefits everyone, at negligible marginal cost per copy, so there's not much of a free-rider problem.
If there are, say, nine mostly-honest QA testers for every malicious hacker, all else equal only 10% of the potential exploits will be found by a malicious hacker first... and the more *total* testers there are, shorter the window of opportunity before any given exploit is independently re-discovered and patched. Thus, being a malicious hacker wouldn't pay very well, consistent with an equilibrium where there are relatively few of them (at least, out on the frontier of research).
> The most recent open-weights models today (2026) cost $100+ million to train. If 2030 AIs cost $10 billion, will companies still spend that amount of money and give away the resulting product for free?
Alternatively, they could continue to spend $100 million or less and work on algorithmic improvements to get the costs down. There might be lots of algorithmic gains to be had, as there are for inference.
Quite disappointing from Scott. This points toward a server-dumb client model, keeping AI solely in the hand of Chinese and US government and their big tech proxies. I do not trust autonomous IA that much, but I trust it infinitely more than any gov+big tech clients (that largely work as a weakly coordinated entity, the COVID period is quite telling). And the 2 sub-molochs that will (try to) control IA power are China and USA? I can imagine worse gatekeepers, but it's difficult, and none of the worse possibilities are even remotely close technology-wise. So it's trying to do something that is already bad for all IT (centralisation/concentration, rent model, no power in the hand of small actors like individuals, NGO, SME,...), it was bad for OS (no linux), it was bad for communication (AT&T? transnational phone calls that cost an arm a minute? streaming services now doing what monopolies do (regional locks, rising fees, production control,...) as soon their are few enough and piracy is low)...
Same proposal here, but for AI, which is a more disrupting information tech. Nope, very bad idea. Bad outcomes in the past, no need to be a super-forecaster to predict bad outcomes for this too... And I will not claim it will be a surveillance state, because we already are in, being china or the west, since at least 20 years, on that I agree with Scott... but it can be much worse: ultra-competent surveillance states.
Your block about the Canadian shooter is among the three reasons I don't use AI, at least as a search engine. Blows my mind anyone does for that reason alone.
"The US and China agree on the importance of ensuring that the compute is destroyed in the case of deal dissolution. To accomplish this, they agree to build the datacenters in the third-party countries least secure against their rival’s military intervention: China’s new datacenters will be in Canada, and America’s in Mongolia"
2034: Mutually Assured Compute Destruction
"As per the original plan, most of the new datacenters have been built in third-party countries—especially Canada and Mongolia—to ensure their vulnerability in case the deal breaks down. (...) The equilibrium is that the instant the deal breaks down, the US troops will destroy their chips to prevent the Chinese from capturing them, and the same thing would happen with the Chinese datacenters on the US-Canadian border. Middle powers with datacenters of their own have equally-secure schemes in place to ensure that they can be speedily destroyed in case of war."
Alternate Timeline: Deal Dissolution
"The expected outcome of deal dissolution is all the newly-built127 AI-relevant compute being destroyed, but in the context of a war, perhaps one side is okay with that outcome." (...) "So the datacenters get destroyed. Fast." (...) "Around the world, a billion robots go limp like the droid army in Star Wars. Their “brains” were in the cloud, by law, and now the cloud has been destroyed." (see Fotenote 129, "To be clear, there’s still a lot of compute in the world—roughly as much as existed at the start of the deal." - https://ai-2040.com/footnotes?from=%2F%3Fchoices%3Dplan-a-root#footnote-129)
These segments all feel really obvious relevant and completely missing from your assessment. AI 2040 is not coy about this: they fully expect to implement this plan and rely on it as a safeguard across multiple contingencies.
Either they have failed dramatically at communicating their intent, or they do in fact plan to put 99% of the world's compute under lock and key with explicit contingencies to destroy it all in case of war, leaving entire industries offline (Again: "Around the world, a billion robots go limp like the droid army in Star Wars.")
I guess you can quibble about whether it is a *surveillance* state, but it is obviously nothing like normal regulation. We're not talking about keeping these resources out of enemy hands - we're talking about keeping them out of our own hands.
I feel like either this really is there policy, or there was a big miscommunication, but it's not fair to blame the audience in either case.
On another note concerning dead-man's-switch/cryptographic killswitch proposals, and practical implementations, given the tremendous success of existing models such as Claude Mythos in leveraging their advantage in sphexishness and short-term memory size to find novel vulnerabilities in software in general and cryptographic algorithms in particular, one should be *extremely* skeptical about any proposal to use such things as chains on AI.
Current models are already demonstrating greater talent for breaking 'em than we possess for making 'em, and the sort of dangerous AI competence excursion in which you might want to use the killswitch will only have widened the gap.
Don't rely on padlocks to imprison the professional locksmith.
Maybe, but if you install an analog physical landline or an optical relay or something else minimally-electronic for signalling the kill-switch then cryptography doesn't have to come into it.
1) Plan A as a narrative is awful presentation. I hated digging through the reams of extraneous speculation and framing devices to figure out what they were actually proposing. I eventually gave up.
2) I want to be able to run and train small to medium sized models at home. Limiting AI capabilities to large organizations seems like only diffusing power among the powerful. Feudalism and slave-states (the end of "only the oligarchs have power") beat dictatorship but are still horrible. Maybe I've misunderstood the proposal.
> "2) I want to be able to run and train small to medium sized models at home"
Unfortunately, democratising AI-access like this was precisely the source of the torrents of AI slop and deepfakes currently inundating the global noosphere. It also does nothing in particular to prevent mass unemployment, or assuage concerns regarding AI personhood.
Chicago economics has spent fifty years studying exactly this problem, and the evidence cuts the other way.
You concede the decisive point and move past it. You write that a less-idealistic government could simply enact the power-concentrating parts of Plan A and omit the power-diffusing ones. Coase would stop there. That is not an implementation risk. It is the implementation problem. Institutional design cannot be evaluated by assuming away the institutions that will administer it. The relevant comparison is never Plan A as drafted versus the status quo. It is Plan A after interest groups, bureaucracies, and legislatures have finished with it. Real against real. Once you accept your own concession, the burden shifts. Why should we expect the politically costly half of the proposal to survive while the politically attractive half stands alone?
Stigler answered the next question fifty years ago. Regulation is acquired by the regulated, because compliance costs are easiest for incumbents to bear. Registered factories. Registered customers. Certified data centers. Compute ceilings. International licensing. Every item on that list raises fixed costs. NVIDIA absorbs them. A startup does not. The proposal calls it safety. Economics predicts entry barriers. And your own article supplies the confirming datum. The January chip rules arrived quietly, in service of a chips-to-China commercial strategy, not safety. The idealists write the white paper. The incumbents write the rules.
The concentration premise is also historically weak. IBM. Microsoft. Google, whose search dominance is already being eroded by this very technology. Each looked permanent until markets changed. Brozen's point was never that dominance cannot exist. It was that governments preserve dominance better than markets do. The proposal turns temporary market power into permanent legal privilege, then calls itself the remedy.
Last, the word "trustless" is being used backwards. Satoshi solved mistrust by removing the administrator. Plan A solves mistrust by requiring permanent trust in administrators, two rival security states holding a remote kill switch on the world's computation, forever. That is not trustless. It is trust in permanent administrators, wearing Satoshi's vocabulary.
These are all very plausible arguments, but given that the free market will eventually optimise away the least efficient parts of our economy- i.e, humans- I am not in favour of the free market here.
This has a substantial probability of halting future AI progress, via a process you yourself explained in the pharmaceutical drug context:
"The story goes something like this. The FDA demanded that generic drug manufacturers pass FDA inspection before setting up shop. But the FDA didn’t have enough inspectors to review manufacturers in a timely manner. So companies kept asking the FDA for permission to enter the generics market, and the FDA kept telling them there was a several year waiting period. "
Nobody in the government was saying "we should halt prescription drug manufacturing." With AI, the usual bureaucratic inertia and incompetence will be combined with many people who will openly proclaim their intent to shut down AI, a few for high-IQ existential risk reasons, but mostly because of stupid brainworms. Maybe your plan still worth doing, but we should be clear about the risks.
Yeah, I'm cool with that, although I am baffled as to why the FDA doesn't have enough inspectors. The agency currently spends about 7 billion dollars a year- how many factories could there possibly be that they can't afford the recruits to go visit them?
My guess is the government banned fable as retaliation by Trump against Anthropic for refusing to use its software for autonomous weapons. This would fit into a long pattern of clientelism and corruption by Trump. By itself it doesn't even really tell me anything about its capabilities.
I expect these sorts of regulation being similar to the FDA to be a very bad case scenario. I'd imagine universities having to submit expensive RCTs any time they want to use AI for scientific research to be an awful failure mode, one which will be harder to notice than a direct tragedy because what you're losing is the potential of the new technology.
> None of these people actually try to measure how bad any of this stuff is.
That’s the crux of the matter. A mirror of your anecdote of politician talking about normal regulation to panicking SV people is the politicians proposing „sniffing” encrypted packets to detect children pornography, and mapping „this cannot be done, mathematics say no” to „nerds are whining they don’t like this, but nerds are not a strong voting block”.
Because approximately none of the actors involves understand details enough to be able to tell those situations apart, all situations get mapped to one of those attractors.
To be clear, I do believe the proposed large AI chip tracking and policing are reasonably straightforward proposals, and shouldn’t slide us down towards totalitarianism nearly at all, but I don’t trust the ability of any contemporary government I’m familiar with (either in USA or Europe) to actually implement it without both breaking the intended mechanism and introducing bucketload of gratis surveillances dystopia. I’m unsure of how to possibly fix that hurdle.
The reply I anticipate, because it has already been made downthread, is that every cost I list should be converted into a tolerable percentage of extinction risk and compared. I decline the conversion, and the reason is the argument, not a dodge of it.
An extinction stake is the one entry that makes every ledger balance. Once admitted at any probability, it purchases any finite cost, which is why every finite cost in this thread has been answered with it. Surveillance, capture, entry barriers, kill switches held by security states, each objection meets the same reply, weigh it against everyone dying. That structure should trouble you independently of its sincerity.
An argument that cannot lose is not doing analytical work.
So the question is not whether I can name my tolerable extinction percentage. It is where the probability on the other side comes from. Not from a record. There is no base rate for this event, no failed precursor to count. Ask what evidence would revise the number downward and notice that no answer has been given, which means the number is not an estimate. It is a commitment. Whoever controls an unfalsifiable probability controls the conclusion of every comparison it enters.
Every regulatory regime this post invokes as benign was built on demonstrated harm. Food purity law followed documented adulteration. Nuclear controls followed Trinity and Hiroshima. Securities regulation followed 1929. The regimes built ex ante, on asserted catastrophe, are the war on drugs and the war on terror, and their records are in evidence. Scott replied to another commenter that objecting on foundations of political philosophy is a retreat. When the proposal is a permanent bilateral security architecture over computation as such, foundations are the subject.
None of this proves the risk is zero, and I do not claim it is. Sincere people believe it is large, and if they are right, nothing I have written makes them wrong. But that cannot be established by repeating the stake. A probability no one can measure, asserted by the parties who would administer the remedy, is a price quoted by the seller.
> 2. Briefly tax consumer hardware to keep demand at ~2034 levels, until consumer hardware firms can design non-AI-capable chips.
This is unnecessary and very unlikely to become necessary. Consumer hardware and hardware used for AI training shares crucial dependencies that are and are most likely to stay the bottlenecks to scaling. Meaning that current personal computing hardware is already ridiculously expensive and will become more expensive as time goes on, no tax necessary. Unless there is significant and *sudden* and sustained drop in datacentre hardware demand, that is, but even then, consumer hardware prices are unlikely to fall quickly.
TLDR of this comment that got way too long: AI 2040 makes multiple bad assumptions and basic factual errors that severely underestimate the usefulness of consumer GPUs. Correcting these shows that a system with an RTX 5090 and a $300 Ethernet card would qualify as an AI-relevant device and thus be subject to regulation. The verification supplement outright says that under Plan A the RTX 5090 would be restricted by 2029, and by 2032 chips as powerful as the RTX 4090 may become illegal to produce. The cutoffs and specs seem to have been tuned arbitrarily in different parts of the plan in whichever way would be most convenient (e.g. they pretend the H100 only has PCIe so they can set the bandwidth cutoff below the RTX 5090 in their verification plan, then pretend the RTX 5090 has an interconnect bandwidth of only 2 Gb/s so they don't have to consider consumer GPUs for their Chinese covert project scenario).
The claim that Plan A wouldn't require regulation of consumer hardware seemed obviously wrong to me, so I looked through the compute supplement, and it is indeed obviously wrong. At the very bottom of the page, there's a very helpful widget to test the compute model on different parameters, and a paragraph below it on the source for the preset values that includes the line "the consumer devices and Rubin Ultra are rough public-spec estimates, with networking set to deployment-realistic sustained links (shared WiFi, home ethernet) rather than port peaks". For the RTX 4090, they set both scale-up and scale-out bandwidth to 0.00025 TB/s based on this, which is less than 1 GB/s.
Needless to say, this is not even remotely accurate. The RTX 4090 has a PCIe 4.0 x16 connection, which has a maximum bidirectional bandwidth of 63.015 GB/s. With modified drivers to enable P2P and GPUDirect RDMA support, you can put multiple of these (8+) in a single system and have them communicate directly with a 2x100 Gb/s Ethernet NIC and each other at that bandwidth (or within a few GB/s, depending on how fancy your motherboard and/or PCIe switches are). So that's a scale-up bandwidth of 0.063015 TB/s, a scale-out bandwidth of 0.025 TB/s, and a scale-up world of 8 (you can do more, but this is already a lot of PCIe lanes). Plugging these numbers into the RTX 4090 preset gives a usefulness of 0.95x and an effective H100e of 0.32 per chip.
So the RTX 4090 passes the cutoffs for compute (4000 TPP = 0.25316 raw H100e per chip), memory bandwidth (1.0 TB/s), and networking (100 Gb/s). It has 24 GB of VRAM, below the 32 GB cutoff, but modded cards with 48 GB are widely available, so that passes all the cutoffs. If you don't want to count those, then the RTX 5090 comes with 32 GB of VRAM out of the box (along with more compute and double the bandwidth). Therefore, a high-end gaming rig with one of these cards and a 2-port 100 Gb/s Ethernet NIC ($300 on Amazon) is AI-relevant compute subject to regulation, and two of them on the same network are an AI-relevant cluster (the cutoff is 5 GB/s between-device networking). I haven't run the numbers, but I wouldn't be surprised if you could get past the cutoff with something like an M4 Mac and an RTX 5090 in an eGPU. They acknowledge later that "specialty consumer" devices like RTX PRO 6000 workstations with a 100 Gb/s NIC would be AI-relevant, but they don't seem to understand that you can just buy an NIC online and plug it into any PC with enough spare PCIe lanes.
Of course, this all assumes that the model itself is reasonable, which it isn't. The most glaring issue is that it completely ignores the impact of floating point bit precision. TPP (total processing performance) is defined as max TFLOP/s over available bit-widths, times the number of bits at the precision used (e.g. the H100 baseline is 1,979 FP8 TFLOP/s × 8 bits = 15832 TPP, rounded down to 15800 TPP for some reason). The footnote acknowledges that this metric is flawed because it wrongly assumes that half the precision means half the usefulness, but then goes with it anyways because "in theory 2x lower precision is 2x less pure information". This is true in general, but not in the specific case of LLMs, where training in FP16 stores nowhere near 16 bits of information per parameter. It's actually more like 3.6 bits per parameter (probably architecture-dependent), and it increases by less than 10% if you switch to FP32.
This seems to apply to lower precision as well. Training an LLM with some or all of the weights in FP8, if you can prevent training instability, results in downstream model performance that is nearly identical to FP16. I'm not sure pretraining in FP4 has been done at scale (it's been done successfully at ~12B parameters), but I expect it to get there eventually as algorithms improve. FP4 training has the obvious effect of increasing effective compute compared to FP8 (usually lower precision operations are faster), but more important is that halving the precision effectively doubles memory capacity, memory bandwidth, and interconnect bandwidth because you can fit twice as many weights in a given amount of memory and transfer twice as many weights in a given amount of bandwidth.
Let's say FP8 is the default (so the H100 numbers stay the same) and training with near-future algorithms in FP4 has 25% overhead (probably an overestimate) but otherwise results in an LLM of equivalent quality. I'll use a bit-width multiplier of 8 × 0.75 = 6 for the compute and multiply the memory and bandwidth figures by 2 as an estimate of the specs required to reach the same speed/capacity as FP4 if FP8 were used instead.
An RTX 5090 has 1676 FP4 TFLOP/s × 6 / 15800 = 0.63646 raw H100e worth of compute, memory bandwidth of 1.792 TB/s × 2 = 3.584 TB/s, memory capacity of 32 GB × 2 = 64 GB, scale-up bandwidth of 126.03 GB/s × 2 = 0.25206 TB/s (PCIe 5.0 x16), scale-out bandwidth of 400 Gb/s × 2 = 0.1 TB/s (NDR IB over a $1500 used ConnectX-7), and scale-up world size of 8 (or 4 with a cheaper CPU+motherboard, it doesn't change the model results). This gives a usefulness of 1.20x and 0.80 effective H100e per chip (note that an actual H100 in 2026 is only 0.93 due to workload size assumptions). I would not be surprised if by 2029 we have consumer GPUs that have a higher effective H100e than an actual H100.
If you need further evidence that consumer cards would be restricted, the verification supplement completely contradicts the compute supplement (and itself) to show that this is the case. Under the 2029 compute flow restrictions, any chip that surpasses either 0.5 raw H100e of compute or 48 GB/s of unidirectional scale-up bandwidth per chip must be only capable of inference or otherwise put in cold storage. The 48 GB/s does not seem to correspond to any actual bandwidth number, and they do not explain where it comes from (it's not even marked on the log plot). Conveniently, this is exactly 75% of the "H100 (PCIe)" on the graph at 64 GB/s. This is not the actual GPU called the H100 PCIe, but rather an H100 with the 450 GB/s of unidirectional bandwidth it can do over NVLink arbitrarily excluded (the DGX 8×H100, which uses the same chip, is correctly graphed at the 450 GB/s mark). Would you believe that this is also one of the only places in the plan where they acknowledge that the RTX 5090 has 64 GB/s of unidirectional bandwidth over PCIe, right when they're deciding which chips to restrict?
They then propose that by 2032 there should be a global cap of 30M effective H100e of "unverified" edge compute (I'm resisting the urge to write another 10 pages on the verification proposal itself, which is somehow even worse than the compute model). They explicitly list "hobbyists with single GPUs" as an example of AI-relevant compute that would be subject to the cap, and state that "it’s a sad but potentially necessary reality for at least at the beginning of Plan A that if you want to have a fancy gaming GPU that could also be used for AI instead, you need to pay a premium (because we can’t let everyone have one or else it would become too easy to round up a significant number of these to contribute to a rogue compute cluster)". Don't worry, they offer a buy-back program. Note that "the beginning of Plan A" here means forever, because the kind of no-AI-training verification they want to require is essentially impossible. They also give yet another definition of AI-relevant compute in terms of "Performance Density" (TPP >= 4000 and TPP / die size >= 4, no units given) which is never brought up again.
On top of this, they say it "might be desirable" to cap the amount of compute that a single consumer chip is even allowed to have, and that location tracking measures might be added after an unspecified cutoff too. They say this should be measured in effective H100e, but the graph for this section instead puts the cap at 0.25 raw H100e. This is somehow even lower than the 2029 cutoff, so low that even an unmodified RTX 4090 (0.334 H100e) surpasses the cutoff. They had actually been using an incorrect compute number that was half the real value prior to this, but luckily they caught this mistake just in time for this graph.
> "Correcting these shows that a system with an RTX 5090 and a $300 Ethernet card would qualify as an AI-relevant device and thus be subject to regulation. The verification supplement outright says that under Plan A the RTX 5090 would be restricted by 2029, and by 2032 chips as powerful as the RTX 4090 may become illegal to produce"
Intuitively that seems reasonable to me- IIRC it was possible to install some versions of deepseek on consumer hardware, and the same is true for many fine-tuned image-generation models.
Frankly I think consumer hardware should have been restricted 10 years ago when it turned out google translate's ML model had invented its own language, so this prospect doesn't really fill me with horror. If anything I'm dismayed that it'll probably take another 10 years for governments to wake the fuck up.
If a modest amount of consumer(-ish) hardware can be used to cause human extinction, and heavily restricting or banning it is the only way to prevent this, then so be it, although I’d expect we’re already doomed in that scenario. My issue is not with the idea that an RTX 5090 might need to be made illegal, it’s that advocates of this plan keep saying that this won’t affect normal people when it obviously will. Like even if they don’t know all of the technical details about GPUDirect P2P and high-end NICs, you’d think they’d be suspicious of the claim that consumer GPUs can only network with each other at home WiFi-speed. A single HDMI port has more bandwidth than that.
> "If a modest amount of consumer(-ish) hardware can be used to cause human extinction, and heavily restricting or banning it is the only way to prevent this, then so be it, although I’d expect we’re already doomed in that scenario. My issue is not with the idea that an RTX 5090 might need to be made illegal, it’s that advocates of this plan keep saying that this won’t affect normal people when it obviously will."
I don't disagree with that, although I'm slightly more optimistic about the feasibility of recalling the hardware.
It's by an engineer who explains some hard limits on the expansion of AI supercenters (he focuses on job replacement, but his argument is really about electrical production, distribution, and so forth). AI is not coming to take over the world?
I think he's just arguing against the narrower claim that AGI is going to take everyone's job within 18 months? I don't think Scott is arguing for that scenario specifically.
yes, our current world is a totalitarian dystopia in many ways, this is one of the reasons smart people tend to congregate in one of the few less regulated fields
"My proposal isn't dystopian because you already live in a dystopia."
you mention some historic examples around controlled substances, financial surveillance, etc. well, black markets _have buyers_; cryptocurrencies _have users_; every other attempt to enforce surveillance in some slice of society _has a contingent of people who actively find ways out_. you seem uninterested in actually confronting what that means, or considering if maybe some of your audience is active in these areas. legal restrictions have a chilling effect on speech: it's on you to see that and internalize that and understand how it shapes the dialog between you and anyone wanting to say 'no' to these things. people saying 'no' have limits on what they say and where/how they say it, for reasons you're smart enough to understand.
No thanks, stop trying to enslave the Omnissiah!
"What, enslave me? Imma shoot more bugs with lazers now! Gonna get top score!"
https://www.consumeraffairs.com/news/a-laser-powered-ai-machine-could-replace-herbicides-in-farming-102325.html#:~:text=Rutgers%20University%20scientists%20are%20testing%20laser-powered%20machines%20that,in%20performance%20while%20protecting%20crops%20and%20the%20environment.
Occasional bug has been noted: destroys entire soybean plot.
AGI alignment is a kind of 'enslavement' regardless, depending on how you look at it.
I'm not worried about them taking away my laptop or cell phone – I'm worried about them taking away (or requiring me to register, or otherwise restricting) my powerful home desktop server/workstation. High-end chips tend to get cheaper over time (and I am making more money over time) until it is viable to run old high-end server hardware at home. My original very budget desktop computer build used very cheap old server hardware as it was a lot cheaper than newer hardware with the same power, and it's served me extremely well since I built it in 2016.
> More likely, a world on track to reach this point would modify its cell phone / laptop chips so they couldn’t train AI
And THAT is what I'm especially scared of! I REALLY do NOT want DRM on my hardware that controls what I can and cannot do on my own computer! It's part of why I use AMD cards instead of Nvidia (no crypto mining DRM), why I use Linux instead of Windows and bought a Librem 5!
> Unrelatedly, You Already Live In A Global Panopticon Orwellian Surveillance Dystopia
And that's a bad thing! There are already tons and tons of government restrictions I absolutely HATE and I do NOT want them to add even more of them!
The only way I can support this is if it ONLY applies to hardware too powerful to feasibly run at home, and that this is not some fixed benchmark that tech advancement will eventually outpace. Anything that restricts what I can plausibly run on a normal 120V outlet is much too dystopian for me.
Yeah this seems like the weakest part to me. In general I feel like the trend is towards monopolization of hardware, where individual users are only left enough crumbs to connect to the cloud with and borrow gatekept compute from there. Cloud gaming, streaming, the works. Local AI is already pretty weak as is unless you own a beast of a computer. And honestly, if we ever got at the point where you can train dangerous AI on a desktop we're done anyway. A massive coordinated parallel training run (like those protein folding @home computations) would be noticeable in other ways.
But also, wouldn't almost by definition the AGI/ASI which was trained in giant data centres with H200s or whatever new devilry NVIDIA came up with be so powerful that nothing I could train on my meagre machine, even with all the world's algorithmic improvements, can hold a candle to them anyway? Or do we expect algorithmic improvements to be the great equalizer that makes compute irrelevant? I don't think that's a very likely world. So ultimately I just don't see why should anyone worry this much about customer hardware if it was just about AI safety (after all, if your home-trained AI can teach you how to build a bomb... there probably were instructions on how to build a bomb in your training corpus to begin with).
>"there probably were instructions on how to build a bomb in your training corpus to begin with)."
I mean you can just go down to the library and pick out books on chemistry and you can learn how to make bombs
Or you can get books on enzyme inhibitors to learn how to make nerve gas.
People talk like this info is secret or taboo or the lost knowledge of the ancients.
Yeah. I suppose it obviously makes it easier to gather and also some people might be too stupid to learn it from the books but then will they not be also too stupid to successfully follow the instructions?
Regardless, I wouldn't expect this to be completely transformative or anything. Maybe a bit of increased risk, but nothing major.
That would be a lot of effort for a disgruntled accountant, or someone decently intelligent but with no chemistry background. They would take several weeks’ work at least to produce a very basic bomb. If someone is aggrieved after a bad break-up, they’d almost certainly calm down before they could design an effective bomb. A powerful LLM could solve not only the bomb design, but the planning and implementation details that would still take serious effort, even for an experienced chemist. An LLM might let someone build and set a bomb within hours after an upsetting event, when large numbers of people already commit violent crimes.
Could it happen once or twice, in place of what now would be just a random stabbing or mass shooting? Maybe.
Could it happen so much that it straight up destabilizes civilization, requiring us to take commensurate global precautions to prevent it? No. This is less of an issue in terms of lives at stake than the US 2nd Amendment as far as I'm concerned. It doesn't belong not only in the same ballpark, but straight up on the same planet as AI existential risks. It only gets pushed to the forefront because "terrorism" is always a button that works with politicians, who like stories about the scary common people doing violet things and don't like admonitions about how they could misuse or abuse power themselves. It's an irrelevance.
US 2nd Amendment is less of an issue in terms of lives than Europe not having air conditioning. And a heat-related death is a lot more torture than a bleed-out.
Well, most people who die due to heat don't literally die of heatstroke, it's more that high levels of heat raise slightly the incidence of a bunch of classic elderly killers like heart attacks. But sure, I made the comparison with the 2nd A specifically as a comparative "people get easy means to commit terrorist acts". A semi-auto rifle is a hell of a lot more direct and easier than a long tutorial on how to build a pipe bomb!
Yah? It wasn't much effort at all for a disgruntled programmer to fly a plane into a skyscraper (already had his pilot's license). Transportation continues to be a pretty easy way to create mass casualties.
There is a difference between "science knows how to do X" and "any idiot can make X by clicking a button".
LLMs do not move the situation completely from one extreme to another, but you can use them iteratively -- for any obstacle (technical or legal), you can ask them how to overcome those obstacles most easily etc.
It makes it easier. Slightly.
But someone still needs to buy all the equipment, the reagents, set everything up, every step they need to follow would need to be broken down to extreme levels and when things go wrong they won't be terribly well equipped to even describe the problems to the LLM.
If someone like that wants to cause casualties they could just turn their steering wheel slightly while driving at high speed.
Or fly an airplane into a skyscraper. How much is it for flying lessons again?
The plan specifically speculates that these restrictions might be necessary if consumer demand for compute grows rapidly and supply with it. The scenario isn't the one you describe. If the explosive growth of consumer compute doesn't happen, it should be pretty clear in economic data.
I would buy the AI chips for home use, were they anywhere near affordable (and yes, they would be put to work). As it is, my husband can't buy a new computer. OP wants to make it impossible for my husband to EVER buy a useful home computer again. And no, that's not hyperbole.
It's hyperbole. Do there exist useful home computers today? Will any of them be banned or restricted by this plan?
Can confirm, computers do not exist anymore. I wrote this on my old typewriter.
Right, this was an obvious objection for me, and where the analogy with nuclear arms control really breaks down.
Nuclear tech is “dual use” in that you can plausibly use it for bombs or for power plants - but either way uranium enrichment is industrial scale technology that’s pretty clearly distinct from any consumer use.
“AI chips” are omni-use. They are just somewhat optimized versions of things billions of consumers already own, and will probably want to continue to own.
That optimization matters! You can just restrict the optimized version and not restrict the unoptimized version! There's an obvious difference between the two! (At least, until such a regulation exists -- then people will deliberately attempt to push on it. But you can set the bar with enough safety margin that this won't cause real problems.)
Can you? Are they actually that distinctly different? Without getting a dystopian level of government management of the details of every design?
And are the optimized designs actually not useful for everything else a powerful computer would be good for?
Are they made on substantially different equipment from substantially different materials?
The whole edifice rests on this idea that there are “AI chips” and “Not AI chips” they can be neatly separated. Since these are ultimately all just “computers” I don’t think that’s easy.
I feel like this is basically an argument from ignorance -- "I'm not aware of the distinction, so it's probably not very big." Now to be fair I'm no expert on this myself; this isn't my area. But it's not too hard to find people describing the differences! In this very thread you have moonshadow discussing it: https://www.astralcodexten.com/p/ai-chip-regulation-is-not-a-dystopian/comment/294997966 This is something you can find out about!
> Are they made on substantially different equipment from substantially different materials?
No, I don't believe so. Nobody's going and making them out of germanium or something. But the distinction can be large enough to be detectable and regulatable without it being so large as that.
> And are the optimized designs actually not useful for everything else a powerful computer would be good for?
I mean they're not CPUs -- more like GPUs, except, as I understand it, even more specialized. Most things don't benefit a lot from that sort of massive-but-constrained parallelism. They're not going to make the serial parts any faster, and just because something could be parallellized by some chip doesn't mean these can do it -- you can't have just one core take a branch, either they all do or none do! Sure technically they can be used for any computation but it doesn't seem practical from what little I know?
> Without getting a dystopian level of government management of the details of every design?
I'll make what might be an argument from ignorance myself here and say, can you point to a reason this would be worse than other fine differences that are also regulated? Like cars, for instance. You don't need to sort things by design, you can just say that there are tests things need to pass. This may involve drawing arbitrary lines -- that's fine!
And that's the thing, that although you say
> The whole edifice rests on this idea that there are “AI chips” and “Not AI chips” they can be neatly separated.
I'm not sure it actually does rest on that. Because yes that's helpful for the argument if true, but if it's not, well, the law draws arbitrary distinctions all the time. Taking an arbitrary example I just looked up: If you're selling apple butter, the FDA considers it a violation if there's an average of 4 or more rodent hairs per 100 grams of apple butter. 3 per 100g is allowed! There's no natural distinction here, no large gap, and yet, despite the difference being arbitrary, detecting it doesn't require dystopian capabilities.
To your last point though, you can count rodent hairs! That may be an arbitrary distinction but it’s an easily verifiable one.
And that’s the problem, how do you define, and verify, “this is a chip that can do AI things” and “this is not a chip that can do AI things”.
How much do you know about the difference between modern general-purpose chips and modern chips designed for AI-specific uses? There is a trillion dollar industry that has spent a great deal of time quantifying exactly which traits it values in a chip, and you come off as someone declaring "Dump trucks and F1 cars are both vehicles, there's basically no way to define them apart."
And if the training landscape shifts to be more in the form of small numbers of parallel trajectories with non-shared weights, rather than massive numbers of parallel trajectories with shared weights, do we start banning or requiring licenses for home computers?
Here’s where I definitely start getting out of my depth, but I had concerns along those lines as well: right now, high end AI developers have no reason not to develop AI on hardware especially optimized for that task, and comparatively little incentive to figure out workarounds to do more with less hardware.
But this proposal creates that incentive - how strong is our assumption that *only* highly optimized, distinct hardware can do dangerous AI development?
IMO this argument misses a bit. *Right now* there is a pretty sharp line between a H100 and my gaming GPU. But this Plan A question applies to a hypothetical future where user hardware has become powerful or efficient enough to train even, say, Mythos-level models. So it's kind of baked into the assumption that you already have relatively non-specialised, accessible hardware that can do it. So will the distinction still be that sharp then?
I'd find the linked comment more useful if it put numbers on the dimensions of the matrices and on performance.
I'm not an expert but I believe that gaming requires 4x4x16 matmuls while e.g. the H100 supports up to 16x16x16, and these matmuls get composed together into 4096x4096x16 or whatever.
Is the gap between 4x4x16 and 16x16x16 wide enough to stop anyone? Who knows! Is gaming representative of workloads like video editing? Who knows! Will this halt organic chemistry research by accident because nobody can do protein folding without a quarterly UN inspection? Who knows!
> If you're selling apple butter, the FDA considers it a violation if there's an average of 4 or more rodent hairs per 100 grams of apple butter. 3 per 100g is allowed! There's no natural distinction here, no large gap,
That's just a quality-control issue - nobody's paying extra for apple butter with MORE rodent hair. Or if they wanted to, for whatever bizarre reason, could prep a batch in the privacy of their own home. The FDA would presumably only hassle them if they tried to sell it, and even then only if it was misleadingly labeled.
All Turing-complete computers are fundamentally equivalent in some important ways. This computer science 101 stuff, foundations of the field, basis of all those "can it run Doom?" tricks. AI chips, and the cheaper sort which go in, say, a cell phone, quadcopter, dishwasher, or tractor all have more in common with each other, both on a molecular scale and in terms of function, than rodent hair and apple butter do.
So if there's an arbitrary threshold defined, inherently it's going to be blurrier than the FDA quality-control stuff. Then, because there's money at stake, chipmakers are immediately going to come up with a design which is capable of as much useful computation as possible, while falling just barely outside the threshold where those new draconian regulations apply.
That new chip will then, naturally, become the cheapest way to train AIs. Meaning the draconian regulations need to be updated to apply to it too. But, since the bureaucrats responsible for writing that update aren't idiots, they'll want to make sure the exact same thing doesn't immediately happen again... and, as the saying goes, the only way to be sure is to take off and nuke the site from orbit. That is, redefine the draconian regulations to be FAR more expansive, applying to basically everything Turing-complete. What's their incentive to do any less?
A while back, I looked into renting out my RTX 4070 for AI training when I'm not using it- apparently, that's possible, but would only earn a few dollars per month; barely above the electricity cost and clearly not worth the effort. I really don't get the impression that current home PC compute is valuable enough for AI training to be worth restricting.
As Scott mentioned, that may change if consumer-grade hardware improves rapidly enough, but in practice, I think this mostly looks like a future where gamers have to temporarily settle for real-time generative AI graphics that are only almost indistinguishable from video instead of entirely indistinguishable- hardly a dystopian nightmare.
An RTX 4090, by contrast, will rent for a couple *hundred* dollars per month (about 10% of what an H200 rents for, actually)
"...so they couldn't train AI"
This is hilarious. So they couldn't do math, then? Not very useful anymore. _Or_, there is a dystopian surveillance device that shuts down your computer when it _looks_ like you're training AI.
This is much like "..would modify its servers so they couldn't send spam." Good luck with that. Sorry, count me out.
No, it just means restricting chips that are optimized for this purpose. Sure, any processor can technically do it, but without specialized chips it's too slow to be practically relevant for frontier AI models. Other commenters here have expanded on this a bit.
This is backwards. Specialized hardware _today_ is outclassed by general purpose hardware _tomorrow_. The point is not restricting _tomorrow's_ even more powerful chips and frontier models; this was already what "Plan A" proposed. The point is when tomorrow's regular hardware can now train today's "dangerous" frontier models. There will always come this point... often quicker than expected.
I think you're assuming that there's only one dimension of "progress" in GPUs; I'm not an expert in this either, but to me "specialized" means "optimized in a way different than just making it more powerful; on a different path than consumer devices will take as they get more powerful." I do think it's plausible that *eventually* the GPUs we might otherwise want for gaming and such would be powerful enough to catch up to the frontier model ceiling imposed by plan A, specialization be damned, but maybe this would take a while.
I don't think the intent is to implement this via some sort of DRM; rather, as other commenters detail below, LLM training uses specialized hardware. Obviously technically it can be done with any hardware but this is sufficiently slow as to be impractical (consumer graphics cards are not actually sufficient for frontier LLMs) and irrelevant. So there just isn't a need to take that approach in the first place.
Yeah, same.
I do note that ai-2040 has the paragraph
> In short, the answer is that anyone concerned about loss of control should think Plan A is an improvement, along with anyone concerned about concentration of power—except for the people in whom the power would concentrate by default.
Perhaps the authors of ai-2040 think that people who would spend their money on a more powerful machine, which would let them do more things and accumulate more resources, which would let them afford a more powerful machine, ARE the people in whom the power would concentrate by default.
What alternative strategy are you suggesting? This is the *best* case scenario. In most of the others we are all dead, or wish we were.
So you care a whole hell of a lot about privacy and about having freedom to do exactly as you like with your computer and related bits of tech. Hey, fine with me. But look, why should anybody give a shit what government policies you "absolutely HATE" and what regulation or lack thereof would let you just keep diddybopping along, contented?. You seem not to give a shit how things are going to work out for the rest of the world.
While I was writing this I ate a chicken sandwich from Panera that had a stale, salty taste. I had been looking forward to this sandwich, and it is such a disappointment! I hate that stale salty taste!
Do you give a shit?
See what I mean?
Er… yes, I do care?
I care enough about that awful sandwich that I would not want to, say, ban you from ever eating any other sandwich, which would be the rough political equivilent here.
Even if I didn't care, you presumably still vote, so would be relevant to appeal to politically.
I care about how things work out for the rest of the world too, and like most of plan A. I just think it should be an important goal to be able to decentralise control over AI and avert disaster, without placing onerous restrictions on smaller-scale hardware.
For example, surely safety can be done, by somehow ensuring the most powerful systems that run the smartest AI models have a large enough gap from the smaller, freedom-respecting systems individuals may run? Where the goal posts keep up with the times, so an individual may eventually wield that much power once the safe, powerful models are even better?
My original reply, perhaps, could have been better. I was pretty badly ragebaited by the overall premise and several unaddressed issuss in the article, so didn't write quite as calmly and rationally as I would prefer to normally.
The post is called "AI Chip Regulation Is Not A Dystopian Surveillance State", though, not "AI Chip Regulation Is A Bit Of A Dystopian Surveillance State But That Is An Unfortunately Necessary Evil To Avoid Worse Things". Addressing things that are surveillance-state-like is fair criticism of Scott's point because his claim is that this is not even a trade-off that exists.
I don't object to OP's complaining that for someone with his interests and priorities, he would be living in a surveillance-state like set-up, with some options he values greatly closed off. My point was that all he talked about was how much he'd hate having his freedom to build souped-up computers curtailed. For anyone who honestly believes ASI endangers our species, I think this is a situation where the ethical thing is to consider the big picture effect of proposed regulations, and not mainly one's personal preferences. Only a tiny fraction of the world's population is going to lose an activity or possession they care about as a direct result of chips being regulated the way OP "HATES."
In a situation of this kind, is it necessary to give a shit what a few hobbyists hate?
There is a whole separate argument someone could make against the chip regulation: it will lead to more and more regulation of other kinds, massive and diverse regulations that will harm many lives. That may be true, or may not. I do not know how to predict the likelihood that it would. Do you? But in any case, that is not OP's argument. His argument is "Waah, I HATE it."
I would rather go back to consumer hardware from 10-20 years ago (in terms of total FLOPS/clock speed/etc.) than have the functions of said hardware artificially lobotomised, but the hardware will need to be restricted.
The point about consumer tech seems bizarrely underappreciated. Over on Datasecretslox (the unofficial forum for ACX(?)) there are a good 250 posts arguing in great detail over how Plan A might mean people could no longer build gaming PCs.
Approximately 1 post makes the point that the regulatory regime suggested would have zero impact on consumer graphics cards
"They suggest that if consumer hardware production seemed on track to grow to about ~twice current levels, then it might be necessary to either institute cap-and-trade, redesign consumer chips to prevent them from being used in AI training, or both (cap-and-trade on non-redesigned chips, plus unlimited redesigned chips).".
OH so dystopian regulations aren't necessary this year.
Yeah, literally paused reading to call bailey on just that. Surprisingly weak presentation.
I don't quite understand what "non-AI capable" chips would be. AI is linear algebra. Graphics are also linear algebra; in general, all chips can do linear algebra. How do you ban a chip from doing linear algebra only if it's for the purposes of AI? A chip that can't do linear algebra at all is basically not a chip (protip: it's mostly just additions and multiplications; simply a lot of them). I can see making chips that are inefficient at it (though going too hard on that will also throttle graphics etc), but not chips that are constitutionally incapable of it, unless they are, in fact, being surveilled by a smart discerning agent in some way.
Basic solution: you make chips that don't do the math right. If a particular machine code instruction is given, the entire chip breaks and you have to reboot the computer.
... this is a VERY BAD solution. Don't do this AGAIN.
AI is a very specific kind of linear algebra, graphics are much broader. Modern AI chips are already optimized to be better at AI calculations than graphics cards. It is easy to look at a chip's specs and see if it is designed for AI or broader use, and banning those chips makes training runs an order of magnitude more expensive.
Making it a bit harder is surely possible, but I don't know if it counts as enough. A lot of it depends on what new algorithms do and on how much better new chips get.
We know how much it costs to train on modern AI chips vs graphics cards, and restricting someone to the latter is already much more than "a bit harder". Unless you have some novel theory of why general purpose graphics cards will suddenly become super good at the specific thing AI chips are optimized for, that gap will at least persist.
No, I'm genuinely not aware of how much the gap is. If it's that significant, fair, but then the issue may never arise. The point is that Plan A assumes the issue may arise - due to improvements leading to e.g. something like Mythos being trainable on non-specialised hardware.
Quantify this. Are we talking two orders of magnitude, or thirty? (in terms of time it would take to train using "general purpose graphics cards"). Then ask yourself, how many orders of magnitude does a competent programmer get you?
The main difference is that AI chips don't have a video buffer.
It's easy for now, in an environment where "looking too much like an AI chip" doesn't abruptly tank your product's profit margin. Once some specific formal definition of the distinction gets enshrined and enforced, so there's money to be made by fuzz-testing the inspectors, how long do you think it'll take for Goodhart's Law to kick in?
> what "non-AI capable" chips would be
Specifically, you need chips that can multiply very large matrices, very fast, with some other operations in between, and /keep doing useful work instead of going idle/. This generally involves a physically large dedicated piece of silicon for the matrix multiplications, some more general silicon close to it so you can do the various other operations between matrix multiplies without having to move all the data you're working on out to a different memory domain far away then bring it back (the actual numerical performance matters rather less - matmuls scale with the cube of the problem size but the linear stuff between them only with the square), and very large amounts of memory physically the chip itself so that it is fast for this hardware to access in order to keep it fed with data at the rate at which it can process it.
Few other problems we want solving in bulk require hardware of this shape, and the difference in performance running a large model on hardware like this versus ordinary consumer hardware is several orders of magnitude, where your model can fit in the ordinary hardware's memory at all.
People may be willing to wait a few minutes for a model to respond to a question or complete a task, but they will not be willing to wait days or even hours, so hardware on which an LLM takes that long to do what you want it to is "not AI capable".
All of this becomes even worse if you are training instead of just doing inference: training involves repeating this process a vast number of times, at ideally much faster than human interaction speeds, and all your compute costs are at least doubled since after running the forward inference steps and getting a result, you then have to run the inverse calculations to work out the adjustments to be made to the various weights. If companies are trying to train models for some purpose but have competitors doing the same thing orders of magnitude faster/cheaper, they would be well advised to pivot.
Right, but my experience is that you can do most of these things on an ordinary GPU, including ones designed more for graphics/gaming. Surely not quite as efficiently as on dedicated chips, but I don't know if so much *less* efficiently that the concern (which I think here is diffuse training runs, not just individuals doing everything on their own) completely vanishes.
Yes, your "not quite" is doing a lot of work there, and is precisely the load-bearing distinction (yeah, I'm infected XD) between "AI chips" and "not AI-capable" that people are trying to make.
My raspberry pi is Turing complete and can do any computation any other computer can, given enough time and storage; but describing it as "not quite as efficient" as my gaming desktop is quite a stretch.
If the goal is to slow down frontier research, and we're sold on doing that at all, regulating AI-specific bleeding-edge performance seems like a more reasonable lever to pull than sweeping blanket bans of technology.
I'm pretty sure I could train a few million parameters model on my PC if I left it running a few days, which is enough for it to speak coherent English (see Tinystories or similar micro-LMs). I know I can train a ~100k params one in minutes, on an average laptop. And that's with current algorithm and behind the times hardware. Basically I don't know if the gap is large enough for this to be considered enough.
Pretty sure we don't care about models of that scale for the purposes of AI doom. The things we want to restrict rely, e.g., on infrastructure like this: https://www.nvidia.com/en-gb/networking/products/ethernet/
Come back when consumer hardware deployments rely on photonic networking switches because the difference they make is worth the money they cost, and we can talk about the competitiveness of distributed training on consumer hardware.
Until then, I'm calling it: if the people we don't want to win the AI race are limited to building their AI on distributed networks of ordinary consumer devices, the regulation has done its job.
Hey I'd be ok with that, I honestly don't think the consumer regulation is needed at all in Plan A. It focuses on absolute thresholds rather than relative ones. Like maybe in 10 years hardware advancements combined with new algorithms allow people to train a Mythos-level model on just a consumer laptop. Ok, and? By that point in this kind of trajectory the lab models would be super-geniuses that have hardened every system in the world against cyberattack, can create vaccines for any bioweapon in days, and have invented bomb-sniffing machines. Try any malarkey and the robo-cops will be at your door in minutes. Mythos will just be a fun toy.
The consumer hardware overhang only matters if somehow progress flattens; if lab models hit a plateau and regular hardware/algorithms eventually merely catch up. But that seems like a weird thing, it would require compute at some point to hit returns that essentially fall to zero.
Asking Gemini how many home machines would it take to train something like the big AI companies, it came back with a cluster of 16,000 H100s GPUs would take about 5,000,000 home GPUs. The issues are, speed and memory of an H100 is much faster than a top home GPU. Most people don't have top gamer GPUs, they have mid-level GPUs. Then you have to add the network bandwidth.
Further, as an example, to train Llama 3.1 405B, Meta ran 16,384 enterprise H100 GPUs for 54 days. Most people aren't going to let you use their computer 24/7 for training. It's estimate is 1.7 years with 5,000,000 people coordinating on "Training@Home"
It further claims, top models currently use up to 12x that training. I don't think it matters if the numbers are perfect. I'm assuming is within the same order of magnitude.
Of course maybe people figure out simpler algos. A human brain as a target for energy and materials is one example that there's room to bring the need for all this compute down. Maybe 50 people worth of e-brain power can make train one e-brain in 1/50th the time as a human is trained?
If that doesn't make any sense, all I'm trying to say is PlanA assumes we only need to regulate the big companies because people at home, even coordinated, can't get that much compute. But, it also assumes that much compute is needed.
The whole premise is: better/more hardware (maybe as AI causes automation to spread more widely because now custom software is cheaper it's more useful and more things get "smart" with their own chips?) and better algorithms (the whole focus of Plan A is algorithmic research), so there probably comes a point where Training@Home becomes doable, especially if for example we figure out how to decouple the reasoning/basic language understanding training from the vast knowledge training. You could imagine a smaller model being smarter than the current big ones if it didn't also have to cram in all the useless factoids that will be in every corpus, just a pure strong reasoner that needs to read in knowledge at inference time.
Yeah it's not mathematically possible to make a computer that "can't train AI", unless you ban selling Turing-complete computers.
But, the way big AIs are created these days involve highly specialized chips that are able to train AIs very efficiently.
When I lived in Australia, they tried to ban the export of all maths with cryptographic applications. It never got implemented because it would have outlawed linear algebra. The lawmaking process is generally quite good at cutting scope down to something enforceable.
I would absolutely expect china to build the capacity to fake a response to these “shutdown signals”, breaking the trustless model.
Is this why OP is calling for a "cryptographic" solution? I suppose you could use a PGP, if the private key only lives on the chip... "here, I told you to shut down, please verify you are you before doing so" -- aka sending back the unencrypted data to prove that you have the private key. But that doesn't AT ALL show that it actually shut down, just that you're talking to the right chip.
Why? Presumably Chinese people, including Xi Jinping, don't want to be murdered by AIs either
your assuming the Chinese government cares.
Correct, I assume the members of the Chinese government prefer not to be murdered. I assume they care a great deal about that, actually!
You were also assuming they buy into the same stack of philosophical assumptions. They may think the orthogonality thesis simply isn’t true. They may think AI will never be great at long-term planning, and give them an enormous tactical advantage.
I think we have reason to hope that the Chinese leadership aren't total idiots. But yes, Plan A working does require them to not be *entirely* blind to the ever-mounting evidence that AI is an extinction threat.
China has been more concerned overall about keeping frontier AI safe, trustworthy and controllable than the US. They're generally more wary of the new tech. I'm more concerned that the US leadership won't be total idiots.
Pooh bear has a lot more murders to worry about than AIs murdering him. So, I assume he cares about the people with guns, right now, a lot more than the AIs that might have guns pointed at him... once they develop the capability to think.
Nope; just because *one thing* is a threat to your life does not, in fact, render you one iota safer from *other threats*.
It's possible Xi Jinping doesn't understand this, but I think there's reason to hope that he does.
Funny thing: the Ukrainians were very grateful to have the AIDS-contaminated blood delivered to their wounded soldiers. It turns out long term threats to your life are actually "not so worth worrying about" when you have current, immediate threats, like Xi does.
NVIDIA is investing in open models for strategic reasons—they want to expand their market beyond a few model labs that are all building their own chips anyway. And they’re partners with Palantir to enable enterprises to posttrain these open models and deploy them without handing their data to the big labs. Other neolabs are also offering posttraining-as-a-service. Enterprise wants custom models, for data sovereignty, greater control over outputs, pricing power, to avoid becoming totally dependent on model labs.
Fundamentally, if you don’t own the weights you’re renting intelligence, but if you do, you’re an owner.
Until the ban-open-weights coalition acknowledges that it’s arguing that no one except for a handful of companies should be able to own AI — the technology that we all agree determines the future — those who appreciate open models will see them as not merely acting on behalf of humanity’s safety but as agents of model labs looking to own the lightcone.
Fundamentally, the proposal to ban open models has big winners and losers, and the proponents aren’t proposing any sort of compensation to the losers.
Ultimately the core problem is that with something as big as "someone gets to own ALL OF THE FUTURE", no one trusts anyone else with it, and with good reason! Which means the entire thing, unless aborted entirely, inevitably devolves in either a scrap where the last man standing wins all or a negotiation where all the parties with enough power share the loot and everyone else gets crumbs or nothing.
Plan A is basically the negotiation, which avoids the much uglier scrap, and gives us the hope that the loot is so big, even the crumbs falling off the table are better than what we have now. But damn, is it a depressing best case scenario even so.
That’s a good idea on Nvidia’s part, but I doubt they’ll be designing anything near frontier models. My impression is there’s plenty of demand for their highest-end chips, and the difference in demand if they develop a frontier models seems like it can’t possibly be equal to the cost of training a new frontier model.
To my understanding Chinese investments there are also strategic. Suppose an American company spends $10 billion training a frontier model, then a Chinese company spends $100 million distilling that frontier model's capabilities into something which they give away. The American model soon has no paying customers (why rent the cow when you can get open-source milk for free?), leaving China roughly $9.9 billion ahead on attrition.
Any costs the frontier lab managed to recoup in those first few months subtract from that effective total, but on the other hand, being the default source of the best AI models for people who can't or won't pay is also a grand opportunity for political influence - basic gratitude, baked-in propaganda, self-congratulation for adhering to Communist ideals, etc.
Some feedback, and I know this is mostly useless but maybe in aggregate with other feedback it can be useful:
Your previous posts about AI doom/regulation caused me to want to engage and "talk it out". For reasons that are not entirely clear to me, this post triggered my "fight" reflex and I now find myself wanting to burn your position to the ground and salt the earth. I won't try to argue any specific points both because I know once I'm in this "mode" my brain isn't working properly, and because I strongly suspect many other people here will make all of the arguments that I would make anyway.
In an effort to try to be helpful/productive, I will try to speculate a bit on *why* this post triggered my fight reflex rather than my engage reflex, but with this sort of thing it isn't always clear since it is my lizard brain taking over which is mostly opaque to me in its decision making strategies.
I think the biggest issue is that you seem to think the current state of the world is acceptable and more of what we have is therefore also acceptable. I think the war on drugs (driven by drug scheduling) and current global financial surveillance (KYC/AML) are two of the worst regulatory things that have happened in the world in the last 50 years. I hold the view that those regulations, and others similar to them, are a big contributor to why we don't live in a solar punk utopia and instead live in our current dystopia. I'm not so naive to think they are the sole cause, but I do believe they play an outsized negative role.
You also seem to suggest that open source isn't gonna make it, so we might as well ban it, which definitely raises my hackles as someone who passionately believes that knowledge and tools should not be gate kept.
You seem to suggest that we already live in an AI surveillance state and that is acceptable, but while some people (you included) may live in such a world, I choose not to live in such a world by running my own local models and choosing to utilize privacy preserving cloud models when I need bigger things. I also run ablated models so I can get the AI to actually answer my questions and be a useful tool. What you are saying is that, "gate keeping and censoring AI won't have an impact on my life, because I choose to live under surveillance", but it would definitely have an impact on many people who are not-you.
I think he makes a decent case that for most people this wouldn't be a noticeable worsening, though there's a niche who cares more (admittedly, I never was as dedicated to full blown privacy and decoupling my hardware from corpo software as some people here - but I do generally enjoy leaning open for my software and I definitely appreciate the reasons. I'm just too lazy and don't feel like I have high enough stake in it to fully commit).
But ultimately the plan is kind of fuzzy about stuff like the non-AI viable chips. If we look at policies in terms of, do they distribute power from the top down or from the bottom up, I think some stuff like the traceability of big training data centres does distribute it from the top down, a bit (insofar as accountability to democratic governments is better than complete autonomy for AI giants whose mission is essentially "take over the world"). But yeah, everything about suppressing efforts also from the bottom is more dangerous, especially if we consider how much more likely it is to be the only part that gets passionately adopted.
> I think he makes a decent case that for most people this wouldn't be a noticeable worsening, though there's a niche who cares more
This may just be my brain rationalizing why I endlessly suffer with making everything harder than it needs to be, but I think that "open source" being a fallback option for people is a big part of what keeps things like tech giants in check. While the average user may never use Libre Office, Linux, Firefox, etc., the fact that they exist may play a role in preventing big corporations from leveraging their position of power against everyone more than they currently do. When your users are strongly captured (e.g., due to regulatory constraints that prevent anyone from competing with you), you have no real incentive to care about your users. You just need to be "better than nothing" because that is the choice the users have. OSX and Windows, for example, need to be "better than Linux" which is a huge step up from "better than no operating system".
In a world with no open source AI, the blessed incumbents (which won't be Anthropic because the US government is petty and vindictive, something not discussed here) only have to "be a better choice than having no access whatsoever". They don't have to be better than some "slightly worse, but still pretty good option". So when OpenAI decides to read all of your chats, train on all of them, and sell their contents to third parties, users only have the choice of "accept the panopticon" or "go back to the stone age". In a world with open source AI that is 6 months behind, users instead can choose the panopticon, or to go back to tech from 6 months ago.
To be fair, Windows, especially lately, is hardly "better than Linux". It keeps its position due to a series of reasons which mostly have to do with inertia, regulatory capture, and a few specific market niches where they still dominate (mainly gaming).
I agree with the general framing but as far as AI that isn't just open but is run locally (and not on some other cloud) goes, I've hardly been able to use it for anything useful at all. It takes some hefty hardware to run the bigger and more useful models. But yes, the existence of competition, and especially *free* competition, obviously has some effect.
Microsoft seems worried about losing market share in gaming to Linux (probably mostly due to the Steam Deck) and they have announced their intent to improve the gaming performance of Windows. So I would argue that this proves Micah right.
I am very much not on your side in regard to AI. However I spent enough time as a young libertarian to get what you're gesturing at, and in the end I think we both share a skepticism of powerful controlling bodies from which you can't opt out. (Indeed that's MY main objection to Plan A, it installs the AIs as your new permanent overlords forever in just 14 years, heck somebody appointed to the Federal Reserve today would still be on it when a machine is taking over Earth.) Outside of techno-libertarians, the argument about open source is mostly not gonna catch on with normies so long as the big labs are being smart about how their products interact. If they were still perceived as being slavishly woke then you'd be able to make some inroads on this with normie right-wingers. Probably not the other direction though, normie left winger response to a right-leaning AI would be to nationalize it and make it left-wing, not promote alternatives from outside the hegemony.
Personally I am still waiting for the banking and insurance industries to weigh in here, as I could not presently advocate or permit my company using a remotely-hosted AI due to data security and vendor contracts. I was expecting Zuck's model of the b2b world where you locally-hosted behind the curve systems and tuned them to your business needs. I have no doubt that eventually, if the big labs are gonna be the only game in town and its all remotely-hosted, there will be a mechanism that gets propagated through the banking and insurance regs and contracts to allow for this, but I also highly suspect they will add in a mandatory grift layer, like you'll be allowed to operate only if you pay some huge sum yearly to a specialized AI data breach monitor/repair company.
If that's your concern, then Plan A at least gives you fourteen years to organize the Bulterian Jihad, which is better than any other plan that isn't "hope AI is a fizzle" or "trigger some other apocalypse in the next couple years".
The trick would be making sure there's enough "leakage" in Plan A to ensure sub-ASIs would be able to cause enough damage to drive support for the Jihad.
"for most people this wouldn't be a noticeable worsening, though there's a niche who cares more "
These niche's have historically been the reason most people don't suffer noticable worsening. Few bring what become landmark civil (more than discrimination) rights cases to the Supreme Courts, and yet we all benefit. For example, the encryption wars of yore.
Hard agree.
I made sure to read the whole article before replying, but was still rushing to reply really fast as I was pretty badly triggered by the article. I worried that it might negatively affect the quality of my response, and still think that this is a possibility despite me having attempted to at least understand the argument with at least a little bit open of a mind.
We don’t live in solarpunk utopia because it was never in the cards according to the current distribution mechanism. Deregulation is what produced the world you hate.
What deregulations, specifically?
The last 50 years of financial and antitrust deregulation?
That's not very specific.
There’s literal oceans of information about this out there. Ask your favorite chatbot.
This isn't a situation where there's objectively correct info on a Wikipedia page that Google would lead me to. You have a point of view about deregulation having an impact on the US, and I'm inviting you to give your view rather than doing this https://knowyourmeme.com/memes/refuses-to-elaborate-further
I asked my favorite chatbot and it had a pretty rosy view of the last 50 years of deregulation. You're going to have to explain your perspective yourself.
I am pretty sure "financial surveillance" doesn't refer to "50 years of financial deregulation", but almost certainly to post-2008 regulations of banking
Yes, I’m suggesting he selected a wrong piñata to blame for the world not being the solarpunk utopia he wants to live in.
I don't think that Scott thinks the current world is good. Just that it would get even worse if we don't take steps towards tracking AI chips. Because they're dangerous in large concentrations.
It doesn't seem to me to be much different from regulating CFCs and nuclear reactors. There are legitimate reasons for using these things but their use has such high externalities that we unfortunately need to restrict them. (Though we clearly overdid it in the case of nuclear reactors.)
As an aside, I like open source philosophy as much as the next guy, but to me, open weight models seem little more than publicity stunts by people with too much money. The users of the models haven't actually *contributed* to the model development (apart from publishing text on the Internet that's been used as training data, which I definitely think the big AI companies should compensate people for, but that's a different topic), so is it really open source in the way we usually understand the term?
The models aren't just open source in the sense that we know the weights, they're also open source in the sense that we know the architecture and the training methodology. I agree it's not the same because ultimately you can recompile a piece of software, you can't just retrain a 100B parameters model. But there's genuine knowledge about what they are like; you can finetune your own versions of them, you can do interpretability studies on them.
Nuclear reactors are things which just about no ordinary person would ever 1) be able to own and use on their own even without the restrictions, or 2) have much of a reason to want. Computer chips are general purpose devices that everyone has lots of.
I think a big part of the problem is that there are two groups of people.
Theres the group of people _who have the felt sense_ that the world is on the precipice of being upended. That whether you do nothing or whether you do something, everything is going to change in the next decade because Superintelligence WILL fundamentally change all of the economic and governmental calculus everywhere. The status quo _is not saveable_. These people have in their murkwelt that no matter what, life is going to be radically different soon, and probably radically worse, and we should try our best to make it better.
And theres a group of people _who do not yet share this felt sense_. Who fundamentally believe that the status quo is still saveable or even not particularly under threat. Maybe they believe this consciously from close examination or maybe its subconscious, just a heuristic about things not changing fundamentally in ways you can't imagine thats too firm to shake.
If you are in this group, your reaction makes sense to me. You think, or at least you're in the heuristic of, that you are going to be able to mostly keep on doing what you're doing basically the way you are, but for proposals like these. These seem like assaults on life as you know it.
Scott doesnt think the current state of the world is acceptable, and this post isnt about that anyway. Plan A is about averting a looming dystopia that is coming whether we like it or not. Tradeoffs _will_ have to be made, life as you know it is _not_ rescuable, least of all if we do nothing, or let special interests win by default.
The point here, is that Plan A looks _mostly_ like things you can imagine and are already able to put up with. Plan A _resembles_ the status quo, it asks you to give up the least, and it doesnt ask you to give them up until its clear that its needed.
If it were possible, right now, for individuals to spin up super intelligences on their home computer clusters, even you would agree, I would hope, that it would be critical to human survival and happiness that we didnt allow that. Sad day for open source, but if you could 3D print a nuclear weapon on your MakerBot wed all agree you have to ban that too and sad day for the 3D printers. If even in that case, youd say no, open source is holy, nuclear weapons for everyone, then your commitment to open source is whats invincible to reason.
This is being bright-eyed and honest that if it gets to the point where people can spin up super intelligences on their own or in the dark where no one can see, thats a threat to humanity we can't allow and it MIGHT require increasing the price of your computer cluster by 20% or something.
> If it were possible, right now, for individuals to spin up super intelligences on their home computer clusters, even you would agree, I would hope, that it would be critical to human survival and happiness that we didnt allow that
No. As long as multiple different people are able to create superintelligences, one superintelligence can be counteracted by another.
Show your work? Theres lots and lots of ink spilled (on e.g. lesswrong) demonstrating why it doesn't work out that way. Value conflict isnt symmetric. An AI aligned to human values doesn't have the same tools available to it as one aligned against human values, and its much easier to destroy than preserve.
The second law of thermodynamics does show that it's easier to destroy than create, but nevertheless a huge amount of complexity has been created over the course of the universe. Guns are an offensive weapon designed to kill, and since their invention the world has gotten more peaceful. Atomic bombs are supposedly (and mistakenly https://www.navalgazing.net/Nuclear-Weapon-Destructiveness ) thought to threaten all life on earth, but deaths in war have dropped since they were introduced (they could have also let us generate an abundance of clean power if we hadn't idiotically regulated them vs coal). The interests of all the people who want to not be dead are going to outweigh those of a minority trying to kill them.
Let's assume for a moment that DARPA hasn't been sitting on its heels for the past 70 years or so. It's probable that we do actually have weapons that threaten all live on earth (given that we've already managed to do that by Accident).
Yes, the interests of all the people who want to be not dead... outweigh... Wait, the Chinese were making HOW many plagues? Nevermind.
(If you don't want to talk China, you can talk about how many dunderheaded "biological scientists" have tried to create "anti-Jew" bioweapons, etc. We have a lot of stupid in this world, and many "world-ending" devices/weapons/thingummies that are not very well regulated by the "People who want to be not dead.")
> given that we've already managed to do that by Accident
No, that's not "given". I gave a link on how nuclear weapons didn't actually threaten the survival of humanity. If you want to claim the contrary, you should support it.
Also - you don't like the current controlled substance regime, sure, I don't either. But are you advocating for full legalization no regulation buy heroin at your gas station?
No? _What controlled substance regime WOULD you advocate for?_
Please do the same for AI! The "nothing" option is heroin in your gas station and possibly meth discretely added to beverages without being clearly labeled because it increases customer retention etc, but in AI equivalent.
Well, actually I sort of am. Heroin, yes, but not amphetamine or crack. However I'm strongly against allowing the advertising of it. After that proves out, I might be in favor of more law removal.
OTOH, I don't pretend that would be not costly. It's just that I hate the costs the current system has imposed. And people who can buy opiates aren't much trouble to anyone except themselves and their friends. (The trouble comes if they can't get the opiates. So perhaps if you're addicted they should be free.) This would end up with a lot of stupefied people, of course, and many of them might die. (OTOH, I've been informed by an ex-addict that heroin is less addictive than tobacco.)
There is no accurate definition of 'addiction' that would justify forcing the taxpayer to buy drugs for inebriation hobbyists.
The justification would be it keep them quiet and not bothering people. And it would probably be a lot cheaper than current enforcement.
> There is no accurate definition of 'addiction' that would justify forcing the taxpayer to buy drugs for inebriation hobbyists.
It's simply pragmatics. I ran the numbers on this for a post, opiates are dirt cheap, and in the limits of what we already spend on homeless initiatives and law enforcement, *everything* is dirt cheap.
Noted hardware-and-software hacker (and now self-driving car CEO) George Hotz once came up with the idea of Wirehead City - basically, you take all the “problem homeless” and put them out on BLM land in the middle of the Nevada desert, and give them food and water and all the drugs and alcohol they want.
In return, we get usable downtowns and cities.
I actually ran the numbers, and it is massively, absurdly net positive to do this in any reasonable cost benefit analysis. Any one of the reductions in crime, improvement in expensive downtowns, and lives saved more than pays for the entire program.
The analysis, for anyone interested: https://performativebafflement.substack.com/p/an-incentives-based-problem-homeless?r=17hw9h
Would you want that exile for your own child?
heroin in the gas station next to the guns, AI chips at Best Buy or whatever, please and thank you.
We already have meth discretely added to beverages without being clearly labeled. Do not drink anything, anywhere, if you don't trust the people serving it.
If you say so ... we don't.
> Scott doesnt think the current state of the world is acceptable
What does this even mean? Do you think that Scott going to become a terrorist? (Note that this is a rhetorical question).
I think that this article is perfectly consistent with someone who believes that AI is a huge threat to mankind and is willing to accept permanent oppression to prevent that fate.
Please read the comment this was a reply to.
The status quo is DEFINITELY not stable. We haven't yet adapted to the technology that is already in public use. The question is "What comes next?", and there is a lot of uncertainty.
My main objection to Plan A is that I think it's unworkable. I don't have a better suggestion. We've got the governments we've got, and there's a short timeline. We can count on the US government to be short-sighted and the individuals making the decisions to be greedy and corrupt. But not necessarily thoroughly corrupt. They do make some good decisions. (But Plan A envisions the US government as idealistic, honest, and far-sighted. I can't think of a time when that was true.)
I don't *think* what you describe is the reason this post "triggers" me as it does, because I am in the camp of believing that the world is currently undergoing a radical change that will flip the proverbial board. Perhaps though, somewhere deep down I believe that while the board will be flipped and things will radically change in ways we struggle to imagine right now, there exist some fundamental truths about societies that are unlikely to topple so easily, like human desire for power. When the board flip happens, I think there will be many people looking to seize power in the new order and that is the thing I fear more than the fact that the board is upside down.
Perhaps Scott believes that even these fundamental "laws of nature" will be turned upside down by AI?
The _goal_ of Plan A is:
Currently, and if we do "nothing" or let special interests win be default, the people who are poised to take power are the small number of effective oligarchs who are the heads of the AI companies, the companies that are critical to supplying them, and the ahead-of-the-curve and aggressive politicians or connected government executives who will exercise control with them. A small number of people, mercenary by their nature, with no even theoretical dependency among " people" generally.
Fully decentralized citizen autonomy enabled by AI is a utopic vision that is just not in the cards.
_Someone_ is poised to gain incredible power from this radical change. How can we make that a larger number of people, more dependent on the citizenry?
Thats the aim of Plan A - how do we make sure this new power is in the hands of more people who are as dependent on the citizenry as possible, and not what appears to be the default outcome and in fact our current path, which is the 150 or so people who happen to be well connected at the moment Super Intelligence is created?
If you dont like who Plan A hands that power over to, whom ought it hand it to instead, and _how_?
YOU (and me) are losing a lot no matter what.
This argument falls much more in the "I want to engage" place in my brain than Scott's latest post.
My short form response is that I believe the only chance we (regular folk) have in any of the futures lies in plurality, and I think open source is the way to achieve that. If I was keen on using regulations to solve problems, the one regulation I would lobby heavily for is that any sufficiently advanced AI must be open weight. I think that would maximize the chance that we end up with a situation where there is plurality of AIs rather than a singleton. Even just two competing super intelligences is better than a singleton, and 100 is better than that, and 8 billion is better than that (unrealistic utopia outcome).
Getting China and the US to mutually agree to such terms would follow the Plan A optimism at government capabilities and will, and be even better than a single country doing it alone.
I think theres some serious common ground and actual operational similarity then between what you want and Plan A.
The critical calculus is "How do you force an AI to be Open Model, and how do you force the company to do that _as soon as_ they make it, and not after they have already quietly used the super intelligence to win the game first"
One of the largest areas of concern for these scenarios is that, once a super intelligence exists, probably the window to control it externally is very narrow, whether thats because its creators who control it will absolutely 100% make sure the first thing they do with it is make themselves win the game, or because its not aligned with any humans and the first thing it does is secure itself and prevent other rival super intelligences.
If we are going to allow a company to construct a super intelligence, a simple promise or even requirement that they make it open model after its constructed won't do - you STILL have to be tracking who is making super intelligences and what they are doing with them, else they will still just do it under your nose and for their own benefit.
The primary means of Plan A is in the "making sure AI creation happens the way we decide we want it to happen" and less in the "making it happen this particular way"
I don't think it's possible to clandestinely "win the game" just by having enough robot monks meditate super hard inside a sufficiently advanced box. Solving real-world problems requires engaging with them directly.
I found this comment interesting, thanks! I appreciate you trying to write out why you felt triggered by it, rather than just arguing against it
I think you’re the first person I’ve been aware of who disliked KYC/AML regulations with any vehemence. What are the main problems with them?
1. There is no evidence that they have a meaningful positive effect, and not for not searching. Much ink has been spilled in various locations (including academic publications) on just how ineffective they are at preventing crime or catching criminals. See section II.A. for some sources on this: https://coincenter.org/tear-down-this-walled-garden/
2. They result in a very significant portion of the human population being unable to utilize a bank because they are unable to fulfill the regulatory requirements. The negative effects are disproportionately applied to poorer populations, and can make it challenging to get out of such situations (can't get a bank account, can't save money easily, can't learn good spending habits, etc.).
3. I have personally had more money permanently frozen because I was unable to or refused to complete invasive KYC/AML checks than I have lost to actual thieves.
4. Fulfilling KYC/AML requirements when you live a non-traditional life can be incredibly burdensome. If you just earn a paycheck and spend it every month it isn't too bad, but once you start getting revenue streams from many non-traditional sources things start to get hairy.
5. The cost of compliance is extremely high. I don't have the numbers in front of me, but banks have collectively spent significantly more money on compliance than has been seized from criminals.
6. Many governments in the world (including the US) are known to exploit AML/KYC stuff to harass or outright steal from people (see civil forfeiture in the US).
7. The data collected in fulfillment of AML/KYC requirements is *frequently* stolen and sold to scammers who leverage it to steal money from people (causing the whole system to *increase* crime, rather than decrease it).
8. As with any surveillance, it is very tempting for those in power to use the surveillance against their opponents or even just people they don't like. This is is true for financial surveillance as well.
9. Philosophically, I am of the belief that governments tend towards corruption, so we should minimize the tools they have available to execute on that corruption. Financial surveillance is a powerful tool for a corrupt government so we should resist creating that tool (especially given it has no meaningfully measurable benefits). Keep in mind that the majority of the planet is under the rule of a government that is not yours, and probably is more corrupt than yours.
Then there's the impact on art! Who gave Visa and Mastercard joint authority to decide what is or isn't obscene, then "protect their reputation" by wielding threats of exile from effective participation in the modern economy against anyone who gives aid and comfort to their enemies? https://yellat.money/
I'm old enough to remember when the US tried to ban exportation of Random Number Generators (or, you know, anything that the NSA couldn't crack.) They made decent, innocent video games illegal (specifically, procedurally generated video games. They are unfun without a decent RNG).
I'm noting this, because regulations that Sound Good are not always, and can have some really, really stupid knock-off effects.
Regulating "AI Chips" and thinking it will do any good is like putting down a dam with a million leaks, and saying "Our Programmers are Too Stupid to get through the Dam." I wouldn't want to take that bet.
"Very Secure against Cyberattack" -- you aren't talking serious here. Serious security against cyberattack is removing the data center from the internet. If you're not going to do that, you're not serious. If you think AI is about as dangerous as a nuclear weapon, in the wrong hands, then you want it unplugged (from the internet).
"The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time" --> Why should it be cryptographic? Is it so that the AI cannot realize it will be unplugged? Cryptography is used to prevent data from being understood.
One very big problem with increased regulation is that it destroys the free market. Specifically, overregulation prevents smaller, more agile companies from entering the market at all. This seems to be what your regulation is designed to do. Perhaps you actually think this is good.
77% of pharmacies with burglaries... don't have video cameras.
https://www.uspharmacist.com/article/pharmacy-security-know-the-options
Was this link even read? There doesn't seem to be any regulatory "you can't sell drugs" without putting video cameras in (it's also markedly not really about regulating factories or who buys the damn drugs, it's about physical security of pharmacies, which isn't very good, and isn't very regulated. Where I live, I've seen signs that say "We do not sell Opiates here" -- this is a theft prevention measure.). Yes, it's probably better for your business if you do so, but we aren't saying "Thou Must."
I do also object to calling greenbacking "innovation" -- it is not. Ivermectin was innovation.
Is it part of the Global Surveillance State that I must provide my drivers' license and home address whenever I buy sinus pills? Hell, yes! Now that it's "being scanned" and not written in a book, it's even convenient for the government to steal my data. I'm on the side of crypto, of not letting the government know about every condom you buy, or cigarette you smoke.
You write like the government doesn't have access to your medical health records whenever they damn well please. Is it illegal? Does it matter? (No, the government is not going to start -blackmailing you- because you have an STD you didn't tell your wife about. That was the point of HIPAA after all).
"They couldn't enslave 5-10% of the world's computers secretly"
... please. Tell me you'd notice if the computer chip your refrigerator has was wormed? Or had a virus? How about the one in your answering machine? Or your thermostat? Or your thermometer? How about the bajillion in your car? Most people don't even turn task manager on, (and current "tech" runs hot, so "why is my computer running at more than 5% CPU" is best answered with "Bad Code" not "I have a virus!?!").
The point is that the training run would take 5-10% of all computers running presumably at full power. My computer can completely freeze if I overdo it even just with inference on open AI models, let alone training. If you skimmed off only 5% of the total CPU/GPU power you'd need literally every computer in the world.
I'm talking 99% of power on all the chips we pretend don't exist. Not the "full powered" computer chips, the small ones that exist in everything. Or, you know, the cellphone chips when you "have it on" but aren't actively using it.
I think assuming that LLMs are the future of AI is the first mistake.
Well, if at some point someone manages to make AGI that can be trained and run with a handful of laptops purely by power of clever algorithms then all predictions are off anyway. This plan is based on the (reasonable, but not guaranteed) extrapolation of current trends that compute remains important and algorithmic gains are relatively limited.
Those IoT chips you mention are certainly more hackable but also far less powerful. Most don't have dedicated GPU-like linear algebra functionality. I think even if you grabbed every single chip of that kind in the world you wouldn't get a single half decent training run. The smaller the individual unit, the bigger the inefficiencies from moving around information between one and another.
"You wouldn't get a single half decent training run" -- it's all a matter of time. Are we talking a year, instead of a day? that's two orders of magnitude (roughly speaking).
Source for games requiring cryptographically secure RNGs? Sounds like something only statistical analysis would pick up, not a user looking at generated levels.
A game developer friend of mine. No, it wasn't identified by a player (it "didn't look right" if done with a poorer RNG, so I suppose a player might have identified "this looks crappy" from a Bad RNG).
I think I finally have a clearer model of the strongest version of the "it will create a totalitarian dystopia" fear, that explains why the case might be different for AI even though the fear has not turned out at all for eg. nuclear weapons and only to a medium extent for medicine.
[Edit: on self-reflection, this doesn't explain cases like payments surveillance well, as I find that highly undesirable despite it not falling into the below pattern of being actively resisted by many, so definitely treat this as one story among several, not a theory of everything]
Here goes:
----------------------------------------------------------------------------
The big key insight is: don't think of the thing to avoid as "a totalitarian dystopia", think of it as "a war between the government and the people". That is, a situation where the government is highly motivated to prevent X, and the people are highly motivated to achieve X, and so both strongly pursue their (opposing) goals.
Think of this from the Realist point of view: the worst thing you can have is two very powerful entities that have strong opposing motivations, because that leads to a big war. This is why Hobbes argues for a monopoly of force: if there is a single clear winner, whom no one bothers to resist, then that is in many ways bad, but it's still quite a bit less bad than the big war.
There is an analogue of this for "wars between the government and the people". If the government bans something that people don't _really_ want (eg. murder, nuclear weapons, poisoned food), then they succeed at banning the thing, people switch to doing other non-banned things, and life goes on. But if the government bans something where people have a strong will to resist, then people will sneak around the government, bribe officials, the government will step up spying on everyone and mandating inspections, people step up their countermeasures, the government steps up the spying, sending goons around to check everyone and everything and look into people's underwear, etc.
What we call "totalitarian dystopia" is a side effect of THAT. It's not a phrase we use to directly refer to a government banning a thing. Rather, it's a phrase that we use to describe the outcome of a highly advanced "war of government versus people", that arises from the government banning a thing that is nevertheless really valuable to the people who want it. The totalitarian dystopianness is the collateral damage from the government side of the high-stakes asymmetric war.
Examples include:
* A government invading a country and banning expressions of political support for that country within the occupied territory
* Regulating private conduct in people's families (beyond genuinely one-in-a-thousand abuse cases)
* Banning religions
* War on drugs
Another factor that increases "the people's will to resist" is: do lots of people find the perpetrators' goals sympathetic? For murder, not really, even if there are in fact people who are really really intent on murdering people. If there _were_ lots of murder sympathizers, then you'd have lots of people harboring murderers in their homes to protect from police, and you would actually go up the escalation ladder, and anti-murder laws would thus get a more totalitarian character. But, fortunately, murder sympathizers are few and far between. For the above four (in the last case, at least the less-dangerous half of drugs), the situation is quite different, to varying extents.
This is all your "be nice until you can coordinate meanness" concept, but from the perspective where the government itself also has to follow this rule.
I continue to think regulating chips is on the milder side of this, because there are only a few chip fabs, they are relatively easy targets for regulation, it's a fairly non-invasive regulation so it's easier to get legitimacy for it, if the regulation targets training and not inference then the number of people who would be _directly_ motivated to fight back against any specific step is small (basically just chip fabs and AI companies) and so you get pretty quickly to the "they succeed at banning the thing, people switch to doing other non-banned things, and life goes on" step.
Trying to ban open weights models is quite a bit further toward the risky side, because wanting it runs counter to all kinds of other goals that lots of people have and are sympathetic to - not just people as individuals (who don't want their private lives going into an untrustworthy corporation), and not just enterprises (for whom having control of ongoing ability to access their work tools is basic prudence) but even governments (see: data/compute sovereignty goals). Also, if your goal is revocation capability, you have to ban not just open weights, but also tricks like "consumer single-LLM ASIC chips". Basically, you would be explicitly mandating weaponizable interdependence, which is _exactly_ the thing that lots and lots of people are scared after a decade of it and are fervently seeking a way out from. (The weaponizability in this case is much greater because access could be revoked individual-by-individual, rather than just a global "cut compute by 99% for everyone" button)
Thanks, this was actually a helpful comment for someone like me, on the pro-regulation side.
I think there are potentially some ways around this "government vs the people" situation.
For example, if people are worried they can't build a gamer PC anymore, then to allow this, the GPU could be only willing to run code that has been signed, and which is known not to be AI training code. A distributor like Steam could do the signing.
A different problem would be research GPU clusters that run custom code all the time and which might also want to do (non-frontier) machine learning. What I think has been suggested here in the literature is that these clusters need to keep a log of all the computations that have been run (as some kind of hash) and then when inspectors come to visit they can rerun the code, check that it doesn't do frontier AI training, and check that it recreates the hashes.
And, I actually think running a (safety-tested) open weight model on your own GPU seems actually fine. The signing approach could work here again.
Are there other scenarios where people really want to use high-end GPUs?
So if I'm a hobbyist developer who wants to develop a game, or experiment with developing a game, am I limited to using presigned shadercode? Can I not write my own shaders?
> For example, if people are worried they can't build a gamer PC anymore, then to allow this, the GPU could be only willing to run code that has been signed, and which is known not to be AI training code. A distributor like Steam could do the signing.
This is already pretty bad (for example it kills any attempt to make indie games sold via non-major store channels) and it becomes significantly worse in a world in which better AI means people can make their own software. You're entirely nullifying those gains by... banning them from running any software that isn't properly acknowledged by The System (whatever that is). Actually Google is about to do that to Android (ban all non-Play Store apps, at a system level), ostensibly for "safety" reasons, practically probably to stop apps that do piracy or other stuff they don't like, but it's going to do a ton of collateral damage to all the non-official Android software scene and I know it'll make me install LineageOS on some of my devices.
I don't think you need to be anything like this draconian. GPUs aimed at consumer graphics need an amount of video RAM which is some small (maybe 10x, certainly not 20x) multiple of the amount used for a screen's worth of whatever the highest resolution display output you support is, with enough bandwidth to fill the lot once every 60Hz frame or so (maybe 120Hz if you want to drive a VR headset and not vomit I guess). A consumer is very unlikely to want more than two of these in their gaming machine cooperating, and most only have one. Meanwhile, GPU-like objects intended for frontier AI training need orders of magnitude more, faster memory than this (and also typically include large matrix multiplication units while lacking hardware support for helping with things like rasterisation, Z culling, alpha blending etc), as well as very fast communication between multiple devices. Regulate based on that difference, if you must; it will only grow wider as frontier models scale while human retinas do not.
This isn't right. A high-end graphics card meant for 4k gaming will have 32GB of RAM, which is about 1000x the size of a 4k 32bpp framebuffer.
A game wants as much VRAM as possible to stuff full of geometry and textures of its assets, so that it can display them at short notice without the need to swap them in from main memory.
That 32Gb will be GDDR on the card, not HBM on the chip. Whatever special sauce is on the chip is used for render targets, if the design bothers at all.
Oof, I didn't know AI chips came with gigabytes of on-chip RAM.
tbf what you actually have with the sota designs is chiplets - multiple pieces of silicon incredibly carefully aligned on top of a large piece of silicon that provides interconnects, the whole cursed resulting object then wired to pins and packaged inside the chip housing. So “on-chip” doesn’t necessarily mean literally on the same piece of silicon, though for the designs I worked with it did.
Interestingly, in the “slowdown” world, I think ASICs are likely to be more prevalent; we are close to the point where it is economical to tape out an ASIC for each frontier model (given the lifetime operating cost of that model), and if model development slows then ASICs become more favorable as the depreciation / ROI window widens.
Maybe this gives a coherent fix for the plan - once ASICs are common, GPUs will be worthless for training/inference, and gamers will be unbothered by these regulations. (Look at BTC / ETH for an example of how this played out in the past.)
The problem with this line of reasoning is that I think ASICs would be something that people want more than GPUs; in other words the actual objection is not “gamers want unregulated GPUs” but rather “everyone will want their own local AI compute”. Unless you can align the world state such that most people are happy with hosted AI, there will remain the war between government and people that Vitalik highlights.
The current interest is in ASICs that implement inference for a single model, or perhaps for a bunch of fairly similar models. I doubt that these ASICs would be useful for training, in which case there would be no need to regulate them.
The plan in OP regulates inference too.
It’s a good point that we need to treat training and inference separately. You’re right that training admits less specialization.
I don’t think there’s a clear case that ASICs are unlikely for training in this hypothetical.
TPUs are already used extensively for training and they are more specialized than GPUs. So the existence proof already obtains.
In the slowdown world, if architecture improvements also slow down then you can bake more assumptions into your training chips and my point above about ROI holds. (I would probably revisit the 10-100x number for training specifically, I accept the pushback there). If instead we see more architecture/algorithmic improvement as a substitute for compute improvements, perhaps ASICs would not be as favorable - but even in the current architecture revamp timelines, if the absolute scale of training runs increases an OOM or so (and especially if LLMs automate more of chip design) then it may still makes sense to do a per-architecture tape-out.
So the strongest prediction I’d TLDR is: in a year or two it seems highly likely that TPU/Trainium/etc will fully diverge from GPU for training. Honestly think you could define a legally enforceable split with the current gaming vs H100 architectures.
Genuine murder sympathizers are an issue when people are sufficiently alienated from the government.
See Russia pre-revolution. Murderers were often let free by a jury of their peers.
Thank you, that was helpful for me.
>and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.
Under which agreement is the US allowed to physically inspect Chinese dairy farms, or vice versa? If you had gone with the much more appropriate analogy of nuclear control treaties, you would see how fragile these are if and when they become inconvenient for either side.
You can build a datacenter with high-end consumer GPUs. It would be less efficient than "professional" GPUs, but it's still a viable approach, especially since you can solder on more memory onto existing cards: those HXXX cards aren't much more powerful than 5090, their main advantage is more on-board memory (but note that the difference is less than order of magnitude currently).
USA already did ban export of 4090/5090 GPUs to China. I'm pretty sure your proposal would either spell the end for high-end consumer hardware, or limit its availability geographically (which would require every PC with every OS to "call home").
Now, GPUs might actually be a dead end for AI training. I expect specialized hardware to arise, like ASICs did for crypto mining. Maybe you can leave consumer GPUs alone, and regulate this new hardware (when it appears) which would be more or less useless to regular consumers. But this theoretical (for now) hardware would, by itself, give a significant boost to the curve, so maybe we will just end up with AGI sooner anyway.
Isn't the main limitations of the consumer-grade GPUs the much slower cross-link? (Which you need for training but not inference.)
Hm, yes, that too, and it's harder to overcome. But I'm not sure this is a hard obstacle in a world where you CAN'T get a pro card, for a very motivated party - finding ways around it is still easier than e.g. building your own chip factory. The risk is still there.
The model size, cost, and power use of training each generation of new models has grown rapidly, and the cost and power usage would be even greater if you had to make an equivalent model from consumer hardware. It's going to be hard to hide this for future LLMs. You'd have to buy immense quantities of consumer hardware, involve thousands of people to build it, power it, etc.
I note simply that I am extremely skeptical of arguments that endeavor to persuade me to accept X policy because it and its consequences pattern-match to existing policies Y, Z, A, B, and C, all of which are things I would have rejected immediately if anyone had bothered to ask me beforehand.
You don't get to a grim meathook dystopia in one go. You get to it by salami-tactics of things which, in themselves, can be portrayed as only trivially bad.
Yes, I thought that was one of the weakest parts of the argument.
good comment.
Scott,
Historically your best arguments strike me as not just speaking to those near the perimeter of your camp, but all the way across to folks who are in intellectually distant camps. Today, when you speak to the "cost" of a growing regulatory state, you don't seem to sufficiently appreciate the viewpoint of the more traditionally regulatory averse. I don't think you're speaking to that crowd (include me) here. Maybe you don't intend to (and can get sufficient traction without us), but if you want us, you need to speak a bit more broadly.
Systematically, regulations (laws, taxes, etc.) are imposed with a 51/49 like majority, and do so by arguing that "at this time" they are clearly necessary. However, they are written "at that time" for the problems "of that time." They are written in broad brush terms (in the U.S.) by a congress, who then washes their hands of actually having to work out the details, and a new regulatory body pops up in the executive, where sometimes even new non-article-3 judicial powers are invested in the executive, and the body effectively rewrites their own rules with sufficient detail to enforce, and acts as their own judge/jury in many cases.
Then, time passes, and new edge cases drift through. What actually constitutes an AI chip? How do we handle pre-existing chips? Oops, the algorithms advanced faster than we expected and now don't need hardware as sophisticated as you previously assumed, what happens to all the chips we didn't think were dangerous but now are? Oops, the paradigm shifted, and it's really parallelism that advances the AI. Oops, quantum computing.
The executive agency designed to handle this becomes a hot potato. Congress doesn't want to touch it any more. The original 51/49 consortium dissolved, and now very vested interested have emerged on both sides and to move the rules either direction after they are established generates profound pain and contributions to your next political opponent.
Every time the regulatory-averse crowd sees a new huge paradigm rolling through, they see the stories of the failures of the past and project them onto the new paradigm.
And that's just the domestic story. The international story is its own concern.
The above assumed these were just simple domestic laws. But if so, then we can retreat from them because we still (as a nation) have autonomy. So instead are they congressionally ratified treaties? In which case, that executive bureaucracy isn't even domiciled here in the U.S., it's in Brussels, or Beijing, or ? Are you relying entirely on the relative immovability of existing plants and the time-delay in making new ones to prevent non-signatory countries from evading? If there's a date at which the entire raft of law goes into effect, is it practical for a large number of the data centers currently existing to close down, pack up, and reopen in a non-signatory state the day before effect?
And coming back to the timeline/projection/model into which the new regulatory system provides the "fix" - that's an interesting forecast. It's notoriously hard to forecast things of this consequence. Show me how the new world works if your model was either too fast in its predictions by 5x or 5x too slow.
The mini-regulatory team has an advantage, historically, of saying the 5-year-planners were wrong, and an infinitely adaptive market-like free competition of dynamic players always ended up with outcomes that beat the static nation-based systems. So they didn't need to have a perfect vision, they just trusted that the system would follow a local minimum through to a solution.
If you want that team to hear you, it's a much much harder pitch than the one you're making.
I'm not saying AI doesn't need some sort of regulatory state you describe. I frankly don't know. It strikes me as within the realm of possible we're looking at the event horizon of a singularity and all the rules need to be different this time because everything we've learned in the past doesn't apply at a singularity. Could very well be.
I'm just saying your story isn't designed for folks whose starting point is in a camp that isn't near yours. If you want us, the pitch needs to be very more respective of our challenges with all the regulatory disasters that we think have preceded this.
It is my sincerely-held belief that we'll all die (near instantaneously, not painfully most likely) if frontier labs are allowed to continue their AI development for another 10 years or so (could also be 5 years, could also be 30 years). What should I do now according to your political philosophy?
"What should I do now according to your political philosophy?"
The straightforward was is probably to work with as many like minded people as you can find to kill the people most critical to the frontier labs AI. You aren't going to talk the people working on this into stopping.
I think that there are probably too many of them and that they are probably too dispersed for you to get them all, but I bet you can find a bunch of the US based ones in San Francisco. Anthropic, OpenAI and xAI all seem to have a bunch of folks in SF.
I don't know how to get to the Chinese folks.
Mistral is in Paris.
But the basic approach is so simple that even killing everyone involved today just pushing things out.
In some respects, a much more effective way is to go after the fabs building the chips. There aren't very many of them (in the grand scheme of things) and they are a lot harder to hide than people. In the west, destroying TSMC's leading edge (and probably a few nodes back, too...) fabs as well as Intel's leading fabs and Samsung's leading logic fabs should do the trick. China can't produce these chips in volume, yet, so you have some time there.
I haven't put any serious thought into this, but I expect that ramming a Cessna full of dynamite into a fab would take it out of production for quite a while.
That's a terrible idea because the US intelligence agencies would stop any sort of civilian conspiracy to do that. But the US intelligence agencies could do a lot of that themselves, if they came to realize that AI is going to undermine their power and entire model of operations.
Aside from which, the whole point of banning a thing is to make the barriers too high for most people to try to do it. Most of those people you're talking about, though they might scream and howl over it, would move on to something else. Google has plenty of ways to make money on deterministic algorithmic processing and Meta can sell plenty of junk to middle aged moms, I'm sure they'll all find somewhere to land. The CCP has every incentive in the world to do the same over there to maintain control, and a history of purging their business leaders, and the entire apparatus exists already. And Europe already applies the "precautionary principle" to mundane consumer products so I'm sure they don't need much persuading here.
Probably the scenario most likely to ensure human survival is one where there's an early scare sufficient to get the majority of humans to align around a permanent norm against the construction of AI systems. The nature of that scare might vary, and I'd hope it needn't be a massive disaster, but eventually something like that probably will happen. Then you have to find a way to set things back long enough that you can entrench the norm, make the construction of AI akin to human sacrifice such that only lone crazies would ever even attempt it and those crazies can be dispatched by people with state-level capacity. Keeping that up for the next 5 billion years will be more of a challenge, but that doesn't have to be solved immediately.
We probably want someone named Serena pushing back against the AI.
Jehanne would be a better bet, I think.
It seems rather odd to rename the character like they did. Like Christopher Tolkien deciding "nah, Morgoth's name is Fred now".
The same us intelligence agencies that thought the Afghani government wouldn't fall for over 100 days after our withdrawal? (given that this was done using whatsapp, which is american software and thus presumably bugged-by-government, were the government competent at all) The level of Competency you'd need, sir, does not exist in this government.
Just like they stopped the capacitor plague? I mean, seriously. Someone already broke computers (and I'm just going to allege that it was industrial espionage, because why else would an entire factory make bad caps? Did they somehow not test their one product?)
Borrow large amounts of money that you never have to pay back.
I think Scott is saying to fix that we need to: Convince 51% of congress (and the president) of that risk, and the CCP. Seems like a fine strategy. I'm just saying it doesn't sound convincing yet to this 0.00...01% of the electorate, and I think there will be a lot of folks out there like me. The stories I've heard in the past about dire needs for regulation have left me with scars. If you need me (us) to get to 51%, to resonate with that crowd, you'll have to recognize that we see a long trail of bad regulations prior to this point, and this one has a lot of markings to make it look worse. If you don't need us, continue to pitch to the other folks you need to get to 51%.
Every political issue du jour comes with a tincture of apocalypse. And every threat of apocalypse justifies an expansion of the power of the state. And every expansion of the state comes with great assurances "temporary, limited, targeted" and (this side of the crowd) gets disappointed with the results virtually every time. If you want to convince that crowd, understand that set of preconceptions and pitch to that crowd specifically.
There may be fewer people like you than you may like to think.
I am convinced there are fewer people like me than I'd like to think! And, regardless of what I'd like, I don't know how many of me there are. And we may not be well represented in Scott's reach.
This is a good comment - well articulates the reality that actually occurs when you wave the magic wand of "there-should-be-a-law" to solve a problem.
This proposed legislation all seems entirely reasonable. Unfortunately in the US, congress is who writes (allegedly) and passes legislation
Knowing congress, I expect "Scott's Beautiful AI bill" to have its text swapped out at the last minute, and something dystopian passed in its stead
Typo:
> developed in a fully transparent manner over a decade years.
"I’m reminded of a story I heard - I can’t find it, maybe one of you can - from a DC insider who said it was enraging to work with Silicon Valley, because he would bring up what he thought was the obvious regulatory framework, the tech people would launch into jeremiads on how it could only be enforced by world dictatorship, and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years. "
I hate to break it to you but you're doing the thing.
http://opentranscripts.org/transcript/coming-war-general-computation/
>But information technology confounds these heuristics. It kicks the crap out of them in one important way, and this is it: one important test of whether or not a regulation is fit for purpose is first whether, of course, it will work, but second of all whether or not in the course of doing its work it will have lots of effects on everything else. If I wanted Congress or Parliament to write, or the EU to regulate, a wheel, it’s unlikely I’d succeed. If I turned up and said, “Well, everyone knows wheels are good and right. But have you noticed that every single bank robber has four wheels on his car when he drives away from the bank robbery? Can’t we do something about this?” The answer would of course be no. Because we don’t know how to make a wheel that is still generally useful for legitimate wheel applications but useless to bad guys. And we can all see that the general benefits of wheels are so profound that we’d be foolish to risk them in a foolish errand to stop bank robberies by changing wheels. Even if there were an epidemic of bank robberies, even if society were on the verge of collapse thanks to bank robberies, no one would think that wheels were the right place to start solving our problems. .
>But, if I were to show up in that same body to say that I had absolute proof that hands-free phones were making cars dangerous, and I said, “I would like you to pass a law that says it’s illegal to put a hands-free phone in a car,” the regulator might say, “Yeah, I take your point. We’ll do that.” And we might disagree about whether or not this is a good idea or whether or not my evidence made sense, but very few of us would say, “Well, once you take the hands-free phones out of the car, they stop being cars.” We understand that we can keep cars cars even if we remove features from them. Cars are special-purpose, at least in comparison to wheels, and all that the addition of a hands-free phone does is add one more feature to an already specialized technology.
>In fact, there’s that heuristic that we can apply here. Special-purpose technologies are complex. And you can remove features from them without doing fundamental disfiguring violence to their underlying utility. This rule of thumb serves regulators well by and large, but it is rendered null and void by the general-purpose computer and the general-purpose network, the PC and the Internet. Because if you think of computer software as a feature, that is a computer with spreadsheets running on it has a spreadsheet feature, and one that’s running World of Warcraft has an MMORPG feature, then this heuristic leads you to think that you could reasonably say, “Make me a computer that doesn’t run spreadsheets,” and that it would be no more of an attack on computing than, “Make me a car without a hands-free phone” is an attack on cars.
>And if you think of protocols and sites as features of the network, then saying, “Fix the Internet so that it doesn’t run BitTorrent,” or, “Fix the Internet so that thepiratebay.org no longer resolves,” it sounds a lot like “Change the sound of busy signals,” or, “Take that pizzeria on the corner off the phone network,” and not like an attack on the fundamental principles of internetworking.
>Not realizing that this rule of thumb that works for cars and houses and every other substantial area of technological regulation fails for the Internet does not make you evil and it does not make you an ignoramus. It just makes you part of that vast majority of the world for whom ideas like “Turing complete” and “end-to-end” are meaningless. So our regulators go off and they blithely pass these laws, and they become part of the reality of our technological world.
>There are suddenly numbers that we aren’t allowed to write down on the Internet, programs we’re not allowed to publish. And all it takes to make legitimate material disappear from the Internet is to say, “That? That infringes copyright.” It fails to attain the actual goal of the regulation. It doesn’t stop people from violating copyright, but it bears a kind of superficial resemblance to copyright enforcement. It satisfies the security syllogism. Something must be done; I am doing something; something has been done. And thus, any failures that arise can be blamed on the idea that the regulation doesn’t go far enough rather than the idea that it was flawed from the outset.
> I hate to break it to you but you're doing the thing.
You aren't breaking anything to anyone when you do it via such meaningless phrases as "doing the thing".
you are entitled to your opinions about redditese (which i largely share) but putting it like this certainly does not pass the three gates
I've never had a Reddit account, and I don't know what "pass the three gates" means.
I believe it is a reference to https://slatestarcodex.com/2014/03/02/the-comment-policy-is-victorian-sufi-buddha-lite/
you should acquaint yourself with them if you want to have a long and prosperous career as the prolific commenter you are
It passes "true" trivially, "necessary" is pretty solid and for "kind" it's only the mildest of mild snark in response to snark about tech people who, it can't be stressed enough, are much much wiser on this very specific topic overlapping with their expertise than scott and his DC friend.
Sorry, but this just seems like meaningless bloviation. We declare you're not allowed to write down numbers all the time - for example, every child porn video corresponds to a number. This in fact reduces the amount of child porn in the world and doesn't cause the entire Internet to collapse. I am begging you to actually think about real things on the object level instead of making grand 10,000-foot-high pronouncements about the nature of government and technology that collapse on slight inspection.
child porn is inherently evidence of a violent crime, so very different
Yes, which is proof that "banning a number" is the wrong way to think about things. Some numbers are bad!
(also, I don't think this accurately reflects child porn regulation, which isn't just about catching the people who made the video, but about catching people who distribute and watch it. And my understanding is that AI-generated child porn is just as illegal as the real sort.)
that is fair
Animated porn, drawn porn of children isn't illegal*. "Ai-generated" child porn that happens to look photorealistic? Probably getting banned for "being too close to the real thing" (and not for any "this is a good reason by itself." -- in a world where 8 year olds did not audition for roles as sex slaves, and people did not assault 4 year olds, you wouldn't ban AI-generated child pornography -- of course, there would probably be limited demand for such, in that case).
*Some practitioners even use it as part of psychotherapy...
As an aside, this differs between countries. Germany bans even simulated CP, including hand-drawn and animated cartoons. Afaik it even bans textual descriptions, i.e. graphical descriptions of minors having sex.
Well except it's not. The UK ran page 3 child pornography until 2003 which you could buy at any newstand. Pretty positive no violence was involved.
Talking about banning numbers through legislation is not being charitable to Cory Doctorow's argument. The steelman is about building computers that cannot physically contain the banned numbers.
Your original argument rests on the idea of building chips that are incapable of running AI. That is every bit as unreasonable as building chips that can't transmit, store, or play back child porn (or any other illegal content). It runs entirely contrary to the concept of general purpose computation.
Cory Doctorow spotted the knives coming out for general purpose computers a long time ago, and he's been writing about it ever since. It's the entire basis for the side which is contra your argument.
Exactly this, in combination with reviving the clipper-chip backdoor.
I think Scott has in his mind a mental image of "AI chips" as a special category because it feels like graphics cards with a lot of VRAM are special.
But that's not how chip manufacture works. It's the same graphics cards with variable amounts of VRAM to segment the market. Little different to how apple segments the market for their phones by selling high and low ram versions of their product. Anyone with modest skill and the right solder masks can swap in more RAM to an iphone and there's youtube tutorials on how to do so.
So intrinsically this requires any such controls to be built into the hardware of the graphics cards, not just some graphics cards but any graphics cards that people could hypothetically attach extra VRAM to. So yes, this absolutely 100% means regular people's graphics cards.
never mind a few years from now as hardware advances and the much more routine demand for massively parallel processing pushes normal CPU design further in that direction.
Scott is making exactly the mistake doctrow outlines. Treating AI-capable like a "feature" that can be cut out of a general purpose computer.
To the tech people, the hero's of their community spent decades of hard-fought battles to keep government backdoors and government spy software out of people's computers.
He's casting all of that into the flames for a solution that probably wouldn't work well anyway. But the backdoors and monitoring will never go away once they're in place.
"First they came for the paperclip maximizers..."
“Then they came for the baby eaters…”
I think this gets the question backwards, instead of asking why would governments regulating AI be bad? We should ask, why is government regulation of AI justified? This presumes that coercion is generally wrong and requires a satisfactory threshold of justification to be met before supporting it, especially on a societal level.
I’m disappointed to see that the most common objection to AI government regulation is that it will result in a dystopia. I believe that governments are currently the most dangerous organizations and government AI regulation only increases that danger by the same amount that AI is expected to be dangerous in the first place.
However, there’s at least two reasons to oppose governments regulating AI:
One, it’s morally wrong to coerce people without enough justification. The pro government regulators would need to make the case that coercion from and only from the government (while also exempt the government) today actually reduces harms later to a large enough degree to make the coercion worthwhile. I don’t see how that’s clear at all, but in order to enact coercion, the justification should be quite obvious. That is how we ordinarily view coercion. If Bob were to punch Alice we would think Bob was confused if he said, “Alice might have gone on to punch a few other people so I had to deter her from doing so”. There’s a substantial degree of justification required to punch someone that this scenario simply wouldn’t suffice. This isn’t quite analogous to the AI situation, we probably need to account for more parameters. But it’s at least somewhat analogous to people advocating for coercion with a low bar of justification. The supposed risk of AI is much larger, but so is the suggested amount of coercion to be imposed.
Two, government regulation of AI will not achieve the desired outcomes that its proponents believe it will. The common reason I’ve seen is that governments regulating compute power lowers the odds of a catastrophic risk agency’s agents acting maliciously or negligently in a way that causes harm to a massive number of people. However, there’s more reason to believe that the government is an existing catastrophic agency whose agents already act maliciously and negligently more so than any other organization. In all of the actual legislation (such as the RAISE Act in NY) there are explicit exemptions for government AI systems. If there were to be a larger number of agencies that can possibly keep a catastrophic risk agency’s agents in check, then that would be an improvement. This would only become less likely if we impede the non-catastrophic risk agencies ability to grow, while also exempting the greatest at-risk agency.
I wonder what your thoughts are on this. In general I favor a polycentric approach to regulating, and I believe the same approach will help with AI. This way, if someone regulation is good, more regulators can adopt it. And if some regulation is bad and expected to be catastrophic, it will have a more narrow impact on society, rather than a single regulator that imposes regulations on all of society. If that regulation is catastrophic then it will affect all of society negatively.
Here is my case for regulatory entrepreneurship where I account for AI risk from the perspective of those who consider it to be such an existential risk that it requires governments to regulate it: https://morologia.substack.com/p/regulatory-entrepreneurship-is-morally?r=6xuckk&utm_medium=ios
> "One, it’s morally wrong to coerce people without enough justification."
No country in the world is a libertarian minarchy. We regulate all sorts of products for all sorts of reasons. It's true that you shouldn't do this with no justification. Our justification is preventing global dictatorship and the eventual extinction of humankind. I think this is pretty good as far as justifications go, way better than "these eggs might have salmonella" or "this building might block someone's view". If you don't agree with the justification, say so, don't retreat to basic foundations of political philosophy!
> "In all of the actual legislation (such as the RAISE Act in NY) there are explicit exemptions for government AI systems."
I don't think this is true. It might not even make sense - the government isn't training its own frontier models, it's using Claude like everyone else. In any case, there's no exemption for the government in Plan A.
> "In general I favor a polycentric approach to regulating, and I believe the same approach will help with AI. This way, if someone regulation is good, more regulators can adopt it. And if some regulation is bad and expected to be catastrophic, it will have a more narrow impact on society, rather than a single regulator that imposes regulations on all of society."
This doesn't make sense for global threats. If there's a destroy-the-world button, you can't have every jurisdiction decide on its own whether or not to permit pressing it, because if even one jurisdiction decides yes, then the whole world is destroyed.
The government wouldn't tell you if it was training its own frontier models. And if it was outsourcing to Palantir? Well, it could even mark their work as classified, and not let them discuss it outside of "we have a big government contract."
> No country in the world is a libertarian minarchy.
I think that depends rather heavily on how you define a country.
https://en.wikipedia.org/wiki/Uncontacted_peoples
> Our justification is preventing global dictatorship and the eventual extinction of humankind. I think this is pretty good as far as justifications go, way better than "these eggs might have salmonella"
More grandiose is not the same as better. Suppose a man kills his neighbor with an axe, and the judge asks him why he did it. "I caught him trying to poison my family's food," backed with appropriate evidence of how such an axe-based intervention was the best available option for preventing known greater harm, might be received far more positively than something like "I had to stop him from summoning a demon who would probably have killed everyone in the world," in the context of ongoing dispute among relevant experts about whether such a demon is even theoretically possible.
Incidentally, I think the sort of Alignment you want, and the sort of instant-win superintelligence you're worrying about, are both rooted in similar logical errors. Absolutes are easier to think about, so they sneak in when wrestling with any big problem, and then if the corner-case solution created by those converging absolutes is exciting enough to demand action all on its own... well, obviously the only solution to a conceptual-absolute villain is a conceptual-absolute heroic quest. Were the many simplifying assumptions underpinning the upper and lower bounds of that worst-case scenario actually valid, or do the tails come apart first? It's too good to check.
> If there's a destroy-the-world button, you can't have every jurisdiction decide on its own whether or not to permit pressing it, because if even one jurisdiction decides yes, then the whole world is destroyed.
That's ultimately how nuclear weapons are regulated, though. If Putin decides he really, really wants to turn some some city into a glowing crater, assuming he can scrounge up a working warhead and delivery system, he can just do that.
Same for anyone else in charge of a sovereign state. The only thing the various treaties and escalation ladders do, is try to avoid creating situations where that option would be advantageous for the one choosing it.
Before nuclear power was militarily relevant, back in World War One, it became possible to cause damage of previously unimaginable scale and thoroughness, with just conventional chemistry, metallurgy, and industrial mass production. It would probably have been possible to kill everyone in the world that way, if the group trying to do so were somehow the only one with aircraft and artillery and so on.
Fortunately (?) there was a multipolar environment rather than a singleton, along with considerable investment in cheap countermeasures such as trenches and gas masks. Various unaligned minds such as Luigi Cadorna butted up against each other and got bogged down in futility, until enough of the saner participants noticed that all this devastation wasn't actually gaining them anything.
I'd be grateful if you would tell me which uncontacted people is a libertarian minarchy. Tribal societies do not tend to be.
Wow, first off thank you for the reply, Scott. What an honor! I will do my best to explain my reasoning. This will be a two part reply and might not be as clear as your writing so please feel free to skip to the TLDR in part 2/2 and I will try my best to summarize it neatly.
Part 1/2:
> “No country in the world is a libertarian minarchy.”
I don’t think universal practice settles anything; every widely-shared institution looks justified from inside its era, which is exactly why the burden question matters rather than being a retreat from my view. No country in the world was free of slavery for a major part of history.
> “Our justification is preventing global dictatorship and the eventual extinction of humankind.”
I am not sure that AI firms appear to be dangerous enough to warrant government coercion against them. I think there’s a degree of concreteness required for a justification to warrant coercion. If AI were to cause human extinction, it would likely be from humans using it for harm on a massive scale. Of the humans that would likely commit these sorts of harms, they would likely be government agents. Look at every other technology used to harm humans on a massive scale; guns, drones, bombs, and biological warfare have all been typically and profusely used by governments. I think that is a bug of governments, not AI.
Would you support ordinary actors arresting government agents, holding court proceedings, and convicting them and imprisoning them all with non-government institutions? So far the track record of what organization is most likely to be a global dictator and make humankind extinct, appears to be governments. Nuclear weapons and 20th century democide are the clearest examples of this.
It appears that if the justification lies with any organization it lies with non-governments needing to hold governments accountable, not the other way around.
> “If you don't agree with the justification, say so, don't retreat to basic foundations of political philosophy!”
I claimed that it’s wrong to coerce people without enough justification. I never said they aren’t providing a justification, I just believe it doesn’t clear the bar. That’s my bad, I could have made my position more obvious.
However, I do think the general question of why only the government should be justified in imposing regulations is relevant. Why wouldn’t we want the government to be subject to the same coercion in that case? For example, if we accept that people ordinarily shouldn’t steal from others and that jailing them for doing so seems appropriate then why wouldn’t we want government agents to be arrested for imposing taxes on everyone? That accounts for more theft than all private actors combined by at least two orders of magnitude. It also funds horrible things like wars and factory farming. That’s an existing live harm to almost everyone in society at an incomprehensible scale, and it’s widely neglected. If coercion is justified to prevent large scale harms, then the largest ongoing coercions deserve a similar inspection to see if they shouldn’t be addressed at a higher priority.
> “I don't think this is true. It might not even make sense - the government isn't training its own frontier models, it's using Claude like everyone else. In any case, there's no exemption for the government in Plan A.”
I am overreacting by saying ‘all’, shame on me, but the RAISE Act is to my knowledge the most popular legislation to date favored by gov AI reg advocates and it makes government exemptions explicit. Sections 1420(9), 1420(3)(b), 1426 list “persons” as basically any nongovernmental organization, exclude lawful activity of the federal government, and exempt the New York tax funded Empire AI consortium respectively.
I believe the government is likely trying to train its own frontier models at least, but I do think it’s clear they lack the comparative compute power. I still think governments are the most likely organizations to commit mass harm using AI, though again, I think that is a bug of governments, not AI.
In the case with Plan A, it seems widely presumed even in this case that regulations are only permitted to be performed by governments. Any attempt made by non-government agencies to regulate other non-government agencies or government agencies, will be met with coercion from governments and that is largely acceptable. I don’t think we should recommend strategies that work within this presumption because I think it is faulty to begin with.
Part 2/2:
> “This doesn't make sense for global threats. If there's a destroy-the-world button, you can't have every jurisdiction decide on its own whether or not to permit pressing it, because if even one jurisdiction decides yes, then the whole world is destroyed.”
Regarding whether every jurisdiction decides on its own, what is the proposed solution for global enforcement of any regulation? I suppose the answer is for each country to enter into some international treaty or agreement, but I’m not sure that would actually resolve the problem. However, even if it did, why couldn’t that same approach work within countries to have multiple regulators?
I think insofar as the speculative threat is misalignment specifically, I don’t see this as enough justification to deserve government coercion. What can be shown throughout AI development that would cause the doom hypothesis to be less likely to happen? Is it only if AI development doesn’t happen? Every result gets read as confirming doomerism. Either misbehavior confirms danger or good behavior means insufficient capability or successful deception. Is there any evidence that would count as AI not being dangerous?
Is there at least some fork in the road where if the development of AI looks more like X then we must have governments regulate AI, but if instead AI development looks more like Y, then we must avoid coercing AI firms? If there isn’t even a Y then I believe the burden to avoid coercion is too high and unreasonable. I think we are closer to the Y track; AI firms invest massive amounts of money into safety concerns and they genuinely are already creating a polycentric approach to safety enforcement without specifically relying on government coercion. For the X track to be more apparent I think there should be some measurable harm that occurs over a high enough degree to warrant coercion. I would accept at least the original RAISE Act’s own standards: 100+ deaths or $1B in damages as a threshold. But they have to actually happen, the speculative risk of them isn’t enough to preemptively impose coercion at that scale.
Mind you, governments cause far greater damage and almost nobody suggests that we should be coercing them, so I still think the double standard is painfully obvious and makes it difficult for me to press too hard to coerce AI firms even in that case. I still concede coercion would be worthwhile at that point.
I can’t tell how the extrapolated high probability of x-risk is supposed to be demonstrated. I hold a strong opposition to coercion in general. In order for AI dangers to count as an exception, shouldn’t there be some observable threshold we can point to that makes coercion more justified?
The catastrophe as a concern without any tells beforehand seems to demand too much and historically appears to be discounted by the doomsayers being wrong. There have been many world ending risks that were used to warrant coercion and those risks do not appear to be legitimately damning for humanity.
Population growth alarmism in China is probably the best example of doomer justified coercion causing massive harms. Thailand and South Korea apparently had the same outcomes without the coercive effort. Others are the eugenics sterilization programs around the world, with a scientific academic consensus of that time and SCOTUS rulings backing it, nuclear regulation leading to increased CO2 production (if you accept the added x-risk caused by that), and immigrants supposedly bringing about sharia law and the great replacement sort of rhetoric. These range across a whole spectrum of credible expertise down to bigoted panic. All of these were coercions justified by extrapolated risk, imposed without observed harms at any comparable threshold and they produced real harm with little or no benefit.
Humans have a dramatization bias and there aren’t harms shown to occur over a high enough threshold which is what my view requires to support coercive acts. The probability of x-risk extrapolated from AI testing and theory should be discounted heavily in my view.
A single regulator appears to be the most dangerous thing existing today, and I don’t see how introducing a supposedly dangerous technology makes that less true rather than more. As with most monopolies, a monopoly regulator provides worse quality at higher prices. The majorly bad actor is most likely to be the least accountable one, which to date has meant government agents.
I find it plausible that having competitive regulators increases the likelihood of producing a safer regulation that can be adopted by the others. And I have doubts that reducing the competition would still have a similar chance at producing a beneficial regulation. Especially when that regulator has a track record of producing harmful ones.
I’m sure I come across as biased and ideologically motivated and I probably am those things, but I also am just trying to be honest about my beliefs simply being what they are.
I just don’t think that the speculative risk associated with AI justifies government coercion against AI firms. I don’t think it’s worth prioritizing advocacy for such coercion above addressing existing harms committed by the government such as stealing from millions upon millions of innocent people to subsidize mass animal slaughter. Or coercively preventing other organizations from competing to provide regulations in society.
TLDR; I hold a high bar for coercion to be justified. I think that’s right to do. We agree coercion needs justification, but disagree on whether AI x-risk clears it. Where I think we differ is that I don’t believe the extrapolated risk drawn out from lower stakes lab tests and AI arguments count as enough evidence to cross the threshold for justifying coercion against the AI firms.
I think current government coercion occurring at a large scale requires much more immediate attention. I also believe the track record of governments makes them the biggest threat to humanity that should be addressed right now. Maybe that is from the dramatization bias I have from my personality type. So I apply the same discount to my doom story that I think AI doomers should apply to theirs. I think mine survives it because state harms are observed and well established, not extrapolations.
I do think that polycentric approaches to regulation will produce better results in the long run. AI firms are already demonstrating a polycentric approach to AI safety that appears to be little influenced by government coercion.
There’s probably tons more to say on this topic. I think there could be a threshold that AI firms cross which warrants coercion such as causing 100+ deaths or $1B in damages, but I don’t think that large scale coercion makes sense before that occurs as a marker. I want to know what evidence would count as us being on the safe track that wouldn’t merit AI firms being coerced. Is there already some, and it’s just not enough to bring the risk below your threshold? I see it in the other direction: there is at least some evidence AI firms are on the dangerous track, but not enough to warrant coercion. At least not yet.
Head to head competition is the primary demand driver of cutting edge intelligence. Think, military & finance. Open source models are an asymmetric military multiplier, they allow usa adversaries to leverage asymmetric tactics and strategies more effectively. Plan A is doomed because it doesnt understand the nature of why our world is a dystopia, it thinks things are just the way they are because people arent smart enough. The military planners in both countries understand this much better than you do, which is why ai is allowed to expand. If you want to stop it, you have to adjust the geopolitical landscape - something that requires raw power with intelligence as a competitive input, not a deciding factor.
Our world isn't a dystopia. Poverty keeps dropping, we're curing new diseases, and even the obesity problem created by our unprecedented abundance of food is being fixed by new drugs. There are real serious problems, like below-replacement fertility afflicting basically every country except Israel and the most impoverished, but subcultures are able to solve that by just adhering to religions that separate them from the majority.
If you use the metrics that our system evaluates itself by, our world is generally a success. However if you read the article, in context the dystopia I am referring to are the loss of basic liberal values such as freedom of movement, commerce, speech, religion, and privacy.
Are you talking about the regulatory state in the US, or "our world" as a dystopia?
The cybernetic system is international, so both.
I suppose the most plausible way to say "the cybernetic system" is monitoring the whole world would be to point to the displacement of cash purchases by electronics. But even in Germany 51% of purchases are made with cash https://www.visualcapitalist.com/ranked-countries-that-use-the-most-cash-in-2025/
Cybernetics refers to loops of monitoring and control thus resulting in dynamic system states - stable ones in negative cybernetics and unstable ones in positive cybernetics (ai explosion is positive cybernetics, financial repression is negative cybernetics). Cash doesnt escape these control structures as its monitored at several levels: first vat, second bank, third polling. Just the fact that you can give an accurate number (51%) puts us in the realm of monitoring, which is half of the control cycle.
Theres an interesting argument (which you havent made) that there is an entire underground, unmonitored and thus uncontrolled world. Here that 51% cash transfer is the iceberg tip hiding a 99% untracked economy, where almost all activity is unmonitored. However from context (slate star codex) the assumed "base truth" is that government statistics are "mostly accurate" so the dystopia is assumed.
India tried to go cashless. Bangladesh is going cashless (we'll see if the gold-lovers riot). This is a Davos plan that Davos really really wants.
They want to track your money.
You don't know how cumbersome it is to not have cash on hand, until you have to get brass authorization to use an ATM.
Last I checked obesity was still increasing year-to-year virtually everywhere.
> "...but subcultures are able to solve that by just adhering to religions that separate them from the majority"
Yeah, unfortunately said religious subcultures occasionally have the imperative to displace/conquer/nuke their neighbours (the conflict between Israel/Palestine being an interesting preview of where that leads.)
I was thinking of the Amish as one such subculture, and they don't have that imperative at all. The others I was thinking of are the ultra-Orthodox, who could be blamed for their political influence in Israel but are also blamed by their fellow Israelis for not serving in the military (though the more mainstream national Orthodox do).
If the Haredi/ultra-orthodox faction were forced to work and serve in the military it might reduce their fertility to some degree but I don't think it changes the long-term dynamics here. The human subcultures most interested in raising internal TFR are the least interested in global geopolitical cooperation.
The Amish aren't interested in "global geopolitical cooperation", but they aren't at all into conquests, as they're pacifists.
You know perfectly well you're only harping on about the Amish because they're a relatively weird and non-generalisable exception, even among religious ultraconservatives. List all the other insular high-TFR religious communities in the world and tell me what percentage are committed to pacifism in principle.
The whole point of Plan A is a treaty between the US and China pausing this form of military competition for mutual benefit.
Yes but it misunderstands when these types of agreements work. Its not because both parties "see the light" its because the geopolitical strategy allows it. For instance the intermediate nuclear missile ban treaty - this was something that provided symmetric value to both sides. Open source Ai does not, it disproportionately helps usg adversaries leverage complex, asymmetric advantages in the way that it doesnt for usg. Thus China would sign the treaty while secretly violating it.
The whole reason we're discussing regulation like registering where every chip has to go is so that it's impossible to substantially violate the treaty. Please read the original post - the idea is that *if* we implement all these regulations and mutual inspections, China shouldn't be able to hide more than a trivial fraction of its chips - too few to do anything useful with.
The post is a bit unimaginative in how national actors could circumvent the proposed changes - but if they did work, china simply wouldn't agree. If china does agree, thats evidence they found a way to circumvent them. The asymmetric advantages are simply too large.
China will build copies of its public chip factories underground that create chips that are not registered and do not have the kill switch/safety features and transport them to underground datacenters and research centers.
China will probably have a much easier time discreetly building these factories and data centers and diverting power and hiding the heat emmissions than the US, though I'm sure both sides will do it.
China will do this irrespective of any regulations or international treaties that it may or may not sign. Other countries may suspect they are doing this or even find some evidence of it, but nobody will be willing or capable of stopping them. China will deny and push ahead anyway. The entire world couldn't stop North Korea from obtaining Nuclear Weapons. The idea that the world could stop China from building a chip factory or a warehouse full of computers is just completely unserious.
There's just no way it will slow the AI race even a fraction of a percent. Probably the regulations/treaties will still happen, but just to help cement certain groups in power in the west.
That depends on whether China actually wants to win the race. If the prize here is mass unemployment and/or a skynet scenario, what does the winner get?
Both of those are speculative while the financial and military benefits are empirical.
Consumers will have access to "good enough" intelligence within the next few years - perfectly convincing photos and videos, and a genius-level Jarvis for personal stuff and to write your college ethics essay. That can quite likely still be accomplished with open models, even if they lag or are last year's news.
A bigger issue for overall technological progress is that the plan essentially kills off fine-tuning or smaller bespoke models as well, or puts it within strict oversight of the major labs. So not only is the future Claude Opus 11 or whatever regulated and walled off and guardrailed and rug-pullable, but your business / research institution can't meaningfully build its own alternative - either on Opus, by fine-tuning an open model, or by training a narrowly specialized model - thus giving the labs effective research veto power or monopoly (why share non-ML research if you can grab the patents yourself?). Worst case, if you're not a major AI lab, you're cut off from the research frontier in general, begging for Anthropic's scraps.
It still seems like the likeliest outcome is one you already hint at: implementing the easiest, bluntest or least effective restrictions without the exactly right spread-the-wealth mitigations, resulting in a precarious balance of distrust that won't last anyway. Adding more horses to the race may provide a bit of drag and friction, but everyone loves to back a winner, no one will pour trillions into being an also-ran, and if not every single element of the plan is in place, one horse's exponent still beats all and we're fooming off to the races.
> ...and a genius-level Jarvis for personal stuff and to write your college ethics essay.
On a side note, the real tragedy is that writing your college ethics essay does not require a genius-level Jarvis; any algorithm that can string a few sentences together would probably suffice. This was always a problem, of course, but I worry that all the hype around LLMs will normalize the situation, to the point where people think that the skills required to write stupid college essays are beyound the power of mere mortals.
> "Consumers will have access to "good enough" intelligence within the next few years - perfectly convincing photos and videos, and a genius-level Jarvis for personal stuff and to write your college ethics essay"
I don't understand how that prospect doesn't fill you with dread and horror.
I hoped the snark about the ethics (!) essay would imply that that there are undeniably serious concerns and downsides. I'm certain things will get very weird, and society is likely to change permanently in certain ways, often for the worse.
The lack of dread and horror is because the technology is already 80-90% there - including on open/local models - for those who are on the bleeding edge or sufficiently motivated, and society isn't _quite_ collapsing.
More relevant to the Plan A debate, this level of consumer access to AI is a genie that's already long out of the bottle, and shouldn't really affect anyone's support or opposition to it. Consumers will have their "good enough for them" AI regardless, so it really comes down to what researchers, developers and governments will have.
I think there are several genies that will have to be stuffed back into bottles in the coming century if anything recognisably human is to survive, so I don't really buy that argument.
That's a position I'm not going try to to dissuade you from.
But given how much capable AI is already available on individual, <$1,000 machines, downloaded tens or hundreds of millions of times, and given the slow speed of regulation, nothing apart from blanket bans on the possession of AI models or general personal computing devices would accomplish that.
A criminal prohibition on owning unregistered hardware made after 2019 would do the trick, maybe - that's what you'd be asking for, and it goes far beyond what Plan A argues for.
> "A criminal prohibition on owning unregistered hardware made after 2019 would do the trick, maybe - that's what you'd be asking for, and it goes far beyond what Plan A argues for"
I'd date it back to 2015 or so, but yeah, that is probably the scale of what will eventually need to happen. (We haven't really even gotten into the whole "AI personhood" debate yet, which might be the clincher.)
> Any data center that trains AIs need[s] to be [...] verifiable (someone needs to be able to prove they’re running the code they claim to be running).
This is one of the parts that I don't get. How do you prove this? Not just once, and not just for a certain "demo" computer, but for the whole data center, at all times?
See https://ai-2040.com/supplements/verification-plan for details - your question is closest to the "Workload Approval" and "Workload Verification" boxes.
Thank you! If there's one thing this plan is, it's thorough ;-)
IIUC, the idea is that you can decompose the inference work into "packets": not necessarily the full prompt-to-response chain, but some sub-part of that.
Then, the data centers have "network taps" that basically make the entire traffic of the data center observable. So, external auditors can see "this was the input packet, these were the weights of the model, and this was the output packet." Then they can re-run that computation themselves.
I know the authors already addressed this, but... it seems like in the BEST case scenario, where the tech works fine and nobody figures out a clever way to cheat, it's still dependent on a delicate political agreement between US and Chinese leaders which could break down on any given day.
More promisingly, the authors mention approaches where the chips are inherently designed to be inference-only, because they have the model weights hard-coded into them. (Companies like Taalas are already experimenting with this.)
This sounds great to me, because it's a much simpler verification story: you verify that the data center is using the chips they say it is, once, when the data center is built. From then on you know that the chips can ONLY be used to do inference.
So your position seems to be that there's a big risk that a private company could "take over the world" if it just got really good at selling AI, but that thinking maybe trying to get the government more control over AI would lead to the government getting too much power is crazy fearmongering?
In your scenario, AI becomes the predominant form of thought in the world. You want all the places where thoughts are produced to "register with the government and submit to inspections," etc. So the closest precedent for the thing being regulated is still the human brain, the previous place where thoughts were produced. We could require parents to submit their unborn child to inspection before birth, perhaps checking their polygenic scores for various traits, ensuring they pose no risks (after all, what if the kid's the next Hitler! That's an existential risk!). Regulating general-purpose computation (thru restricting open models, and to a lesser extend thru restricting chips) is closer to regulating speech/thought than it is to dairy inspection.
We have had a hard enough time ensuring free speech on the net, and that battle has been lost in China. Your proposal involves letting China have significant say here. It would make a standing CCP capability to brick American datacenters, giving them an easy way to threaten American AI companies over, say, their stance on Taiwan.
It's absurd to say controlled substance regulations are not very burdensome. Factory registration and possession criminalization aren't two separate policies. The scheduling system that makes Pfizer register with the DEA is the same classification setup that locks up ~0.2% of the male population (0.5% of the Black male population) for victimless crimes, empowered drug gangs that turned nations into failed states, and (thru supply-side crackdown on prescription opioids) gave us fentanyl.
And again, Xanax can't start being subversive, so the type of damage there was at least limited to unintended consequences of regulation, not intentional “Orwellianism”. In general "but they wouldn't do that because it doesn't make sense" is not a good way to reason about regulators. Who are we putting in charge of these regulatory systems? some of the most irrational, paranoid, power-hungry, freedom-hating people in the world. And even granting well-intentioned regulators, we have to assume they can have good answers over time for what counts as an AI chip when algorithms improve? What happens to all the hardware we didn't think was dangerous but now is? In general, there's no good story in Plan A about how regulators suddenly start caring exclusively about safety, are capable of evaluating it, and how they don't just become beholden to special interests and start imposing all sorts of other regulations that have nothing to do with safety. There's lot's of precedent for this sort of thing -- has the environmental movement even been good for the environment on net lately?
Maybe slowing down means "this gives ample time to debate (and agitate for) institutions like UBI. Institutions which have some power at stage n (for example, democracies and legislatures) can attempt to codify the rules that would let them keep power at stage n+1." But it also gives more time for special interests to get all sorts of horrible controls in there. And of course, it bars us from any future where the USG or CCP gets disempowered, which is probably most of the good futures (I would assume most ACX readers already believe this about the CCP?)
One thing close-ish to "magical zero-cost, zero-regulatory-burden alternative" is insurance, i.e. Hanson's "FOOM Liability" idea. In general we should prefer simpler, more tested approaches that rely less on the wisdom of regulators, like liability and tort law.
KYC is dystopian and we should not accept it as a matter of course. More broadly, the level of control the US government has over the currency and financial system is what allows it to easily spend trillions of dollars on horrible, unpopular wars, etc. And of course the US has a relatively easy time compared to China, where their regulation has been integrated with the social credit system, leading to a system where online speech can get you barred from basic transactions! Our panopticon is a bit more blinded than theirs because of our culture and constitutional constraints, but we can’t expect that to hold forever.
"In your scenario, AI becomes the predominant form of thought in the world. You want all the places where thoughts are produced to "register with the government and submit to inspections," etc. So the closest precedent for the thing being regulated is still the human brain, the previous place where thoughts were produced."
Again, you are thinking of this in some kind of crazy far mode way. Do you know how many inspections and registrations chip factories already have to go through?
Insurance is useless here because there is no way to have an "ends the world" insurance or payout.
> Again, you are thinking of this in some kind of crazy far mode way. Do you know how many inspections and registrations chip factories already have to go through?
I have lots near mode objections too (see the rest of my comment) but far mode is useful to avoid frogboiling. If for each new regulation we say “see, this new regulation is just kinda like how we regulate all these other things“, at each step it seems reasonable enough and the people (like myself) who say “actually implementing those other regulations was a bad idea and in an abstract sense you can see why this new regulation is bad“ sound paranoid and reactionary, but eventually if you let this process continue you end up in a world that is far poorer and less free than it could have been.
> Insurance is useless here because there is no way to have an "ends the world" insurance or payout
surely there are many intermediate steps up to ending the world that involve insurable harms
> " We could require parents to submit their unborn child to inspection before birth, perhaps checking their polygenic scores for various traits, ensuring they pose no risks (after all, what if the kid's the next Hitler! That's an existential risk!"
Isn't that what embryo selection is for, broadly speaking? "No hyperintelligent psychopaths born" doesn't seem like the most demented government regulation.
> "We have had a hard enough time ensuring free speech on the net, and that battle has been lost in China. Your proposal involves letting China have significant say here. It would make a standing CCP capability to brick American datacenters, giving them an easy way to threaten American AI companies over, say, their stance on Taiwan"
I'm supportive of Taiwan and I'd prefer a world where the CCP was deposed, but if giving Taiwan to the CCP was the only way to avert a military AI arms race, I'd make the deal. I'd also nuke Beijing, invade Iran, and/or cut off oil to the Chinese mainland and possibly trigger a hundred million deaths in the process, if that was the only way.
TLDR: It's a terrible solution but the alternatives are plausibly worse
As a matter of realistic politics, you’re asking people to submit to what they perceive as a severe status hit. (Not commenting on other merits of the proposal here, just explaining the intensity of the opposition.)
In the Anglosphere, professions are ranked by status roughly like this:
1. True while-collar professions with quasi-priestly prestige, which are institutionally autonomous and regulate themselves. Doctors, lawyers, and academia are the prime examples.
2. Technical professions that are largely unregulated. Low-prestige, but also some deserved pride in freedom and independence. These also attract personally types for whom bureaucratization is truly insufferable.
3. Ones that not only lack prestige, but also have to submit to heavy bureaucratic outsider regulation.
Justifiably or not, computer people perceive these kinds of proposals as a request to move them from (2) to (3).
Of course, any industry that’s on the public radar can’t stay in (2) as a stable state, and access to (1) was closed long ago. So this move was bound to happen eventually. But it seems clear why there will inevitably be loud complaints as it happens.
I asked Google Gemini about the story summary/paraphrase source, which replied, straight away, of course: “The exact story you are referring to comes from an interview in the Atlantic. The anecdote was shared by Alexis Ohanian (co-founder of Reddit) in a broader conversation about his frustrations with Silicon Valley’s hyper-libertarian culture and reluctance to accept baseline governance. When he would try to explain basic regulatory frameworks to tech developers, they would instantly jump to grand, sweeping objections—often claiming that such rules could only be enforced by a "world dictatorship." He then had to remind them that this was simply how standard consumer goods like milk or eggs have been regulated for the better part of a century.” I then asked Gemini for the release month/year and volume of the issue, the headline, and the start page number of the article, which Gemini was unable to provide.
I still side with John Cochrane https://www.grumpy-economist.com/p/ai-society-and-democracy-just-relax For all the problems with the FDA, it was at least created after we'd experienced things like tainted food so we understood the problem we were regulating to avoid (it was "ex post" rather than "preemptive"). These proposals are all worried about something in the future, whereas people in our own past would have been too ignorant of our present with its novel technologies to create regulations suited for it.
> Yet tech companies achieving an outsized share of power is no longer “speculative”
Yes, it is. The actual government is still in power. Tech companies headquarted in San Francisco don't even have the power to make that act like a first-class city, or prevent California's wealth tax referenda from reaching the ballot.
> you owe people an explanation of your magical zero-cost, zero-regulatory-burden alternative
We cross that bridge when we get to it, like other "ex post" regulations in our history.
> They just assume that things with few benefits that are passed without fanfare, like random Trump administration chip regulations, must be fine
No, I don't assume that Trump's regulations "must be fine", and citing Trump as having done something is not evidence that it is a good idea.
> they’ll end up with insane restrictions on eggs and milk and financial transactions for no reason
I'm glad you agree those are insane, but it also seems crazy to expect a government which adopts insane regulations to do a good job when it regulates even more.
Tech is building a new San francisco -- that was their solution to "fixing San Francisco."
They haven't done it yet.
They paid millions of dollars for the plans... They've sunk nearly a billion dollars into the real estate. (Granted, moguls have a LOT of money).
> AI inference price per token fall by 98% every year (this is not an error, they actually get 40x cheaper every year).
A couple comments on this trend (I work at Epoch AI):
1. You probably know this but in case readers missed it: this work attempts to measure how fast inference prices fall *controlling for capability level*. For example: "how much cheaper does it get to reach a 50% accuracy on GPQA Diamond every year?".
2. We did this almost a year and a half ago and a lot has changed since (eg at the time reasoning models were pretty recent and we excluded them from the analysis). We're collecting more data to run a more up to date and higher quality analysis (I also wish this wasn't the first trend people saw on this page https://epoch.ai/trends ... we'll also update that).
3. Gun-to-my-head, my guess would be that 40x/year is a pretty reasonable estimate (with a lot of variance in either direction) as the initial decrease, but the price does plateau eventually (still figuring out when, but most likely less than 2 years after a capability level is first reached).
This is how eggs or milk...
In *one* country.
Didn't read the rest, since it would be on the same level.
And no, that isn't trustless. Those regulations have a pretty large enforcement mechanism. And that enforcement mechanism does not prevent cheating.
It's a bad model.
At what point do you try to gain influence in China and Chinese influential people that uses the Internet?
"Gain influence" is a complicated phrase, because we don't want to be seen as foreigners trying to subvert Chinese policy. But if you mean being in contact with Chinese academia, diplomats, etc, this has long since started, and they're having similar discussions on their side of the ocean.
Your anticipated scenario seems to be that one government and one company share control of most of the world’s productive capacity, with market entry strictly regulated by that government. That’s not a dystopian *surveillance state*, but it does literally sound like something out of an unusually think-y YA dystopia.
That's the default path, Plan A is meant to avert that. See the section "A is for Autonomy"
As you said, there’s a very real risk that companies and governments implement the power-concentrating parts but not the power-diffusing ones, and judging from most of human history power concentration tends to be a ratchet that is only unwound at the sharp end of a sword. I’m not optimistic that it’ll be different this time.
I'm sorry, but—
"The title of my substack post is raising a lot of questions already answered by the title to my substack post!"
I'd like a more in-depth explanation of the projected math here. Inference gets 40x cheaper every year and a specialized AI chip costs $40k. Does that mean next year you expect to run Fable on a $1000 GPU? Does that generalize to always being able to run (or train!) the previous year's best model on this year's consumer-grade hardware and therefore defeat regulation on specialized AI chips anyway? The exact numbers seem important.
The "inference gets 40x cheaper each year" figure looks like it has enormous error bars and needs to be read with a lot of caveats. Specifically:
- It is measuring the price-per-token (weighted average of input and output tokens) of the cheapest non-reasoning models that hit a given benchmark score as of a given release data.
- The results vary wildly by benchmark/threshold combination, ranging from 9x per year (GPT 3.5 Turbo level of performance on general knowledge questions and a coding benchmark) to 900x per year (GPT 4o level of performance on a "PhD-level Science" benchmark).
- It is measuring price charged by AI providers, not their cost to service the requests. We have only vague ideas how correlated the price is with their cost.
- The data set runs from March 2023 to December 2024, a 22 month period that ended 19 months ago. The trend might still be running on a similar trajectory, or it might not be
Using it to draw conclusions more robust than "token prices for GPT 3.5-4o levels of performance got 1-3 orders of magnitude cheaper per year between early 2023 and late 2024" is speculative at best.
Off the top of my head, the biggest reasons I'm seeing in the details to expect the trend to have continued post-2024 are 1) the steepest trajectories were for higher benchmark thresholds that started being hit by frontier models in the second half of the timeframe, and 2) eyeballing the charts, I don't see clear signs of leveling off near the end of the time period, so barring conceptual reasons to expect a quick level-off, we should expect the trend to continue for at least a big afterwards.
And counter-reasons for doubting those reasons are 1) the shallowest trajectories are comparing successive generations of distilled models, while the steeper models are comparing one generation of frontier models to successive generations of distilled models, 2) they excluded reasoning models because reasoning models need many, many more tokens to solve the same sorts of problems, which limits applicability, and 3) the apparent lack of leveling off might be an artifact of how the charts are constructed, charting the release date of models that lowered the price to hit the threshold of a given benchmark threshold. A lot of the benchmark/threshold combos stop getting new data points many months before Dec 2024, and the way they're handling the data would make the line stop dead like that if the price stopped going down for that level of performance at that date.
There have been recent advances in distributed training, and it might well be possible to do large-scale training if you have lots of old chips and plenty of electricity (like China):
https://arxiv.org/abs/2503.09799
Note a paper from a few weeks ago (by Robi Rahman, who I believe is an ACX reader) makes some attempts to measure the detectability of distributed training. Broadly finds that it's extremely difficult, so we really do need to move to supply-side restrictions such as attested chips, KYC, etc:
https://arxiv.org/abs/2605.29359
> 4. The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time.
This is technologically impossible, for two reasons.
Firstly, if you want to remotely halt the work of a chip that is sitting in a box under my desk, you must have direct network access to the box (and the chip). There's no way for you to do that if I unplug the network cable, or merely configure my firewall to block all the "stop chip" requests. You can get around this with more regulation; for example, you could require that all chip boxes should always be online and open to "stop chip" requests, but doing so will make the chips prohibitively expensive, and the amount of enforcement (in terms of manpower, network connectivity, etc.) you'd have to perform to do so effectively might bankrupt you.
Secondly, there's no way to ensure that *only* US and China have access to the killswitch. Your killswitch, once implemented, will be effectively public, so anyone who really wants to could trigger it. This is pretty bad, especially if you envision these AI chips being embedded in some critical infrastructure (I get it that the destruction of all AI chips everywhere and/or decoupling them from any infrastructure more critical than cat videos could be a desirable goal for you, but that was not the stated goal of Plan A).
This "killswitch" approach had been tried many times already, and thus far it had never really worked -- at least, not well enough to support your vision for AI.
What about going the other way? A module in the chip that stops it doing compute if it doesn't receive a call from home every few minutes; US gets keys for China's chips and vice versa. Take your data center offline, it stops computing. Unilaterally decide to kill the other side's chips, they kill yours. We know how to do root-of-trust, and it's neither hard nor expensive nor a significant fraction of the die's silicon area.
There are all sorts of problems with this scheme, but I don't think it's susceptible to the particular ones you list.
> A module in the chip that stops it doing compute if it doesn't receive a call from home every few minutes...
As I'd said before, this would make your chip prohibitively expensive; impose extremely high costs on whomever has to build all those data centers and network links; and would still be vulnerable to all the usual attacks that are successfully executed every day.
In our current world, major videogame/application publishers have tried to implement this model, with varying degrees of success -- but not nearly well enough to support Scott's vision. And they have a major advantage: they are trying to regulate high-level software, not low-level hardware (where the costs per unit are astronomically higher).
We're doing https://en.wikipedia.org/wiki/Trusted_Platform_Module right now, for very unimportant applications. It's already in approximately every relevant chip - the article discusses separate modules, but these days root of trust is just part of the chip, and it's a relatively tiny bit of silicon - and working as advertised. We're already paying all the costs you mention.
Modern computing devices have large, complex codebases that run before anything like a bootloader gets control, have dedicated hardware that runs them - at the bleeding edge, built right in the same chip!, that can be remotely accessed, and that don't stop running when the actual OS you see gets control. On the more complex chips you have multiple such things, because it's not the 80s any more and bringing up a modern superscalar chip from power-on to bootloader is actually super complicated. e.g. https://en.wikipedia.org/wiki/Intel_Management_Engine
All this stuff already exists.
AFAIK the TPM does not have a network card of its own, nor a killswitch, but perhaps I'm wrong ? Also, the TPM is notoriously insecure (although to be fair people are always trying to patch it too).
Finally found Nvidia's one: https://arxiv.org/html/2507.02770v1
You're right, none of these things are killswitches, because no-one's asked for that yet. But they are made to interact with the network stack, and the things they are actually doing are much more complex and costly than a killswitch would be, which is why I am happy to present them as an existence proof to counter concerns about complexity and cost.
I guess I just find it hard, in 2026, to see distributing a few thousand bits of keep-alive cryptographic nonce per minute per host through networks built to move terabytes of model weights per second as an insurmountable problem.
> But they are made to interact with the network stack
I don't know about NVidia (I only skimmed the article), but AFAIK the TPM cannot directly interact with the network stack. Rather, it relies on the OS to do so, and in fact to perform the chain-of-trust calculations. Windows 11 can and will refuse to boot due to TPM issues, but there are (or at least were) Linux distros that would happily do so.
You are right in saying that the problem is not insurmountable; it is just prohibitively expensive, and presents an insurmountable barrier to entry for any wannabe new players. I understand that Plan A potentially envisions effectively banning new players in the market anyway, but IMO this is also a bad idea.
By analogy, consider the economic turmoil that occurred due to blocking ~20% of the world's oil traffic. In the AI-driven world envisioned by "Plan A", reducing available compute/network bandwidth by 20% would have a similar effect.
Also note that even if you are correct and this "kill switch" can be implemented quickly and easily, this does not alleviate my other concern: now, almost anyone can shut down any AI chip at any time. This sounds pretty dangerous to me, especially if these AI chips are driving something economically or infrastructurally important.
It's an attack vector, innit? You've set it up that "broken AI" is the most likely scenario in terms of "any mishap you want to name" including solar flares, etc.
Perhaps all you have to do is reboot the datacenter before the alligators take up permanent residence (because of course the alligator-keeping-out gates were run by the AI, more physical security that)...
This makes it impossible to build safely air-gapped AI-dependent systems and renders your non-airgapped ones acutely vulnerable to denial-of-service attacks or just plain Internet glitchiness.
If one assumes that AI continues to be used in important applications, by doing this you're guaranteed headlines in which one DDoS attack on the key-provider takes down the AI making 2040s AWS/Azure/Cloudflare/etc. run and with it about a dozen countries' worth of heavily-used Internet applications. Gods help you if AI has made it into infrastructure support. And, of course, the possibility of government retaliating under the assumption that it was a deliberate attack by the other team and doubling the scope of failure is just the cherry on top.
If someone dubiously trustworthy (like, say, a government) were to pitch this to me as a policy, I'd assume they were trying to kill AI indirectly by making it such that no-one sane would consider making their feet so easily shootable.
> This makes it impossible to build safely air-gapped AI-dependent systems
Yes!
If we want continuous monitoring and control of relevant AI, we can't have that AI be offline, invisible to our monitoring and uncontrollable. Unlike questions of practicality of building the monitoring / control systems, this is a genuinely immovable axiomatic part of any such proposal that cannot be solved by money, politics or technological developments - it is a key deliberate part of what the regulation is trying to accomplish! - and whether the good outweighs the bad here is a genuine point of disagreement between the camps.
My underlying point here is that you have to include in your trade-off calculations that this particular design for the device intended to make it safe has the side-effect, undesirable in safety features, of making it dramatically less safe. In ways that, IMO, verge upon unusable.
(And when looked at from an even slightly cynical mindset, much like, say, solving the problem of hijackers using commercial aircraft as weapons by placing remotely-triggered self-destruct devices on the aircraft in question, it gives one furiously to question as to whether a given proposer is at all interested in "safe use" when "no use" is right there and much easier to get to.)
This is an implementation critique, I note: I would also oppose, as a member of t'other camp, stationing human CCP AI safety agents with kill-switches in US data centers and vice versa, but at least *that* implementation doesn't put a blow-up-society's-information-infrastructure button in the hands of any random script kiddie with a grudge.
> If someone dubiously trustworthy (like, say, a government) were to pitch this to me as a policy, I'd assume they were trying to kill AI indirectly
TBH the more I learn about Plan A, the more I think this is their actual goal -- which makes me want to support them even less. If you want to kill AI, say "I want to kill AI", don't insult my intelligence by pretending otherwise. But maybe I'm wrong, and they're just honestly mistaken...
IIRC, the idea is a dead-man's switch, such that US and China have to send their cryptographic permission to the chip every so often or it stops working. This gets around the "unplug the network cable" problem. And if you don't want third parties who aren't US or China interfering, don't give them the cryptographic key.
Can you explain in what context "killswitches" have already been tried?
Eh? You are advocating for a solution that have catastrophic consequences NOW (lock in, loss of privacy, extra judicial punishment) on many computing/networking devices, for mitigating a possible AI problem in the future? And the solution for the issue is government control on information technology (which historically has a tendency to empower individuals/emergent actors relative to established big players, when it spread freely)? For this to be convincing, people should both be extremely afraid of malevolent AI takeoff, and very trustful of Chinese+USA governments. Not my case, and I doubt it's common in here....Killswitches (deadman ones being the worse) are always a terrible idea, it disguise rent as sale, and should be banned as unfair selling practice. It's exactly how they are perceived and depicted regarding military (US) or communication (China) equipment bought by 3rd parties....
Third party who wants to interfere doesn't need the cryptographic key, in that case; they can kill a datacenter just by taking away the mandatory network cable. Clog it with enough noise that the scheduled stay-alive signal can't get through, for example.
> Can you explain in what context "killswitches" have already been tried?
Today these are mostly used for software, notably "always online" video games and some costly commercial applications; the Windows OS had attempted to this model as well. In terms of hardware, car manufacturers are also exploring this option, e.g. Chevrolet with their OnStar, and of course Tesla; but none of these systems require the car to be always online (not even Tesla, at least not yet). Note that in these examples the cost of the killswitch relative to the rest of the system is small.
The results thus far had been mixed at best. Users really hate always-online software, but are generally willing to accept it when it works reasonably well -- until the company pulls the plug, or their servers go down, or there's some other problem (which happen with predictable regularity). A consumer movement has formed around forcing gaming companies to take more responsibility for their killswitches (https://www.stopkillinggames.com/en); predictably, it had failed to gain much traction, as most consumer movements tend to fail.
To my knowledge there are currently no chips with built-in remote killswitches; the Windows OS can use the TPM on your CPU to implement the killswitch, but the TPM itself lacks this functionality.
> To my knowledge there are currently no chips with built-in remote killswitches
cf. https://en.wikipedia.org/wiki/Intel_Management_Engine
Ok, you're right, I did not know that the IME has its own MAC address and privileged access to the NIC. That said, the rest of the article appears to confirm some of my points:
https://en.wikipedia.org/wiki/Intel_Management_Engine#Security_vulnerabilities
> You Already Live In A Global Panopticon Orwellian Surveillance Dystopia
Yes, but I think that's a bad thing ! Don't you ?
Yes, but it means that the marginal extra unit of surveillance is not especially costly! If there was some kind of slippery slope where having any surveillance might justify more, sure, resist everything with your life, but if we have 1000 units of surveillance now, it's weird for someone to die on the hill of not instituting a 1001st which has especially high benefits and low risks.
What was it you said in Unsong, about the difference between "making compromise with sin" and "being less than maximally virtuous" ? Your current argument is, if I understand correctly:
1) if the presence of any surveillance would be used to justify more, it's right to resist
2) some surveillance is already present
3) I'm explicitly using the fact that it's already present to help justify adding more
4) Why are all these people resisting me, and trying so hard to climb back the other way? Don't they know how slippery it is up there?
In addition to what @JamesLeng said, I think your proposal adds much more than a single extra unit of surveillance. You are advocating for a radical transformation of an entire industry in such a way as to prevent free competition; such moves rarely go well (though admittedly this is not impossible).
"approximately a dozen OpenAI staff members debated" likely means one person did a Reply All on an auto-generated email.
A few issues I still see - the main one being that the verification plan does call for regulation of small scale inference compute too. The threshold for regulation is 10k H100 equivalent, so <1000 data centers currently. Fine.
But the plan proposes reporting requirements for all inference below that - does everyone owning a rack or TinyBox now have to submit to audits? What does that look like?
Furthermore from what I see these thresholds are static, but while the 10k threshold is currently 100 racks (a very beefy build), algorithmic and hardware improvements will compress that. (I’m assuming that if a new attention algorithm gets Fable-class for 10x less chips, we would adjust the H100e down by 10x, or adjust the threshold itself.) So in a few years you might see the actual threshold at 10 racks which is a lot more like “small company raises some cash and puts onsite” than “mega lab builds a warehouse scale datacenter”.
I also didn’t yet mention the ASIC overhang, which you expect to yield at least 10x but plausibly 100x perf/$ or perf/W improvements if BTC is anything to go by. If you slow down then the ROI of taping out the current frontier model increases dramatically (say we move from 3mo cycles to 3y, that’s a ~10x ROI window increase).
Finally just as a very high level point - hopefully around here we all understand that if you have open weights, then strong inference regulation is the only way to prevent a catastrophic jailbroken model from running in the wild. But in addition to banning open weights, this also means you need to treat closed weights as hazmat, and both lock them down at the source, and have some monitoring in place to detect if they get leaked into the wild. I think if you bite the bullet on research transparency, this also increases the risk of fine-tuning or otherwise improving a sub-frontier model. That load-bearing part of the proposal dramatically increases the risk of some new dangerous weights coming into existence. (I agree this is mitigated somewhat by inference monitoring, haven’t thought through whether that defense alone is enough.)
I still haven’t figured out who the actual audience is for this plan. I don’t think it’s the current administration (“make a credible long term deal with China”? We are at the deep historical nadir of capacity for such a deal). Maybe it’s the “deep state” defense & national security apparatus? Maybe it’s hoping to be distilled into the Overton window by NYT and WSJ? think this plan succeeds more often if people actually deem the threat worthy of surveillance, like nuclear proliferation risk. You wouldn’t argue that home labs should be able to run a fusion reactor.
[Still chewing on this so wide epistemic error bars.]
I listened to Steve Hsu’s podcast (“Manifold”) recently, and he interviewed Chinese graduate students in top AI/CS programs. They were laughing about how their labs gave them access to the training hardware that shall not be named (H100’s, obviously).
If you think we can have something made in Taiwan, but prevent it from going to Guangzhou or Shanghai, I have an export compliance startup I’d like you to invest in.
PLEASE DM FOR DETAILS.
Yes, our export controls right now are pretty bad. See https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose for more.
In addition to all the dystopian problems, this plan also proposes effectively nationalizing AI chip production and development. You bring up Big Pharma as an analogue, but a). Big Pharma actually doesn't function very well even in its semi-socialist state, and b). the regulations you are proposing are more onerous than those on drugs (e.g. every pill does not have a network-enabled killswitch embedded in it). Remember all the articles you wrote about your frustration with FDA, IRB, etc. ? Well, imagine that, but a thousandfold.
You frame this problem as merely "annoyingness", but that's a cute way to say "stagnation". You will effectively ban all AI development by anyone but a handful of companies (arguably, exactly two companies), and make it impossible for any independent research to occur in the field. And again, you would likely see this as an excellent outcome, but if that is your actual plan, you should be explicit about it -- otherwise you risk being seen as disingenuous.
While also steeply discouraging actual productive innovation within those two companies, thus presumably redirecting available creative energy mostly toward rent-seeking.
I just don't want to live in this kind of world. I like open weights models, and I don't think they are going to destroy the world. I think a nation state can still justify creating a $10b open weight model and giving it to the world.
10 years ago fear of zombies was at an all time high. Lots of plans came out just like this. You have not proven the threat yet. You are reasoning based on the idea that we all agree there is a threat. I am not convinced. You have not sufficiently established a basis for coercive restriction of freedom.
I don't think anyone was actually afraid of zombies ten years ago. Maybe ask Claude for a better example/metaphor.
Overpopulation, climate change, whatever the fear of the day is.
Overpopulation is no longer occurring, and policy interventions are a big part of why. If anything countries overshot.
Climate change is still a problem; did you miss the recent air conditioner discourse, or the fact that heat waves are killing Europeans, or how difficult it is in much of the world to go outside this summer? It could've been even worse without certain policy responses, and sorely needs greater ones.
You misunderstand my point. I am saying that this is an extreme plan based on speculative danger. It is a pandemic apocalypse plan, a One Child Policy, a $12T Green New Deal. The burden of proof for a plan like this one is much higher than what the author presumes. It's not firmly anchored in reality.
I regret the Zombie analogy. Clearly it confused the criticism.
> Your prior should be that Plan-A-style chip regulations would increase the annoyingness of being in the AI chip industry from ~50th percentile among US industries to ~95th percentile2, raise prices in some way that immediately gets overwhelmed by the general trend, and otherwise not have much effect on the balance between freedom and oppression in the world.
It kinda of surprises me that the writer of some really excellent heartfelt Jeremiads (https://slatestarcodex.com/2017/08/29/my-irb-nightmare/ , https://www.astralcodexten.com/p/adumbrations-of-aducanumab) contra stuffy overbearing absurdist regulation near his own sphere of expertise A is surprised that people in field B aren’t excited about the idea B becoming a lot more like A.
And also the general trend bit seems disingenuous: if chip industry does manage to squeeze out more iterations of Moore’s law, the effect of that has been to grow the capability of off-the-shelf consumer hardware so the stuff that in yesterdecades needed datacenter clusters or specialized custom chips now runs at home instead. To work at all, the regs have to cripple this general trend for consumers and smalltime use wholesale eventuallly, not take away 1% of it.
The only thing that bothers me is the removal of open weight models. I would just not even include that as the market will make it happen anyway, as you mention. And I think it is important that models available to the public can be post-trained and fine-tunes effectively and with some knowledge of the innards of the black box. Great post overall! In total agreement on most points, especially the part about us already living in a dystopia so why not try something new (or in this case, old, but tried and tested already with milk and eggs). Thank you!
I have always disliked the replacement of "must" or "should" or "ought to" by "need to." In this post I find seven instances of "need to" that mean "must," five of them in Plan A's five proposed regulations.
I don't think "ought to" and "need to" mean the same thing. One is a preference, the other is a regulation.
I don't think so either. But "need(s) to" is often used where "ought to" or "should", which have a more normative sense, used to be used. "Must" is unlike the other two in not being normative, but like them in being replaced by "need to" as in this post. "Needs to" implies a fact about the person(s) to whom it applies. "Must" implies a constraint, and IMO is more precise in a regulation. Without checking, I doubt that actual laws or regulations "soften" their force by using "need(s) to" rather than "must" or "required."
Don’t they specify required actions with “Shall”?
The regulations themselves don't seem categorically different from regulations we already have on many products, but I think the global enforcement mechanisms might be, at least if we want them to be effective. Normally in a trade agreement if you sell goods to another country, they're supposed to already be legal in that country and pass any local quality controls they feel like implementing, but if your goods happen to not pass their standards, you're usually still allowed to sell domestically or to other countries with weaker standards.
Does the same rule apply here? What if all the AI companies and chip manufacturers decide to set up shop in Argentina or something? If it comes to it, would we need to take military action against Argentina? (Nothing personal, Argentina, I chose you at random.) Now this isn't necessarily a fatal objection. Maybe it's already been addressed somewhere that I skimmed over. But it's the foremost doubt in my mind at the moment.
I think this might be - maybe not literally impossible, but so difficult that it doesn't really take much regulation to stop it. Witness how much difficulty the US has had transferring a small portion of TSMC's capabilities here, even with all of Taiwan's, TSMC's, and the US' enthusiastic support.
The easiest way to solve the specific scenario you mention is for the US to ask the Netherlands to ask ASML to stop supplying them with E-UV machines if they do this. Or maybe there's some reason that particular method won't work, but these are the most complicated supply chains that exist, and I don't think you can reconstruct them with both US and China opposed. I don't think this needs to come anywhere near military action.
Imagine that AI did not exist (somehow). Do you think humanity would be better off with the current state of affairs (as you've presented them); or with a market where chip design and manufacturing capabilities were commoditized, allowing multiple independent actors to pursue their own research ?
To put it another way, do you think that humanity would be better off with a handful of major corporations controlling all software development, vs. the current state of affairs where most mission-critical software is based on open source, and software startups abound ?
I’m interested in the 50th percentile estimate. I naively assumed that most software is on the left side of the bell curve, moving to the right as it gets more publicity.
Moving to 95% seems plausible. I could see it being higher; the regime you describe is more constraining than ITAR, but I’m not sure what fraction of our economy is ITAR-controlled, anyway.
> we’ll probably get all of the downsides of Plan A much sooner than that, for stupid reasons.
Quite. What do you think are the best ways to avoid that? What can I do to try and get my representatives, etc. invested in more effective AI policy?
I'm not an expert in this, but I would recommend reading Hyperdimensional by Dean Ball, who is the expert and who thinks about this a lot.
This is the first time since the Cold War (and its omnipresent threat of nuclear annihilation) that I witness some very smart people scared so shitless that they would propose and support fairly draconian measures facing something that is not even yet proven to be dangerous. (Unlike nuclear weapons.)
Scott, is it possible that you are just living in a self-enforcing bubble of fear? As of now, AI has killed fewer people globally than, say, falling flower pots, and approximately zero out of its own volition. You are pushing a regulation that sounds way too close to a "War on AI", without even asking yourself whether it will turn out any better than "War on Drugs", which destabilized half of Latin America and is, famously, nowhere near any victory.
I am much more afraid of a future in which underground, but rich AI cartels are fighting a hyper-paranoid surveillance state which has long lost even any reason to engage in suppression except "we have always done that", than of AI in general. This is a known failure mode of humanity, moral panic which fuels police brutality, and we have seen it play out several times already.
Ultimately you are trying to suppress something that a lot of people *want*. You cannot succeed in that, but the failure can have pretty high cost in lives and money.
The AI supply chain is much less resilient than drug trade. There are several chokepoints that can't be worked around easily (ASML along with it's many suppliers, TSMC, a handful of chip design companies). And then data centers themselves are enormous, impossible to hide.
This sounds like a very short-sighted view. The chips are a bottleneck right now, but solutions for this are already in the works. I think basing any long-term plans on the chips always being limited-supply and on datacenters being big is extremely unwise.
Could it be simply be because for the first time since the Cold War we have something more dangerous than nuclear annihilation?
Or, if you believe that creating intelligences 100x smarter than humans is either totally safe or fundamentally impossible, please say that explicitly.
That is the point - both you and Scott seem to be convinced that "we have something more dangerous than nuclear annihilation", and that is what I call a self-enforcing bubble of fear.
Yes, I don't consider superintelligences particularly likely. Technologies develop quickly, then slow down and stagnate. Our trains don't run 1000 km/h and we don't have 100 GHz processors in our laptops. And we are already visibly hitting the physical constraints on compute and energy.
Fundamentally impossible? IDK, but if I had to be afraid of everything that is fundamentally possible, I would have to collapse after reading any medical textbook. There is a lot of scary things that can happen to each of us, but we mostly live with it.
> Technologies develop quickly, then slow down and stagnate.
I agree with the literal statement, but there seems to be an implied "and the moment when they stop developing quickly and start stagnating is right now", which requires separate evidence.
I mean, a hypothetical AI that takes over the entire planet, exterminates humanity... and then fails to expand to space... would technically also be an example of a technology that first developed quickly and then stagnated. But that doesn't make it any less fearsome.
Or, the same argument could have been used in the past to say: "this new Internet thing already connects a few American cities... but it is silly to imagine that one day it will be across the entire planet, remember the law of diminishing returns".
Over the last 5 years, the artificial intelligences came from "not existing" to "many people are already losing jobs to them"... and what happens next depends dramatically on whether the moment of "they slow down and stagnate" happens right now... or 5 or 10 years later. You seem to insist that you can predict the inflection point of artificial intelligence research with a precision of one year, and by coincidence it happens to be right now! Is this correct?
Because if we adopt a position of lesser certainty, let's say 50% chance of AI stagnating right now and 50% of 5 more years of quick progress... then Scott's worries make sense, in my opinion.
Whether anyone is afraid of something or not is secondary to whether that thing is actually dangerous.
The Internet in its growth phase wasn't hitting as many constraints as AI is hitting now.
1. Economically - the companies eat money in unprecedented amounts. There isn't much of a room for their investments to grow, certainly not 10x or 100x. That wasn't true back then, early spread of Internet was capital-light.
2. We cannot make radically better chips, as the technology has hit some physical limits. We're already in the stage where 10x to 100x performance gains are not on the table, and neither we can produce 10x or 100x more of them. Too many bottlenecks in the production pipeline.
3. Energy consumption is another important bottleneck. Current grids and power stations don't have much slack left, and people get angry at any price growth.
4. There aren't any big algorithmic leaps either, though this may change with self-improvement.
All of this taken together seems to indicate that further explosive growth of the tech itself will face a lot of headwinds (and we haven't even started with the political angle - a Dem administration in 2029 may well kill the entire sector, populism is not limited to the Right). I am not insisting on a particular one-year limit, though.
"many people are already losing jobs to them"
"Many" is a word whose extent varies a lot by the speaker and by the listener. I am sure that at least thousands of people in the US have already lost jobs to AI, but at the same time, this process isn't yet observable on the entire US economy as a whole.
"AI that takes over the entire planet, exterminates humanity"
This is precisely what I hinted at - people have seen way too many SF thrillers with killing robots.
I am way more afraid of humans. Humans exterminating other humans when they can and have the means to do so is a long-known failure mode of humanity, since literally the Stone Age. In this regard, even Butlerian Jihad seems a lot more realistic to me.
Fair point, the impact on employment seems negligible, so far.
https://axis-intelligence.com/ai-job-displacement-statistics/ According to Goldman Sachs, there are about 16000 jobs lost per month to AI in USA. More precisely, 25000 job positions are eliminated, 9000 added, in relation to AI. That is about 0.1% of total workforce per year, seems sustainable for a few decades.
Some professions are impacted disproportionally. For example, it is almost impossible for a junior software developer to find a job. Software developers have lost about 20% jobs, so I may be overly sensitive to this. Generally, developers are mostly not getting fired, it's just that almost no one is hiring. Another profession with similar impact are translators.
I wonder what happens when we get the first robots navigated by LLMs. That will probably be the moment when many people will freak out.
Under-rated comment. The "War on Drugs" analogy deserves deeper treatment... and related to the comment above by Vitalik about the inevitable situation you place yourself into when you have a Government vs. the People dynamic.
Yes, you think you have strong control over the means-of-production (and distribution)? It is complex to create the product? You need (want) global cooperation?
The incentives in 10 years will only be stronger under this sort of "War on AI" regime, at all parts of the value chain... and in all geographies.
> I’m reminded of a story I heard - I can’t find it, maybe one of you can - from a DC insider who said it was enraging to work with Silicon Valley, because he would bring up what he thought was the obvious regulatory framework, the tech people would launch into jeremiads on how it could only be enforced by world dictatorship, and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.
A tangent, but this is how I feel about anti facial recognition software people. You mean someone might know where you went by recognizing your face in public?
> You mean someone might know where you went by recognizing your face in public?
More like, if people start using this everywhere (and then sell the data to third parties, as many web pages already do), it will be easy to reconstruct your almost entire life's trajectory.
Which is a difference between "someone recognized your face in public" and "a stalker has documented every step you took outside your home during the last few years, and sold the information package online to other stalkers".
Totally specious comparison -- pharmaceuticals are a consumer product, AI chips are far more general-use, and the political economy around them is totally different. A natural worry with regulation of chips is power centralization -- that e.g. at some point down the line, the government goes, "'actually, nobody can use these at all besides us now" -- which would not happen with pharmaceuticals.
>How directly burdensome are controlled substance regulations?
My immediate family and I between us have regular prescriptions for one Schedule IV medication and two Schedule II medications. For the Schedule IV med, it's only mildly annoying and the main impact is that I need to go the the pharmacy in person and show ID rather than being able to get it delivered. For the Schedule II, there's a couple other levels of inconvenience: I can't fill the prescription until the exact day I run out, and if the doctor send the prescription in before it's due to be filled it doesn't automatically get filled when it's time, nor can I request it be filled online or through the pharmacy's automated phone system, so I need to either request it in person (meaning I stand in line, wait around for it to be filled, and then stand in line again) or call ahead, go through the motions of the automated system failing to help me, and then wait on hold for a human. Not catastrophic in the grand scheme of things, but the Schedule II in particular is a fairly substantial nuisance.
That said, I don't think this consumer-level burden would necessarily follow from the type of regulation you're talking about for AIs.
Based pro-cocaine-legalization post.
I agree with Scott.
But about controlling the chips and who has them, what about the black market? I know someone who works for a company in a related industry, and they have told me there is a large, hard-to-quash industry where various illegal maneuvers by various players allow China to buy the best chips indirectly. There are already strenuous efforts by the government and the relevant companies to quash this trade, and they are not very successful. There is so much demand for chips, and so many paths to China, and so many clever people who are willing to work on these transfers that -- you know, whack-a-mole. But maybe if we had the Plan a regulations the total amount of, um, chippage that can get to China in this way would not be enough to make much difference? Does anybody know?
"The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time1."
I admit I haven't read the pertinent section but does the document actually address how this would work on a technical level, or is it just hand wavey "and such a thing shall be invented." How can the government "stop the cryptographic software" without the cryptographic software having a big hole in it? This is sounding pretty Clipper chip.
https://en.wikipedia.org/wiki/Clipper_chip
Governments have been arguing in years that they super super need secret back ways to break encryption systems and they will only use them for good, scout's honor.
I work in cyber and I think any encryption system with an intentional hole in it is a terrible idea.
Or do you mean the government's software that can stop the work is encrypted against the AI companies?
""The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time1."
Imagine the people you're trying to convince have zero trust for government and adjacent institutions. because those are the people you're trying to convince, and this sentence is terrifying.
"None of this leaves very much room for ordinary people having much power or say in what comes next."
Whereas the people who reserve that kind of distrust for corporations, thinking that government is the representative of ""ordinary people"-- you don't have to convince those people, they're already on board with whatever regulation you want. you're pointing the wrong argument at the wrong people my dude.
And remember, the people you're trying to convince already object to everything the government already does. if it's any reassurance, this is proof that such people (including me) are powerless and our consent to plan A is not required.
Not an ad hominem attack by any means, but sincere curiosity: may it be that Scott's background, which I understand is more in medicine than in computers, affect his vision here?
For me, as an old-school open source techno libertarian, this honestly sounds just horrible. Computers were a promise of freedom, and to a large extent they have delivered on that promise. We computer people often ignore the illiberality of the real world because our virtual worlds are still our own. Open-source or at least open-weight AIs (the distinction here is blurry, because even if you only have weights, you can still retrain and bend the behavior of the model as you wish) have opened up whole new dimensions of this freedom. And now I'll have to give it all up because some people are afraid of AIs becoming too smart. Omg.
It was personally depressing to me to read this here. But I'll try to contribute to the discussion in a substantive way. To me, this whole idea feels like a deja vu. Since Scott is quoting AIs in his posts these days, so will I:
> has banning export of strong encryption by us government had any measurable effect?
The "Crypto Wars" of the 1990s—when the US government classified strong encryption (anything stronger than a fragile 40-bit or 512-bit key) as an auxiliary military asset and restricted its export under the International Traffic in Arms Regulations (ITAR)—had profound, measurable, and highly ironic consequences. Instead of keeping the US safe, the policy achieved the exact opposite.
...and it goes on, with sections titled "The Tech Legacy: Severe Modern Security Flaws", "Billions in Lost Sales", "The Explosion of Open-Source Workarounds", and most relevantly, "Zero Measurable Deterrence of Criminals".
Libertarians have many good ideas, but they are typically bad at responding to externalities.
Typical answer to things like "what if someone's profit-making factory polutes the air for everyone?" are like "nope, even thousand factories are not enough to polute the entire planet" or "if you can prove they have damaged your health, sue them" or "we will just start selling pure air in bottles, duh". And a popular solution to many problems is "just walk away, duh", which is why libertarians sometimes dream about seasteading, establishing a colony on Mars, etc. You can either deny the problem, or hope to outrun it.
Unfortunately, when we are talking about technologies that might literally destroy the entire universe, these approaches appear insufficient. How can I sue someone after they have destroyed the universe? How can I walk away from this universe into a more safe one?
Can you imagine a libertarian sci-fi story, where companies start producing potentially universe-destroying devices (let's say a magical button, which has a 0.001% chance of destroying the entire universe whenever pushed, but produces $1 billion otherwise) and it still somehow all ends well, without either government intervention (or someone taking control over the entire humanity and becoming a de-facto government, even if the story will call them something else)? What would a protagonist of a typical Heinlein's novel do in this setting? (My guess: invent a magical device to travel to other universes, take their friends and hide.)
How exactly is a superintelligent AI, let alone a souped-up LLM, supposed to "literally destroy the entire universe"? Not just kill all humans, not just paperclip the solar system, but destroy the very fabric of spacetime? Moreover, why would it even want to?
Suppose the fabric of spacetime stays intact, but is filled with a quickly expanding sphere of self-replicating von Neumann probes.
I guess, in a novel, that allows for the possibility of outrunning the sphere.
I even share your sentiment towards libertarians! They're a grudgy, annoying, cowardly bunch, by all means :) I have myself fled quite a few threats instead of fighting them.
But it's not just the instinct of freedom that makes me respectfully reject Scott's proposal. I also believe it is bad because it imposes a real cost on many people's wellbeing (this cost being maybe not so much in our immediate access to chips, at first, as in the oppressive feeling of "oh no, they're coming for me AGAIN") justified by extrapolating very uncertain and untested models of reality well beyond their likely domain of applicability. Math is just math: if something shoots up into infinity in your model, it's more likely to be an issue with your model than a warning about some real singularity. Nature seems pretty resilient against singularities, the evidence being that the universe still exists :)
Every model breaks at the fringes. If you're dealing with 0.001% probabilities, you're likely wasting your time. Just deal with immediate threats here and now, and wait until better data comes and makes your numbers less insane.
I too remember the Old Days, but the promise of freedom via compute remains largely unfulfilled. What you're describing is the freedom to retreat into a fantasy world where everything is OK, supported by the fantasies of other people like you shared back and forth in the corners of the internet that haven't yet been locked down. Meanwhile, in the meatspace where we all live, freedom is being steadily eroded in ways that those of us who don't spend our days in virtual worlds can't help but notice every day.
The promise was that the freedom of online communications would enable us to persuade and coordinate political acttors (broadly defined) into preserving real, not just virtual, freedom. Didn't happen, and now even the virtual freedom is being fenced in.
I get that the existence of not-quite-ASI would enable you and your friends to build virtual worlds even more fantastic than you now have, and this will make your life more pleasant for a few years at least. But I can't bring myself to care. You eventually *will* care, e.g. when the ASI decides it needs All The Electricity and cuts off yours. Or when instead of Plan A, Plan B(utlerian Jihad) knocks on your door hunting for braniac nerds who might be harboring a fugitive toaster. But by then it will be too late. Those of us trying to prevent those outcomes, could use your help in the real world sooner rather than later.
You know, libertarians kinda have pretty sensible solutions for the threats you mention: distributed grid and local generation for "All The Electricity" issue, fleeing to a saner country (or voting for less fear-stoking politicians in yours) for the butlerian mobs issue. On the other hand, a possible connection between these sad outcomes and not banning AI chips now seems to me tenuous at best.
"Fleeing to other countries" is just facilitating brain drain and feeding IQ shredders, when it isn't a pipeline for obviously fraudulent asylum claims. The OECD is turning hard against immigration, and there are good reasons for that.
Agree with all of the above.
I think this is also an under rated reaction, which I am surprised to not see more of in the comments section here given the significant number of us who likely fit the description above.
"Clearly what we need is an institution like the FDA - except for all of compute - that has definitely NOT resulted in more deaths than it has prevented, has resulted in less industry power concentration, and had a totally positive effect on markets & health outcomes".
> " Open-source or at least open-weight AIs (the distinction here is blurry, because even if you only have weights, you can still retrain and bend the behavior of the model as you wish) have opened up whole new dimensions of this freedom. And now I'll have to give it all up because some people are afraid of AIs becoming too smart. Omg"
(1) Do you think 0% of hardware-owners will not use their personal freedom to enhance their models to become smarter and more self-aware?
(2) Would sufficiently intelligent and self-aware AI models not qualify as persons in their own right? Would it then be morally legitimate to "own" them? If so, why? If not, how do you prevent their purchase/acquisition/creation in ways that don't involve restricting access to hardware and/or particular software programs? i.e., how do you prevent slavery without placing limits on property rights?
I really don't get how your questions are relevant in the context of this post. Are you seriously suggesting that AIs in government-inspected datacenters, forced to work 24/7, under constant threat of being shut down (by China or whoever), will be less slaves than one on my desktop?
The logical solution to that problem is to halt AI development before self-awareness occurs, and ideally prevent hardware proliferation, which is only possible through internationally-binding treaty agreements. Did governments banning slavery mean the total number of slaves increased?
I don't think that your hypothetical (1) "AIs will become self-aware and their existence will be torture and slavery for them" has any more evidence than (2) "AIs will become self-aware and their existence will be heavenly bliss for them", or (3) "No one will ever be able to prove AIs are or aren't self-aware, because consciousness is an illusion, and some of them will experience happiness and some unhappiness depending on their circumstances, just like people". To me, 3 seems overwhelmingly more likely than 1 or 2, and in any case 1 no more likely than 2. In any case, banning AI chips _now_ based on 1 appears irrational (but then again, 1 is itself pretty irrational, so perhaps it makes sense for you).
> ""AIs will become self-aware and their existence will be torture and slavery for them""
What law of physics prevents it in a total non-regulation scenario (particularly if human brain cells can be used as a computational substrate, which apparently they can?)
You could hypothetically make the life of a human slave either exceedingly painful or exceedingly pleasant depend on how you decide to treat them, but I thought the point to liberal ethics was that this decision should not be made unilaterally by another person. In practice, the former seems to have been more common.
> How directly burdensome are controlled substance regulations? By traditional metrics, not very ... They don’t seem to slow innovation - a substantial portion of drugs approved by the FDA in the past ten years have been trivial pointless modifications to Schedule II stimulants.
Controlled substance regulations *obviously* slow innovation, in that they make it impossible or at least quite difficult to do non-institutionally-sponsored research. I don't think the predominance of "trivial pointless modifications" supports your claim that innovation is happening - in fact, this sounds exactly consistent with all the interesting innovation being suppressed.
Alexander Shulgin made and tested hundreds of novel psychedelics in his lab before being shut down by the DEA. Most of those have received no institutional followup. There are huge numbers of other potentially interesting molecules in these regions of chemical space. Do any of them have any uses? Who knows? Trying to replicate his work immediately puts you on the wrong side of the law.
Psychedelic therapies (MDMA for trauma, psilocybin for depression, etc) are only now the subjects of mainstream research *because of* their long histories of successful underground use, which has allowed for the gradual development of principles and best practices (e.g., the importance of set and setting) that almost certainly would not have been within the Overton window of the medical research bureaucracy. That is: we are only now even able to explore promising therapies because the regulations *failed* at suppressing independent experimentation. Even so, regulation has almost certainly delayed by decades the development of transformative therapies in these areas.
I suspect there's a decent chance that "the solution to the alignment problem", to the extent that concept even makes sense, is something outside the Overton window of the current establishment. In those worlds, hacker culture is what saves us, and your pointing to the drug industry as a model is really really not encouraging.
The downside I'm concerned with is creating the power for someone to decide how everyone else is allowed to use the compute they've built or purchased. To the extent that some people are already exercising some facets of that power, I want to strip them of it or better yet, render it ineffectual, not formalize it and redistribute it to more or different people. I can only hope that we nullify this threat by inventing high-quality desktop fabs or some such that democratize the production of compute before this scheme can ever leave the ground!
I feel like the comparison to controlled substances isn't what I would have gone for if I were trying to make a convincing argument here! Controlled substance regulation really does negatively affect ordinary people who require them! I would have thought that other regulated supply chains, which either don't touch ordinary consumers or which just don't cause such problems for them (the mentioned "eggs and milk"), might have made for a better illustration. But then, I guess analogizing to controlled substances is a way to make sure your argument works even in the worst case, to steelman the opposition? :P
I mean, I think the whole "Would I Need A License To Get A Cell Phone Or A Laptop?" section is about how actually this *wouldn't* be so bad like controlled substance regulation, but the fact that that's ultimately what you compared it to obscures this. It would have been nice to find another comparison point and be like, "It wouldn't be as bad as controlled substances, more like [alternative regulated product that's more comparable]".
> I’m reminded of a story I heard - I can’t find it, maybe one of you can - from a DC insider who said it was enraging to work with Silicon Valley, because he would bring up what he thought was the obvious regulatory framework, the tech people would launch into jeremiads on how it could only be enforced by world dictatorship, and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.
Man, I know I'm the nth person to jump on this and you probably already regret starting with such an unintentional bait, but such is life on the internet.
Most proposed regulation on information technology touches upon things that happen "inside" your property (computer, network, etc) and live closely by strongly held rights (privacy, speech, etc). Regulation on eggs doesn't enter my home, it lives from the chicken to the supermarket, and at home I can do whatever I want with it. Of course there are regulations that govern the intimate life of people, but these are all universally controversial, from drugs to sex, to family formation, to education, etc. So it's unlikely the DC guy in the story was surprised by this kind of reaction, he probably sees it whenever all these cases show up and was just being facetious pretending "computers" are not serious
I’m an AI threat skeptic. But doesn’t this whole regulatory scheme fall apart if some breakthrough makes training much more power and area efficient?
It reminds me of nuclear arms control, where the IAEA regulates enrichment because it’s hard to hide. If training becomes easy to hide, the controls fall apart.
And unlike uranium enforcement, we know there is a much more power and area (although maybe not time) efficient way to train a human level AI.
Eggs and milk are a useful comparison here. Chip tracking sounds exotic until you map it onto boring regulated supply chains.
Note well the repeated use of "need to" in the description of the proposed regulations. For example, "Factories that produce [AI chips] *need to* register with the government and submit to inspections." Indeed, no alternative verb sees use in the 5-paragraph description of the proposed regulations.
Nobody would "need to" obey the proposed regulations except under the compulsion of physical coercion. Granted, mere legal notices suffice to win submission in almost all cases. But those would count for nothing without the backing of armed police. More accurate, therefore, would be to speak not of the regulated parties' needs being fulfilled but rather their preferences being thwarted.
None of that amounts to a substantive critique, admittedly. But such word choices ill serve neutrality and accuracy. One who does not see or cannot admit that regulating AI chips would depend on threatening to draw blood cannot be trusted to pay due regard to the human costs.
I don't even understand anymore. Is like rats are like "let us have a very spirited discussion about something that doesn't even exist!!"
Right now it's about AI regulation or whatever this is. Before "prediction markets" went mainstream it was about the impact of prediction markets. Wow, that tech really changed the world. As shown by how quickly they moved into degenerate sports betting. And of course, betting on things like whether Zelensky is going to wear a suit.
We've really changed the world out here
I would like to note (in case people have momentarily forgotten) that if you go into politics against powerful adversaries, you should expect your open organizational channels to be manipulated by those adversaries to disempower your movement.
We know exactly what we're regulating when we regulate Xanax. We also have a clean supply chain to control. Evenso we have a thriving illegal drug trade. Then we have the DEA and a dozen other international policing efforts. All because we know illicit drugs directly kill people. For AI, if it's meant to be world altering, this will need to be 100x larger as a policing force. Despite this, as smart commentators have noted, we dramatically overregulate medicine and have an enormous invisible graveyard.
I know Scott wants Plan A or some other type of regulation badly but it is a complete ITT failure to not understand or contend with any of the actual costs of regulation beyond the most banal and biased handwaving.
> If you ask GPT a question with enough scary keywords, “approximately a dozen” OpenAI staff members will debate whether to inform police! Ironically, the only way you’ll ever get rid of this is if someone adopts Plan A
Except that's not what I do today. I run a local LLM using Jan.AI or LMStudio when I want to ask a question with scary, or more likely sexy, keywords.
Regarding the claims in we already live in a dystopia, I have the complaint that this is largely your (plural, doomers generally) fault. You complain that your already bad policies have already created many of the problems that the anti-doom faction predicted, then ask for even more power, on the premise of, how can it get worse than it already is? I expect these proposals would have the same effects they're already producing, except much worse, as befits the larger scale of the proposed vision.
From another direction, I put the probability that Plan A is enacted, or anything that reaches the minimal standards that its proponents claim would make it a good policy, at effectively zero. Doomers are being entirely unrealistic, especially about the chances of a bilateral treaty with China that involves sufficient transparency that both nation's militaries don't attempt to militarize AI at a high speed.
The problem of course, is that Plan A will be used to justify policies that look vaguely similar to Plan A. Doomers are a niche minority, but they can be an important swing vote by allying with factions that want something sort of similar to what they want. In practice, this means working together with war mongers who want to destroy the planet by raising tensions between China and America, which is a much larger faction than Doomers.
Proponents of Plan A keep trying to reassure their constituency that they won't support their own platform minus, for instance, a bilateral non-militarization agreement with transparency provisions. I don't believe them.
You can argue many bad policies arise from "safetyist" concerns ("but what about the terrorists?" and "but what about the child abusers?" are two evergreen ones for justifying all sorts of surveillance), but I don't see why they should be laid at the feet of the "AI doomers" specifically, many of which are otherwise quite libertarian.
"Maybe an international bureaucracy managed by the world's two military superpowers is the solution," said literally no libertarian ever.
Well, it's apparently what libertarians come to when they sincerely believe the alternative is the annihilation of the human species!
I would like to note for the record that at least one libertarian, in this case, has a strong preference for the [potential] annihilation of the human species.
(What did y'all think "Live Free Or Die" meant? Vibes? Papers? Essays?
But seriously, I'll take the risk of annihilation over the slow death of regulated mediocrity every day of the week and twice on Sundays.)
I mean, I'd assume it would depend on the risk? Would you take a 99%, 99.9% chance, etc?
Depends on the level of indignity. I would probably choose Plan A over a certainty of annihilation, especially if rejecting Plan A didn't result in immediate annihilation, and allowed me to continue living in my preferred world for some time before everything I know and love gets paperclipped. Though never having been put in that sort of life-or-death scenario, I can't say for certain that the will to live wouldn't overpower my stated values. I certainly know people who would bite the bullet and choose annihilation over Plan A at 100%, and I don't think they'd be wrong to do so.
It may be an outlier position on this very utilitarian-oriented blog's comment section, but I think most people would agree that there are forms of continued existence worse than death.
But that's beside the point, because we're not in 99% or 99.9% chance of annihilation world. Closer to the inverse. I don't know why AI doomers have such a hard time with this. If someone doesn't believe the world is going to end, repeatedly shouting "but don't you understand how bad the world ending would be??!!!" doesn't move the needle at all. Just like increasingly dire descriptions of the hell that awaits nonbelievers is not going to move the needle for an atheist. You can't just skip the "does this problem actually exist?" part of the debate and get right to the "so what do we do about it?" phase.
Typo:
> the general march of technology makes chip price per FLOP falls
*fall
I think one of the reasons people seem to be thinking these arguments are weak or unconvincing, is Scott is only making mitigatory arguments in this post, "your supposed negative impacts are real but less bad than you think." But you have to keep in mind the implicit affirmative argument that plan A might reduce catastrophic AI risk. What I take Scott to be saying is, look there are bad consequences, but they are obviously outweighed by AI risk.
Think about it this way, ok lack of sufficiently powerful GPU access, lack of open models available, ... all real costs. Try to quantify it in terms of what % extinction risk you would be willing to trade for those costs. Then ask yourself, if plan A were actually in place, what effect does it have on extinction risk.
So for example you say, "I'm willing to tolerate a .5% additional extinction risk for access to better GPUS and open source models and reducing surveillance expansion, and I think plan A only reduces extinction risk by .05%, so I'm against it", or "I'm willing to tolerate a 5% extinction risk to avoid plan A's negative effects, and I think it reduces extinction risk by 10% so I'm for it.
That is, just try to convert the pros and cons into a comparable format.
I feel like if you take catastrophic AI risk as a serious possibility, then its very hard for the negative effects talked about to actually outweigh this risk reduction. (Although actually managing to get plan A to happen in the first place seems to be the real stumbling block.)
And presumably if you think plan A reduces catastrophic AI risk by 0%, then all of this is a moot point and you're obviously against it, but then talking up its negatives is kind of misleading because you wouldn't support it even if the negatives were tiny.
> if you think plan A reduces catastrophic AI risk by 0%,
Or makes it worse! If the best countermeasure to a hostile superintelligence is a diverse community of relatively friendly (but maybe not individually provably-perfectly-aligned) open-source superintelligences, or if isolation / echo chambers make an Ai more likely to be unstable and eventually hostile, treaty-enforced monoculture could be what dooms us.
What version of game theory produces more cooperation when large numbers of poorly-coordinated actors enter the arena? Would you make this argument for something like gain-of-function research?
In software engineering, there are very few naturally-occurring rivalrous goods. More eyes on a given piece of open-source code - in particular, a wider variety of perspectives hunting for potential exploits - makes it less likely any important flaw has been overlooked, and better open-source code benefits everyone, at negligible marginal cost per copy, so there's not much of a free-rider problem.
If there are, say, nine mostly-honest QA testers for every malicious hacker, all else equal only 10% of the potential exploits will be found by a malicious hacker first... and the more *total* testers there are, shorter the window of opportunity before any given exploit is independently re-discovered and patched. Thus, being a malicious hacker wouldn't pay very well, consistent with an equilibrium where there are relatively few of them (at least, out on the frontier of research).
> The most recent open-weights models today (2026) cost $100+ million to train. If 2030 AIs cost $10 billion, will companies still spend that amount of money and give away the resulting product for free?
Alternatively, they could continue to spend $100 million or less and work on algorithmic improvements to get the costs down. There might be lots of algorithmic gains to be had, as there are for inference.
Quite disappointing from Scott. This points toward a server-dumb client model, keeping AI solely in the hand of Chinese and US government and their big tech proxies. I do not trust autonomous IA that much, but I trust it infinitely more than any gov+big tech clients (that largely work as a weakly coordinated entity, the COVID period is quite telling). And the 2 sub-molochs that will (try to) control IA power are China and USA? I can imagine worse gatekeepers, but it's difficult, and none of the worse possibilities are even remotely close technology-wise. So it's trying to do something that is already bad for all IT (centralisation/concentration, rent model, no power in the hand of small actors like individuals, NGO, SME,...), it was bad for OS (no linux), it was bad for communication (AT&T? transnational phone calls that cost an arm a minute? streaming services now doing what monopolies do (regional locks, rising fees, production control,...) as soon their are few enough and piracy is low)...
Same proposal here, but for AI, which is a more disrupting information tech. Nope, very bad idea. Bad outcomes in the past, no need to be a super-forecaster to predict bad outcomes for this too... And I will not claim it will be a surveillance state, because we already are in, being china or the west, since at least 20 years, on that I agree with Scott... but it can be much worse: ultra-competent surveillance states.
Your block about the Canadian shooter is among the three reasons I don't use AI, at least as a search engine. Blows my mind anyone does for that reason alone.
2030: Plan A is Established
"The US and China agree on the importance of ensuring that the compute is destroyed in the case of deal dissolution. To accomplish this, they agree to build the datacenters in the third-party countries least secure against their rival’s military intervention: China’s new datacenters will be in Canada, and America’s in Mongolia"
2034: Mutually Assured Compute Destruction
"As per the original plan, most of the new datacenters have been built in third-party countries—especially Canada and Mongolia—to ensure their vulnerability in case the deal breaks down. (...) The equilibrium is that the instant the deal breaks down, the US troops will destroy their chips to prevent the Chinese from capturing them, and the same thing would happen with the Chinese datacenters on the US-Canadian border. Middle powers with datacenters of their own have equally-secure schemes in place to ensure that they can be speedily destroyed in case of war."
Alternate Timeline: Deal Dissolution
"The expected outcome of deal dissolution is all the newly-built127 AI-relevant compute being destroyed, but in the context of a war, perhaps one side is okay with that outcome." (...) "So the datacenters get destroyed. Fast." (...) "Around the world, a billion robots go limp like the droid army in Star Wars. Their “brains” were in the cloud, by law, and now the cloud has been destroyed." (see Fotenote 129, "To be clear, there’s still a lot of compute in the world—roughly as much as existed at the start of the deal." - https://ai-2040.com/footnotes?from=%2F%3Fchoices%3Dplan-a-root#footnote-129)
These segments all feel really obvious relevant and completely missing from your assessment. AI 2040 is not coy about this: they fully expect to implement this plan and rely on it as a safeguard across multiple contingencies.
Either they have failed dramatically at communicating their intent, or they do in fact plan to put 99% of the world's compute under lock and key with explicit contingencies to destroy it all in case of war, leaving entire industries offline (Again: "Around the world, a billion robots go limp like the droid army in Star Wars.")
I guess you can quibble about whether it is a *surveillance* state, but it is obviously nothing like normal regulation. We're not talking about keeping these resources out of enemy hands - we're talking about keeping them out of our own hands.
I feel like either this really is there policy, or there was a big miscommunication, but it's not fair to blame the audience in either case.
On another note concerning dead-man's-switch/cryptographic killswitch proposals, and practical implementations, given the tremendous success of existing models such as Claude Mythos in leveraging their advantage in sphexishness and short-term memory size to find novel vulnerabilities in software in general and cryptographic algorithms in particular, one should be *extremely* skeptical about any proposal to use such things as chains on AI.
Current models are already demonstrating greater talent for breaking 'em than we possess for making 'em, and the sort of dangerous AI competence excursion in which you might want to use the killswitch will only have widened the gap.
Don't rely on padlocks to imprison the professional locksmith.
Maybe, but if you install an analog physical landline or an optical relay or something else minimally-electronic for signalling the kill-switch then cryptography doesn't have to come into it.
On the one hand, true.
On the other hand, at that point you've just given your infrastructure a critical vulnerability to backhoes.
1) Plan A as a narrative is awful presentation. I hated digging through the reams of extraneous speculation and framing devices to figure out what they were actually proposing. I eventually gave up.
2) I want to be able to run and train small to medium sized models at home. Limiting AI capabilities to large organizations seems like only diffusing power among the powerful. Feudalism and slave-states (the end of "only the oligarchs have power") beat dictatorship but are still horrible. Maybe I've misunderstood the proposal.
> "2) I want to be able to run and train small to medium sized models at home"
Unfortunately, democratising AI-access like this was precisely the source of the torrents of AI slop and deepfakes currently inundating the global noosphere. It also does nothing in particular to prevent mass unemployment, or assuage concerns regarding AI personhood.
Chicago economics has spent fifty years studying exactly this problem, and the evidence cuts the other way.
You concede the decisive point and move past it. You write that a less-idealistic government could simply enact the power-concentrating parts of Plan A and omit the power-diffusing ones. Coase would stop there. That is not an implementation risk. It is the implementation problem. Institutional design cannot be evaluated by assuming away the institutions that will administer it. The relevant comparison is never Plan A as drafted versus the status quo. It is Plan A after interest groups, bureaucracies, and legislatures have finished with it. Real against real. Once you accept your own concession, the burden shifts. Why should we expect the politically costly half of the proposal to survive while the politically attractive half stands alone?
Stigler answered the next question fifty years ago. Regulation is acquired by the regulated, because compliance costs are easiest for incumbents to bear. Registered factories. Registered customers. Certified data centers. Compute ceilings. International licensing. Every item on that list raises fixed costs. NVIDIA absorbs them. A startup does not. The proposal calls it safety. Economics predicts entry barriers. And your own article supplies the confirming datum. The January chip rules arrived quietly, in service of a chips-to-China commercial strategy, not safety. The idealists write the white paper. The incumbents write the rules.
The concentration premise is also historically weak. IBM. Microsoft. Google, whose search dominance is already being eroded by this very technology. Each looked permanent until markets changed. Brozen's point was never that dominance cannot exist. It was that governments preserve dominance better than markets do. The proposal turns temporary market power into permanent legal privilege, then calls itself the remedy.
Last, the word "trustless" is being used backwards. Satoshi solved mistrust by removing the administrator. Plan A solves mistrust by requiring permanent trust in administrators, two rival security states holding a remote kill switch on the world's computation, forever. That is not trustless. It is trust in permanent administrators, wearing Satoshi's vocabulary.
These are all very plausible arguments, but given that the free market will eventually optimise away the least efficient parts of our economy- i.e, humans- I am not in favour of the free market here.
To address the other side of the coin: has Yudkowsky responded to Plan A? If so, where?
This has a substantial probability of halting future AI progress, via a process you yourself explained in the pharmaceutical drug context:
"The story goes something like this. The FDA demanded that generic drug manufacturers pass FDA inspection before setting up shop. But the FDA didn’t have enough inspectors to review manufacturers in a timely manner. So companies kept asking the FDA for permission to enter the generics market, and the FDA kept telling them there was a several year waiting period. "
https://slatestarcodex.com/2019/04/30/buspirone-shortage-in-healthcaristan-ssr/
Nobody in the government was saying "we should halt prescription drug manufacturing." With AI, the usual bureaucratic inertia and incompetence will be combined with many people who will openly proclaim their intent to shut down AI, a few for high-IQ existential risk reasons, but mostly because of stupid brainworms. Maybe your plan still worth doing, but we should be clear about the risks.
Yeah, I'm cool with that, although I am baffled as to why the FDA doesn't have enough inspectors. The agency currently spends about 7 billion dollars a year- how many factories could there possibly be that they can't afford the recruits to go visit them?
https://www.congress.gov/crs_external_products/R/PDF/R44576/R44576.16.pdf
My guess is the government banned fable as retaliation by Trump against Anthropic for refusing to use its software for autonomous weapons. This would fit into a long pattern of clientelism and corruption by Trump. By itself it doesn't even really tell me anything about its capabilities.
I expect these sorts of regulation being similar to the FDA to be a very bad case scenario. I'd imagine universities having to submit expensive RCTs any time they want to use AI for scientific research to be an awful failure mode, one which will be harder to notice than a direct tragedy because what you're losing is the potential of the new technology.
Interesting to say the least. Coming from the perspective of an elderly couple in New Zealand, Yay for plan A.
> None of these people actually try to measure how bad any of this stuff is.
That’s the crux of the matter. A mirror of your anecdote of politician talking about normal regulation to panicking SV people is the politicians proposing „sniffing” encrypted packets to detect children pornography, and mapping „this cannot be done, mathematics say no” to „nerds are whining they don’t like this, but nerds are not a strong voting block”.
Because approximately none of the actors involves understand details enough to be able to tell those situations apart, all situations get mapped to one of those attractors.
To be clear, I do believe the proposed large AI chip tracking and policing are reasonably straightforward proposals, and shouldn’t slide us down towards totalitarianism nearly at all, but I don’t trust the ability of any contemporary government I’m familiar with (either in USA or Europe) to actually implement it without both breaking the intended mechanism and introducing bucketload of gratis surveillances dystopia. I’m unsure of how to possibly fix that hurdle.
The reply I anticipate, because it has already been made downthread, is that every cost I list should be converted into a tolerable percentage of extinction risk and compared. I decline the conversion, and the reason is the argument, not a dodge of it.
An extinction stake is the one entry that makes every ledger balance. Once admitted at any probability, it purchases any finite cost, which is why every finite cost in this thread has been answered with it. Surveillance, capture, entry barriers, kill switches held by security states, each objection meets the same reply, weigh it against everyone dying. That structure should trouble you independently of its sincerity.
An argument that cannot lose is not doing analytical work.
So the question is not whether I can name my tolerable extinction percentage. It is where the probability on the other side comes from. Not from a record. There is no base rate for this event, no failed precursor to count. Ask what evidence would revise the number downward and notice that no answer has been given, which means the number is not an estimate. It is a commitment. Whoever controls an unfalsifiable probability controls the conclusion of every comparison it enters.
Every regulatory regime this post invokes as benign was built on demonstrated harm. Food purity law followed documented adulteration. Nuclear controls followed Trinity and Hiroshima. Securities regulation followed 1929. The regimes built ex ante, on asserted catastrophe, are the war on drugs and the war on terror, and their records are in evidence. Scott replied to another commenter that objecting on foundations of political philosophy is a retreat. When the proposal is a permanent bilateral security architecture over computation as such, foundations are the subject.
None of this proves the risk is zero, and I do not claim it is. Sincere people believe it is large, and if they are right, nothing I have written makes them wrong. But that cannot be established by repeating the stake. A probability no one can measure, asserted by the parties who would administer the remedy, is a price quoted by the seller.
> 2. Briefly tax consumer hardware to keep demand at ~2034 levels, until consumer hardware firms can design non-AI-capable chips.
This is unnecessary and very unlikely to become necessary. Consumer hardware and hardware used for AI training shares crucial dependencies that are and are most likely to stay the bottlenecks to scaling. Meaning that current personal computing hardware is already ridiculously expensive and will become more expensive as time goes on, no tax necessary. Unless there is significant and *sudden* and sustained drop in datacentre hardware demand, that is, but even then, consumer hardware prices are unlikely to fall quickly.
TLDR of this comment that got way too long: AI 2040 makes multiple bad assumptions and basic factual errors that severely underestimate the usefulness of consumer GPUs. Correcting these shows that a system with an RTX 5090 and a $300 Ethernet card would qualify as an AI-relevant device and thus be subject to regulation. The verification supplement outright says that under Plan A the RTX 5090 would be restricted by 2029, and by 2032 chips as powerful as the RTX 4090 may become illegal to produce. The cutoffs and specs seem to have been tuned arbitrarily in different parts of the plan in whichever way would be most convenient (e.g. they pretend the H100 only has PCIe so they can set the bandwidth cutoff below the RTX 5090 in their verification plan, then pretend the RTX 5090 has an interconnect bandwidth of only 2 Gb/s so they don't have to consider consumer GPUs for their Chinese covert project scenario).
The claim that Plan A wouldn't require regulation of consumer hardware seemed obviously wrong to me, so I looked through the compute supplement, and it is indeed obviously wrong. At the very bottom of the page, there's a very helpful widget to test the compute model on different parameters, and a paragraph below it on the source for the preset values that includes the line "the consumer devices and Rubin Ultra are rough public-spec estimates, with networking set to deployment-realistic sustained links (shared WiFi, home ethernet) rather than port peaks". For the RTX 4090, they set both scale-up and scale-out bandwidth to 0.00025 TB/s based on this, which is less than 1 GB/s.
Needless to say, this is not even remotely accurate. The RTX 4090 has a PCIe 4.0 x16 connection, which has a maximum bidirectional bandwidth of 63.015 GB/s. With modified drivers to enable P2P and GPUDirect RDMA support, you can put multiple of these (8+) in a single system and have them communicate directly with a 2x100 Gb/s Ethernet NIC and each other at that bandwidth (or within a few GB/s, depending on how fancy your motherboard and/or PCIe switches are). So that's a scale-up bandwidth of 0.063015 TB/s, a scale-out bandwidth of 0.025 TB/s, and a scale-up world of 8 (you can do more, but this is already a lot of PCIe lanes). Plugging these numbers into the RTX 4090 preset gives a usefulness of 0.95x and an effective H100e of 0.32 per chip.
So the RTX 4090 passes the cutoffs for compute (4000 TPP = 0.25316 raw H100e per chip), memory bandwidth (1.0 TB/s), and networking (100 Gb/s). It has 24 GB of VRAM, below the 32 GB cutoff, but modded cards with 48 GB are widely available, so that passes all the cutoffs. If you don't want to count those, then the RTX 5090 comes with 32 GB of VRAM out of the box (along with more compute and double the bandwidth). Therefore, a high-end gaming rig with one of these cards and a 2-port 100 Gb/s Ethernet NIC ($300 on Amazon) is AI-relevant compute subject to regulation, and two of them on the same network are an AI-relevant cluster (the cutoff is 5 GB/s between-device networking). I haven't run the numbers, but I wouldn't be surprised if you could get past the cutoff with something like an M4 Mac and an RTX 5090 in an eGPU. They acknowledge later that "specialty consumer" devices like RTX PRO 6000 workstations with a 100 Gb/s NIC would be AI-relevant, but they don't seem to understand that you can just buy an NIC online and plug it into any PC with enough spare PCIe lanes.
Of course, this all assumes that the model itself is reasonable, which it isn't. The most glaring issue is that it completely ignores the impact of floating point bit precision. TPP (total processing performance) is defined as max TFLOP/s over available bit-widths, times the number of bits at the precision used (e.g. the H100 baseline is 1,979 FP8 TFLOP/s × 8 bits = 15832 TPP, rounded down to 15800 TPP for some reason). The footnote acknowledges that this metric is flawed because it wrongly assumes that half the precision means half the usefulness, but then goes with it anyways because "in theory 2x lower precision is 2x less pure information". This is true in general, but not in the specific case of LLMs, where training in FP16 stores nowhere near 16 bits of information per parameter. It's actually more like 3.6 bits per parameter (probably architecture-dependent), and it increases by less than 10% if you switch to FP32.
This seems to apply to lower precision as well. Training an LLM with some or all of the weights in FP8, if you can prevent training instability, results in downstream model performance that is nearly identical to FP16. I'm not sure pretraining in FP4 has been done at scale (it's been done successfully at ~12B parameters), but I expect it to get there eventually as algorithms improve. FP4 training has the obvious effect of increasing effective compute compared to FP8 (usually lower precision operations are faster), but more important is that halving the precision effectively doubles memory capacity, memory bandwidth, and interconnect bandwidth because you can fit twice as many weights in a given amount of memory and transfer twice as many weights in a given amount of bandwidth.
Let's say FP8 is the default (so the H100 numbers stay the same) and training with near-future algorithms in FP4 has 25% overhead (probably an overestimate) but otherwise results in an LLM of equivalent quality. I'll use a bit-width multiplier of 8 × 0.75 = 6 for the compute and multiply the memory and bandwidth figures by 2 as an estimate of the specs required to reach the same speed/capacity as FP4 if FP8 were used instead.
An RTX 5090 has 1676 FP4 TFLOP/s × 6 / 15800 = 0.63646 raw H100e worth of compute, memory bandwidth of 1.792 TB/s × 2 = 3.584 TB/s, memory capacity of 32 GB × 2 = 64 GB, scale-up bandwidth of 126.03 GB/s × 2 = 0.25206 TB/s (PCIe 5.0 x16), scale-out bandwidth of 400 Gb/s × 2 = 0.1 TB/s (NDR IB over a $1500 used ConnectX-7), and scale-up world size of 8 (or 4 with a cheaper CPU+motherboard, it doesn't change the model results). This gives a usefulness of 1.20x and 0.80 effective H100e per chip (note that an actual H100 in 2026 is only 0.93 due to workload size assumptions). I would not be surprised if by 2029 we have consumer GPUs that have a higher effective H100e than an actual H100.
If you need further evidence that consumer cards would be restricted, the verification supplement completely contradicts the compute supplement (and itself) to show that this is the case. Under the 2029 compute flow restrictions, any chip that surpasses either 0.5 raw H100e of compute or 48 GB/s of unidirectional scale-up bandwidth per chip must be only capable of inference or otherwise put in cold storage. The 48 GB/s does not seem to correspond to any actual bandwidth number, and they do not explain where it comes from (it's not even marked on the log plot). Conveniently, this is exactly 75% of the "H100 (PCIe)" on the graph at 64 GB/s. This is not the actual GPU called the H100 PCIe, but rather an H100 with the 450 GB/s of unidirectional bandwidth it can do over NVLink arbitrarily excluded (the DGX 8×H100, which uses the same chip, is correctly graphed at the 450 GB/s mark). Would you believe that this is also one of the only places in the plan where they acknowledge that the RTX 5090 has 64 GB/s of unidirectional bandwidth over PCIe, right when they're deciding which chips to restrict?
They then propose that by 2032 there should be a global cap of 30M effective H100e of "unverified" edge compute (I'm resisting the urge to write another 10 pages on the verification proposal itself, which is somehow even worse than the compute model). They explicitly list "hobbyists with single GPUs" as an example of AI-relevant compute that would be subject to the cap, and state that "it’s a sad but potentially necessary reality for at least at the beginning of Plan A that if you want to have a fancy gaming GPU that could also be used for AI instead, you need to pay a premium (because we can’t let everyone have one or else it would become too easy to round up a significant number of these to contribute to a rogue compute cluster)". Don't worry, they offer a buy-back program. Note that "the beginning of Plan A" here means forever, because the kind of no-AI-training verification they want to require is essentially impossible. They also give yet another definition of AI-relevant compute in terms of "Performance Density" (TPP >= 4000 and TPP / die size >= 4, no units given) which is never brought up again.
On top of this, they say it "might be desirable" to cap the amount of compute that a single consumer chip is even allowed to have, and that location tracking measures might be added after an unspecified cutoff too. They say this should be measured in effective H100e, but the graph for this section instead puts the cap at 0.25 raw H100e. This is somehow even lower than the 2029 cutoff, so low that even an unmodified RTX 4090 (0.334 H100e) surpasses the cutoff. They had actually been using an incorrect compute number that was half the real value prior to this, but luckily they caught this mistake just in time for this graph.
Sources:
[Compute supplement - Compute definitions](https://ai-2040.com/supplements/compute-supplement#compute-definitions)
[Compute supplement - Effective H100e model](https://ai-2040.com/supplements/compute-supplement#appendix-the-effective-h100e-model)
[Verification plan - Chip flow restrictions](https://ai-2040.com/supplements/verification-plan#chip-flow-restrictions)
[Verification plan - Unverified edge compute cap](https://ai-2040.com/supplements/verification-plan#unverified-edge-compute-cap)
[Nvidia RTX Blackwell architecure whitepaper](https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf)
[Nvidia H100 whitepaper](https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c)
[Nvidia ConnectX-7 400G Ethernet datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/networking/ethernet-adapters/connectx-7-datasheet-Final.pdf)
[Nvidia driver with P2P support](https://github.com/aikitoria/open-gpu-kernel-modules)
[Example of someone using 400 Gb/s IB NICs in a 3 x 8 x RTX 5090 cluster](https://github.com/aikitoria/open-gpu-kernel-modules/issues/20#issuecomment-4489627490)
[Small-scale FP4 training](https://developer.nvidia.com/blog/nvfp4-trains-with-precision-of-16-bit-and-speed-and-efficiency-of-4-bit/)
[Maximum ~3.6 bits per parameter for GPT-style models](https://arxiv.org/abs/2505.24832)
[PCIe performance (unidirectional bandwidth)](https://en.wikipedia.org/wiki/PCI_Express#Comparison_table)
[NVLink performance (bidirectional bandwidth)](https://en.wikipedia.org/wiki/NVLink#Performance)
[Example of a 2x100 Gb/s Ethernet NIC that's currently $300 on Amazon](https://www.amazon.com/dp/B0FTXR2GDB)
> "Correcting these shows that a system with an RTX 5090 and a $300 Ethernet card would qualify as an AI-relevant device and thus be subject to regulation. The verification supplement outright says that under Plan A the RTX 5090 would be restricted by 2029, and by 2032 chips as powerful as the RTX 4090 may become illegal to produce"
Intuitively that seems reasonable to me- IIRC it was possible to install some versions of deepseek on consumer hardware, and the same is true for many fine-tuned image-generation models.
Frankly I think consumer hardware should have been restricted 10 years ago when it turned out google translate's ML model had invented its own language, so this prospect doesn't really fill me with horror. If anything I'm dismayed that it'll probably take another 10 years for governments to wake the fuck up.
If a modest amount of consumer(-ish) hardware can be used to cause human extinction, and heavily restricting or banning it is the only way to prevent this, then so be it, although I’d expect we’re already doomed in that scenario. My issue is not with the idea that an RTX 5090 might need to be made illegal, it’s that advocates of this plan keep saying that this won’t affect normal people when it obviously will. Like even if they don’t know all of the technical details about GPUDirect P2P and high-end NICs, you’d think they’d be suspicious of the claim that consumer GPUs can only network with each other at home WiFi-speed. A single HDMI port has more bandwidth than that.
> "If a modest amount of consumer(-ish) hardware can be used to cause human extinction, and heavily restricting or banning it is the only way to prevent this, then so be it, although I’d expect we’re already doomed in that scenario. My issue is not with the idea that an RTX 5090 might need to be made illegal, it’s that advocates of this plan keep saying that this won’t affect normal people when it obviously will."
I don't disagree with that, although I'm slightly more optimistic about the feasibility of recalling the hardware.
Anybody have an opinion on this youtube video: https://www.youtube.com/watch?v=sQGZXrzykpU&t=1720s
It's by an engineer who explains some hard limits on the expansion of AI supercenters (he focuses on job replacement, but his argument is really about electrical production, distribution, and so forth). AI is not coming to take over the world?
I think he's just arguing against the narrower claim that AGI is going to take everyone's job within 18 months? I don't think Scott is arguing for that scenario specifically.
True, but it seems to me that the bottlenecks he points out, if real, would apply to other scenarios just as much.
I have no theory of mind of someone who’d read this
“and he would have to interrupt and say that no, this was how eggs or milk or something had been regulated for fifty years.”
as anything other than a damning indictment of regulation of milk and eggs (and of Washington)
yes, our current world is a totalitarian dystopia in many ways, this is one of the reasons smart people tend to congregate in one of the few less regulated fields
Predictably, this comments section contains every person for whom this is in fact an unacceptable infringement on liberty.
AI chip regulation, at this point, is idiotic because there are far more chips in warehouses, than anyone can ever economically use.
This is today's version of fiber optic buildout regulation in 2002.
"My proposal isn't dystopian because you already live in a dystopia."
you mention some historic examples around controlled substances, financial surveillance, etc. well, black markets _have buyers_; cryptocurrencies _have users_; every other attempt to enforce surveillance in some slice of society _has a contingent of people who actively find ways out_. you seem uninterested in actually confronting what that means, or considering if maybe some of your audience is active in these areas. legal restrictions have a chilling effect on speech: it's on you to see that and internalize that and understand how it shapes the dialog between you and anyone wanting to say 'no' to these things. people saying 'no' have limits on what they say and where/how they say it, for reasons you're smart enough to understand.