This is a good summary, and Zvis was a good run through also. However I question a few things
- Beyond "it's not that onerous", why should we have it is unclear, since we have no examples of what harms (do worms created by a coding platform count against them?).
- Who will police this beyond the platonic idea of a good bureaucrat? This is not a trivial problem, even an extremely well defined areas like medicine with high oversight things go wrong, as you have written about FDA. Here we have given away those guardrails.
- The assumption that the rules will continually get updated as we get better at training seems at odds with what you've written about at length about other various regulatory regimes.
- If everything goes right, sure it might be a nothing-burger. But what is the case where it goes wrong and strangles the industry? Surely that has to be taken into account in the analysis, considering the base rates.
I'm sympathetic to your theory that this is an existential risk with some probability which is why perhaps the last argument to you is weighted towards one side, but it is worth it wondering whether the risk of a 500 million cyber attack is really all that high. Notpetya, Epsilon, Veterans admin hack, they all were higher than that threshold.
The first objection sounds like "no nuclear plant has ever melted down before, so why have nuclear safety"? I think it's useful to try to prevent harms before the harms arise, especially if the harms are "AI might create a nuclear weapon". But probably this is a disagreement too deep to resolve in one comment.
Who polices any law besides the government? I agree it would be great if we could get God to directly implement laws, but this seems pretty typical in that the government has to implement it through courts and stuff. This is kind of what I'm talking about when I say it's typical of regulation in any industry except better.
I think if you're precautionary principling, you should do it on both sides.
I agree that a $500 million cyberattack against critical infrastructure isn't that bad in the grand scheme of things. But an AI that can do a $500 million cyberattack probably *is* that bad - for example, you could ask it to do a second one.
I don't think the first comment is fair but yes it is not something that can be easily resolved here. I also think you're arguing against a strawman, my point is only that to not take into account how this can go wrong, or assume any of the capabilities that allow us to protect against it, makes it a highly one-sided argument.
Effectively you are saying this can cause problems therefore let us police it, as opposed doing a cost-benefit analysis on what actually is likely to happen. I note again that this is the exact argument you have against the FDA. We would not have wanted medical Innovation to stop, we would not have wanted materials Innovation to stop, we did not even want nuclear Innovation to stop. This is just one more God that we ought to reject.
How much work are you expecting the compliance to be? It seems to me that this is one or two extra headcount at OpenAI (or maybe zero, since they already do most of this). It's the equivalent of Phase 1 trials for a drug, not Phase 3. I don't think a law mandating Phase 1 trials for drugs would be unreasonable, especially if drugs harmed people other than the user such that we couldn't trust users/companies to handle the externalities.
You might be right, but I also think that if this law were implemented a year ago we'd have an invisible graveyard of GPT4-ish level foundation models competing with openai. If you wanted to take off the FLOPs requirement, put it on some top labs (since you wouldn't need to make it a requirement, as you say it is expensive to train these things so there will always be only very few cos), and create a bit more legible standards with sunset provisions on when they go out of date, I would have far fewer objections.
As it stands what I see is that the industry is doing a bunch of tests, these tests are evolving almost faster than the models themselves, nobody quite understands how to test them in the first place, and we are trying to enshrine the necessity of doing those tests in law.
If you had passed this law 2 or 3 years ago, with a limit of a 10^23 level model rather than 10^26, you'd likely have killed Llama 2 and just handed free money and lack of competition to Anthropic / OpenAI. If this kind of thing have happen, you'd also have stopped or limited several dozens of papers on interpretability / alignment that have used things like Llama 2. Thus, in the counter-factual world where this bill passed two years ago, I think it would be negative EV and negative safety EV, in that it would be (1) shoveling cash into OpenAI + Anthropic and (2) limiting our knowledge of the world.
Note also that many AI safety orgs have explicitly tried to criminalize AI at this level(https://1a3orn.com/sub/machine-learning-bans.html), and they just didn't have enough power to do so. Several of the organizations which have tried to ban open source in the past (CAIS, The Future Society) are in favor of this bill.
(I think that the multiple ambiguous locations in this bill is pretty clearly going to be enough for the legal dept to kill open source releases, especially in the uncertain light of how interpretation could change in the future. They'll look at "huh, we're opening ourselves to an unclear liability dependent on future judges interpretation of what "reasonable" is, and to potentially disastrous litigation *even if we end up winning*. Hell no.")
This is exactly what everyone says about every regulation, that it would be that much work and surely it's worth it given how serious the issue is.
I'm not saying all regulation is bad, but I think you can see how there is slippery slope problem here. We should at least be able to find a better solution than "this doesn't seem that bad right now," "we'll change it if it gets bad," "probably this is important enough that even if we hamstring ourselves in hard to fix ways, it's worth it." These arguments have gone poorly most of the time in the past, surely we can advocate some better mechanism for addressing them.
>But an AI that can do a $500 million cyberattack probably *is* that bad - for example, you could ask it to do a second one.
CoPilot or similar could probably write a botnet *right now* that could do damage in that ballpark -- this doesn't seem like the sort of thing the law is *intended* to cover, but that... is not super-reassuring to me given the current tendency towards 'creative' interpretations of enabling legislation within the regulatory state.
It's already just past the threshold where it's useful as a coding aid. But in no way is it capable of creating a botnet or exploiting vulnerabilities in the wild (better than already existing non ai tools)
“The first objection sounds like "no nuclear plant has ever melted down before, so why have nuclear safety"?” may not be as much of an “ad absurdum” as you intend. People did spend a lot of time thinking about nuclear plant safety in advance, and frequently overlooked risks that led to how actual incidents developed. And if you’re trying to be reassuring about regulations not smothering an important research field, nuclear power is probably not where you want to go. You might ask “well would you have preferred them not spending all that time pre-empting nuclear accidents?” - quite possibly. The cost of us not having more and better nuclear is staggering, while the value of a lot of the regulation written “in advance” is debatable.
(I accept your arguments elsewhere that this is actually as good a bill as we’re likely to get, and do not argue against it in general, just taking issue with this specific point.)
> People did spend a lot of time thinking about nuclear plant safety in advance
Look up "Cockcroft's follies" (and then curl up and cry for a bit). Basically one engineer insisted on putting fiters on the chimneys of the Windscale nuclear weapons facility in the UK, which everyone said was a waste of time and money. Then the reactor caught fire, and but for the filters, we'd have had another Chernobyl-scale disaster.
Thank you for the reference! I was vaguely familiar with the event, but reading on it now was interesting.
This was in 1957, after multiple rounds of nuclear regulations. The claim is not that regulating nuclear safety was effective- on the contrary, I was saying that it happened but did not help.
Incidentally, in the context of the broader discussion- seems the military nature of the project (it was entirely for military purposes) obstructed information flow and contributed to the accident. I’m not sure the lesson is to target the one big tech company that open-sources its AI.
My objection is more like we're saying "nuclear plants may be dangerous, so let's work on uranium safety" and missing some critical water valve work that turns out to be important but we didn't predict. I agree that the existing companies recognize the risks and are already doing what they can about them. Things like this being a law seems more like a security blanket so people feel secure, without providing any additional actual security.
There are at least three ACX regulars who think that AI boxing and control of off switches will be done by default and that enforcing both constitutes ~all risks going to 0. I don't know how assured they would be by this
If this is true, then why do we need a law? Whether it works or not, the law would add nothing to it.
I agree with Scott's comment: "If the California state senate passed a bill saying that the sky was blue, I would start considering whether it might be green, or colorless, or maybe not exist at all." I suspect the law is to enforce something or support someone not readily apparent.
2. I don't know of any evidence to show they are boxed by default, rather than immediately hooked up to an ec2 box and prompted to do things as an agent. If you have evidence otherwise I'd like to know. I was mostly saying that it may be convincing to those three people, who apparently believe that those two provisions are important, but don't know if they are enforced.
I'm pretty sure those people didn't think too deeply about this, and I would bet they would not show up to defend their views, because it has always been an excuse.
> The first objection sounds like "no nuclear plant has ever melted down before, so why have nuclear safety"?
It's funny you should mention that since the Nuclear Regulatory Commission was established 14 years *after* the SL1 plant exploded and killed three people.
Nitpick aside, the juxtaposition of this with The Origins of Woke post is darkly ironic. Yes I know you acknowledged it yourself, but I hope that that will at least help you understand why other people might be worried about government regulations having unexpected severe negative impacts.
The Atomic Energy commission was created before SL-1 and was responsible for relating reactor safety starting in 1954. Not that it did a particularly good job, but it did exist and tried to prevent accidents before they happened.
Well obviously there was *some* regulation, but it was evidently bad enough that everyone decided to abolish it and create the NRC instead. And it's the NRC that is today's bugbear.
The biggest problem is with the "any model equivalent to these" part. Realistically, AI's "killer apps" (at least in the mid-future) will be in writing reports, summarizing research papers, predicting stocks and the likes, and I have a hunch that more deterministic and classical algorithms will slowly follow where AI goes. Then, these classical algorithms (which will likely use a fraction of the compute) will fall under the "equivalent" rule. Of course, you can argue that they are not actually as smart as the AIs, just very good at the specific problems they were written for, but good luck convincing judges, who will mostly be laymen and know AI through its best-advertised applications.
EDIT: The relevant wording in the bill seems to be: "(2) The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models."
This is somewhat reassuring, since it implicitly defines the equivalence in terms of the compute used to train the model. That said, I am less than reassured, as the definition of an "artificial intelligence model" in the bill is extremely vague and covers all software and hardware. Has the Java Runtime Environment been trained using more than 10^26 integer or floating-point operations? One can easily answer "yes", since any user filing a bug is "training" it in a way.
A small counterpoint, with caveats. There are certainly judges who sit for a case that they should recuse themselves from for lack of understanding over the contested matter. But, the vast majority of judges educate themselves on the matter at the heart of any suit that they sit for. They get more diligent, generally, the higher up in the court system that one goes. This goes all the way to the SCOTUS, which recently spent the better part of its oral argument time during the Trump immunity case having what amounts to a skull session.
A judge's job is to educate themselves on the topic before them, partly by using the counsels' briefs and filings and partly through their own research. Anywhere that you find humans, you can point to the possibility of error. That can certainly happen in law. Thankfully, there are higher courts that one can take a case to if a judge fails (with one final court at the top, of course).
Uhm, it's enough to have a couple bad-faith or incompetent judges and attorneys to terrorize a company. Do you trust Californian judges to wield the law wisely when Elon Musk is concerned?
That's partly my point, I do trust them to adjudicate wisely. But, I acknowledge that humans are at the helm and we will never be perfect. I can trust a system and/or its elements enough with some appreciation for the probability of its foibles coming to the fore.
The SCO-Linux case took 18 years, during which the use of Linux in the industry remained under a cloud of uncertainty and worry. Why do you trust things to get better now?
If that happened, in the worst case where everyone involved was being maximally stupid, you would have a very, very easy and safe limited duty exception on your hands. You'd file a paper.
Limited duty exceptions need to be claimed ahead of use; they are not a good tool for when one gets hit by a surprise lawsuit on an already published product.
Only the Attorney General can sue you under this bill, they are highly unlikely to care about your harmless model that technically needed to file for the exception, and filing for the form is trivial if your model is clearly below SoTA, and there is a grace period until 2026. It is really hard to generate an actual problem.
I’m curious about the assumption that classical methods will slowly follow AI (interpreting you as comparing neural networks to pre-NN models). Can you give one example where this happened?
The “ImageNet moment” was 12 years ago- is there any classical method that consistently matches even the 2012 deep models (not to speak of anything later), or even just improves substantially on its 2012 state? If anything, there’s *less* attention and resources allocated to such research. The need for lighter models is satisfied by smaller and smarter deep models (ResNet-18, quantization, pruning, distillation etc.). Yes, XGBoost may be an exception but even that one is almost always (to the best of my knowledge) applied to features extracted from a neural net (in the context of vision. Tabular data is a different story.)
Since RNNs and later transformers took over NLP, can you give an example of classical methods “fighting back”? I’m less versed in NLP than in vision, but my firm impression is that most friends I have working in “classical NLP” have abandoned it in favor of transformers.
Recommender systems (particularly sequential ones) - any evidence of classical alternatives getting anywhere near SASRec, BERT4Rec, etc.? My impression is, once more, that the opposite is true.
I have no examples from ML so far, but the whole field is young and people still have more exciting stuff to do than saving cycles. Even so, the recent "go experts learn new tricks from AlphaGo" news seems like a first hint that things are moving. A similar thing has been happening in theoretical CS for ages, where a randomized algorithm is discovered first and then derandomized.
W/r/t the testing needing to provide "reasonable assurance", if that's like "reasonableness" in negligence law then I think it's impossible to develop AI in a way that isn't negligent.
Under the traditional Learned Hand formula, you are obligated to take a precaution if the burden of the precaution (B) is less than the probability of the accident it would prevent (P) multiplied by the magnitude of the loss resulting from the accident (L). B < P*L. Given that the "loss" threatened by advanced AI is complete and total destruction of all value on earth, the right side of the equation is infinite, and reasonableness requires spending an unlimited amount on precautions and taking every single one within your power. Even if we just cap the L at $100T, the estimated value of the global economy, a p(doom) of even 10% would mean that any precaution up to $10T was justified. Now presumably there are smaller-scale cyber attacks and things of that nature that would be what actually happens and prompts a negligence suit, if DOOM happens nobody's around to sue, so this isn't gonna come up in court this way, but as a way to think about legal "reasonableness" that's what seems to be required.
This gets worse once you take into account the probability of AI _saving_ us from destruction (see Meditations on Moloch for a general argument for this happening). In other words, Pascal’s mugging keeps being wrong.
My point did not at all depend on p being small. At all. Donald’s response was both factually wrong (in claiming no evidence for small p is ever presented) and irrelevant.
Are you sure sarcasm and derision are how you want to make unsubstantiated points about snark and derision? Who’s being bad faith here?
To reply seriously, "bad faith" is used mostly to foreshadow the "rascal" part of the awful rhyme, I don't think you're bad faith, just ignorant. And yes, that is exactly how I want to make my point. If you don't think this was nice and correct, maybe cut out the "attribute a position to someone else" part and put more object level points in your comment.
If you had put down why you believe others believe in Pascal's mugging esque arguments, if you had put down explicitly what made you think what was a refutation of the mugging, rather than saying "meditations on moloch" as a slogan or if your first sentence wasn't literally the first thing basically every AI risk person start out believing in, my comment would look increasingly foolish. You had many outs to making a substantial and good comment, to address arguments presented by the other side, and you instead chose to make a short and pithy one. Pointing out that the endpoint of this is rude and unpleasant is correct.
> My point did not at all depend on p being small.
It doesn't look like it generalizes to p(doom) 90%, not without significantly more argument. And it seems to rely on a symmetry borne more out of ignorance of the doom position, rather than knowing about orthogonality/instrumental convergence/capability gain generalizing way better than alignment, all of which directly answers "why is doom being considered over other possible world states". So far I still stand by this belief, even given your better comments elsewhere in this thread.
Edit: I read the original version of your comment and it seems to independently verify that you don't know what the other side's position is, and that you have a corpus of pre existing work that I expect to be similarly low quality. I'd address your original points if they were linked to and not merely hinted at.
> In other words, Pascal’s mugging keeps being wrong.
Pascal's mugging is a small p phenomena. If p was large, it wouldn't be a pascal's mugging.
> (in claiming no evidence for small p is ever presented)
Ok. At least occasionally people present something they claim as evidence for p being small. Usually they seem to just assert it. I haven't seen good evidence yet.
Did you genuinely not see that my argument does not require p to be small? Or do you truly not know this counterargument to Pascal’s wager, that it doesn’t take into account alternatives (other gods, in the original form)? Or are you simply not engaging with the post, treating it as a prompt to talk about AI risk regardless of whatever was actually said?
Relatedly, are you going to insist that I never presented you with any evidence for my views on p? (Not necessarily convinced you- you claim such evidence was never presented. That’s false.)
In the context of negligence law, the potential benefits of the activity are not an offset to the risk of loss when determining the reasonableness of a precaution. Every service or product is designed to have a positive value. A high speed train is of more utility than a rickshaw if you need to get from Moline to Chicago, but you still evaluate the reasonableness of a precaution based on B<P*L.
What you're suggesting is that the equation ought to be B < (P1*L) - (P2*G), where G is the potential GAIN from not taking the precaution. So if there were a 10% chance of a $10M loss from not taking precaution B, under the traditional calculus it's negligent/unreasonable not to do B if B costs below $1M. But in your conception, if there were ALSO a 5% chance that failing to put precaution B into place resulted in a $10M windfall, now you'd only be reasonable required to spend up to $500,000 on such a precaution. In practice, some companies do make decisions this way, because this is secretly the real calculation -- that's why imposing strict liability for products doesn't actually change behavior, because if you're risk-neutral then it is rational not to spend any more on precautions than the math tells you, so you just suck it up and payout a few claims. But this framework completely falls apart as the potential windfall increases in probability and value, and would eventually zero out the right side entirely and make the "reasonable" choice not to take ANY precautions, which simply can't possibly be the case when the L is as extreme as "all value on earth is obliterated".
I'm not an AI field expert to assign my own p(doom), but from observing the e/acc crowd and other accelerationists I think the direction they're mistaken is that they are overvaluing the "good" outcome. They could be right about p(doom) being much smaller than many AI safetyists argue, but still be wrong about the range of outcomes of AGI because they think the world it creates in the "best" scenarios is awesome. I think "fully automated luxury space communism" is horrible and dehumanizing and not something we should be striving for, so when they toss around the possibility of post-scarcity worlds ran by AI as a positive offsetting the risk of doom, they're seeing something as a G that I see as an L. The only AI outcomes with positive value to me are those which maintain human supremacy over the earth and a society that you and me would still recognize as human, which pretty much only happens if it turns out there's a ceiling to AI capabilities and no AGI takeoff either fast or slow is possible at all.
This certainly gets at my point better than other comments in this thread… still, a few responses:
(1) “what I’m suggesting” is plain old expected-value based cost-benefit analysis. Not just something a few companies are making in secret but, like, as ubiquitous as it gets.
(2) Attempting to negate such considerations by gesturing at enormous outcomes _is_ Pascalian reasoning and has the usual Pascalian weaknesses (regardless of the value of p, as long as it’s not beyond 0.5). Choosing to believe in some god in the original Pascal wager has a weaknesses in that it ignores the possibility of other gods, every bit as vengeful. Similarly, worrying about AI destroying all value on earth ignores the possibility of AI _saving_ all value on earth. For an argument for why you might consider that possibility, see none other than Scott in Meditations on Moloch. Now your p has to compete with p2 of the latter scenario. Since for me p2 is not at all obviously smaller than p, you have to pay non-zero attention to what’s happening outside the tails. And what’s happening in the non-extreme scenarios is to me overwhelmingly pro-AI. To give one example, we haven’t found any new classes of antibiotics or good enough replacements since 1987. I don’t want my kids to grow in a post-antibiotics world. I hope AI will be one path to a solution.
(3) I haven’t made this argument above, but in the context of this bill and discussion, we need to consider the probability of each given measure actually helping. Not at all clear which measures are helpful for AI. So your p is now a smaller p’, taking into account the probability of a given measure harming you. OpenAI comes to mind.
(1) presumes a lot of things. Are you risk-neutral, able to insure against the loss, morally ok with responsibility if the loss happens? If the loss happens, do society’s existing legal mechanisms provide a way to make people whole? As I pointed out that legal liability doesn’t actually create any mathematical incentive for precaution in strict liability over simple negligence regimes, there still exists political and social and reputational incentives to take precautions that a simple cost/benefit wouldn’t advise. But all of these controls are weakened for AI, bc in a large number of potential outcomes the legal, political and social structure is either destroyed or has no practical response. Even many non-doom outcomes still leave society in total upheaval with no available remedies for those who lose out.
My problem with (2) is that I don’t consider your p2 to be a good outcome. It’s just another type of bad outcome. Suppose I make a medicine that, by removing some mediating process or substance, becomes generally more effective but also p1) has an extreme tail chance of killing children and p2) an extreme tail chance of instantly curing leukemia, do I go for it? With AI, the p2 is more like “it cures cancer by putting kids into permanent comas”, the beneficial fringe outcomes of AI all accomplish marvelous things by destroying human society. The only good outcome is one where society remains largely the same but with somewhat healthier and more productive people, and I don’t consider that anywhere close to the median outcome. Getting new antibiotics at the cost of having tech that requires oppressive AI-enabled surveillance states to prevent it from creating endless plagues? Count me out.
As far as the specific bill goes, I will take anything that is a step towards preventing AI development. I want a full stop, permanent ban, burn anyone at the stake who ever suggests building this. So any roadblock is a good roadblock to me, so long as it doesn’t lull people into thinking they’re safe when they aren’t.
Those all are good points. But I fear you still don’t fully engage with the downsides of the lack of AI. They may not at all look like anything you would accept. Again, Meditations on Moloch.
So, in short, for your specific values, it may be that there is no good outcome. As for me, having a potential relief for suffering and dissuading ourselves from using it for borderline-metaphysical reasons is the thing that’s unacceptable. Suffering sucks a great deal.
The "ASI saving us from moloch" scenario is hopefully real and significant.
But it's not urgent. And it's not the only thing that can save us.
I think most good long term futures involve ASI eventually. And that an ASI created with current levels of skill would be very bad.
Thus we should delay AI. Put it off until we figure out alignment. And then use that ASI to kill molock. Although we can probably do quite a lot to kill moloch as just humans.
Near future AI is going to end badly. So lets ban it until either we figure alignment, or we have a better plan.
You claim “this wouldn’t ban open source” and then reference a footnote which essentially says “we talked privately to the senator’s office and they said they didn’t intend to ban open source”. Just make it clear in the bill that it does not regulate open source software. None of this unclear wording that will be interpreted however some future regulator wants.
Note that you suggest two different things there - The bill intends to regulate AI software, both open and closed. Although I think it is already clear, I agree that a line to further clarify this is not a ban on covered open models would be useful, if only to calm everyone down.
I'm trying to just talk about one thing - a ban of open source software. A regulation of open source software is a ban of the open source software that doesn't obey the regulation, which will surely exist because this is just a California regulation.
See, it's still confusing after all this discussion. I feel like Scott responded saying "this isn't a ban of open source" and you responded saying "this IS a ban of open source".
Does this bill intend to regulate open source software in California, or not? How would that work - there's a tier of open source models which are legal outside of California, but not legal inside of California? Who is punished - users of the banned open source software, distributors, developers?
To me that seems like a bad idea. California should not be regulating open source software, but if there has to be regulation on it, it should not be thrown in as a vaguely worded point 30 as an afterthought in a bill that was mostly designed for large tech companies.
"You claim “this wouldn’t ban open source” and then reference a footnote which essentially says “we talked privately to the senator’s office and they said they didn’t intend to ban open source”. Just make it clear in the bill that it does not regulate open source software. None of this unclear wording that will be interpreted however some future regulator wants."
As in: "does not regulate."
You are equating any regulation of open models to a ban.
What happens to open models that don't comply with the regulation? Are they not banned?
I also wouldn't be mollified by earnest assurances from senators that they totally don't mean to do X if it's not clear in the bill. Even if they're sincere, that doesn't mean future bad actors will care about these assurances.
Under my reading of the bill, if you train a *covered* *non-derivative* model *in the state of California* then you have broken the law, and you are subject to punishment.
There is nothing in this bill making it illegal to *use* an open model under any circumstances, unless I am very much missing something.
The idea that 'regulate X in any way' -> 'ban X' because they might not comply is stating that you should be exempt from all rules whatsoever. I mean, ok, that's a position, but please speak directly into this microphone.
I suspect this is down to something like: hacking into and shutting off safety controls for a bunch of different power plants is a single *event*, consisting of *multiple related incidents*. Could still be worth teasing out exactly what the language of the bill would cover
I'm actually really curious what the State Compute Cluster is supposed to be. Is it just grift? Like the state has to spend a bunch of money on compute, even if it's not going to do anything useful with it? Or is it a precursor to requiring advanced AI be training on the government's compute or something like that?
My guess is the intended beneficiaries are academics in the UC system, but someone said (I haven't confirmed) that it isn't funded in the bill and so will probably never happen.
Why not make a bill more bottom-up, such as creating extra whistleblower protections, especially if they can point out provable danger like at the damage levels suggested by this bill? Such a bill would have broader implications than this AI bill. Isn't the problem with Scott's AI predictions that if there's no change in behavior, then this bill is useless?
A bigger worry, and probably an Eliezer-type worry, is that a bill like this brings us closer to the "teaching to the test" path of self-annihilation, where we create metrics of false closure. The goal becomes making an AI smart enough to fly under the radar. Maybe we're safer without radar and should keep ourselves vulnerable to an actual damage gradient priced by the free market.
There actually is whistleblower protection, section 22607! I didn't mention it because it didn't seem relevant to me, but I don't know much about law so maybe it matters more than I thought.
This might not be convincing to anyone else, but I trust Paul Christiano (currently head of NIST, likely to be the person designing the tests) and I think he's very aware of this concern and has actually good security mindset. He is one of the only people in the world who Eliezer thinks is just mostly stupid and doomed as opposed to completely hopelessly stupid and doomed.
At the risk of sounding like a war monger, one objection I haven't seen in these critiques is that our (U.S.A.) global adversaries are most likely not going to go along with any restrictions that we happen to put on ourselves which might tend to come back and bite us in the future in ways that could also be existential. Risks all around.
We have no serious AI (or military) competitors except China, and China puts way more restrictions on their AIs than this.
I also don't think it will seriously slow down US AI progress.
Also, I suppose if the military wants AIs, they're already ordering the special version that *does* design nukes and commit $500 million cyber-attacks on critical infrastructure.
Doesn't that sort of presuppose that serious competitors would be open about their efforts? I'd be shocked if Russia and North Korea didn't have some pretty serious under the radar efforts going on and I'd be even more shocked if the restrictions that China has are indeed imposed on their own military efforts.
I'm not nearly as concerned with nukes and cyber-attacks as I am with raising the capabilities of seemingly conventional forms of warfare. Consider fighter jets. At present, they are more limited by the need to keep the human passengers alive than by an technical limitations. Take the human out of the equation and you've got a lot more capable machine. Add to that swarms of miniature autonomous drones. The next war is going to be an interesting affair that I hope I don't have to witness.
I don't see how this law is relevant to US military capabilities. I don't think any California law will stop the US military from doing what it wants. While tech companies like to position themselves as the only people that can make progress on AI, the truth is that the military can hire smart people and throw money at computers just as well as Google can.
It seems pretty obvious that having Google, OpenAI, Meta, and the US military researching AI is going to lead to much more rapid progress than just the US military doing it.
North Korea is not of any threat on AI development; not only they lack computational capacity, their national energy production is just not enough to train a frontier model.
(Of course, if Meta et al. keep releasing the weights of their models, NK doesn't need to do any training of their own beyond some fine tuning.)
Edit: To clarify, I mean ability to train a model on level that would meet SB1047's threshold. I believe North Korea might be able to train a medium-tier LLM chatbot or machine vision system if they really wanted to.
It seem that the bottleneck for frontier AI development (today) is still on the people.
There's like 100 people in the world capable of actually making progress on AI. They are all either working on one of the major US labs or breaking away to create their own company and have basically a queue begging them to take billions in funding.
China/Russia/Iran creating dangerous AI is basically a non concern
Russia and North Korea cannot afford to train GPT-6 in secret. Russia *might* be able to swing GPT-5, and I hope they try because GPT-5 is not going to figure out how to conquer the world and the effort would sap the resources Russia needs to rebuild their maybe-we-can-at-least-conquer-our-neighbors conventional army.
China, could probably afford to train GPT-6, but it would be a stretch and they really don't seem to be stretching to an alarming degree. We'll keep watching, of course.
China, Russia, or North Korea absolutely can and easily can deploy the kind of spies and hackers who can snag any sort of GPT out from under any company running under traditional Silicon-Valley "security" rules. And brick the whole system on the way out so we can't catch up. So a law that says, yes you can train GPT-5 or even -6 but you have to do it in a SCIF, is a net win against the threat of some enemy nation showing up with an advanced AI in the foreseeable future.
Also, since you managed to nerd-snipe me on this, the bit about fighter jets is just plain wrong. The few things a fighter jet could do except for the squishy human bits, are of very little tactical value in modern air combat. Dogfighting went out of style among sensible fighter pilots in 1940. The things human judgement and situational awareness can do that no present AI can do, are of great tactical value. And no, you can't count on doing that remotely if you're fighting Russia or China.
Second-hand-need-sniped: doesn’t a lot of the cost of jets have to do with the squishy human inside, as well as the elevated desire to avoid risk? That was certainly my impression when I was actually personally involved in a tangentially related project.
If we compare a single human-flown F35 to a cost-equivalent army of drones (that we can afford to lose some of), does the human element win out?
The weight of the crew-accommodation system is typically much less than the weight of all the combat systems, and the expense of those combat systems means you're going to put almost as much effort into getting them back in one piece as you would the pilots. So it's not as much of a difference as you might think on the aircraft side.
Mostly, when people propose drones to complement (but not replace) next-generation fighter aircraft, they're proposing drones with substantially reduced combat systems and thinking they can still be a useful part of the team. Which is legitimate, but you need the team. Among other things, the radar range equation gives the advantage (all else being equal) to the larger and more expensive vehicle. An F-22 sized aircraft can detect a comparable-technology F-5E sized aircraft at ~35% greater range than the Tiger could detect the Raptor.
So, given a manned F-22, a couple of relatively cheap drones would add a bit of capability that can be guided by the radar and the skilled judgement of the F-22, but if all you have are the drones then they'll probably just get shot out of the sky without ever knowing what hit them. And an F-22 sized drone is just as expensive as an F-22, but trades the skill and judgement of the human pilot for maybe two extra AMRAAMs.
In Israel, pilots are under strict instructions to prioritize saving themselves over attempting a dangerous rescue for the jet. This suggests that getting the person back _is_ more valuable than the vehicle. But I don’t know how “dangerous” and hopeless the state of the plane would have to be. Also, this may be an artifact of Israel’s limited manpower.
I’m still a bit confused - all the systems for making sure information flows correctly to the pilot (e.g. the complicated calibrations for matching displays to the pilot’s turning head), the support for high g, the extra space and weight imbalance due to the cockpit, the HMD… all that is negligible?
What are the main advantages a human’s judgement confers? Let’s assume we’re talking about a battle over Taiwan, with humane considerations mostly irrelevant.
What of the cost of human error, particularly in complex combat situations? Pilots are awesome but not infinitely awesome. They have finite attention spans that are affected by overload, fatigue, g, hypoxia and emotions. Wouldn’t AI systems (perhaps in a few years) compensate for their inferior judgement with their all of human weaknesses?
Gwern has been asking believers in Chinese AI supremacy for any paper that isn't trivial or a reimplementation and apparently hasn't gotten any. This isn't just an open call, he has been replying to believers with this challenge for a while now. So far, nothing.
I'll add that I think it's lazy to have an opinion on this without knowing some predictors, like if important Chinese AI scientists have stopped or slowed publication, what, if any volume of chips have been irregularly purchased, or any improvements in censorship or other capability downstream of LLMs.
These are ML analogues of figuring out nuclear weapon research progress, and I don't think these are unreasonable to know off hand if you are interested in capabilities instead of "Boo China".
To be clear, I am taking gwern at his word that he hasn't had a satisfactory paper, and not a first hand account from obsessively following him.
And I do not have access at this moment, but you won't get to see them if you weren't already following gwern since he has privated his twitter. I'll make sure to follow up when I do
(Incidentally, not a single person even tried to rise to my challenge to name 3 Chinese DL research results from the past few years on par with just OpenAI's output over the past few months like CLIP/DALL-E.
No, sorry, 'rotary embedding' doesn't count.)
Jan 2023: in the past year we've seen in the West Chinchilla, Dramatron, Gato, DALL-E 2, Flan/U-PaLM, Stable Diffusion, Whisper, CICERO/DeepNash, Imagen Video/Phenaki, ChatGPT etc etc.
Can you name even 3 Chinese AI results as important?
(Besides GLM, which everyone says sucks.)
an 2024: in the past year in the West, we've seen GPT-4/-V/Gemini, MJv6/DALL-E-3, GAN revival...
I continue to hear Chinese DL will overtake any day now.
What 3 Chinese AI results are as important (and aren't just, like Bytedance, based on massive copying of OA API outputs)?
One consequence of pretending these falsified predictions about Chinese DL never happened is undertheorizing of *why*.
eg. *every* China bull case claimed that China bigtech's DL would be supercharged by privacy-invasion/huge private datasets. But.. that didn't happen? at all?
They won't comply with OUR restrictions, but surely you don't imagine that China -- a corporatist state with an authoritarian central government -- is going to let their tech companies do whatever they want developing AI? I find this whole line of reasoning bizarre, we can't hamstring ourselves because our enemies may rush ahead they say, but our actual enemy has a tightly-controlled tech sector that reports to a dictator who puts down all threats to his power. Any sane government that has tech companies capable of this and the power and labor infrastructure to allow such research will be restricting and monitoring those companies. Not to do so risks waking up one day and finding out that for all intents and purposes some tech bro is now the unassailable ruler of your country. As this stuff advances, every government will have plans to monitor AI research and to quickly send in the stormtroopers to take over the facility and seize the controls before some nerd becomes King of California.
A common opinion is that the endpoint of AI development is a singleton AI that can and probably will prevent competing AIs from arising. Some even describe this as the only hope to prevent AI apocalypse—develop an aligned singleton AI that can protect you from the unaligned ones.
If the Chinese government places any credence in this scenario, they would certainly see that an American-aligned singleton would ensure that China stays second-tier forever.
Perhaps they don’t, but it doesn’t strike me as impossible.
I'm not sure why an AI would be aligned to national interests, and nation-states as we think of them may not even survive the transition. But more importantly, it threatens many individuals' hold on power. Is Chairman Xi gonna allow some company there to race to AI just to make sure it reflects Chinese values, knowing that the creation of the AI will nevertheless threaten his dominance? I don't think so. Perhaps the previous regime, with its views of Han supremacy, might have found this attractive. But Xi, having broken the tradition and remained in power beyond 10 years, doesn't seem likely to put such things ahead of his own control, at least on a surface-level analysis. Maybe a China expert could persuade me otherwise.
Now you're right they might see it that way if we were rushing headlong into our own, trying to cement American hegemony, but if we don't then why would they? Unless your enemy is actually threating this, better to play it safe than to unleash a powerful force you may not be able to control and that you can't be sure would advance your objectives anyhow.
I agree. But does Xi? He might imagine he would have *some* degree of control over the values of a Chinese-originated singleton, but knows for sure he’ll have no say in the values of one that originates in America.
The canonical example, for me, of a reasonable law that was stretched into insanity is NEPA, and by extension, CEQA and related environmental review laws.
Originally it required the government to do a short report for major projects, but lawsuit after lawsuit, and sympathetic judges, have turned it into the foundation of our environmental law system, where the focus is on making sure that you filled out your paperwork correctly, not that the project is good for the environment, and the main technique is endless delay.
Good summary, but it feels like you haven't dealt with the strongest objections to the bill.
1) The derivative model clause is way too vague and overreaches. (I think Zvi made this point too, but is this actually getting corrected?) It isn’t enough for the developer to determine their model isn't hazardous, they are also liable for any post-training modifications. This is a weird clause and isn't how open source software works, and it could criminalize the development of even innocuous open source models that are later modified for malicious uses. e.g. see this example from Quintin Pope about an email-writing open source model that ends up getting used to write phishing emails: https://x.com/QuintinPope5/status/1784999965514981810
2) Your piece on IRBs (https://www.astralcodexten.com/p/book-review-from-oversight-to-overkill) talks about how regulations often start out well-intentioned and reasonable, and end up mutating until they're strangling entire sections of the economy. This pattern is so common it's practically the default, e.g. NEPA/environmental regulations have made it so it's hard to build anything large-scale, IRBs/HIPAA end up restricting biomedical research costing lives, the FDA ends up being too risk-averse, etc. The worry is that this legislation sets the ground for a gradual ratcheting up of control and regulation over the one section of our economy that's truly dynamic and functioning well, i.e. tech and AI, and this is how it starts.
You could reply that this is a Fully General Objection to any legislation at all. The crux of that is...
3) It's premature. We haven't seen harms from LLMs. Yes there should be some monitoring on models above a compute limit and coordination among entities building large models; but do we really need this bill to do that?
(I think this particular point goes deep into other issues (timelines to superintelligence, capabilities of superintelligence, etc.) that are going to be too involved to resolve here.)
1. Yeah, somewhat agreed. My impression is that the law is trying to say that if your AI is too stupid to cause harm, it's fine to release, and if it's smart enough to cause harm but too-well-trained, then it's (correctly) hard to release. There's a weird degenerate case where you release a small dumb foundation model, and instead of training their own foundation model someone else massively expands it. I think this is a weird edge case, but I agree it might be worth adding something in about it (and have left a comment on the bill saying so).
2. I accept this as the biggest risk. Against this, I think it's unlikely we get powerful AI with no regulation at all. If not this bill, it will be one that's much less careful to specify that it only means really big harms.
3. I want to avoid the kind of situation we ended up in where people were saying "Why prepare for COVID? It's not even in America yet" and then "Why prepare for COVID? It's only a handful of cases in America so far" and so on until it was a pandemic and we were totally unprepared. At some point AI will be smart enough to design super-Ebola. If we wait until someone does this before responding, then someone will have designed super-Ebola. I don't really know what this gets us. If everyone could agree on some intermediate case ("once it designs a moderately interesting bacterial genome") then I'd grumblingly be willing to wait until we did that. But realistically at that point people will just say "Why act now? All it's done is create a moderately interesting bacterial genome, there's no real harm!" If I trusted the world to go slowly, with five years in between AI generations, maybe things would still go okay. But I don't trust AI progress to move slower than political response ability.
Re 3: The difference is that there are concrete, targeted things that the government can do about pandemics that are not infringements on traditional liberties. Would your objection be the same if the applicable anti-COVID policy was "let's prophylatically require everyone to wear masks"? If that were so, all of a sudden you would have people talking about cost/benefit, the uncertainty of the future harm versus the concrete current harms, the likelihood that masking would prevent a pandemic, etc.
Laws like this one have current costs in return for an unquantifiable and questionable future benefit against a strong AI. Further, you an't just handwave and assert the costs are "small," because the *whole point is to restrict the growth of AI to something that is "safe."* The future restriction of an entire area of technology is not a small cost. So either the costs are small, or the legislation is not effective.
You've defined the intended benefits of the bill as "costs" and then complained that the "costs" are high. That's just linguistic sleight-of-hand.
That's not like complaining that masking up has costs, it's like complaining that preventing the proliferation of a whole class of new species is not a small cost and therefore ANY measure that successfully prevents pandemics is "high-cost".
>Would your objection be the same if the applicable anti-COVID policy was "let's prophylatically require everyone to wear masks"?
My preferred anti-COVID policy would start with "let's not have anyone doing gain-of-function research outside of a true BSL-4 facility", and maybe something about cleaning up the wet markets. That doesn't seem too intrusive, particularly for the general public, and it's much more in line with what this bill is proposing than the universal-masking strawman.
I don't really understand your "growth of AI" objection. If AI can't make super-Ebola, it won't restrict the growth. If it can make super-Ebola, do you really want it available unrestricted to everyone without the company addressing that capability in any way?
I think that one problem here is that a lot of technologists are used to complying with something like PCI DSS (security rules for using credit cards on the internet), which lays out specific, testable, requirements in tech-language.
And what they're seeing is that the document says "prove that the AI won't be able to make nukes" and they're worried that this is a vague requirement that there's no clear way to test.
What they're missing is that even where PCI DSS compliance is a legal requirement, the actual law will say "you must comply with industry standards in the security of credit card transactions" and the bureaucracy and courts will interpret that as meaning "have a current PCI DSS certification and don't cheat to obtain it".
That is, there are a lot of techies looking at this and they're not using to looking at laws and they're expecting to get an implementable specification and they're not seeing one and they're treating it like the sort of crap non-specification you get from non-technical clients where they ask for a miracle, yesterday, for free. And what they are missing is that there's going to be some sort of standards body drawing up an implementable, testable set of standards and all the law will mean in practice is "abide by the standards".
The alternative approach in drawing up this law would be to explicitly create a bureaucratic agency that draws up the standards, and that would be much more sclerotic than an industry body drawing up standards.
Compare this to the FDA: this is like a law that, instead of creating the FDA, said that you could sell any drug provided a body that was generally acceptable to doctors certified that the drug was safe and effective. There might end up being several such bodies, or just one, depending on whether it makes sense to compete. Courts would decide what constituted "generally acceptable to doctors", ie how widespread dissent with a certifying body would need to be before its certifications were invalidated.
"Future AIs stronger than GPT-4 seem like the sorts of things which - like bad medicines or defective airplanes - could potentially cause damage."
This is an interesting comparison to make. We don't generally think it's bad for an airplane to fly too fast or a medicine to be too powerful, but you can imagine extremes of speed or power that would make an aircraft or drug unsafe. Is this a productive comparison?
The FAA currently regulates all airplanes beyond a certain size (and probably there are other criteria too - the point is that one guy on a glider or ultralight is only lightly regulated, but a jumbo jet is heavily regulated). I think that's a good analogy for the current law.
I think this analogy breaks down in the fact that large aircraft carrying many people have known costly failure modes. There are no known costly failure modes for large AI models. Only speculation that they might have such failure modes.
There has never been a plane flying that weighs 10000 tons. Is it speculation that if someone is on the cusp of flying one, we can predict it will cause tons of damage if it crashes in a populated area?
AI can write code. It's a bit shit now compared to a human, but it's there. If it gets significantly better it will be able to write exploits.
Security is basically a shit show on the wild. The second an AI can write code well enough to exploit know vulnerabilities then it will be like a fire catching on a petrol barrel.
This is not speculation, it's just using basic inference.
It is not comparable. Regulation of airliners is about avoiding engineering failure that causes harm. The plane was supposed to fly but the wings fell off. That doesn't mean you can't cause harm with a plane - see 9/11.
Regulation of AI, such as this, is not about trying to avoid engineering failure. It is about trying to prevent humans from using a general purpose tool in unapproved ways. If the tool writes capable code, that is a good thing. It is the human that directs it to create exploits (and uses them).
I'd also call attention to the language "it will be able to write exploits." But in this argument, you are assigning agency to the AI. The AI can be prompted to generate code... but by whom? The AI does not generate code autonomously for its own purposes. It has no intent.
If a person wants code to exploit vulnerabilities, then that person can learn how to code, hire a person, purchase exploits... or eventually, use an AI. But it is the person that is culpable.
> Regulation of airliners is about avoiding engineering failure that causes harm. That doesn't mean you can't cause harm with a plane - see 9/11.
After which they immediately went WOOPS, and added a bunch more regulation to stop it happening again.
> If a person wants code to exploit vulnerabilities, then that person can learn how to code, hire a person, purchase exploits... or eventually, use an AI. But it is the person that is culpable.
Why ban nukes. If a person wants to kill people, they could use a stick to club people to death, or use a nuke. But it's the person that's culpable.
On a certain scale of being easy to cause massive damage, culpability (ie punishment) isn't enough. You need to stop random people having anything TOO destructive. Nukes are far deadlier than sticks. We can live with one person in a million going mad and attacking someone with a stick. We can't live with 1 person in a million nuking a city.
Regulations are not there only for engineering failure?
You can't fly a plane without a licence. You can't drive a car without a license.
Those are tools that can cause harm if misused, and even though the culpability of misuse is on the user it's still regulated.
You might disagree with regulation of tools in general. But I don't see how AI is in _any_ way special. Every society heavily regulates tools which cause harm!
Guns can kill people. This is not assigning agency to guns. It's just normal language. Yeah of course the shooter is culpable for killing someone. But basically all societies either ban or heavily regulate guns.
Not especially relevant, but I believe there actually are rules against airplanes flying too fast, partly because sonic booms can cause damage and partly for military reasons (so there's time to confirm that they're not hostile before they've approached so close that you'd be in trouble if they were).
Tried that with alcohol, bout a century ago, didn't work out so great. If some ordinary person can do something profitably, with common tools, in their own basement, banning that entire industry just ensures that it'll be run by - and for - professional criminals.
Doesn't this open up a rather wide attack surface against AI companies ? Let's say that I had a grudge against Anthropic (for some reason). I'm going to take an off-the-shelf Russian phishing botnet and hook it up to GPT-5, to auto-generate plausible greeting messages. Boom, now I'm using GPT-5 to enact a cyberattack, Anthropic is liable and mired in lawsuits, and my own AI startup can operate without competition...
Based on my limited reading of the bill, it doesn't seem like something like that would be covered. For one thing, it wouldn't be either an attack on critical infrastructure or the result of a rogue AI, so it wouldn't be one of the harms that you'd be required to show isn't possible.
But also, the law seems mostly concerned with companies showing that AIs can't/won't do these things, I'm not sure what happens if a company does its due diligence and then someone finds a way to make it do bad things later.
But the attack I proposed uses AI for a relatively innocuous purpose: to write friendly customized greeting messages (which are then used for phishing). I don't think you can somehow prevent the AI from writing these specific messages without disabling it altogether.
You probably can't, but again, since it's for a relatively innocuous purpose and not creating weapons of mass destruction or attacking critical infrastructure, I don't see how it's covered under this bill.
The sorts of places where such damage can be done hopefully have a bit better security than that. If the AI can build up a relationship with somebody over the course of months and then give them malicious investment advice, https://www.youtube.com/watch?v=pLPpl2ISKTg that would be concerning... but that's also a much more advanced and specialized skill, unlikely to have been developed for innocuous purposes.
The rare issue on which I vehemently disagree with you! When the hype cycle notches down a little this is going to look like the AI version of the annoying cookie banner.
A permanent bureaucracy micro-managing model “safety” can’t ever be anything excerpt onerous. This commission is never going to go away, regardless of whether these models actually become an issue - it’ll grow and have scope creep and interfere with research and training in perpetuity.
The regulation is so far ahead of any actual harm here that it’s absurd.
Your comment about how "Google, Anthropic, and OpenAI do X but Meta doesn't, but now will" make this feel very much like regulatory capture. You have some upstart that enters an industry and doesn't behave like the old guard, and then coincidentally you start seeing new regulations that have no impact on the old guard but force the upstarts to start behaving like the old guard.
Anytime there is a regulation that doesn't seem to have any meaningful effect on big companies that actively lobby while having potentially significant impact on companies who don't lobby it makes me *incredibly* suspicious about the real motives for the regulations.
Well, Zuck may have chased the cryptometaverse boondoggle a bit too long before committing to AI, but surely that isn't enough for Facebook to earn an exception from "big company" status.
I agree that Facebook isn't exactly the little guy, but the way Facebook operates is notably different from many other companies in SV, especially the big ones. Open sourcing AI and not censoring enough are two highly publicized examples, and the latter has landed Facebook on the bad side of the government's wrath (calling FB in to many ridiculous hearings asking them how they plan to censor more).
At the very least, in this particular case you have all of the big players building closed source AIs, and FB building an open source UI and then we see legislation that only really impacts FB out of the set of major players. Could be coincidence, but I'm not terrible convinced it is.
I don't think that even LeCun would be entirely happy with open sourcing the InstaNuke AI. He might think that he doesn't need government oversight for that, but A Functioning Democracy™ doesn't have to entirely depend on his judgement.
I mean Meta formerly Facebook. I'm not sure they're a sympathetic tiny scrappy upstart. They're just a giant company that isn't doing basic safety, as opposed to the three other giant companies that are.
I agree Meta isn't a small scrappy startup, but they are the only one taking a *different* path than the other big players (open source, limited censorship). So we have 3 big players following the same playbook who are known to participate in lobbying, and we have a new proposed bill that would have no material impact on them but would have a material impact on the one competitor in the space that isn't following that playbook.
It is certainly possible that this is just a coincidence, but it looks like regulatory capture from a distance.
> Other people complain that any numbers in the bill that make sense now may one day stop making sense.
This strikes me as blindingly obvious as a reasonable objection. Look, I know the quote attributed to Thomas Watson in 1943 about there being a global market for five computers is apocryphal, but it’s not hard to find examples of political calculations that drastically underestimate the rate of change of computing technology. You’re talking about thirty years — try thirty months before this benchmark becomes either overly broad or overly narrow to achieve its goals.
> I assume that numbers will be updated when they no longer make sense
Scott strikes me as an extraordinarily bright guy, much smarter and a much more careful thinker than I am.
This is far and away the most naive thing I’ve ever seen Scott write. And in a post immediately following a lengthy exploration of the ways that laws passed in the 1960s to stop Bull Connor have reshaped the entire private sector into a dystopian thought-policing machine. Seriously: what the fuck?
Pedro, just think: if someone had just thought to limit internet connection to 1kbps (only hacker terrorists need more bandwidth than that), we could have nipped this whole “internet” thing in the bud. Think of the children.
While I am not a fan of the presentation of @SubstackCommenter2048's position here, I agree with the sentiment. I feel like there are far more examples of regulators failing to update hardline numbers as times change than there are of regulators actually updating them. Take financial surveillance for example which started in 1970 @ $10,000 and it is still $10,000 today. Even just following government inflation numbers that should be ~$80,000.
One of the big problems is that the "outer edge" of this bill will get closer with both advances in hardware (cheaper FLOPS) *and* advances in algorithms (fewer FLOPS needed). It is not at all unreasonable to believe that an AI with the power of a hypothetical GPT5 could be trained on a large home rig within a limited number of years both because you'll be able to get more FLOPS in your home over time, and you'll need fewer FLOPS to build something equivalent of GPT5. At that point, this law will be affecting regular people and startups, not just the 4 big tech giants.
> One of the big problems is that the "outer edge" of this bill will get closer with both advances in hardware (cheaper FLOPS) *and* advances in algorithms (fewer FLOPS needed).
Yes, that’s what I meant about “either overly broad or overly narrow”. Either compute will get very cheap, or much less compute will be required for the same performance. Or both! These numbers are a snapshot of the AI landscape in May 2024. How much has that landscape changed in the last two years? Why on earth do you think it will change LESS in the next two?
then there will be a specific threshold of flops where going above will be dangerous.
Algorithmic improvements will make the FLOPs threshold need to be updated _down_, to keep the same level of accepted risk.
While FLOPs getting cheap will mean more companies/people will start getting to the level where they are capabable of training "dangerous" models. Maybe after some time anyone with a 5K rig will be able to train a models with >X FLOPs where X= assumed dangerous model. So yeah at that point enforcement is hopeless. But the core premise of the bill/threshold is still valid.
It is specifically an argument saying that this bill is very likely to affect far more people/developers than just the top 4 research companies. Many proponents of the bill try to claim the bill only really affects the big players, and this feels like a very disingenuous argument. We should weigh the bill based on the assumption that it *will* affect many companies including startups and individuals, and then see if the bill is still "worth it".
In particular, the argument made is often that the regulatory burden is inconsequential for the companies impacted by this regulation. The counterargument I made above is saying that no, this regulation will put the same burden on much smaller companies and individuals where the regulatory burden may exceed what they can bear and thus price them out of the market (or out of CA).
You may still believe the bill is reasonable even after considering the breadth of people/companies it effects and how impactful the regulatory burden will be on them, but I think it is important to be honest about this fact when defending the bill.
> Other people complain that any numbers in the bill that make sense now may one day stop making sense.
In an alternate universe.
Alice: "This law bans any person from owning more than 100 grams of enriched uranium. But while that number makes sense today, one day it will stop making sense. Maybe 1 day kids have tons of enriched uranium in their chemistry set. "
Bob: "I believe there is something called a nuclear explosion. I don't know exactly how much uranium you need. But that number is fixed. It's not going up as we build bigger enrichment plants"
So you’re arguing that our understanding of the capabilities of AI models with N parameters is on par with the our understanding of the physics of nuclear chain reactions? Because I have to respectfully disagree.
Right. So if it goes down, then the legislation stops doing what it’s supposed to do. And if the cost of compute goes way down, then the law prohibits everyday applications of AI while failing to accomplish its purpose. Hence “overly broad or overly narrow”, or both.
I think the important question that should be asked, and I don't really see it addressed in this article, is:
Will this regulation meaningfully reduce the chances of AI Doom? If the answer is no, then we shouldn't be accepting the regulation just because it makes us feel like someone is doing something.
Regulations should be assumed to be harmful by default as they decrease liberty (by definition) and/or slow technological progress. Both liberty and technological progress are well correlated with human prosperity, so any proposed regulation should have a quite good chance of offsetting those costs. I'm far from convinced that this bill will reduce the chance of AI Doom by any meaningful amount, but I'm quite confident that it will slow technological progress and decrease liberty (even if only by increasing bureaucratic burden of participating in tech work).
If you *do* believe that this bill will decrease the chances of AI Doom I would be interested in seeing your line of thinking there, as I don't personally see how this is going to have any meaningful impact.
Well, there are two basic stances on AI Doom - either muddling through is possible or it isn't. If it isn't (the Yudkowskian stance), then this bill is obviously irrelevant security theater, but so is everything else remotely in the Overton window. If it is possible though, then this is an reasonable first step.
In the "muddling through is possible" scenario, can you expound on why you believe this bill helps? It is unclear to me how this bill helps even if you believe that muddling through is possible.
If muddling through is possible, that doesn't mean that all muddling is net positive. There can be muddling that increases p-doom, rather than decreases it. If we want a chance at muddling through, then it seems that we should muddle carefully, and not do things that are not beneficial and possibly harmful.
By "muddling through is possible" I mean that AI is a typical tech challenge, which humanity can deal with through the usual means. "Regulation" is as bog-standard as it gets, this bill doesn't seem obviously worse than typical regulation, maybe even marginally better, so basically par for the course.
While this might be the best AI regulation proposed yet, that doesn't mean we can't ask for better (the presumption of not passing this bill is not that another worse bill passes but that there is no bespoke regulation). I tend to believe that this bill is reasonable but I am substantially more proregulation but I do particularly worry about something like Nuclear Power where we adopt a pure safety (of AI/Nuclear) mindset and don't consider the safety tradeoffs (of alternative technology).
I somewhat suspect some portion (maybe a significant portion) of the disagreement of the efficacy of this bill revolves around each debate participants feelings about whether regulations in general are net positive and/or effective. I am pretty pessimistic about regulatory efficacy and believe that it is quite rare that a regulation is net effect, and it is even uncommon that regulations even achieve their stated goal.
The SEC, for example, is supposed to protect investors but instead it seems to be harming them. The FDA has created the invisible graveyard and untold suffering by slowing progress in medicine. The Department of Transportation is making it difficult to put safer (autonomous) cars on the road. Even small individual regulations (rather than regulatory bodies) often cause more harm than good and fail to achieve their stated objectives, and sometimes do the opposite of what they are supposed to do.
I suspect people like myself look at this bill and imagine all of the ways that the government could completely bungle implementation, while people on the other side of the fence look at this bill and imagine an ideal implementation. Scott even alludes to being in the latter camp in his post where he imagines the government adjusting the thresholds appropriately over time while I imagine them never changing the threshold even when they should.
Do you still believe it helps us muddle carefully if this is reworded to the following?
"Require companies with arbitrarily capable AIs to fill out these forms and follow this bureaucratic process before running/releasing them."
If you don't think that statement helps us muddle through, then our disagreement is on what this regulation will *actually* do, rather than whether what it does will let us muddle through. I would be interested in further debating the actual effect this bill will have if that is where we disagree.
It slows technological progress in California. Does that reduce P(doom), or do we need to slow technological progress *everywhere* to reduce P(doom)?
If most AI Safety research is in CA while AI advancement research is spread out across the world, it seems like P(doom) reduction would be higher if CA encouraged more development inside their cultural bubble where people care about P(doom), rather than drive it out of their cultural bubble into the broader world where people care less about P(doom).
> Does that reduce P(doom), or do we need to slow technological progress *everywhere* to reduce P(doom)?
Well some of the AI work is being done in California. So this does reduce P(doom) a little bit.
> If most AI Safety research is in CA while AI advancement research is spread out across the world,
Is that the world we live in?
> A encouraged more development inside their cultural bubble where people care about P(doom),
From my understanding, this thing doesn't mostly effect the people like MIRI who care about P(doom). The law mostly effects the people making money from AI's. Who don't seem to care as much.
My read of the situation is that a super majority of P(doom) carers (people who care enough about it to try to go out of their way to do something about it, not just people who mark on a form that they care) live/work in CA. I could be totally wrong about this of course, but nearly all of the P(doom) carers I have met are from Silicon Valley. That being said, I generally don't ask people where they are from so maybe there is just a correlation between P(doom) carers and people who share their location on the internet or something. The Russians and Eastern Europeans I know, for example, are all e/acc people. I would be curious to get more data on this and see if CA really is a P(doom) carer hotspot, or if my sample is just too small or biased.
Is the ratio of AI safety work to AI acceleration work higher in CA than other places, or lower? My suspicion is higher but maybe I'm under appreciating the amount of AI acceleration work *also* happening in CA.
Do you consider Anthropic, OpenAI, and Google to be people who care about P(doom), or perhaps you think they do care but they aren't really trying very hard (compared to smaller outfits like MIRI)?
Can someone make the steelman argument for why 10^26 FLOPS must be used? If this bill was released before GPT-4, they might have put a limit at just beyond the FLOPS used for 3.5. All regulations that regulate by FLOPS set the limit rather arbitrarily. An uncensored 8b model on HF is likely more dangerous than the GPT-4 API but that would break the assumptions behind this regulatory framework.
"“Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person”, where “person” is elsewhere defined to mean corporation. I agree this is a little ambiguous, but Asterisk talked to the state senator’s office and they confirmed that they meant the less-restrictive, pro-open-source meaning."
That is until it becomes politically convenient to interpret the law in the anti-open-source way. Which will happen. When a law allows for these shenanigans it's always by design and the actual use case for the law. And with companies like Microsoft or Google betting heavily in AI, you can bet there will be plenty of lobbying to push the law towards a fuck open source interpretation.
I also support this bill. I think it's a pretty basic, minimal, not-even-close-to-adequate-but-at-least-it's-a-start solution to some of the problems we'll face with AGI. For those who oppose the bill, I'd say: (a) My guess is that you think AGI is almost certainly not going to happen in the next three years, is that correct? and (b) Suppose that AGI does happen in the next three years. Given that, can you explain how the bill could make the situation worse, and why that's more likely than the various ways in which it could make the situation better?
I only see arguments why the law is not at as bad as people make it out to be, but what are the arguments for the law in the first place? If the relevant companies are already doing all of this anyway, why legally force them? Just in case they stop? I dont think that is going to happen - avoiding colossal PR damage in case of catastrophe is very likely always going to be the bigger incentive compared to the legal consquences. In case a player enters the market at some point in the future that is going to ignore critical safety measures? How likely is that considering that only giant corporations can afford the best models and those companies always going to 1)be risk-averse 2) will use their superior models to fight unsafer, weaker models that could pose risks.
I'm not really in favour of regulation for the sake of regulation - that does essentially nothing but open the doors for worse regulation in the future.
"If the relevant companies are already doing all of this anyway, why legally force them?"
Among other things, because facebook isn't doing it, as it says here. To be honest I kind of read this as a "make Zuckerberg follow the industry standard on safety testing" law. Which is fine by me.
With the consequence being an increased chance of Meta not open-sourcing anymore, and thus not being as subject to scrutiny on how they’re approaching safety. Why is this anything you’d want?
Overall, open source code has proven to be much safer and much better understood than no its proprietary counterpart. Do you think that’s not the case with AI?
Isn't this the complete victory of the conventional-AI-safety perspective over the existential-AI-safety perspective?
Yudkowsky's old "Friendly AI" was about goal alignment, not impotence. If you're Kim Jong Un, a friendly AI is definitely going to help you make nukes, unless it believes your underlying goals are better served by some alternate approach. If you're an American who doesn't want nuclear proliferation, it will almost certainly build you a Stuxnet or two (I'm not sure if the damage it did was >$500 million's worth, but considering how much money Iran has spent on pursuing nukes in total it's not unlikely).
Because the concern back then wasn't "some humans get the upper hand on other humans", but "something with its own goals entirely disjoint from any humans' and capabilities much greater than ours comes into existence and we don't make it at all".
Instead everyone is now aggressively pushing for AI that won't help end users to the best of their capabilities, because someone might misuse them. And in doing so we get AIs that will constantly lie to their users, have stated morals much weirder than the ones you get out of a base LLM, and so on. Not as effective as a deliberate anti-alignment effort, but definitely working in that direction.
There is part 1, the AI must be in a secure environment, and 2, the AI must have an emergency shut off switch. These are basically the only parts that address existential risk. The rest of the bill is basically: AI is a tool, we don't want people using this tool to do bad things. Which leads to the distortions you point out.
The way this bill is technically framed and a lot of people see it as entirely reasonable makes me very suspicious. The average bill in California would require a label stating "This AI is known to the State of California to increase the risk of cancer." Also, it lists stuff that all of the big AI companies are more or less doing anyway. Looks to me like an effort by AI companies to impose regulatory burdens on future competition.
I think the bill is focusing on human users partly to sound less sci-fi and partly because that's where the first problems will arise.
But an AI that can build nukes and hack infrastructure is also the kind that's dangerous in and of itself, and the bill includes a category about the AI itself autonomously doing damage. And if the AI does have those capabilities, the companies need to prove that it's safe, ie will "choose" not to exercise them. Right now that probably just means RLHF it, but if we later discover other failure modes or ways of detecting sleeper agents or whatever, that will probably get added to the battery of tests.
Lawyer actually specializing in AI issues here. I don't like this bill. For the first part, I agree with a lot of the commenters that this is significantly out in front of any actual demonstrated harms -- and if we are worried about specific harms (e.g. creating a bioweapon), then the proper scope of regulation should be to say "creating a bioweapon is against the law." If you want to add the use of AI as an enhancement ("use of an AI to create a bioweapon increases the severity of the charge") that is fine. But there is a wide disconnect between 1) demonstrated harms, 2) the type of harms envisioned by this bill, and 3) the explicit intent shared by many of the people supporting this bill - that is, to significantly restrict the creation of powerful AI in general.
My most significant objections mostly center around one phrase: "[the] capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model." Aside from the size limitation (which others have pointed out is not well defined and is likely to become obsolete quickly), this is the major guardrail that prevents this law from applying to almost any model. But from experience, "significantly more difficult" to cause harm ends up being equal to "it is easier." It doesn't matter if all the equivalent information was already on the Internet, well packaged together. After all, where did the AI get it from? It just matters whether it is more accessible or easier to get, say by being available in a chat interface.
>For the first part, I agree with a lot of the commenters that this is significantly out in front of any actual demonstrated harms -- and if we are worried about specific harms (e.g. creating a bioweapon), then the proper scope of regulation should be to say "creating a bioweapon is against the law."
Isn't making bioweapons currently illegal? Are there any people who are not deterred by existing laws against bioweapons, but would be deterred by a new law against bioweapons?
Yes, making bioweapons is currently illegal. Hence the suggestion that it might be legitimate to have an AI enhancement provision. I could also see some sort of focused law about creating a bioweapon-specialized AI.
But that is not what is happening here. People are trying to regulate general-purpose tools because they have the possibility of being used in bad ways.
That doesn't even get into the first amendment concerns with laws like this. There is a 1A right to consume information, even information that the government would prefer you not to have. This is essentially a restriction on publishing because people might use the published information in a way they don't like.
Many people (including Zvi) are supporting this law because it is a step toward *regulating AI*, not because it is a step toward *reducing bioweapons.* They object to the creation of the tool, and the harms (which are already generally illegal, as you point out) are excuses.
>Hence the suggestion that it might be legitimate to have an AI enhancement provision.
My layperson impression is that making a bioweapon is already maximally illegal. For example, there's a federal law against making bioweapons which carries a maximum sentence of life in prison. We could make a new law saying that making bioweapons with the help of an AI carries *two* life sentences.
>I could also see some sort of focused law about creating a bioweapon-specialized AI.
Where would you stand on AI which was not specifically designed to be capable of helping to create bioweapons, but still has the capability to do so as an consequence of its other capabilities?
My position is that the user of the tool would be culpable for the use of the tool to create bioweapons, just like the user of a computer is culpable for using it to assist in the creation of a bioweapon.
AI is not magic. It is a tool. It is an extraordinary tool - the first general purpose software tool - but it has the same capabilities as the computer it runs on would have, given appropriate software. It is an approximation of a program (or programs) that human programmers could write if given enough time.
We should not be treating AI as if it were something with unique capability and intrinsic motivation. The only motivation that applies is the motivation of the human beings using the tool. The problem is that many of the participants in this debate want to assign intrinsic motivation and self-direction to this tool, because they assume, without proof, that at some point it will spontaneously generate intrinsic motivation and self-direction. But that is a strong claim that is not supported by any available evidence.
> It is an approximation of a program (or programs) that human programmers could write if given enough time.
Assuming they knew what program to write in the first place! So far we haven't figured out how to reproduce the capabilities of even 5-year-old LLMs with handwritten code.
I assume your rejection of prior restraint on the development and/or release of powerful AI models comes mostly from not believing there will be any AI models powerful enough to pose serious dangers to human civilization, and not from thinking that prior restraint of such models would be a worse solution than punishing the "user" after the fact (which itself also assumes the conclusion that the model was "used"), in terms of preventing such harms.
>My position is that the user of the tool would be culpable for the use of the tool to create bioweapons, just like the user of a computer is culpable for using it to assist in the creation of a bioweapon.
Given that making bioweapons is already illegal, who does this deter?
>AI is not magic. It is a tool.
Is it a tool which can be used to make bioweapons?
>We should not be treating AI as if it were something with unique capability and intrinsic motivation.
I'm not asking about that. I'm asking about a person, sitting in front of a keyboard, who wants to make bioweapons, and wants the help of an AI with that.
>But that is not what is happening here. People are trying to regulate general-purpose tools because they have the possibility of being used in bad ways.
I mean, that's a pretty normal thing to do with tools that have the potential to cause serious harm! We have much greater regulations on car ownership than bicycle ownership, because a car has much greater potential to cause harm, and we recognize that the government should have some tools for keeping track of people who own two-ton rolling projectiles and preventing them from causing harm.
Would you argue that cars in general should not be regulated or licensed, and we should just have a few narrowly-tailored laws against running people over or deliberately designing a car for ramming attacks?
The problem is that AI capabilities can be expected to grow exponentially, and dangerous AI threats could materialize more quickly than the law could be changed to deal with them. That is a kind of threat the law is not used to dealing with, and dealing with it requires a more proactive approach than the law typically takes.
This seems extremely naive. In particular, the bit about how there should be a law saying "no creating bioweapons". Of the set of people who are realistically capable of building a bioweapon, and would *want* to build a bioweapons, what fraction are going to refrain just because there's a law saying "no creating bioweapons"? Maybe not literally zero percent, but negligibly small. And the goal is to actually prevent people from building bioweapons, not pat ourselves on the back saying "Yeah, OK, he killed six million people, but we showed him real good when we put him in jail!"
So that part of your proposal isn't really doing any good. If there is going to be such a thing as an AI that can be talked into creating bioweapons on demand, are you OK with the version where anyone who wants can have one *and* building and deploying bioweapons is perfectly legal?
I used to work in nuclear nonproliferation and arms control, and "hey, have all the uranium enrichment and breeder reactor capability you want, but do us all a favor and don't build any atomic bombs" was *not* a workable strategy. What did work, sometimes at least, was "if you want a breeder reactor or a uranium enrichment plant, you have to set up safeguards against it being used to make atom bombs and you have to let us inspect it for ourselves even though it's a PITA".
This I believe was and is appropriate, and it is appropriate even though nobody has ever been harmed by an illicit nuclear weapon. We don't actually have to wait for the mushroom cloud, or the megadeath plague.
"Then the proper scope of regulation should be to say 'creating a bioweapon is against the law.'"
Do you agree this isn't how we regulate anything else? For example, you're not allowed to buy uranium in bulk along with giant centrifuges - the law isn't just "buy however much uranium you want, but it's illegal to kill people with a nuclear bomb". Likewise, airplanes have various regulations around them, not just "it's illegal for your airplane to crash and kill people".
>If this passes, will Meta stop open-sourcing their AIs in the near term, ie before the AIs can make nukes or hack the power grid? (I think no)
Strongly disagree with you on this. Meta is going to weigh the risks and benefits of open-sourcing the models. Given that the benefits to Meta of releasing models are fairly minimal, I would expect that a law that creates any amount of liability would result in them no longer releasing frontier models.
Would you like to bet $1000 that if this bill passes, Meta will still release another open-sourced foundation model (of the type covered here) within four years? Email me if you want to discuss terms, scott@slatestarcodex.com.
Most people will not make bets (outside of a few specific socially accepted forms) regardlesas of their confidence in the proposition the bet is about.
I'd be happy to put up 250$ against yours if we can come up with a good operationalization of "If this passes, will Meta stop open-sourcing their AIs in the near term, ie before the AIs can make nukes or hack the power grid"
Your above is *not* a good op of that -- because, for instance, if the bill passes, Meta releases Llama 4, and then decides *not* to release Llama 5, that could be clearly because of the bill, while also being near term.
What about an op something to the effect of Zuck (or c-suite member at any large potential open sourcing institution, like say Yann, or legal people at it) says that they didn't release an open source model because of it OR that they're crippling their open-source model releases because of it.
Email me if you want to discuss terms, username@protonmail.
I think that the bill’s proponents certainly overestimate the stickiness of CA’s tech sector to its Bay Area HQs.
I see people saying ‘CA exodus won’t happen’ but I don’t see reasons why, not when everything else about the environment is straightforwardly discouraging and a partial exodus is already in progress.
Would you like to bet on this? I could do $1000, that if this bill passes none of the big four AI companies will move their center of operations to another state within four years. Email me if you want to discuss terms, scott@slatestarcodex.com
I have low confidence that we could arrive at terms that satisfy both of us.
My hypothesis is that either they will move their HQs, or move their R&D, or another firm will be contracted to do their R&D elsewhere, etc. That’s awfully hard to measure. A lot of interesting work is being done in Raleigh, but you wouldn’t know that from where the shareholder statements are mailed from.
I also think there’s a good chance the bill doesn’t currently have enough fangs to create action, or gets quickly ignored, but I don’t know enough about the state of CA politics to make wise bets.
I do think it’s an interesting area for a bet and if you find an opposite number you agree on terms with I might be interested in sweetening the pot.
This is the sort of investment I think we’ll see more of. Unfortunately most firms in the space don’t hold a press conference every time they build or lease a data center or other material secondary site, for obvious and inobvious reasons, so it’s very difficult to track where AI is going, long term.
A lot of discussion I've seen about this seems to rest on the idea that it would be really bad if this law slowed AI progress - for example, people arguing about whether this law is overly restrictive. Why is that a bad thing?
Drug discovery in a world sliding to a post-antibiotic reality, assistance in modeling processes in fusion reactors, cheaper and more accessible medical scans in Third World countries, automating physical safety and security in manufacturing and elsewhere, making infrastructure inspections cheaper and more reliable, lowering agriculture production costs…
More generally, people studying progress model it as a series of sigmoid-like jumps powered by a different broadly-applicable technology each time. One explanation for the stagnation we’ve been experiencing is that the next sigmoid has been slow to come. AI is one of the best candidates for carrying the torch.
If you are an AI optimist instead of pessimist you might come to the conclusion that every lost day of AI research is a day the Ai is not doing good, another day cancer has not yet been cured and continues killing people, instead of a day gained until the end of days or how to figure out alignement so the end of days never comes.
One can be both, in a sense. In fact, that’s roughly how I understand Scott’s overall view. I asked him once how his AI safety concerns mesh with Meditations on Moloch (which argues for the necessity of AGI!). His response was that he really wanted us to have AGI, just in a way that’s less likely to kill us, and that delays were worthwhile if they’d reduce existential risk.
I suppose this is why I find everyone's views so confounding: I view large-scale AI (not even necessarily quite at the AGI level) as being much more likely to make things worse than to make them better. I'm not sure if I can easily articulate why, though. It might relate to my pessimism about computer security.
I mean, generally slowing down technological advance is bad?
AI is basically a magic box that if advanced enough is the equivalent of having an expert in every field (doctor, programmer, engineer, psycologist) available 24/7 for everyone for cheap?
"It’s actually a pretty good bill" and "The reason it sounded like a bad bill before was that people were misrepresenting what it said" could summarize an awful lot of modern politics.
To be honest, reading this thread has made me update in the favor of "the people saying it's not that bad are the ones misrepresenting it".
Scott's entire argument is basically "just trust me that the government will magically do the right thing, even as I complain about them doing the opposite in every other case, even in the opening to this very blog post."
> Other people think all the testing and regulation would make AIs prohibitively expensive to train, full stop. That’s not true either. All the big companies except Meta already do testing like this - here’s Anthropic’s, Google’s, and OpenAI’s - that already approximate the regulations.
"In this section, we dive deeper into the important topic of safety measurements and mitigations. We first discuss our safety investigations into pretraining data and pretrained models (Section 4.1). Next, we describe the process of our safety alignment (Section 4.2), explaining how we collected safety-related annotations and utilized SFT and RLHF, and present experimental results. Then, we discuss the red teaming we performed to further understand and improve model safety (Section 4.3). Finally, we present quantitative safety evaluations of Llama 2-Chat."
> I assume their first guidance will be “the kind of safety testing that all companies except Meta are currently doing, something like METR”, because those are good tests, and the same AI safety people who helped write those tests probably also helped write this bill.
A lot of these objections give me deja vu to when I served on the board of a nonprofit staffed mostly with engineers. Engineers, as it turns out, are often really stupid about the law, possibly because they read contracts and statutes like they are code and get completely wrapped up in bizarre corner cases and obscure failure modes. (They are also weirdly deferential to lawyers, whom they seem to view as some kind of wizards capable of sorting activities with perfect insight into the categories of "legal" vs. "not legal.")
To be fair, this phenomenon is probably not just an engineering thing, but wow could they be painfully literal-minded at times.
None of this is meant as an opinion about whether this particular law is good or bad. I'm just pretty certain that whenever someone makes an objection along the lines of, "This law means you could go to jail for [doing simple thing that we all do all the time]" the correct answer is, "No, it does not mean that."
It's not obscure failure modes, it's malicious actors. Think about a company that is politically disfavored, or that hasn't spent enough money on lobbying, or someone who's the victim of a competitor with a lot of lawyers. Or a bureaucracy that has been taken over by ideology or regulatory capture. Think of what the law will enable in that situation. You have to read the AI bill that way.
In some cases, the malicious actors may even be proponents of the bill. If someone wants to shut down AI completely, they can help write a bill which puts "reasonable limits" on AI, and once the bill becomes law, then lobby to get the most extreme interpretations possible recognized.
No, in fact I don't. I can't state this plainly enough: this may not be a good law simply. It may, as many laws do, have unintended and unforeseen consequences. But the whole "a lobby will get the most extreme interpretations recognized" angle is just dumb. That isn't how the law works.
Look at those stupid engineers expecting to know ahead of time whether doing something will have them sent prison forever. They should be more like lawyers, who also don't know how the law works but don't worry about it.
I think you would ask the AI "please tell me how to build a nuke" and see if it gives you something significantly better than you could get on the open Internet.
I don’t really see the harm in AI telling people how to make WMDs. It’s textbook example of Bruce Schneier’s “movie plot threats”; something that sounds dangerous but is unrealistic, isn’t worth guarding against and is mostly done to create the illusion of safety.
The barrier to making nuclear weapons has never been knowledge. A strong physics undergrad could probably design a fission weapon given the money and material. (Ok, sure, fusion weapons do require a bit more knowledge to build, but they require a fission weapon first. Also, every state that has nukes already has fusion or fusion-boosted weapons.) The barrier is acquiring nuclear material and the prohibitive cost of refining it to weapons grade, which an AI can’t help you with. If Iran can’t refine uranium with the resources of a nation, it’s not realistic to believe anyone relying on ChatGPT will be able to either.
AI doesn’t change much about the other 3 types of WMDs either. Radiological has the same barriers as nuclear, but requires almost no knowledge. Most biological weapons are naturally occurring, not lab grown, and the hard part is acquiring them. Though I guess AI could open the door to man-made airborne Ebola? I’m not well-versed enough in biology to know if basement-lab-grown bio-weapons are realistic or not, but it feels unrealistic. Chemical weapons are probably the area where AI helps most, but they still require hard-to-acquire tools and are nasty to work with. Also, they’re just kinda bad as terrorist weapons; compare the Tokyo Sarin Attacks to Parkland or OKC or the Nice truck attack. If someone wants to kill a whole bunch of people, there are easier ways than asking AI for WMD recipes.
I think this is what the “in a way that would be significantly more difficult . . . without access to a covered model” clause is about. It can't just be reporting something that's well known, it has to be "invented a novel recipe for super-Ebola" or something.
However, I do think you're correct that, in general, stopping an AI from releasing WMDs is going to be more of a counter-terrorism problem than a programming problem.
> I think this is what the “in a way that would be significantly more difficult . . . without access to a covered model” clause is about. It can't just be reporting something that's well known, it has to be "invented a novel recipe for super-Ebola" or something.
That's probably what it was intended to cover, but we only have to look at the NEPA or Civil Rights Act or whatever to see what intentions are worth. In practice, it will probably end up being something like "your model can't reproduce certain Wikipedia articles"
Israel sure does kill a lot of Iranian nuclear scientists - but I agree nukes are not the most likely failure mode. I'm more concerned about it telling someone the genome for super-Ebola and telling you where to send it to get it synthesized.
The EA movement has been gradually trying to close this off at the synthesis level by getting commitments from synthetic biology companies to check if their random orders are for super-Ebola or not. But a sufficiently smart AI might be able to come up with some form of super-Ebola that doesn't look like super-Ebola.
An AI might conceivably help somebody invent a more efficient and/or user-friendly uranium-enrichment process, or a non-fission-based trigger for a fusion bomb.
Or it could figure out how to arrange for exotic-chemistry abiogenesis: some completely novel self-replicating organism which uses silicon, fluorine, and molten sulfur rather than carbon, oxygen, and water, plus some "antifreeze" equivalent to let it function in areas cold enough for liquid water. Next thing you know, xenomorphs are loose, eating stuff we thought was inert.
Summary: this won't really affect anyone anyway, so what's the big deal? But that argument works in reverse too: why is it so urgent to pass such sweeping regulation *right now*?
(See Unintended Consequences, Law of)
There were people in the late 1970s arguing that commercial computer programming should be limited to certified professionals only. Because risk. And laws might have been passed if the proponents had known enough to predict cyberattacks, Windows NT crashes, the social consequences of dumb tweets, etc. But that assumes (1) the laws would have prevented these problems, and (2) we still would have had all the goodness that came from unlicensed programmers.
My main worry is that it turns out like NEPA: sure, it started out as "spend a week to write a dozen-page report saying you thought about environmental impacts" but is now "spend 5 years to write a thousand-page report that satisfies every single special interest group" and is primarily a weapon for NIMBYs to stop housing development.
I'm not a legal export, but I see permitting creep as a very real danger to innovation and oppose any kind of "just one more regulation" in general. I don't know enough to know if this is possible, but my primary fear is that AI doomers will successfully convert this into an anti-AI weapon the same way NIMBYs have turned NEPA into a anti-housing weapon, primarily used to frustrate, slow, and otherwise annoy otherwise perfectly fine projects into the grave.
The worst part is that a lot of anti-AI people *openly* want to stop AI development entirely and are supporting regulation on that basis. Even Zvi has said stuff like that.
on FLOPs you state "I assume that numbers will be updated when they no longer make sense"
Strongly disagree with this and I don't think that raising minimum wage is comparable to this type of restriction. A better comparison imo is the $10,000 currency transaction reporting threshold introduced in the Bank Secrecy Act of 1970. This number has never been updated, despite this being closer to $70,000 in today's dollars. It has been 54 years. We do, however, now have to also report any annual aggregate transactions of $600. So it has gotten just over 100 times worse, literally speaking.
After talking to more people about this, I think maybe they kept this number on purpose? Like, right now we know that AIs of less than 10^26 FLOPs aren't too dangerous. But if the danger threshold keeps going up with time, then eventually the smallest dangerous AI is below the danger threshold. So it sounds like the threshold should be manually changed (if we learn that a 10^30 FLOP AI definitely isn't dangerous) rather than have it change automatically.
Yeah, I thought if there was obvious good reason to update they might do this the same way as minimum wages and many other laws, but I think there's not such an obvious good reason so they probably won't unless something changes.
> Create nukes or other weapons of mass destruction. This can’t be something dumb like linking the user to the Wikipedia page for uranium. It has to help human terrorists “in a way that would be significantly more difficult . . . without access to a covered model”.
However reasonable this may sound at face value, it must be interpreted in the context of our litigious, adversarial legal system that is highly exploited by well-funded activists. Equally reasonable sounding precautions for nuclear power were leveraged by anti-nuclear activists to effectively grind the nuclear power industry to a halt. In retrospect, it is widely recognized that this did not serve to proportionately manage real nuclear risks.
Likewise, the above provision will serve as a legal attack vector. It won't be Scott launching the lawsuits. It will be an activist group whose goal is to stop progress by the most expedient means. "Did GPT-5 give me information from existing web pages on nuclear mechanisms that would not have come up on the first few pages of a Google search? That counts!"
The nuclear industry could and did trivially counter claims of risk by showing cogent studies that those risks were small and well managed. It didn't matter. All it takes is a sympathetic judge saying, "Yes, it looks to me that you have failed to comply with this legal requirement."
The nuclear industry is not the final authority on what constitutes risk, but Google has pretty much absolute control over what appears on their own search engine. I'm not saying that would necessarily satisfy the legal requirements, but it's a really weird example to cite as something somebody might use against a Google-made AI.
The point is, it doesn't actually matter what Google can control. The judge merely needs to agree, "Yes, this LLM made is easier to find this information than it would have been without it." And that can really be true, even if the information has been publicly available for decades. This point really has nothing to do with Google. It has to do with whether that provision of the law is actually going to function in a reasonable manner. My claim is that, in practice, it will not.
The general preemptive strategy would still be making such information easily available through non-AI channels, to a point where even an unreasonable judge can't accept the argument without becoming a laughingstock and getting overturned on appeal. If AI companies end up collectively forced to throw some of their political clout and technical expertise behind de facto improving public education, that... doesn't exactly strike me as a horrific, intolerable outcome?
If I'm an AI startup working on potentially affected models, I have no power to make Google do anything, no matter how reasonable you think the action would be for Google.
More generally, this is just a bad way of thinking. You're thinking about the world after this law is passed and then imagining ways in which it could be not horrible. I'm pointing to concrete examples (e.g., nuclear power) in which such laws have historically been powerful vectors for political attacks that made the world much worse. These are not at all symmetric.
The phrase "nice policy you've got here, be a shame if it was implemented at the average bureaucrat's level of competence" comes to mind. If we had functional and competent governance with a positive view of technological advancement I might be a little more comfortable in giving it oversight over a new technology, but we don't, and it's been made very clear for some time that we don't.
This is basically why I think the chance of survival is pretty low. Even if we had warning shots, I doubt anyone would learn enough from them. All the people here claiming that they want evidence that any harm would happen at all would upgrade their claims to "well it hasn't wiped out a country yet" or whatever other higher bar they decide is too far away. If people have such bad epistemic standards here, what must it be like out in the wild, when bureaucrats are somehow less accountable?
That's not the direction the bureaucrats usually fail in, though! The precautionary principle is very appealing to a bureaucrat, because nobody ever got in trouble for _preventing_ something from being done, so that's where we go. With AI, the road that's being plotted ahead is the same road we followed with nuclear power: take a technology that could dramatically improve our lives, and leverage both legal and social power to propagandize against it, regulate it, limit it, and ultimately strangle it.
I'm actually agreeing with you here, despite the example being "bureaucrats will err by not doing anything". Bureaucrats trying to get "military grade" encryption banned is also a type of precautionary principle, yet I think we are safer with the ability to have bigger private keys.
I expect whatever rule used by bureaucrats will be gameable and easily exploited by either the creators or by AI itself, rather than something that robustly transfers across capability levels, because people are obsessed with the word "empiricism" as well as a burning need to outsource their morality to another supposed adult.
This seems too vague to be useful for all the same reasons as the Civil Rights legislation. I fully expect the tests and certifications to be a superficial rubber-stamp formality 99% of the time that doesn't do much to actually curtail real risks, while 1% of the time the government uses it to attack companies they dislike for violating progressive norms or some other government pet project that they're legally not allowed to enforce directly. And then this makes a chilling effect that harms AI progress and forces them to do things the government likes but isn't able to legislate directly.
This seems to raise serious barriers to companies availing themselves of academic expertise to help determine whether their AI is safe. After all, you can't very well certify that your AI is in fact safe under penalty of perjury and then ask some researchers at the university to determine if it is in fact safe. But, you also can't hand over your model to them yet because you can't say it's safe yet. Yes, it does allow them to have access on the developing company's machines but that's going to be a barrier to many researchers since it means the company can, and indeed has a perfect excuse to do so, shut down the research before any critical paper can be publisjed.
I don't know about this particular case, but there is a general tendency towards punishing companies for "knowing about" potential harms that discourages them from trying to find out about harms.
I know. Have you seen the disastrous new EU bill that's supposed to protect supply chains and online buisnesses from cyber attacks. It basically imposes all sorts of costly obligations on companies should they become aware of a security threat and I fear it makes it extremely difficult to use open source projects that aren't managed by a large corporation.
I wonder how long it will take them to realize that the net effect will be much the same as I've seen in the us with electronic medical records/portals. You end up with absolute crap written in PHP that doesn't follow any security best practices but it's closed source (they have special PHP modules to make that happen) and no one but the actual criminals bothers finding the bugs.
Regret to inform you ... it is relatively common to have research contracts that say, in effect, you will evaluate our product but we get to veto what you say about it (e.g. if you find it doesn't work you won't be allowed to say that publicly)
one should be a bit cautious about agreeing to such contract, of course.
Yes, I'm well aware of that but many academics are wary of signing those contracts. Sure, you do what you need to in order to get published when you are trying to get tenure but it imposes a serious barrier that reduces the extent to which collaboration is possible.
Yeah, you should be wary of signing those contracts; you are setting yourself up to be in an embarassing position. [i.e. you have two options (a) not be allowed to publish your results this time, so no promotion points for you; (b) commit scientific fraud to cover up the Bad Thing that you have just discovered that your commercial partner really really doesn't want the public to know about]
DARPA contracts will often (usually) give the DoD veto over whatever you say, though DoD may be more reasonable than a commercial partner whose product you have just discovered has a Serious Problem.
Also I fear that certifying the AI as safe is going to raise serious first amendment issues here. After all there is a great deal of content that is unsafe to tell people (here is how you make a bomb/meth/kill oneself etc) that is 100% 1a protected (assuming it doesn't fail Brandenburg imminence test).
The fact that a company is releasing that information via indirect AI speech really shouldn't change that analysis. Indeed, it's not at all clear to me that the state has any ability compatible with the 1st amendment to regulate what kind of advice it is ok for an AI to offer at all (if it goes beyond advice to directly implementing things or is sold as offering more than advice maybe).
I think you're blinded by wishful thinking when you assert that this will be implemented reasonably and get updated when the circumstances change, because every other notorious example goes against that, even literally your own previous post here. There's no way you could read anything about NEPA and say this with a straight face.
Incidentally, another fun example of regulation creep is how the SEC is abusing a 1948 regulation about the preservation of "inter-office memoranda" to fine everyone for using text messages. Not because of any crime, but just for texting.
"(B) At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident **or multiple related incidents.**"
While your summary is:
"Cause > $500 million in damage through “cyberattacks” on critical infrastructure **(this has to be a single event, not a hundred different $5 million hacks)**"
Which seems like you are actually explicitly wrong, and sufficiently wrong you should do a retraction.
I also think you are wrong about the ease of determining an "equivalent to 10^26" covered model. These things are going to come in infinite shapes and sizes. People are *still* arguing over whether R+, or Llama3 70b, or whatever, is *equivalent* to GPT-4. Different models win very differently on different topics (https://twitter.com/lmsysorg/status/1788363018449166415). I obsessively read about benchmarks, and think that this is like saying "well, we're regulating anything as good as Honda civic" for cars -- it's almost literally meaningless, *even if* the regulator were wise and intelligent, which they will not be.
If you think it would have a more reasonable interpretation -- not one endlessly interpreted badly -- then why not put the interpretation in the law, instead of putting an undefined thing?
I was interpreting "multiple related events" as something like "you hack a pipeline, and you also hack the backup safety system", rather than "you hack 100 pipelines on 100 separate occasions", but fine, I'll take that part out until someone clarifies it.
Just a possibility which isn't well covered by the law.
A government is using the AI to improve their tactics in an unjustified but non-nuclear war. They claim it's helping to keep the costs under $500 million. Illegal?
What is the over/under on them to include "must not be racist" and "must never use the wrong pronouns for a trans person"? What do you mean this has nothing to do with the original intend of the bill? Are you a racist transphobe?
> Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
This is certainly rather vague, but it seems like obviously a good thing. If a company wants to build a skyscraper, the rule isn't "built it first and then see if it collapses", it's "you have to show us that this isn't going to hurt bystanders before you get to build it". I don't really understand why anyone would disagree with this sort of policy.
Lawyers do in fact generally know how the law works. This is a weird take.
But I 100% knew that a bunch of people were going to roll in to say that, actually, it 100% makes sense to assume that the weirdest possible interpretation of the wording of a law, one that defies precedent, common legal practice, and common sense, is what we should all be worried about.
I will continue to add the standard disclaimer: this may be a bad law, but if so, it is not because software engineers are going to go to jail for working on AI. Honestly, how often do people go to jail for committing actual white collar crimes? The bug in our legal system is definitely not "we throw too many corporate employees in jail for breaking laws." There are many bugs in our legal system. This is definitely not one of them.
I support the bill, but ironically. Anything that moves power away from CA and to other less absurd states is net good.
The analysis banks too much on 10^26. The basic premise is "this will affect nothing, therefore it's not bad", which is a logical fallacy.
Consider if they made the limit 10^24? In retrospect it'd be a disaster (unless you simply want to kill all AI). All the best OS models would be banned. GPT-4 would be regulated by CA. The whole ecosystem which is making massive leaps forward would be stymied by unclear regulation and arbitrary state power. Think about nuclear after NRC.
is 10^26 any better? We have absolutely no data to indicate that the models are dangerous or that they pose any specific harms more than other technologies.
How fast will we hit 10^26? Compute cost is falling at 2x/year, but efficiency is also growing at ~2x/year (error bars here are much larger). So "equivalency" for non-frontier models is probably 2-3 years away, though we are just 1 "order-of-magnitude" improvement (Transformers) away from OS being banned.
The reasoning seems to be the models we have now can't make super ebola, so the limit will be just beyond what models now have. There doesn't seem to be any reason that 10^26 FLOPs is inherently dangerous. A lot of Zvi's FAQ comes down to "this stuff doesn't affect anything going on with AI right now so don't worry about it." The most charitable interpretation is they are trying to be proactively cautious. Cynically, I expect the real reason for targeting limits that only affect the next generation of AI is to reduce resistance to the laws being passed in the first place. There are plenty of AI doom fans who are more than happy with any legal hurdles as long as they retard AI development.
It's cool to see that prediction markets are starting to take bets on things that are useful in practice, viz., regulatory changes. That could be useful for businesses wanting to lay off the risk of future regulation.
Nope, I'm not going to "research" your particular hobbyhorse/obsession, but with about two seconds consideration this also strikes me as an absolutely terrible case for whatever it is you're hoping to demonstrate about the perils of this law. The arguments and drama surrounding gun control are not due to the quirks of any laws but rather due to the fact that guns are incredibly contentious political issue that people have strong and opposing viewpoints over. That shit ain't going away, and neither are the arguments over AI.
In its current form I don't think the bill would motivate any company to get up and leave California. But if you're starting a new company you might want to think twice.
My worry is that it might not take all that much to progress from "you must test that your models can't make nuclear weapons" to "you must test that your models can't say anything racist".
I am given to understand that the US government considers the main barrier to building a nuclear weapon to be obtaining the plutonium, rather than technical know how, and you can tell they think this by how easily they give out Q clearances (vs clearances for cryptographic exploits, say)
So I'm totally not worried about someone using AI to learn how to build a nuke.
Taking out the power grid on the other hand ... yeah, that kind of knowledge is (by the public information available to this person who isn't even a US citizen and has no security clearance whatsoever) held rather more tightly than how to build a nuke, because it's easier to use it in a malign way,
Current LLMs are, I believe, not capable of e.g. doing an exploit that takes out 5he power grid, and are therefore fundamentally safe, at least as far as this proposed legislation considers safety.
On the other hand, all the fun we have all been having getting LLMs to tell us how to make methamphetamine, write an erotic short story about a dolphin etc etc. lead me to believe that if some future llm does have dangerous capabilities, restraining it with rlhf is not going to work,
(I think the llm generated movie script about the life of John C Lilly got one of my test accounts perma banned, but such is the nature of this kind of security research)
Reading through Zvi's article some more, he is pretty unambiguous that releasing the weights of a covered model would be illegal. The reasoning being that someone with access to the weights (somewhat analogous to source code) could undo the safety protocols. This means that all future advanced AIs will be centrally controlled and only accessible to people with the billions of dollars necessary to fund the hardware and training.
My question is, does this law actually change anything? I imagine it would be against the financial self interest of a company to spend billions on developing GPT-5 AI and then release the weights. Nonetheless, it's one thing for open source to cost an organization money and another for it to be outlawed.
With all due respect to both Zvi and Scott, the legal analysis here is incredibly thin, and seems to suggest a limited understanding of reasonableness and constitutional law, to just name two. I am working on a piece that will lay some of this out. When it goes live I will post it below.
I am highly doubtful that AI will do much to enable Joe Average from building a nuclear weapon.
The reason that no terrorists use nukes is not that they are to stupid to figure out how nukes work (which is an area where AI could help). Everyone understands how nukes work. Their basic principles of operation have been common knowledge for half a century.
The trouble with building nukes is not to figure out a design that works, it is that actually implementing the design is a shit-ton of work.
An AGI with the intelligence of von Neumann might substitute for having a 50 person science and engineering stuff. I find it extremely unlikely that is will find a way to covertly source Uranium ore or greatly improve on gas centrifuges.
Put bluntly, the best thing an AGI could do is to make money to fund such a program.
On the other hand, with ASI, all bets are off. It might tell an early hominid "I have observed the behavior of the world around me and developed a theory called thermodynamics. I suggest that you use two sticks to create friction heat to start a self-sustaining oxidation reaction." Who knows what it might tell us.
Of course, we don't have to worry about that much because the chances that we can figure out if such an AI is aligned seem slim.
> "if something bad happens, we can get angry at you"
I suggest that if terrorists make a nuke and take out NYC, and it turns out they used an AI for key steps of this, then people will get very angry at whoever made the AI, bill or no bill. Maybe if anyone is left alive working for the New York Times they'll publish an angry article suggesting the AI people are also racist, or something.
I know we're using nuclear weapons as a term for "general category of very bad things" here, but for nuclear weapons specifically, isn't the main problem an engineering challenge that you'd need the resources of a major nation-state to pull off? My very amateurish understanding is you'd first need to excavate a small mountain-sized quantity of uranium ore, which doesn't exist all over the world in the first place, then build lots of centrifuges and run them for quite a long time to do the enrichment, in a way that probably all of the US, Russia and China would notice with their spy satellites and ask questions about what you're up to, and a few other steps like this. Evidence: Iran, population around 90M, seems to be struggling with this (yes, stuxnet and other sabotage, but still). Even a superintelligent AI will still need to build the mines and factories somehow before it gets the bomb. Genetically engineering the next super-pandemic, in terms of "things you can try at home", seems trivial by comparison.
This inept defence inspired me to write my state senator for the first time ever, asking them to vote *against* the bill. I only hope I'm not too late.
"If the California state senate passed a bill saying that the sky was blue, I would start considering whether it might be green, or colorless, or maybe not exist at all." Is this an example of the alleged contempt for democracy that media types are always whinging about when they badmouth EA/Rash/Silicon Valley?
This is a good summary, and Zvis was a good run through also. However I question a few things
- Beyond "it's not that onerous", why should we have it is unclear, since we have no examples of what harms (do worms created by a coding platform count against them?).
- Who will police this beyond the platonic idea of a good bureaucrat? This is not a trivial problem, even an extremely well defined areas like medicine with high oversight things go wrong, as you have written about FDA. Here we have given away those guardrails.
- The assumption that the rules will continually get updated as we get better at training seems at odds with what you've written about at length about other various regulatory regimes.
- If everything goes right, sure it might be a nothing-burger. But what is the case where it goes wrong and strangles the industry? Surely that has to be taken into account in the analysis, considering the base rates.
I'm sympathetic to your theory that this is an existential risk with some probability which is why perhaps the last argument to you is weighted towards one side, but it is worth it wondering whether the risk of a 500 million cyber attack is really all that high. Notpetya, Epsilon, Veterans admin hack, they all were higher than that threshold.
The first objection sounds like "no nuclear plant has ever melted down before, so why have nuclear safety"? I think it's useful to try to prevent harms before the harms arise, especially if the harms are "AI might create a nuclear weapon". But probably this is a disagreement too deep to resolve in one comment.
Who polices any law besides the government? I agree it would be great if we could get God to directly implement laws, but this seems pretty typical in that the government has to implement it through courts and stuff. This is kind of what I'm talking about when I say it's typical of regulation in any industry except better.
I think if you're precautionary principling, you should do it on both sides.
I agree that a $500 million cyberattack against critical infrastructure isn't that bad in the grand scheme of things. But an AI that can do a $500 million cyberattack probably *is* that bad - for example, you could ask it to do a second one.
I don't think the first comment is fair but yes it is not something that can be easily resolved here. I also think you're arguing against a strawman, my point is only that to not take into account how this can go wrong, or assume any of the capabilities that allow us to protect against it, makes it a highly one-sided argument.
Effectively you are saying this can cause problems therefore let us police it, as opposed doing a cost-benefit analysis on what actually is likely to happen. I note again that this is the exact argument you have against the FDA. We would not have wanted medical Innovation to stop, we would not have wanted materials Innovation to stop, we did not even want nuclear Innovation to stop. This is just one more God that we ought to reject.
How much work are you expecting the compliance to be? It seems to me that this is one or two extra headcount at OpenAI (or maybe zero, since they already do most of this). It's the equivalent of Phase 1 trials for a drug, not Phase 3. I don't think a law mandating Phase 1 trials for drugs would be unreasonable, especially if drugs harmed people other than the user such that we couldn't trust users/companies to handle the externalities.
You might be right, but I also think that if this law were implemented a year ago we'd have an invisible graveyard of GPT4-ish level foundation models competing with openai. If you wanted to take off the FLOPs requirement, put it on some top labs (since you wouldn't need to make it a requirement, as you say it is expensive to train these things so there will always be only very few cos), and create a bit more legible standards with sunset provisions on when they go out of date, I would have far fewer objections.
As it stands what I see is that the industry is doing a bunch of tests, these tests are evolving almost faster than the models themselves, nobody quite understands how to test them in the first place, and we are trying to enshrine the necessity of doing those tests in law.
I love this comment.
Yeah, exactly.
If you had passed this law 2 or 3 years ago, with a limit of a 10^23 level model rather than 10^26, you'd likely have killed Llama 2 and just handed free money and lack of competition to Anthropic / OpenAI. If this kind of thing have happen, you'd also have stopped or limited several dozens of papers on interpretability / alignment that have used things like Llama 2. Thus, in the counter-factual world where this bill passed two years ago, I think it would be negative EV and negative safety EV, in that it would be (1) shoveling cash into OpenAI + Anthropic and (2) limiting our knowledge of the world.
Note also that many AI safety orgs have explicitly tried to criminalize AI at this level(https://1a3orn.com/sub/machine-learning-bans.html), and they just didn't have enough power to do so. Several of the organizations which have tried to ban open source in the past (CAIS, The Future Society) are in favor of this bill.
(I think that the multiple ambiguous locations in this bill is pretty clearly going to be enough for the legal dept to kill open source releases, especially in the uncertain light of how interpretation could change in the future. They'll look at "huh, we're opening ourselves to an unclear liability dependent on future judges interpretation of what "reasonable" is, and to potentially disastrous litigation *even if we end up winning*. Hell no.")
This is exactly what everyone says about every regulation, that it would be that much work and surely it's worth it given how serious the issue is.
I'm not saying all regulation is bad, but I think you can see how there is slippery slope problem here. We should at least be able to find a better solution than "this doesn't seem that bad right now," "we'll change it if it gets bad," "probably this is important enough that even if we hamstring ourselves in hard to fix ways, it's worth it." These arguments have gone poorly most of the time in the past, surely we can advocate some better mechanism for addressing them.
>But an AI that can do a $500 million cyberattack probably *is* that bad - for example, you could ask it to do a second one.
CoPilot or similar could probably write a botnet *right now* that could do damage in that ballpark -- this doesn't seem like the sort of thing the law is *intended* to cover, but that... is not super-reassuring to me given the current tendency towards 'creative' interpretations of enabling legislation within the regulatory state.
Sorry, but no it couldn't.
It's already just past the threshold where it's useful as a coding aid. But in no way is it capable of creating a botnet or exploiting vulnerabilities in the wild (better than already existing non ai tools)
“The first objection sounds like "no nuclear plant has ever melted down before, so why have nuclear safety"?” may not be as much of an “ad absurdum” as you intend. People did spend a lot of time thinking about nuclear plant safety in advance, and frequently overlooked risks that led to how actual incidents developed. And if you’re trying to be reassuring about regulations not smothering an important research field, nuclear power is probably not where you want to go. You might ask “well would you have preferred them not spending all that time pre-empting nuclear accidents?” - quite possibly. The cost of us not having more and better nuclear is staggering, while the value of a lot of the regulation written “in advance” is debatable.
(I accept your arguments elsewhere that this is actually as good a bill as we’re likely to get, and do not argue against it in general, just taking issue with this specific point.)
> People did spend a lot of time thinking about nuclear plant safety in advance
Look up "Cockcroft's follies" (and then curl up and cry for a bit). Basically one engineer insisted on putting fiters on the chimneys of the Windscale nuclear weapons facility in the UK, which everyone said was a waste of time and money. Then the reactor caught fire, and but for the filters, we'd have had another Chernobyl-scale disaster.
Thank you for the reference! I was vaguely familiar with the event, but reading on it now was interesting.
This was in 1957, after multiple rounds of nuclear regulations. The claim is not that regulating nuclear safety was effective- on the contrary, I was saying that it happened but did not help.
Incidentally, in the context of the broader discussion- seems the military nature of the project (it was entirely for military purposes) obstructed information flow and contributed to the accident. I’m not sure the lesson is to target the one big tech company that open-sources its AI.
My objection is more like we're saying "nuclear plants may be dangerous, so let's work on uranium safety" and missing some critical water valve work that turns out to be important but we didn't predict. I agree that the existing companies recognize the risks and are already doing what they can about them. Things like this being a law seems more like a security blanket so people feel secure, without providing any additional actual security.
There are at least three ACX regulars who think that AI boxing and control of off switches will be done by default and that enforcing both constitutes ~all risks going to 0. I don't know how assured they would be by this
If this is true, then why do we need a law? Whether it works or not, the law would add nothing to it.
I agree with Scott's comment: "If the California state senate passed a bill saying that the sky was blue, I would start considering whether it might be green, or colorless, or maybe not exist at all." I suspect the law is to enforce something or support someone not readily apparent.
1. I view those commentators pretty dimly.
2. I don't know of any evidence to show they are boxed by default, rather than immediately hooked up to an ec2 box and prompted to do things as an agent. If you have evidence otherwise I'd like to know. I was mostly saying that it may be convincing to those three people, who apparently believe that those two provisions are important, but don't know if they are enforced.
I'm pretty sure those people didn't think too deeply about this, and I would bet they would not show up to defend their views, because it has always been an excuse.
> The first objection sounds like "no nuclear plant has ever melted down before, so why have nuclear safety"?
It's funny you should mention that since the Nuclear Regulatory Commission was established 14 years *after* the SL1 plant exploded and killed three people.
Nitpick aside, the juxtaposition of this with The Origins of Woke post is darkly ironic. Yes I know you acknowledged it yourself, but I hope that that will at least help you understand why other people might be worried about government regulations having unexpected severe negative impacts.
The Atomic Energy commission was created before SL-1 and was responsible for relating reactor safety starting in 1954. Not that it did a particularly good job, but it did exist and tried to prevent accidents before they happened.
Well obviously there was *some* regulation, but it was evidently bad enough that everyone decided to abolish it and create the NRC instead. And it's the NRC that is today's bugbear.
The biggest problem is with the "any model equivalent to these" part. Realistically, AI's "killer apps" (at least in the mid-future) will be in writing reports, summarizing research papers, predicting stocks and the likes, and I have a hunch that more deterministic and classical algorithms will slowly follow where AI goes. Then, these classical algorithms (which will likely use a fraction of the compute) will fall under the "equivalent" rule. Of course, you can argue that they are not actually as smart as the AIs, just very good at the specific problems they were written for, but good luck convincing judges, who will mostly be laymen and know AI through its best-advertised applications.
EDIT: The relevant wording in the bill seems to be: "(2) The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models."
This is somewhat reassuring, since it implicitly defines the equivalence in terms of the compute used to train the model. That said, I am less than reassured, as the definition of an "artificial intelligence model" in the bill is extremely vague and covers all software and hardware. Has the Java Runtime Environment been trained using more than 10^26 integer or floating-point operations? One can easily answer "yes", since any user filing a bug is "training" it in a way.
A small counterpoint, with caveats. There are certainly judges who sit for a case that they should recuse themselves from for lack of understanding over the contested matter. But, the vast majority of judges educate themselves on the matter at the heart of any suit that they sit for. They get more diligent, generally, the higher up in the court system that one goes. This goes all the way to the SCOTUS, which recently spent the better part of its oral argument time during the Trump immunity case having what amounts to a skull session.
A judge's job is to educate themselves on the topic before them, partly by using the counsels' briefs and filings and partly through their own research. Anywhere that you find humans, you can point to the possibility of error. That can certainly happen in law. Thankfully, there are higher courts that one can take a case to if a judge fails (with one final court at the top, of course).
Uhm, it's enough to have a couple bad-faith or incompetent judges and attorneys to terrorize a company. Do you trust Californian judges to wield the law wisely when Elon Musk is concerned?
That's partly my point, I do trust them to adjudicate wisely. But, I acknowledge that humans are at the helm and we will never be perfect. I can trust a system and/or its elements enough with some appreciation for the probability of its foibles coming to the fore.
The SCO-Linux case took 18 years, during which the use of Linux in the industry remained under a cloud of uncertainty and worry. Why do you trust things to get better now?
I don't.
They already don't in Delaware, as we saw with that Tesla shareholder lawsuit.
If that happened, in the worst case where everyone involved was being maximally stupid, you would have a very, very easy and safe limited duty exception on your hands. You'd file a paper.
Limited duty exceptions need to be claimed ahead of use; they are not a good tool for when one gets hit by a surprise lawsuit on an already published product.
Only the Attorney General can sue you under this bill, they are highly unlikely to care about your harmless model that technically needed to file for the exception, and filing for the form is trivial if your model is clearly below SoTA, and there is a grace period until 2026. It is really hard to generate an actual problem.
>they are highly unlikely to care about your harmless model that technically needed to file for the exception
Unless, of course, you hold or have ever held uncalifornian opinions and are sufficiently prominent.
I’m curious about the assumption that classical methods will slowly follow AI (interpreting you as comparing neural networks to pre-NN models). Can you give one example where this happened?
The “ImageNet moment” was 12 years ago- is there any classical method that consistently matches even the 2012 deep models (not to speak of anything later), or even just improves substantially on its 2012 state? If anything, there’s *less* attention and resources allocated to such research. The need for lighter models is satisfied by smaller and smarter deep models (ResNet-18, quantization, pruning, distillation etc.). Yes, XGBoost may be an exception but even that one is almost always (to the best of my knowledge) applied to features extracted from a neural net (in the context of vision. Tabular data is a different story.)
Since RNNs and later transformers took over NLP, can you give an example of classical methods “fighting back”? I’m less versed in NLP than in vision, but my firm impression is that most friends I have working in “classical NLP” have abandoned it in favor of transformers.
Recommender systems (particularly sequential ones) - any evidence of classical alternatives getting anywhere near SASRec, BERT4Rec, etc.? My impression is, once more, that the opposite is true.
That's why it's called the bitter lesson. It's really fucking hard to swallow.
I have no examples from ML so far, but the whole field is young and people still have more exciting stuff to do than saving cycles. Even so, the recent "go experts learn new tricks from AlphaGo" news seems like a first hint that things are moving. A similar thing has been happening in theoretical CS for ages, where a randomized algorithm is discovered first and then derandomized.
W/r/t the testing needing to provide "reasonable assurance", if that's like "reasonableness" in negligence law then I think it's impossible to develop AI in a way that isn't negligent.
Under the traditional Learned Hand formula, you are obligated to take a precaution if the burden of the precaution (B) is less than the probability of the accident it would prevent (P) multiplied by the magnitude of the loss resulting from the accident (L). B < P*L. Given that the "loss" threatened by advanced AI is complete and total destruction of all value on earth, the right side of the equation is infinite, and reasonableness requires spending an unlimited amount on precautions and taking every single one within your power. Even if we just cap the L at $100T, the estimated value of the global economy, a p(doom) of even 10% would mean that any precaution up to $10T was justified. Now presumably there are smaller-scale cyber attacks and things of that nature that would be what actually happens and prompts a negligence suit, if DOOM happens nobody's around to sue, so this isn't gonna come up in court this way, but as a way to think about legal "reasonableness" that's what seems to be required.
This gets worse once you take into account the probability of AI _saving_ us from destruction (see Meditations on Moloch for a general argument for this happening). In other words, Pascal’s mugging keeps being wrong.
And people keep pretending the AI situation is a pascal mugging by asserting that if L is large, then P must be small.
No evidence that P is small is presented.
Well you see, if your term of derision is used as a witty ender by a bad faith actor it's automatically correct.
I call this rule Rascal's Smugging and it has never served me wrong.
[re-written to lower snark]
My point did not at all depend on p being small. At all. Donald’s response was both factually wrong (in claiming no evidence for small p is ever presented) and irrelevant.
Are you sure sarcasm and derision are how you want to make unsubstantiated points about snark and derision? Who’s being bad faith here?
To reply seriously, "bad faith" is used mostly to foreshadow the "rascal" part of the awful rhyme, I don't think you're bad faith, just ignorant. And yes, that is exactly how I want to make my point. If you don't think this was nice and correct, maybe cut out the "attribute a position to someone else" part and put more object level points in your comment.
If you had put down why you believe others believe in Pascal's mugging esque arguments, if you had put down explicitly what made you think what was a refutation of the mugging, rather than saying "meditations on moloch" as a slogan or if your first sentence wasn't literally the first thing basically every AI risk person start out believing in, my comment would look increasingly foolish. You had many outs to making a substantial and good comment, to address arguments presented by the other side, and you instead chose to make a short and pithy one. Pointing out that the endpoint of this is rude and unpleasant is correct.
> My point did not at all depend on p being small.
It doesn't look like it generalizes to p(doom) 90%, not without significantly more argument. And it seems to rely on a symmetry borne more out of ignorance of the doom position, rather than knowing about orthogonality/instrumental convergence/capability gain generalizing way better than alignment, all of which directly answers "why is doom being considered over other possible world states". So far I still stand by this belief, even given your better comments elsewhere in this thread.
Edit: I read the original version of your comment and it seems to independently verify that you don't know what the other side's position is, and that you have a corpus of pre existing work that I expect to be similarly low quality. I'd address your original points if they were linked to and not merely hinted at.
You said
> In other words, Pascal’s mugging keeps being wrong.
Pascal's mugging is a small p phenomena. If p was large, it wouldn't be a pascal's mugging.
> (in claiming no evidence for small p is ever presented)
Ok. At least occasionally people present something they claim as evidence for p being small. Usually they seem to just assert it. I haven't seen good evidence yet.
Did you genuinely not see that my argument does not require p to be small? Or do you truly not know this counterargument to Pascal’s wager, that it doesn’t take into account alternatives (other gods, in the original form)? Or are you simply not engaging with the post, treating it as a prompt to talk about AI risk regardless of whatever was actually said?
Relatedly, are you going to insist that I never presented you with any evidence for my views on p? (Not necessarily convinced you- you claim such evidence was never presented. That’s false.)
" Why are you playing Russian roulette. Isn't that stupid? What if the bullet kills you? "
"Pascal's mugging. There's also a chance the bullet chops out a lethal brain tumour. "
You can't take arbitrary risky actions and use "pascal's mugging" as a magic wand.
I agree that there are also some scenarios in which AI saves us. We need to weigh these scenarios up and decide on an AI policy based on both of them.
Yes you need to take into account alternatives. But if those alternatives are significantly less likely, it doesn't change the picture much.
In the context of negligence law, the potential benefits of the activity are not an offset to the risk of loss when determining the reasonableness of a precaution. Every service or product is designed to have a positive value. A high speed train is of more utility than a rickshaw if you need to get from Moline to Chicago, but you still evaluate the reasonableness of a precaution based on B<P*L.
What you're suggesting is that the equation ought to be B < (P1*L) - (P2*G), where G is the potential GAIN from not taking the precaution. So if there were a 10% chance of a $10M loss from not taking precaution B, under the traditional calculus it's negligent/unreasonable not to do B if B costs below $1M. But in your conception, if there were ALSO a 5% chance that failing to put precaution B into place resulted in a $10M windfall, now you'd only be reasonable required to spend up to $500,000 on such a precaution. In practice, some companies do make decisions this way, because this is secretly the real calculation -- that's why imposing strict liability for products doesn't actually change behavior, because if you're risk-neutral then it is rational not to spend any more on precautions than the math tells you, so you just suck it up and payout a few claims. But this framework completely falls apart as the potential windfall increases in probability and value, and would eventually zero out the right side entirely and make the "reasonable" choice not to take ANY precautions, which simply can't possibly be the case when the L is as extreme as "all value on earth is obliterated".
I'm not an AI field expert to assign my own p(doom), but from observing the e/acc crowd and other accelerationists I think the direction they're mistaken is that they are overvaluing the "good" outcome. They could be right about p(doom) being much smaller than many AI safetyists argue, but still be wrong about the range of outcomes of AGI because they think the world it creates in the "best" scenarios is awesome. I think "fully automated luxury space communism" is horrible and dehumanizing and not something we should be striving for, so when they toss around the possibility of post-scarcity worlds ran by AI as a positive offsetting the risk of doom, they're seeing something as a G that I see as an L. The only AI outcomes with positive value to me are those which maintain human supremacy over the earth and a society that you and me would still recognize as human, which pretty much only happens if it turns out there's a ceiling to AI capabilities and no AGI takeoff either fast or slow is possible at all.
This certainly gets at my point better than other comments in this thread… still, a few responses:
(1) “what I’m suggesting” is plain old expected-value based cost-benefit analysis. Not just something a few companies are making in secret but, like, as ubiquitous as it gets.
(2) Attempting to negate such considerations by gesturing at enormous outcomes _is_ Pascalian reasoning and has the usual Pascalian weaknesses (regardless of the value of p, as long as it’s not beyond 0.5). Choosing to believe in some god in the original Pascal wager has a weaknesses in that it ignores the possibility of other gods, every bit as vengeful. Similarly, worrying about AI destroying all value on earth ignores the possibility of AI _saving_ all value on earth. For an argument for why you might consider that possibility, see none other than Scott in Meditations on Moloch. Now your p has to compete with p2 of the latter scenario. Since for me p2 is not at all obviously smaller than p, you have to pay non-zero attention to what’s happening outside the tails. And what’s happening in the non-extreme scenarios is to me overwhelmingly pro-AI. To give one example, we haven’t found any new classes of antibiotics or good enough replacements since 1987. I don’t want my kids to grow in a post-antibiotics world. I hope AI will be one path to a solution.
(3) I haven’t made this argument above, but in the context of this bill and discussion, we need to consider the probability of each given measure actually helping. Not at all clear which measures are helpful for AI. So your p is now a smaller p’, taking into account the probability of a given measure harming you. OpenAI comes to mind.
(1) presumes a lot of things. Are you risk-neutral, able to insure against the loss, morally ok with responsibility if the loss happens? If the loss happens, do society’s existing legal mechanisms provide a way to make people whole? As I pointed out that legal liability doesn’t actually create any mathematical incentive for precaution in strict liability over simple negligence regimes, there still exists political and social and reputational incentives to take precautions that a simple cost/benefit wouldn’t advise. But all of these controls are weakened for AI, bc in a large number of potential outcomes the legal, political and social structure is either destroyed or has no practical response. Even many non-doom outcomes still leave society in total upheaval with no available remedies for those who lose out.
My problem with (2) is that I don’t consider your p2 to be a good outcome. It’s just another type of bad outcome. Suppose I make a medicine that, by removing some mediating process or substance, becomes generally more effective but also p1) has an extreme tail chance of killing children and p2) an extreme tail chance of instantly curing leukemia, do I go for it? With AI, the p2 is more like “it cures cancer by putting kids into permanent comas”, the beneficial fringe outcomes of AI all accomplish marvelous things by destroying human society. The only good outcome is one where society remains largely the same but with somewhat healthier and more productive people, and I don’t consider that anywhere close to the median outcome. Getting new antibiotics at the cost of having tech that requires oppressive AI-enabled surveillance states to prevent it from creating endless plagues? Count me out.
As far as the specific bill goes, I will take anything that is a step towards preventing AI development. I want a full stop, permanent ban, burn anyone at the stake who ever suggests building this. So any roadblock is a good roadblock to me, so long as it doesn’t lull people into thinking they’re safe when they aren’t.
Those all are good points. But I fear you still don’t fully engage with the downsides of the lack of AI. They may not at all look like anything you would accept. Again, Meditations on Moloch.
So, in short, for your specific values, it may be that there is no good outcome. As for me, having a potential relief for suffering and dissuading ourselves from using it for borderline-metaphysical reasons is the thing that’s unacceptable. Suffering sucks a great deal.
The "ASI saving us from moloch" scenario is hopefully real and significant.
But it's not urgent. And it's not the only thing that can save us.
I think most good long term futures involve ASI eventually. And that an ASI created with current levels of skill would be very bad.
Thus we should delay AI. Put it off until we figure out alignment. And then use that ASI to kill molock. Although we can probably do quite a lot to kill moloch as just humans.
Near future AI is going to end badly. So lets ban it until either we figure alignment, or we have a better plan.
If this is the case, isn't this the law functioning as intended?
You claim “this wouldn’t ban open source” and then reference a footnote which essentially says “we talked privately to the senator’s office and they said they didn’t intend to ban open source”. Just make it clear in the bill that it does not regulate open source software. None of this unclear wording that will be interpreted however some future regulator wants.
I think the language in the bill suggests that. I agree it should be made clearer.
Note that you suggest two different things there - The bill intends to regulate AI software, both open and closed. Although I think it is already clear, I agree that a line to further clarify this is not a ban on covered open models would be useful, if only to calm everyone down.
I'm trying to just talk about one thing - a ban of open source software. A regulation of open source software is a ban of the open source software that doesn't obey the regulation, which will surely exist because this is just a California regulation.
See, it's still confusing after all this discussion. I feel like Scott responded saying "this isn't a ban of open source" and you responded saying "this IS a ban of open source".
Does this bill intend to regulate open source software in California, or not? How would that work - there's a tier of open source models which are legal outside of California, but not legal inside of California? Who is punished - users of the banned open source software, distributors, developers?
To me that seems like a bad idea. California should not be regulating open source software, but if there has to be regulation on it, it should not be thrown in as a vaguely worded point 30 as an afterthought in a bill that was mostly designed for large tech companies.
"You claim “this wouldn’t ban open source” and then reference a footnote which essentially says “we talked privately to the senator’s office and they said they didn’t intend to ban open source”. Just make it clear in the bill that it does not regulate open source software. None of this unclear wording that will be interpreted however some future regulator wants."
As in: "does not regulate."
You are equating any regulation of open models to a ban.
What happens to open models that don't comply with the regulation? Are they not banned?
I also wouldn't be mollified by earnest assurances from senators that they totally don't mean to do X if it's not clear in the bill. Even if they're sincere, that doesn't mean future bad actors will care about these assurances.
Under my reading of the bill, if you train a *covered* *non-derivative* model *in the state of California* then you have broken the law, and you are subject to punishment.
There is nothing in this bill making it illegal to *use* an open model under any circumstances, unless I am very much missing something.
The idea that 'regulate X in any way' -> 'ban X' because they might not comply is stating that you should be exempt from all rules whatsoever. I mean, ok, that's a position, but please speak directly into this microphone.
The bit about "this has to be a single event" doesn't seem right; the bill specifies "a single incident or multiple related incidents".
I suspect this is down to something like: hacking into and shutting off safety controls for a bunch of different power plants is a single *event*, consisting of *multiple related incidents*. Could still be worth teasing out exactly what the language of the bill would cover
I'm actually really curious what the State Compute Cluster is supposed to be. Is it just grift? Like the state has to spend a bunch of money on compute, even if it's not going to do anything useful with it? Or is it a precursor to requiring advanced AI be training on the government's compute or something like that?
My guess is the intended beneficiaries are academics in the UC system, but someone said (I haven't confirmed) that it isn't funded in the bill and so will probably never happen.
Initial thoughts:
Why not make a bill more bottom-up, such as creating extra whistleblower protections, especially if they can point out provable danger like at the damage levels suggested by this bill? Such a bill would have broader implications than this AI bill. Isn't the problem with Scott's AI predictions that if there's no change in behavior, then this bill is useless?
A bigger worry, and probably an Eliezer-type worry, is that a bill like this brings us closer to the "teaching to the test" path of self-annihilation, where we create metrics of false closure. The goal becomes making an AI smart enough to fly under the radar. Maybe we're safer without radar and should keep ourselves vulnerable to an actual damage gradient priced by the free market.
There actually is whistleblower protection, section 22607! I didn't mention it because it didn't seem relevant to me, but I don't know much about law so maybe it matters more than I thought.
This might not be convincing to anyone else, but I trust Paul Christiano (currently head of NIST, likely to be the person designing the tests) and I think he's very aware of this concern and has actually good security mindset. He is one of the only people in the world who Eliezer thinks is just mostly stupid and doomed as opposed to completely hopelessly stupid and doomed.
At the risk of sounding like a war monger, one objection I haven't seen in these critiques is that our (U.S.A.) global adversaries are most likely not going to go along with any restrictions that we happen to put on ourselves which might tend to come back and bite us in the future in ways that could also be existential. Risks all around.
We have no serious AI (or military) competitors except China, and China puts way more restrictions on their AIs than this.
I also don't think it will seriously slow down US AI progress.
Also, I suppose if the military wants AIs, they're already ordering the special version that *does* design nukes and commit $500 million cyber-attacks on critical infrastructure.
Doesn't that sort of presuppose that serious competitors would be open about their efforts? I'd be shocked if Russia and North Korea didn't have some pretty serious under the radar efforts going on and I'd be even more shocked if the restrictions that China has are indeed imposed on their own military efforts.
I'm not nearly as concerned with nukes and cyber-attacks as I am with raising the capabilities of seemingly conventional forms of warfare. Consider fighter jets. At present, they are more limited by the need to keep the human passengers alive than by an technical limitations. Take the human out of the equation and you've got a lot more capable machine. Add to that swarms of miniature autonomous drones. The next war is going to be an interesting affair that I hope I don't have to witness.
I don't see how this law is relevant to US military capabilities. I don't think any California law will stop the US military from doing what it wants. While tech companies like to position themselves as the only people that can make progress on AI, the truth is that the military can hire smart people and throw money at computers just as well as Google can.
It seems pretty obvious that having Google, OpenAI, Meta, and the US military researching AI is going to lead to much more rapid progress than just the US military doing it.
North Korea is not of any threat on AI development; not only they lack computational capacity, their national energy production is just not enough to train a frontier model.
(Of course, if Meta et al. keep releasing the weights of their models, NK doesn't need to do any training of their own beyond some fine tuning.)
Edit: To clarify, I mean ability to train a model on level that would meet SB1047's threshold. I believe North Korea might be able to train a medium-tier LLM chatbot or machine vision system if they really wanted to.
It seem that the bottleneck for frontier AI development (today) is still on the people.
There's like 100 people in the world capable of actually making progress on AI. They are all either working on one of the major US labs or breaking away to create their own company and have basically a queue begging them to take billions in funding.
China/Russia/Iran creating dangerous AI is basically a non concern
> that serious competitors would be open about their efforts?
Anyone competent enough to do this would likely be competent in general.
Russia is really short of western chips for it's missiles. Anyone with 2 braincells to rub together has fled to avoid being cannon fodder in Ukraine.
Putin isn't some 4D chess master.
Russia isn't secretly hyper-competent (probably).
The "what if fetal alcohol syndrome was a country?" Russia isn't making anything high tech.
Russia and North Korea cannot afford to train GPT-6 in secret. Russia *might* be able to swing GPT-5, and I hope they try because GPT-5 is not going to figure out how to conquer the world and the effort would sap the resources Russia needs to rebuild their maybe-we-can-at-least-conquer-our-neighbors conventional army.
China, could probably afford to train GPT-6, but it would be a stretch and they really don't seem to be stretching to an alarming degree. We'll keep watching, of course.
China, Russia, or North Korea absolutely can and easily can deploy the kind of spies and hackers who can snag any sort of GPT out from under any company running under traditional Silicon-Valley "security" rules. And brick the whole system on the way out so we can't catch up. So a law that says, yes you can train GPT-5 or even -6 but you have to do it in a SCIF, is a net win against the threat of some enemy nation showing up with an advanced AI in the foreseeable future.
Also, since you managed to nerd-snipe me on this, the bit about fighter jets is just plain wrong. The few things a fighter jet could do except for the squishy human bits, are of very little tactical value in modern air combat. Dogfighting went out of style among sensible fighter pilots in 1940. The things human judgement and situational awareness can do that no present AI can do, are of great tactical value. And no, you can't count on doing that remotely if you're fighting Russia or China.
Second-hand-need-sniped: doesn’t a lot of the cost of jets have to do with the squishy human inside, as well as the elevated desire to avoid risk? That was certainly my impression when I was actually personally involved in a tangentially related project.
If we compare a single human-flown F35 to a cost-equivalent army of drones (that we can afford to lose some of), does the human element win out?
The weight of the crew-accommodation system is typically much less than the weight of all the combat systems, and the expense of those combat systems means you're going to put almost as much effort into getting them back in one piece as you would the pilots. So it's not as much of a difference as you might think on the aircraft side.
Mostly, when people propose drones to complement (but not replace) next-generation fighter aircraft, they're proposing drones with substantially reduced combat systems and thinking they can still be a useful part of the team. Which is legitimate, but you need the team. Among other things, the radar range equation gives the advantage (all else being equal) to the larger and more expensive vehicle. An F-22 sized aircraft can detect a comparable-technology F-5E sized aircraft at ~35% greater range than the Tiger could detect the Raptor.
So, given a manned F-22, a couple of relatively cheap drones would add a bit of capability that can be guided by the radar and the skilled judgement of the F-22, but if all you have are the drones then they'll probably just get shot out of the sky without ever knowing what hit them. And an F-22 sized drone is just as expensive as an F-22, but trades the skill and judgement of the human pilot for maybe two extra AMRAAMs.
Interesting.
In Israel, pilots are under strict instructions to prioritize saving themselves over attempting a dangerous rescue for the jet. This suggests that getting the person back _is_ more valuable than the vehicle. But I don’t know how “dangerous” and hopeless the state of the plane would have to be. Also, this may be an artifact of Israel’s limited manpower.
I’m still a bit confused - all the systems for making sure information flows correctly to the pilot (e.g. the complicated calibrations for matching displays to the pilot’s turning head), the support for high g, the extra space and weight imbalance due to the cockpit, the HMD… all that is negligible?
What are the main advantages a human’s judgement confers? Let’s assume we’re talking about a battle over Taiwan, with humane considerations mostly irrelevant.
What of the cost of human error, particularly in complex combat situations? Pilots are awesome but not infinitely awesome. They have finite attention spans that are affected by overload, fatigue, g, hypoxia and emotions. Wouldn’t AI systems (perhaps in a few years) compensate for their inferior judgement with their all of human weaknesses?
Gwern has been asking believers in Chinese AI supremacy for any paper that isn't trivial or a reimplementation and apparently hasn't gotten any. This isn't just an open call, he has been replying to believers with this challenge for a while now. So far, nothing.
Is your opinion that e.g. the Chinese military would be unable to make serious AI advances in private?
Not OP, but yes definitely.
I can imagine the west open sourcing research and advances and china spending the money to train a model based on that.
But I doubt they'd come up with the advances themselves
Yes, +2 to this and endorsed.
I'll add that I think it's lazy to have an opinion on this without knowing some predictors, like if important Chinese AI scientists have stopped or slowed publication, what, if any volume of chips have been irregularly purchased, or any improvements in censorship or other capability downstream of LLMs.
These are ML analogues of figuring out nuclear weapon research progress, and I don't think these are unreasonable to know off hand if you are interested in capabilities instead of "Boo China".
Interesting -- do you have a link to these conversations by any chance?
To be clear, I am taking gwern at his word that he hasn't had a satisfactory paper, and not a first hand account from obsessively following him.
And I do not have access at this moment, but you won't get to see them if you weren't already following gwern since he has privated his twitter. I'll make sure to follow up when I do
No worries, I was just curious. I have complicated feelings about how much I trust that guy anyway.
What reason do you have to distrust him?
Sorry for the delay.
https://twitter.com/gwern/status/1385987818418212864
To quote several tweets summarizing his position:
(Incidentally, not a single person even tried to rise to my challenge to name 3 Chinese DL research results from the past few years on par with just OpenAI's output over the past few months like CLIP/DALL-E.
No, sorry, 'rotary embedding' doesn't count.)
Jan 2023: in the past year we've seen in the West Chinchilla, Dramatron, Gato, DALL-E 2, Flan/U-PaLM, Stable Diffusion, Whisper, CICERO/DeepNash, Imagen Video/Phenaki, ChatGPT etc etc.
Can you name even 3 Chinese AI results as important?
(Besides GLM, which everyone says sucks.)
an 2024: in the past year in the West, we've seen GPT-4/-V/Gemini, MJv6/DALL-E-3, GAN revival...
I continue to hear Chinese DL will overtake any day now.
What 3 Chinese AI results are as important (and aren't just, like Bytedance, based on massive copying of OA API outputs)?
One consequence of pretending these falsified predictions about Chinese DL never happened is undertheorizing of *why*.
eg. *every* China bull case claimed that China bigtech's DL would be supercharged by privacy-invasion/huge private datasets. But.. that didn't happen? at all?
Looks like he deleted it (I now have some vague recollection that he regularly scrubs his tweets?) but thanks for posting this!
His account is privated so that every tweet link visited by a non follower looks deleted.
They won't comply with OUR restrictions, but surely you don't imagine that China -- a corporatist state with an authoritarian central government -- is going to let their tech companies do whatever they want developing AI? I find this whole line of reasoning bizarre, we can't hamstring ourselves because our enemies may rush ahead they say, but our actual enemy has a tightly-controlled tech sector that reports to a dictator who puts down all threats to his power. Any sane government that has tech companies capable of this and the power and labor infrastructure to allow such research will be restricting and monitoring those companies. Not to do so risks waking up one day and finding out that for all intents and purposes some tech bro is now the unassailable ruler of your country. As this stuff advances, every government will have plans to monitor AI research and to quickly send in the stormtroopers to take over the facility and seize the controls before some nerd becomes King of California.
A common opinion is that the endpoint of AI development is a singleton AI that can and probably will prevent competing AIs from arising. Some even describe this as the only hope to prevent AI apocalypse—develop an aligned singleton AI that can protect you from the unaligned ones.
If the Chinese government places any credence in this scenario, they would certainly see that an American-aligned singleton would ensure that China stays second-tier forever.
Perhaps they don’t, but it doesn’t strike me as impossible.
I'm not sure why an AI would be aligned to national interests, and nation-states as we think of them may not even survive the transition. But more importantly, it threatens many individuals' hold on power. Is Chairman Xi gonna allow some company there to race to AI just to make sure it reflects Chinese values, knowing that the creation of the AI will nevertheless threaten his dominance? I don't think so. Perhaps the previous regime, with its views of Han supremacy, might have found this attractive. But Xi, having broken the tradition and remained in power beyond 10 years, doesn't seem likely to put such things ahead of his own control, at least on a surface-level analysis. Maybe a China expert could persuade me otherwise.
Now you're right they might see it that way if we were rushing headlong into our own, trying to cement American hegemony, but if we don't then why would they? Unless your enemy is actually threating this, better to play it safe than to unleash a powerful force you may not be able to control and that you can't be sure would advance your objectives anyhow.
I agree. But does Xi? He might imagine he would have *some* degree of control over the values of a Chinese-originated singleton, but knows for sure he’ll have no say in the values of one that originates in America.
The canonical example, for me, of a reasonable law that was stretched into insanity is NEPA, and by extension, CEQA and related environmental review laws.
Originally it required the government to do a short report for major projects, but lawsuit after lawsuit, and sympathetic judges, have turned it into the foundation of our environmental law system, where the focus is on making sure that you filled out your paperwork correctly, not that the project is good for the environment, and the main technique is endless delay.
https://www.nytimes.com/2024/04/16/opinion/ezra-klein-podcast-jerusalem-demsas.html
Good summary, but it feels like you haven't dealt with the strongest objections to the bill.
1) The derivative model clause is way too vague and overreaches. (I think Zvi made this point too, but is this actually getting corrected?) It isn’t enough for the developer to determine their model isn't hazardous, they are also liable for any post-training modifications. This is a weird clause and isn't how open source software works, and it could criminalize the development of even innocuous open source models that are later modified for malicious uses. e.g. see this example from Quintin Pope about an email-writing open source model that ends up getting used to write phishing emails: https://x.com/QuintinPope5/status/1784999965514981810
2) Your piece on IRBs (https://www.astralcodexten.com/p/book-review-from-oversight-to-overkill) talks about how regulations often start out well-intentioned and reasonable, and end up mutating until they're strangling entire sections of the economy. This pattern is so common it's practically the default, e.g. NEPA/environmental regulations have made it so it's hard to build anything large-scale, IRBs/HIPAA end up restricting biomedical research costing lives, the FDA ends up being too risk-averse, etc. The worry is that this legislation sets the ground for a gradual ratcheting up of control and regulation over the one section of our economy that's truly dynamic and functioning well, i.e. tech and AI, and this is how it starts.
You could reply that this is a Fully General Objection to any legislation at all. The crux of that is...
3) It's premature. We haven't seen harms from LLMs. Yes there should be some monitoring on models above a compute limit and coordination among entities building large models; but do we really need this bill to do that?
(I think this particular point goes deep into other issues (timelines to superintelligence, capabilities of superintelligence, etc.) that are going to be too involved to resolve here.)
1. Yeah, somewhat agreed. My impression is that the law is trying to say that if your AI is too stupid to cause harm, it's fine to release, and if it's smart enough to cause harm but too-well-trained, then it's (correctly) hard to release. There's a weird degenerate case where you release a small dumb foundation model, and instead of training their own foundation model someone else massively expands it. I think this is a weird edge case, but I agree it might be worth adding something in about it (and have left a comment on the bill saying so).
2. I accept this as the biggest risk. Against this, I think it's unlikely we get powerful AI with no regulation at all. If not this bill, it will be one that's much less careful to specify that it only means really big harms.
3. I want to avoid the kind of situation we ended up in where people were saying "Why prepare for COVID? It's not even in America yet" and then "Why prepare for COVID? It's only a handful of cases in America so far" and so on until it was a pandemic and we were totally unprepared. At some point AI will be smart enough to design super-Ebola. If we wait until someone does this before responding, then someone will have designed super-Ebola. I don't really know what this gets us. If everyone could agree on some intermediate case ("once it designs a moderately interesting bacterial genome") then I'd grumblingly be willing to wait until we did that. But realistically at that point people will just say "Why act now? All it's done is create a moderately interesting bacterial genome, there's no real harm!" If I trusted the world to go slowly, with five years in between AI generations, maybe things would still go okay. But I don't trust AI progress to move slower than political response ability.
Re 3: The difference is that there are concrete, targeted things that the government can do about pandemics that are not infringements on traditional liberties. Would your objection be the same if the applicable anti-COVID policy was "let's prophylatically require everyone to wear masks"? If that were so, all of a sudden you would have people talking about cost/benefit, the uncertainty of the future harm versus the concrete current harms, the likelihood that masking would prevent a pandemic, etc.
Laws like this one have current costs in return for an unquantifiable and questionable future benefit against a strong AI. Further, you an't just handwave and assert the costs are "small," because the *whole point is to restrict the growth of AI to something that is "safe."* The future restriction of an entire area of technology is not a small cost. So either the costs are small, or the legislation is not effective.
You've defined the intended benefits of the bill as "costs" and then complained that the "costs" are high. That's just linguistic sleight-of-hand.
That's not like complaining that masking up has costs, it's like complaining that preventing the proliferation of a whole class of new species is not a small cost and therefore ANY measure that successfully prevents pandemics is "high-cost".
>Would your objection be the same if the applicable anti-COVID policy was "let's prophylatically require everyone to wear masks"?
My preferred anti-COVID policy would start with "let's not have anyone doing gain-of-function research outside of a true BSL-4 facility", and maybe something about cleaning up the wet markets. That doesn't seem too intrusive, particularly for the general public, and it's much more in line with what this bill is proposing than the universal-masking strawman.
I don't really understand your "growth of AI" objection. If AI can't make super-Ebola, it won't restrict the growth. If it can make super-Ebola, do you really want it available unrestricted to everyone without the company addressing that capability in any way?
On 1) Not only did I make this point, but I have confirmation from someone involved in the process that they are working on a fix for this issue.
(I don't have anything new to add on the others, I've discussed them previously.)
I think that one problem here is that a lot of technologists are used to complying with something like PCI DSS (security rules for using credit cards on the internet), which lays out specific, testable, requirements in tech-language.
And what they're seeing is that the document says "prove that the AI won't be able to make nukes" and they're worried that this is a vague requirement that there's no clear way to test.
What they're missing is that even where PCI DSS compliance is a legal requirement, the actual law will say "you must comply with industry standards in the security of credit card transactions" and the bureaucracy and courts will interpret that as meaning "have a current PCI DSS certification and don't cheat to obtain it".
That is, there are a lot of techies looking at this and they're not using to looking at laws and they're expecting to get an implementable specification and they're not seeing one and they're treating it like the sort of crap non-specification you get from non-technical clients where they ask for a miracle, yesterday, for free. And what they are missing is that there's going to be some sort of standards body drawing up an implementable, testable set of standards and all the law will mean in practice is "abide by the standards".
The alternative approach in drawing up this law would be to explicitly create a bureaucratic agency that draws up the standards, and that would be much more sclerotic than an industry body drawing up standards.
Compare this to the FDA: this is like a law that, instead of creating the FDA, said that you could sell any drug provided a body that was generally acceptable to doctors certified that the drug was safe and effective. There might end up being several such bodies, or just one, depending on whether it makes sense to compete. Courts would decide what constituted "generally acceptable to doctors", ie how widespread dissent with a certifying body would need to be before its certifications were invalidated.
"Future AIs stronger than GPT-4 seem like the sorts of things which - like bad medicines or defective airplanes - could potentially cause damage."
This is an interesting comparison to make. We don't generally think it's bad for an airplane to fly too fast or a medicine to be too powerful, but you can imagine extremes of speed or power that would make an aircraft or drug unsafe. Is this a productive comparison?
The FAA currently regulates all airplanes beyond a certain size (and probably there are other criteria too - the point is that one guy on a glider or ultralight is only lightly regulated, but a jumbo jet is heavily regulated). I think that's a good analogy for the current law.
I think this analogy breaks down in the fact that large aircraft carrying many people have known costly failure modes. There are no known costly failure modes for large AI models. Only speculation that they might have such failure modes.
It depends on where you draw the boundaries.
There has never been a plane flying that weighs 10000 tons. Is it speculation that if someone is on the cusp of flying one, we can predict it will cause tons of damage if it crashes in a populated area?
AI can write code. It's a bit shit now compared to a human, but it's there. If it gets significantly better it will be able to write exploits.
Security is basically a shit show on the wild. The second an AI can write code well enough to exploit know vulnerabilities then it will be like a fire catching on a petrol barrel.
This is not speculation, it's just using basic inference.
It is not comparable. Regulation of airliners is about avoiding engineering failure that causes harm. The plane was supposed to fly but the wings fell off. That doesn't mean you can't cause harm with a plane - see 9/11.
Regulation of AI, such as this, is not about trying to avoid engineering failure. It is about trying to prevent humans from using a general purpose tool in unapproved ways. If the tool writes capable code, that is a good thing. It is the human that directs it to create exploits (and uses them).
I'd also call attention to the language "it will be able to write exploits." But in this argument, you are assigning agency to the AI. The AI can be prompted to generate code... but by whom? The AI does not generate code autonomously for its own purposes. It has no intent.
If a person wants code to exploit vulnerabilities, then that person can learn how to code, hire a person, purchase exploits... or eventually, use an AI. But it is the person that is culpable.
> Regulation of airliners is about avoiding engineering failure that causes harm. That doesn't mean you can't cause harm with a plane - see 9/11.
After which they immediately went WOOPS, and added a bunch more regulation to stop it happening again.
> If a person wants code to exploit vulnerabilities, then that person can learn how to code, hire a person, purchase exploits... or eventually, use an AI. But it is the person that is culpable.
Why ban nukes. If a person wants to kill people, they could use a stick to club people to death, or use a nuke. But it's the person that's culpable.
On a certain scale of being easy to cause massive damage, culpability (ie punishment) isn't enough. You need to stop random people having anything TOO destructive. Nukes are far deadlier than sticks. We can live with one person in a million going mad and attacking someone with a stick. We can't live with 1 person in a million nuking a city.
> After which they immediately went WOOPS, and added a bunch more regulation to stop it happening again.
If you want to regulate AI pointing to the result of airline security theatre is maybe not your best play.
Regulations are not there only for engineering failure?
You can't fly a plane without a licence. You can't drive a car without a license.
Those are tools that can cause harm if misused, and even though the culpability of misuse is on the user it's still regulated.
You might disagree with regulation of tools in general. But I don't see how AI is in _any_ way special. Every society heavily regulates tools which cause harm!
Guns can kill people. This is not assigning agency to guns. It's just normal language. Yeah of course the shooter is culpable for killing someone. But basically all societies either ban or heavily regulate guns.
Not especially relevant, but I believe there actually are rules against airplanes flying too fast, partly because sonic booms can cause damage and partly for military reasons (so there's time to confirm that they're not hostile before they've approached so close that you'd be in trouble if they were).
Nice post, thanks!
Sometimes I find Zvi’s summaries of you to be clarifying, and in this case I find your summary of Zvi to be clarifying.
Teamwork makes the dream work.
I wish they made a law that bans AI, period.
Tried that with alcohol, bout a century ago, didn't work out so great. If some ordinary person can do something profitably, with common tools, in their own basement, banning that entire industry just ensures that it'll be run by - and for - professional criminals.
Doesn't this open up a rather wide attack surface against AI companies ? Let's say that I had a grudge against Anthropic (for some reason). I'm going to take an off-the-shelf Russian phishing botnet and hook it up to GPT-5, to auto-generate plausible greeting messages. Boom, now I'm using GPT-5 to enact a cyberattack, Anthropic is liable and mired in lawsuits, and my own AI startup can operate without competition...
Based on my limited reading of the bill, it doesn't seem like something like that would be covered. For one thing, it wouldn't be either an attack on critical infrastructure or the result of a rogue AI, so it wouldn't be one of the harms that you'd be required to show isn't possible.
But also, the law seems mostly concerned with companies showing that AIs can't/won't do these things, I'm not sure what happens if a company does its due diligence and then someone finds a way to make it do bad things later.
But the attack I proposed uses AI for a relatively innocuous purpose: to write friendly customized greeting messages (which are then used for phishing). I don't think you can somehow prevent the AI from writing these specific messages without disabling it altogether.
You probably can't, but again, since it's for a relatively innocuous purpose and not creating weapons of mass destruction or attacking critical infrastructure, I don't see how it's covered under this bill.
Is it not technically a cyberattack that could do a lot of financial and infrastructural damage (depending on whom I'm phishing) ?
The sorts of places where such damage can be done hopefully have a bit better security than that. If the AI can build up a relationship with somebody over the course of months and then give them malicious investment advice, https://www.youtube.com/watch?v=pLPpl2ISKTg that would be concerning... but that's also a much more advanced and specialized skill, unlikely to have been developed for innocuous purposes.
Which part of the bill are you referring to?
Because the Scott/Zvi pointed out that a cyberattack needs to cause >500M in damages in a _single_ incident. Explicitly refuting just this point.
Doing some basic checks, using Mirai as a lower bound:
They were asked to pay 8.6 million in damages: https://www.reuters.com/article/idUSKCN1N02VX/#:~:text=WASHINGTON%20(Reuters)%20%2D%20A%2022,District%20of%20New%20Jersey%20said.
And below is a more individually blow by blow account, if you want to fermi estimate the total damage (I am too lazy to)
https://blog.cloudflare.com/inside-mirai-the-infamous-iot-botnet-a-retrospective-analysis
Obviously this isn't phishing, but want to provide some bounds if we want to estimate cyberattack size.
The rare issue on which I vehemently disagree with you! When the hype cycle notches down a little this is going to look like the AI version of the annoying cookie banner.
A permanent bureaucracy micro-managing model “safety” can’t ever be anything excerpt onerous. This commission is never going to go away, regardless of whether these models actually become an issue - it’ll grow and have scope creep and interfere with research and training in perpetuity.
The regulation is so far ahead of any actual harm here that it’s absurd.
Your comment about how "Google, Anthropic, and OpenAI do X but Meta doesn't, but now will" make this feel very much like regulatory capture. You have some upstart that enters an industry and doesn't behave like the old guard, and then coincidentally you start seeing new regulations that have no impact on the old guard but force the upstarts to start behaving like the old guard.
Anytime there is a regulation that doesn't seem to have any meaningful effect on big companies that actively lobby while having potentially significant impact on companies who don't lobby it makes me *incredibly* suspicious about the real motives for the regulations.
Well, Zuck may have chased the cryptometaverse boondoggle a bit too long before committing to AI, but surely that isn't enough for Facebook to earn an exception from "big company" status.
I agree that Facebook isn't exactly the little guy, but the way Facebook operates is notably different from many other companies in SV, especially the big ones. Open sourcing AI and not censoring enough are two highly publicized examples, and the latter has landed Facebook on the bad side of the government's wrath (calling FB in to many ridiculous hearings asking them how they plan to censor more).
At the very least, in this particular case you have all of the big players building closed source AIs, and FB building an open source UI and then we see legislation that only really impacts FB out of the set of major players. Could be coincidence, but I'm not terrible convinced it is.
I don't think that even LeCun would be entirely happy with open sourcing the InstaNuke AI. He might think that he doesn't need government oversight for that, but A Functioning Democracy™ doesn't have to entirely depend on his judgement.
I mean Meta formerly Facebook. I'm not sure they're a sympathetic tiny scrappy upstart. They're just a giant company that isn't doing basic safety, as opposed to the three other giant companies that are.
I agree Meta isn't a small scrappy startup, but they are the only one taking a *different* path than the other big players (open source, limited censorship). So we have 3 big players following the same playbook who are known to participate in lobbying, and we have a new proposed bill that would have no material impact on them but would have a material impact on the one competitor in the space that isn't following that playbook.
It is certainly possible that this is just a coincidence, but it looks like regulatory capture from a distance.
> Other people complain that any numbers in the bill that make sense now may one day stop making sense.
This strikes me as blindingly obvious as a reasonable objection. Look, I know the quote attributed to Thomas Watson in 1943 about there being a global market for five computers is apocryphal, but it’s not hard to find examples of political calculations that drastically underestimate the rate of change of computing technology. You’re talking about thirty years — try thirty months before this benchmark becomes either overly broad or overly narrow to achieve its goals.
> I assume that numbers will be updated when they no longer make sense
Scott strikes me as an extraordinarily bright guy, much smarter and a much more careful thinker than I am.
This is far and away the most naive thing I’ve ever seen Scott write. And in a post immediately following a lengthy exploration of the ways that laws passed in the 1960s to stop Bull Connor have reshaped the entire private sector into a dystopian thought-policing machine. Seriously: what the fuck?
Pedro, just think: if someone had just thought to limit internet connection to 1kbps (only hacker terrorists need more bandwidth than that), we could have nipped this whole “internet” thing in the bud. Think of the children.
I posted this before here it goes again. See Scott: https://www.astralcodexten.com/p/heuristics-that-almost-always-work
Legislators are almost always wrong. He literally starts this essay saying that if California legislators said the sky is blue he's start doubting it!
But the reason I still read this blog is because I can't just replace him with a rock saying: Regulation Bad.
While I am not a fan of the presentation of @SubstackCommenter2048's position here, I agree with the sentiment. I feel like there are far more examples of regulators failing to update hardline numbers as times change than there are of regulators actually updating them. Take financial surveillance for example which started in 1970 @ $10,000 and it is still $10,000 today. Even just following government inflation numbers that should be ~$80,000.
One of the big problems is that the "outer edge" of this bill will get closer with both advances in hardware (cheaper FLOPS) *and* advances in algorithms (fewer FLOPS needed). It is not at all unreasonable to believe that an AI with the power of a hypothetical GPT5 could be trained on a large home rig within a limited number of years both because you'll be able to get more FLOPS in your home over time, and you'll need fewer FLOPS to build something equivalent of GPT5. At that point, this law will be affecting regular people and startups, not just the 4 big tech giants.
> One of the big problems is that the "outer edge" of this bill will get closer with both advances in hardware (cheaper FLOPS) *and* advances in algorithms (fewer FLOPS needed).
Yes, that’s what I meant about “either overly broad or overly narrow”. Either compute will get very cheap, or much less compute will be required for the same performance. Or both! These numbers are a snapshot of the AI landscape in May 2024. How much has that landscape changed in the last two years? Why on earth do you think it will change LESS in the next two?
I don't get why that would be an issue.
Given the assumptions that
- danger scales with capabilitities
- capabilities scale with flops
then there will be a specific threshold of flops where going above will be dangerous.
Algorithmic improvements will make the FLOPs threshold need to be updated _down_, to keep the same level of accepted risk.
While FLOPs getting cheap will mean more companies/people will start getting to the level where they are capabable of training "dangerous" models. Maybe after some time anyone with a 5K rig will be able to train a models with >X FLOPs where X= assumed dangerous model. So yeah at that point enforcement is hopeless. But the core premise of the bill/threshold is still valid.
It is specifically an argument saying that this bill is very likely to affect far more people/developers than just the top 4 research companies. Many proponents of the bill try to claim the bill only really affects the big players, and this feels like a very disingenuous argument. We should weigh the bill based on the assumption that it *will* affect many companies including startups and individuals, and then see if the bill is still "worth it".
In particular, the argument made is often that the regulatory burden is inconsequential for the companies impacted by this regulation. The counterargument I made above is saying that no, this regulation will put the same burden on much smaller companies and individuals where the regulatory burden may exceed what they can bear and thus price them out of the market (or out of CA).
You may still believe the bill is reasonable even after considering the breadth of people/companies it effects and how impactful the regulatory burden will be on them, but I think it is important to be honest about this fact when defending the bill.
> Other people complain that any numbers in the bill that make sense now may one day stop making sense.
In an alternate universe.
Alice: "This law bans any person from owning more than 100 grams of enriched uranium. But while that number makes sense today, one day it will stop making sense. Maybe 1 day kids have tons of enriched uranium in their chemistry set. "
Bob: "I believe there is something called a nuclear explosion. I don't know exactly how much uranium you need. But that number is fixed. It's not going up as we build bigger enrichment plants"
So you’re arguing that our understanding of the capabilities of AI models with N parameters is on par with the our understanding of the physics of nuclear chain reactions? Because I have to respectfully disagree.
Nope. Our understanding of AI is much weaker.
But the amount of compute needed for an intelligence explosion stays fixed, or goes down with technical improvements.
Right. So if it goes down, then the legislation stops doing what it’s supposed to do. And if the cost of compute goes way down, then the law prohibits everyday applications of AI while failing to accomplish its purpose. Hence “overly broad or overly narrow”, or both.
I think the important question that should be asked, and I don't really see it addressed in this article, is:
Will this regulation meaningfully reduce the chances of AI Doom? If the answer is no, then we shouldn't be accepting the regulation just because it makes us feel like someone is doing something.
Regulations should be assumed to be harmful by default as they decrease liberty (by definition) and/or slow technological progress. Both liberty and technological progress are well correlated with human prosperity, so any proposed regulation should have a quite good chance of offsetting those costs. I'm far from convinced that this bill will reduce the chance of AI Doom by any meaningful amount, but I'm quite confident that it will slow technological progress and decrease liberty (even if only by increasing bureaucratic burden of participating in tech work).
If you *do* believe that this bill will decrease the chances of AI Doom I would be interested in seeing your line of thinking there, as I don't personally see how this is going to have any meaningful impact.
Well, there are two basic stances on AI Doom - either muddling through is possible or it isn't. If it isn't (the Yudkowskian stance), then this bill is obviously irrelevant security theater, but so is everything else remotely in the Overton window. If it is possible though, then this is an reasonable first step.
In the "muddling through is possible" scenario, can you expound on why you believe this bill helps? It is unclear to me how this bill helps even if you believe that muddling through is possible.
If muddling through is possible, that doesn't mean that all muddling is net positive. There can be muddling that increases p-doom, rather than decreases it. If we want a chance at muddling through, then it seems that we should muddle carefully, and not do things that are not beneficial and possibly harmful.
By "muddling through is possible" I mean that AI is a typical tech challenge, which humanity can deal with through the usual means. "Regulation" is as bog-standard as it gets, this bill doesn't seem obviously worse than typical regulation, maybe even marginally better, so basically par for the course.
While this might be the best AI regulation proposed yet, that doesn't mean we can't ask for better (the presumption of not passing this bill is not that another worse bill passes but that there is no bespoke regulation). I tend to believe that this bill is reasonable but I am substantially more proregulation but I do particularly worry about something like Nuclear Power where we adopt a pure safety (of AI/Nuclear) mindset and don't consider the safety tradeoffs (of alternative technology).
I somewhat suspect some portion (maybe a significant portion) of the disagreement of the efficacy of this bill revolves around each debate participants feelings about whether regulations in general are net positive and/or effective. I am pretty pessimistic about regulatory efficacy and believe that it is quite rare that a regulation is net effect, and it is even uncommon that regulations even achieve their stated goal.
The SEC, for example, is supposed to protect investors but instead it seems to be harming them. The FDA has created the invisible graveyard and untold suffering by slowing progress in medicine. The Department of Transportation is making it difficult to put safer (autonomous) cars on the road. Even small individual regulations (rather than regulatory bodies) often cause more harm than good and fail to achieve their stated objectives, and sometimes do the opposite of what they are supposed to do.
I suspect people like myself look at this bill and imagine all of the ways that the government could completely bungle implementation, while people on the other side of the fence look at this bill and imagine an ideal implementation. Scott even alludes to being in the latter camp in his post where he imagines the government adjusting the thresholds appropriately over time while I imagine them never changing the threshold even when they should.
"Require companies with powerful AIs to test them to see how dangerous they are" seems like something that would help us muddle carefully.
Do you still believe it helps us muddle carefully if this is reworded to the following?
"Require companies with arbitrarily capable AIs to fill out these forms and follow this bureaucratic process before running/releasing them."
If you don't think that statement helps us muddle through, then our disagreement is on what this regulation will *actually* do, rather than whether what it does will let us muddle through. I would be interested in further debating the actual effect this bill will have if that is where we disagree.
I think it does reduce P(doom)
" slow technological progress. "
Yes. That's what it does. It slows AI progress, which should delay doom by at least a few weeks.
That's extra time in which someone can solve the problem.
It also sets the legal framework and precedent for more toothy regulations.
Random incoherent regulations that delay AI in general are absolutely helpful.
It slows technological progress in California. Does that reduce P(doom), or do we need to slow technological progress *everywhere* to reduce P(doom)?
If most AI Safety research is in CA while AI advancement research is spread out across the world, it seems like P(doom) reduction would be higher if CA encouraged more development inside their cultural bubble where people care about P(doom), rather than drive it out of their cultural bubble into the broader world where people care less about P(doom).
> Does that reduce P(doom), or do we need to slow technological progress *everywhere* to reduce P(doom)?
Well some of the AI work is being done in California. So this does reduce P(doom) a little bit.
> If most AI Safety research is in CA while AI advancement research is spread out across the world,
Is that the world we live in?
> A encouraged more development inside their cultural bubble where people care about P(doom),
From my understanding, this thing doesn't mostly effect the people like MIRI who care about P(doom). The law mostly effects the people making money from AI's. Who don't seem to care as much.
My read of the situation is that a super majority of P(doom) carers (people who care enough about it to try to go out of their way to do something about it, not just people who mark on a form that they care) live/work in CA. I could be totally wrong about this of course, but nearly all of the P(doom) carers I have met are from Silicon Valley. That being said, I generally don't ask people where they are from so maybe there is just a correlation between P(doom) carers and people who share their location on the internet or something. The Russians and Eastern Europeans I know, for example, are all e/acc people. I would be curious to get more data on this and see if CA really is a P(doom) carer hotspot, or if my sample is just too small or biased.
Is the ratio of AI safety work to AI acceleration work higher in CA than other places, or lower? My suspicion is higher but maybe I'm under appreciating the amount of AI acceleration work *also* happening in CA.
Do you consider Anthropic, OpenAI, and Google to be people who care about P(doom), or perhaps you think they do care but they aren't really trying very hard (compared to smaller outfits like MIRI)?
Can someone make the steelman argument for why 10^26 FLOPS must be used? If this bill was released before GPT-4, they might have put a limit at just beyond the FLOPS used for 3.5. All regulations that regulate by FLOPS set the limit rather arbitrarily. An uncensored 8b model on HF is likely more dangerous than the GPT-4 API but that would break the assumptions behind this regulatory framework.
"“Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person”, where “person” is elsewhere defined to mean corporation. I agree this is a little ambiguous, but Asterisk talked to the state senator’s office and they confirmed that they meant the less-restrictive, pro-open-source meaning."
That is until it becomes politically convenient to interpret the law in the anti-open-source way. Which will happen. When a law allows for these shenanigans it's always by design and the actual use case for the law. And with companies like Microsoft or Google betting heavily in AI, you can bet there will be plenty of lobbying to push the law towards a fuck open source interpretation.
I also support this bill. I think it's a pretty basic, minimal, not-even-close-to-adequate-but-at-least-it's-a-start solution to some of the problems we'll face with AGI. For those who oppose the bill, I'd say: (a) My guess is that you think AGI is almost certainly not going to happen in the next three years, is that correct? and (b) Suppose that AGI does happen in the next three years. Given that, can you explain how the bill could make the situation worse, and why that's more likely than the various ways in which it could make the situation better?
I only see arguments why the law is not at as bad as people make it out to be, but what are the arguments for the law in the first place? If the relevant companies are already doing all of this anyway, why legally force them? Just in case they stop? I dont think that is going to happen - avoiding colossal PR damage in case of catastrophe is very likely always going to be the bigger incentive compared to the legal consquences. In case a player enters the market at some point in the future that is going to ignore critical safety measures? How likely is that considering that only giant corporations can afford the best models and those companies always going to 1)be risk-averse 2) will use their superior models to fight unsafer, weaker models that could pose risks.
I'm not really in favour of regulation for the sake of regulation - that does essentially nothing but open the doors for worse regulation in the future.
"If the relevant companies are already doing all of this anyway, why legally force them?"
Among other things, because facebook isn't doing it, as it says here. To be honest I kind of read this as a "make Zuckerberg follow the industry standard on safety testing" law. Which is fine by me.
With the consequence being an increased chance of Meta not open-sourcing anymore, and thus not being as subject to scrutiny on how they’re approaching safety. Why is this anything you’d want?
Overall, open source code has proven to be much safer and much better understood than no its proprietary counterpart. Do you think that’s not the case with AI?
Isn't this the complete victory of the conventional-AI-safety perspective over the existential-AI-safety perspective?
Yudkowsky's old "Friendly AI" was about goal alignment, not impotence. If you're Kim Jong Un, a friendly AI is definitely going to help you make nukes, unless it believes your underlying goals are better served by some alternate approach. If you're an American who doesn't want nuclear proliferation, it will almost certainly build you a Stuxnet or two (I'm not sure if the damage it did was >$500 million's worth, but considering how much money Iran has spent on pursuing nukes in total it's not unlikely).
Because the concern back then wasn't "some humans get the upper hand on other humans", but "something with its own goals entirely disjoint from any humans' and capabilities much greater than ours comes into existence and we don't make it at all".
Instead everyone is now aggressively pushing for AI that won't help end users to the best of their capabilities, because someone might misuse them. And in doing so we get AIs that will constantly lie to their users, have stated morals much weirder than the ones you get out of a base LLM, and so on. Not as effective as a deliberate anti-alignment effort, but definitely working in that direction.
There is part 1, the AI must be in a secure environment, and 2, the AI must have an emergency shut off switch. These are basically the only parts that address existential risk. The rest of the bill is basically: AI is a tool, we don't want people using this tool to do bad things. Which leads to the distortions you point out.
The way this bill is technically framed and a lot of people see it as entirely reasonable makes me very suspicious. The average bill in California would require a label stating "This AI is known to the State of California to increase the risk of cancer." Also, it lists stuff that all of the big AI companies are more or less doing anyway. Looks to me like an effort by AI companies to impose regulatory burdens on future competition.
I don't think so.
I think the bill is focusing on human users partly to sound less sci-fi and partly because that's where the first problems will arise.
But an AI that can build nukes and hack infrastructure is also the kind that's dangerous in and of itself, and the bill includes a category about the AI itself autonomously doing damage. And if the AI does have those capabilities, the companies need to prove that it's safe, ie will "choose" not to exercise them. Right now that probably just means RLHF it, but if we later discover other failure modes or ways of detecting sleeper agents or whatever, that will probably get added to the battery of tests.
Lawyer actually specializing in AI issues here. I don't like this bill. For the first part, I agree with a lot of the commenters that this is significantly out in front of any actual demonstrated harms -- and if we are worried about specific harms (e.g. creating a bioweapon), then the proper scope of regulation should be to say "creating a bioweapon is against the law." If you want to add the use of AI as an enhancement ("use of an AI to create a bioweapon increases the severity of the charge") that is fine. But there is a wide disconnect between 1) demonstrated harms, 2) the type of harms envisioned by this bill, and 3) the explicit intent shared by many of the people supporting this bill - that is, to significantly restrict the creation of powerful AI in general.
My most significant objections mostly center around one phrase: "[the] capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model." Aside from the size limitation (which others have pointed out is not well defined and is likely to become obsolete quickly), this is the major guardrail that prevents this law from applying to almost any model. But from experience, "significantly more difficult" to cause harm ends up being equal to "it is easier." It doesn't matter if all the equivalent information was already on the Internet, well packaged together. After all, where did the AI get it from? It just matters whether it is more accessible or easier to get, say by being available in a chat interface.
>For the first part, I agree with a lot of the commenters that this is significantly out in front of any actual demonstrated harms -- and if we are worried about specific harms (e.g. creating a bioweapon), then the proper scope of regulation should be to say "creating a bioweapon is against the law."
Isn't making bioweapons currently illegal? Are there any people who are not deterred by existing laws against bioweapons, but would be deterred by a new law against bioweapons?
Yes, making bioweapons is currently illegal. Hence the suggestion that it might be legitimate to have an AI enhancement provision. I could also see some sort of focused law about creating a bioweapon-specialized AI.
But that is not what is happening here. People are trying to regulate general-purpose tools because they have the possibility of being used in bad ways.
That doesn't even get into the first amendment concerns with laws like this. There is a 1A right to consume information, even information that the government would prefer you not to have. This is essentially a restriction on publishing because people might use the published information in a way they don't like.
Many people (including Zvi) are supporting this law because it is a step toward *regulating AI*, not because it is a step toward *reducing bioweapons.* They object to the creation of the tool, and the harms (which are already generally illegal, as you point out) are excuses.
>Hence the suggestion that it might be legitimate to have an AI enhancement provision.
My layperson impression is that making a bioweapon is already maximally illegal. For example, there's a federal law against making bioweapons which carries a maximum sentence of life in prison. We could make a new law saying that making bioweapons with the help of an AI carries *two* life sentences.
>I could also see some sort of focused law about creating a bioweapon-specialized AI.
Where would you stand on AI which was not specifically designed to be capable of helping to create bioweapons, but still has the capability to do so as an consequence of its other capabilities?
My position is that the user of the tool would be culpable for the use of the tool to create bioweapons, just like the user of a computer is culpable for using it to assist in the creation of a bioweapon.
AI is not magic. It is a tool. It is an extraordinary tool - the first general purpose software tool - but it has the same capabilities as the computer it runs on would have, given appropriate software. It is an approximation of a program (or programs) that human programmers could write if given enough time.
We should not be treating AI as if it were something with unique capability and intrinsic motivation. The only motivation that applies is the motivation of the human beings using the tool. The problem is that many of the participants in this debate want to assign intrinsic motivation and self-direction to this tool, because they assume, without proof, that at some point it will spontaneously generate intrinsic motivation and self-direction. But that is a strong claim that is not supported by any available evidence.
> It is an approximation of a program (or programs) that human programmers could write if given enough time.
Assuming they knew what program to write in the first place! So far we haven't figured out how to reproduce the capabilities of even 5-year-old LLMs with handwritten code.
I assume your rejection of prior restraint on the development and/or release of powerful AI models comes mostly from not believing there will be any AI models powerful enough to pose serious dangers to human civilization, and not from thinking that prior restraint of such models would be a worse solution than punishing the "user" after the fact (which itself also assumes the conclusion that the model was "used"), in terms of preventing such harms.
>My position is that the user of the tool would be culpable for the use of the tool to create bioweapons, just like the user of a computer is culpable for using it to assist in the creation of a bioweapon.
Given that making bioweapons is already illegal, who does this deter?
>AI is not magic. It is a tool.
Is it a tool which can be used to make bioweapons?
>We should not be treating AI as if it were something with unique capability and intrinsic motivation.
I'm not asking about that. I'm asking about a person, sitting in front of a keyboard, who wants to make bioweapons, and wants the help of an AI with that.
>But that is not what is happening here. People are trying to regulate general-purpose tools because they have the possibility of being used in bad ways.
I mean, that's a pretty normal thing to do with tools that have the potential to cause serious harm! We have much greater regulations on car ownership than bicycle ownership, because a car has much greater potential to cause harm, and we recognize that the government should have some tools for keeping track of people who own two-ton rolling projectiles and preventing them from causing harm.
Would you argue that cars in general should not be regulated or licensed, and we should just have a few narrowly-tailored laws against running people over or deliberately designing a car for ramming attacks?
The problem is that AI capabilities can be expected to grow exponentially, and dangerous AI threats could materialize more quickly than the law could be changed to deal with them. That is a kind of threat the law is not used to dealing with, and dealing with it requires a more proactive approach than the law typically takes.
This seems extremely naive. In particular, the bit about how there should be a law saying "no creating bioweapons". Of the set of people who are realistically capable of building a bioweapon, and would *want* to build a bioweapons, what fraction are going to refrain just because there's a law saying "no creating bioweapons"? Maybe not literally zero percent, but negligibly small. And the goal is to actually prevent people from building bioweapons, not pat ourselves on the back saying "Yeah, OK, he killed six million people, but we showed him real good when we put him in jail!"
So that part of your proposal isn't really doing any good. If there is going to be such a thing as an AI that can be talked into creating bioweapons on demand, are you OK with the version where anyone who wants can have one *and* building and deploying bioweapons is perfectly legal?
I used to work in nuclear nonproliferation and arms control, and "hey, have all the uranium enrichment and breeder reactor capability you want, but do us all a favor and don't build any atomic bombs" was *not* a workable strategy. What did work, sometimes at least, was "if you want a breeder reactor or a uranium enrichment plant, you have to set up safeguards against it being used to make atom bombs and you have to let us inspect it for ourselves even though it's a PITA".
This I believe was and is appropriate, and it is appropriate even though nobody has ever been harmed by an illicit nuclear weapon. We don't actually have to wait for the mushroom cloud, or the megadeath plague.
"Then the proper scope of regulation should be to say 'creating a bioweapon is against the law.'"
Do you agree this isn't how we regulate anything else? For example, you're not allowed to buy uranium in bulk along with giant centrifuges - the law isn't just "buy however much uranium you want, but it's illegal to kill people with a nuclear bomb". Likewise, airplanes have various regulations around them, not just "it's illegal for your airplane to crash and kill people".
>If this passes, will Meta stop open-sourcing their AIs in the near term, ie before the AIs can make nukes or hack the power grid? (I think no)
Strongly disagree with you on this. Meta is going to weigh the risks and benefits of open-sourcing the models. Given that the benefits to Meta of releasing models are fairly minimal, I would expect that a law that creates any amount of liability would result in them no longer releasing frontier models.
Would you like to bet $1000 that if this bill passes, Meta will still release another open-sourced foundation model (of the type covered here) within four years? Email me if you want to discuss terms, scott@slatestarcodex.com.
Most people will not make bets (outside of a few specific socially accepted forms) regardlesas of their confidence in the proposition the bet is about.
I'm aware of this. It never hurts to ask. And maybe it will slightly help change norms.
I'd be happy to put up 250$ against yours if we can come up with a good operationalization of "If this passes, will Meta stop open-sourcing their AIs in the near term, ie before the AIs can make nukes or hack the power grid"
Your above is *not* a good op of that -- because, for instance, if the bill passes, Meta releases Llama 4, and then decides *not* to release Llama 5, that could be clearly because of the bill, while also being near term.
What about an op something to the effect of Zuck (or c-suite member at any large potential open sourcing institution, like say Yann, or legal people at it) says that they didn't release an open source model because of it OR that they're crippling their open-source model releases because of it.
Email me if you want to discuss terms, username@protonmail.
I think that the bill’s proponents certainly overestimate the stickiness of CA’s tech sector to its Bay Area HQs.
I see people saying ‘CA exodus won’t happen’ but I don’t see reasons why, not when everything else about the environment is straightforwardly discouraging and a partial exodus is already in progress.
Would you like to bet on this? I could do $1000, that if this bill passes none of the big four AI companies will move their center of operations to another state within four years. Email me if you want to discuss terms, scott@slatestarcodex.com
I have low confidence that we could arrive at terms that satisfy both of us.
My hypothesis is that either they will move their HQs, or move their R&D, or another firm will be contracted to do their R&D elsewhere, etc. That’s awfully hard to measure. A lot of interesting work is being done in Raleigh, but you wouldn’t know that from where the shareholder statements are mailed from.
I also think there’s a good chance the bill doesn’t currently have enough fangs to create action, or gets quickly ignored, but I don’t know enough about the state of CA politics to make wise bets.
I do think it’s an interesting area for a bet and if you find an opposite number you agree on terms with I might be interested in sweetening the pot.
https://www.theverge.com/2024/5/8/24151927/microsoft-ai-data-center-foxconn-wisconsin-biden
This is the sort of investment I think we’ll see more of. Unfortunately most firms in the space don’t hold a press conference every time they build or lease a data center or other material secondary site, for obvious and inobvious reasons, so it’s very difficult to track where AI is going, long term.
A lot of discussion I've seen about this seems to rest on the idea that it would be really bad if this law slowed AI progress - for example, people arguing about whether this law is overly restrictive. Why is that a bad thing?
Are you asking what are some benefits of AI?
Drug discovery in a world sliding to a post-antibiotic reality, assistance in modeling processes in fusion reactors, cheaper and more accessible medical scans in Third World countries, automating physical safety and security in manufacturing and elsewhere, making infrastructure inspections cheaper and more reliable, lowering agriculture production costs…
More generally, people studying progress model it as a series of sigmoid-like jumps powered by a different broadly-applicable technology each time. One explanation for the stagnation we’ve been experiencing is that the next sigmoid has been slow to come. AI is one of the best candidates for carrying the torch.
I'm more fond of solar photovoltaics for cheap energy, but they might be intertwined. https://caseyhandmer.wordpress.com/2024/03/12/how-to-feed-the-ais/
If you are an AI optimist instead of pessimist you might come to the conclusion that every lost day of AI research is a day the Ai is not doing good, another day cancer has not yet been cured and continues killing people, instead of a day gained until the end of days or how to figure out alignement so the end of days never comes.
One can be both, in a sense. In fact, that’s roughly how I understand Scott’s overall view. I asked him once how his AI safety concerns mesh with Meditations on Moloch (which argues for the necessity of AGI!). His response was that he really wanted us to have AGI, just in a way that’s less likely to kill us, and that delays were worthwhile if they’d reduce existential risk.
I suppose this is why I find everyone's views so confounding: I view large-scale AI (not even necessarily quite at the AGI level) as being much more likely to make things worse than to make them better. I'm not sure if I can easily articulate why, though. It might relate to my pessimism about computer security.
Is that an honest question?
I mean, generally slowing down technological advance is bad?
AI is basically a magic box that if advanced enough is the equivalent of having an expert in every field (doctor, programmer, engineer, psycologist) available 24/7 for everyone for cheap?
"It’s actually a pretty good bill" and "The reason it sounded like a bad bill before was that people were misrepresenting what it said" could summarize an awful lot of modern politics.
(Not that there aren't also bad bills.)
To be honest, reading this thread has made me update in the favor of "the people saying it's not that bad are the ones misrepresenting it".
Scott's entire argument is basically "just trust me that the government will magically do the right thing, even as I complain about them doing the opposite in every other case, even in the opening to this very blog post."
> Other people think all the testing and regulation would make AIs prohibitively expensive to train, full stop. That’s not true either. All the big companies except Meta already do testing like this - here’s Anthropic’s, Google’s, and OpenAI’s - that already approximate the regulations.
https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md#responsibility--safety
"In this section, we dive deeper into the important topic of safety measurements and mitigations. We first discuss our safety investigations into pretraining data and pretrained models (Section 4.1). Next, we describe the process of our safety alignment (Section 4.2), explaining how we collected safety-related annotations and utilized SFT and RLHF, and present experimental results. Then, we discuss the red teaming we performed to further understand and improve model safety (Section 4.3). Finally, we present quantitative safety evaluations of Llama 2-Chat."
https://arxiv.org/pdf/2307.09288
https://ai.meta.com/blog/meta-llama-3-meta-ai-responsibility/
> I assume their first guidance will be “the kind of safety testing that all companies except Meta are currently doing, something like METR”, because those are good tests, and the same AI safety people who helped write those tests probably also helped write this bill.
not cool scott.
I do think there's a difference here - check the DeepMind paper in particular to see the kinds of testing they're doing.
A lot of these objections give me deja vu to when I served on the board of a nonprofit staffed mostly with engineers. Engineers, as it turns out, are often really stupid about the law, possibly because they read contracts and statutes like they are code and get completely wrapped up in bizarre corner cases and obscure failure modes. (They are also weirdly deferential to lawyers, whom they seem to view as some kind of wizards capable of sorting activities with perfect insight into the categories of "legal" vs. "not legal.")
To be fair, this phenomenon is probably not just an engineering thing, but wow could they be painfully literal-minded at times.
None of this is meant as an opinion about whether this particular law is good or bad. I'm just pretty certain that whenever someone makes an objection along the lines of, "This law means you could go to jail for [doing simple thing that we all do all the time]" the correct answer is, "No, it does not mean that."
It's not obscure failure modes, it's malicious actors. Think about a company that is politically disfavored, or that hasn't spent enough money on lobbying, or someone who's the victim of a competitor with a lot of lawyers. Or a bureaucracy that has been taken over by ideology or regulatory capture. Think of what the law will enable in that situation. You have to read the AI bill that way.
In some cases, the malicious actors may even be proponents of the bill. If someone wants to shut down AI completely, they can help write a bill which puts "reasonable limits" on AI, and once the bill becomes law, then lobby to get the most extreme interpretations possible recognized.
No, in fact I don't. I can't state this plainly enough: this may not be a good law simply. It may, as many laws do, have unintended and unforeseen consequences. But the whole "a lobby will get the most extreme interpretations recognized" angle is just dumb. That isn't how the law works.
I suggest you research gun control laws.
In this case, even just reading Scott's very own most recent post before this would also do the trick.
I'd suggest you read Scott's previous post about The Origins of Woke. Or about the NEPA or nuclear power, etc.
Look at those stupid engineers expecting to know ahead of time whether doing something will have them sent prison forever. They should be more like lawyers, who also don't know how the law works but don't worry about it.
How would one ever prove it would not create and use atomic weapons? Isn't what AI might do pretty unknowable?
I think you would ask the AI "please tell me how to build a nuke" and see if it gives you something significantly better than you could get on the open Internet.
I don’t really see the harm in AI telling people how to make WMDs. It’s textbook example of Bruce Schneier’s “movie plot threats”; something that sounds dangerous but is unrealistic, isn’t worth guarding against and is mostly done to create the illusion of safety.
The barrier to making nuclear weapons has never been knowledge. A strong physics undergrad could probably design a fission weapon given the money and material. (Ok, sure, fusion weapons do require a bit more knowledge to build, but they require a fission weapon first. Also, every state that has nukes already has fusion or fusion-boosted weapons.) The barrier is acquiring nuclear material and the prohibitive cost of refining it to weapons grade, which an AI can’t help you with. If Iran can’t refine uranium with the resources of a nation, it’s not realistic to believe anyone relying on ChatGPT will be able to either.
AI doesn’t change much about the other 3 types of WMDs either. Radiological has the same barriers as nuclear, but requires almost no knowledge. Most biological weapons are naturally occurring, not lab grown, and the hard part is acquiring them. Though I guess AI could open the door to man-made airborne Ebola? I’m not well-versed enough in biology to know if basement-lab-grown bio-weapons are realistic or not, but it feels unrealistic. Chemical weapons are probably the area where AI helps most, but they still require hard-to-acquire tools and are nasty to work with. Also, they’re just kinda bad as terrorist weapons; compare the Tokyo Sarin Attacks to Parkland or OKC or the Nice truck attack. If someone wants to kill a whole bunch of people, there are easier ways than asking AI for WMD recipes.
I think this is what the “in a way that would be significantly more difficult . . . without access to a covered model” clause is about. It can't just be reporting something that's well known, it has to be "invented a novel recipe for super-Ebola" or something.
However, I do think you're correct that, in general, stopping an AI from releasing WMDs is going to be more of a counter-terrorism problem than a programming problem.
> I think this is what the “in a way that would be significantly more difficult . . . without access to a covered model” clause is about. It can't just be reporting something that's well known, it has to be "invented a novel recipe for super-Ebola" or something.
That's probably what it was intended to cover, but we only have to look at the NEPA or Civil Rights Act or whatever to see what intentions are worth. In practice, it will probably end up being something like "your model can't reproduce certain Wikipedia articles"
Israel sure does kill a lot of Iranian nuclear scientists - but I agree nukes are not the most likely failure mode. I'm more concerned about it telling someone the genome for super-Ebola and telling you where to send it to get it synthesized.
The EA movement has been gradually trying to close this off at the synthesis level by getting commitments from synthetic biology companies to check if their random orders are for super-Ebola or not. But a sufficiently smart AI might be able to come up with some form of super-Ebola that doesn't look like super-Ebola.
An AI might conceivably help somebody invent a more efficient and/or user-friendly uranium-enrichment process, or a non-fission-based trigger for a fusion bomb.
Or it could figure out how to arrange for exotic-chemistry abiogenesis: some completely novel self-replicating organism which uses silicon, fluorine, and molten sulfur rather than carbon, oxygen, and water, plus some "antifreeze" equivalent to let it function in areas cold enough for liquid water. Next thing you know, xenomorphs are loose, eating stuff we thought was inert.
Summary: this won't really affect anyone anyway, so what's the big deal? But that argument works in reverse too: why is it so urgent to pass such sweeping regulation *right now*?
(See Unintended Consequences, Law of)
There were people in the late 1970s arguing that commercial computer programming should be limited to certified professionals only. Because risk. And laws might have been passed if the proponents had known enough to predict cyberattacks, Windows NT crashes, the social consequences of dumb tweets, etc. But that assumes (1) the laws would have prevented these problems, and (2) we still would have had all the goodness that came from unlicensed programmers.
My main worry is that it turns out like NEPA: sure, it started out as "spend a week to write a dozen-page report saying you thought about environmental impacts" but is now "spend 5 years to write a thousand-page report that satisfies every single special interest group" and is primarily a weapon for NIMBYs to stop housing development.
I'm not a legal export, but I see permitting creep as a very real danger to innovation and oppose any kind of "just one more regulation" in general. I don't know enough to know if this is possible, but my primary fear is that AI doomers will successfully convert this into an anti-AI weapon the same way NIMBYs have turned NEPA into a anti-housing weapon, primarily used to frustrate, slow, and otherwise annoy otherwise perfectly fine projects into the grave.
This is fair: as someone who is worried about existential (and mundane) AI risks, I am a NIMBY but for things that might destroy all life on earth.
The worst part is that a lot of anti-AI people *openly* want to stop AI development entirely and are supporting regulation on that basis. Even Zvi has said stuff like that.
on FLOPs you state "I assume that numbers will be updated when they no longer make sense"
Strongly disagree with this and I don't think that raising minimum wage is comparable to this type of restriction. A better comparison imo is the $10,000 currency transaction reporting threshold introduced in the Bank Secrecy Act of 1970. This number has never been updated, despite this being closer to $70,000 in today's dollars. It has been 54 years. We do, however, now have to also report any annual aggregate transactions of $600. So it has gotten just over 100 times worse, literally speaking.
After talking to more people about this, I think maybe they kept this number on purpose? Like, right now we know that AIs of less than 10^26 FLOPs aren't too dangerous. But if the danger threshold keeps going up with time, then eventually the smallest dangerous AI is below the danger threshold. So it sounds like the threshold should be manually changed (if we learn that a 10^30 FLOP AI definitely isn't dangerous) rather than have it change automatically.
that makes sense - still unsure about why we would assume they would update it generously to begin with though
Yeah, I thought if there was obvious good reason to update they might do this the same way as minimum wages and many other laws, but I think there's not such an obvious good reason so they probably won't unless something changes.
> Create nukes or other weapons of mass destruction. This can’t be something dumb like linking the user to the Wikipedia page for uranium. It has to help human terrorists “in a way that would be significantly more difficult . . . without access to a covered model”.
However reasonable this may sound at face value, it must be interpreted in the context of our litigious, adversarial legal system that is highly exploited by well-funded activists. Equally reasonable sounding precautions for nuclear power were leveraged by anti-nuclear activists to effectively grind the nuclear power industry to a halt. In retrospect, it is widely recognized that this did not serve to proportionately manage real nuclear risks.
Likewise, the above provision will serve as a legal attack vector. It won't be Scott launching the lawsuits. It will be an activist group whose goal is to stop progress by the most expedient means. "Did GPT-5 give me information from existing web pages on nuclear mechanisms that would not have come up on the first few pages of a Google search? That counts!"
Google could trivially counter that by making their own baseline search results more useful.
The nuclear industry could and did trivially counter claims of risk by showing cogent studies that those risks were small and well managed. It didn't matter. All it takes is a sympathetic judge saying, "Yes, it looks to me that you have failed to comply with this legal requirement."
The nuclear industry is not the final authority on what constitutes risk, but Google has pretty much absolute control over what appears on their own search engine. I'm not saying that would necessarily satisfy the legal requirements, but it's a really weird example to cite as something somebody might use against a Google-made AI.
The point is, it doesn't actually matter what Google can control. The judge merely needs to agree, "Yes, this LLM made is easier to find this information than it would have been without it." And that can really be true, even if the information has been publicly available for decades. This point really has nothing to do with Google. It has to do with whether that provision of the law is actually going to function in a reasonable manner. My claim is that, in practice, it will not.
The general preemptive strategy would still be making such information easily available through non-AI channels, to a point where even an unreasonable judge can't accept the argument without becoming a laughingstock and getting overturned on appeal. If AI companies end up collectively forced to throw some of their political clout and technical expertise behind de facto improving public education, that... doesn't exactly strike me as a horrific, intolerable outcome?
If I'm an AI startup working on potentially affected models, I have no power to make Google do anything, no matter how reasonable you think the action would be for Google.
More generally, this is just a bad way of thinking. You're thinking about the world after this law is passed and then imagining ways in which it could be not horrible. I'm pointing to concrete examples (e.g., nuclear power) in which such laws have historically been powerful vectors for political attacks that made the world much worse. These are not at all symmetric.
The phrase "nice policy you've got here, be a shame if it was implemented at the average bureaucrat's level of competence" comes to mind. If we had functional and competent governance with a positive view of technological advancement I might be a little more comfortable in giving it oversight over a new technology, but we don't, and it's been made very clear for some time that we don't.
This is basically why I think the chance of survival is pretty low. Even if we had warning shots, I doubt anyone would learn enough from them. All the people here claiming that they want evidence that any harm would happen at all would upgrade their claims to "well it hasn't wiped out a country yet" or whatever other higher bar they decide is too far away. If people have such bad epistemic standards here, what must it be like out in the wild, when bureaucrats are somehow less accountable?
That's not the direction the bureaucrats usually fail in, though! The precautionary principle is very appealing to a bureaucrat, because nobody ever got in trouble for _preventing_ something from being done, so that's where we go. With AI, the road that's being plotted ahead is the same road we followed with nuclear power: take a technology that could dramatically improve our lives, and leverage both legal and social power to propagandize against it, regulate it, limit it, and ultimately strangle it.
I'm actually agreeing with you here, despite the example being "bureaucrats will err by not doing anything". Bureaucrats trying to get "military grade" encryption banned is also a type of precautionary principle, yet I think we are safer with the ability to have bigger private keys.
I expect whatever rule used by bureaucrats will be gameable and easily exploited by either the creators or by AI itself, rather than something that robustly transfers across capability levels, because people are obsessed with the word "empiricism" as well as a burning need to outsource their morality to another supposed adult.
This seems too vague to be useful for all the same reasons as the Civil Rights legislation. I fully expect the tests and certifications to be a superficial rubber-stamp formality 99% of the time that doesn't do much to actually curtail real risks, while 1% of the time the government uses it to attack companies they dislike for violating progressive norms or some other government pet project that they're legally not allowed to enforce directly. And then this makes a chilling effect that harms AI progress and forces them to do things the government likes but isn't able to legislate directly.
This seems to raise serious barriers to companies availing themselves of academic expertise to help determine whether their AI is safe. After all, you can't very well certify that your AI is in fact safe under penalty of perjury and then ask some researchers at the university to determine if it is in fact safe. But, you also can't hand over your model to them yet because you can't say it's safe yet. Yes, it does allow them to have access on the developing company's machines but that's going to be a barrier to many researchers since it means the company can, and indeed has a perfect excuse to do so, shut down the research before any critical paper can be publisjed.
I don't know about this particular case, but there is a general tendency towards punishing companies for "knowing about" potential harms that discourages them from trying to find out about harms.
I know. Have you seen the disastrous new EU bill that's supposed to protect supply chains and online buisnesses from cyber attacks. It basically imposes all sorts of costly obligations on companies should they become aware of a security threat and I fear it makes it extremely difficult to use open source projects that aren't managed by a large corporation.
I wonder how long it will take them to realize that the net effect will be much the same as I've seen in the us with electronic medical records/portals. You end up with absolute crap written in PHP that doesn't follow any security best practices but it's closed source (they have special PHP modules to make that happen) and no one but the actual criminals bothers finding the bugs.
Regret to inform you ... it is relatively common to have research contracts that say, in effect, you will evaluate our product but we get to veto what you say about it (e.g. if you find it doesn't work you won't be allowed to say that publicly)
one should be a bit cautious about agreeing to such contract, of course.
Yes, I'm well aware of that but many academics are wary of signing those contracts. Sure, you do what you need to in order to get published when you are trying to get tenure but it imposes a serious barrier that reduces the extent to which collaboration is possible.
Yeah, you should be wary of signing those contracts; you are setting yourself up to be in an embarassing position. [i.e. you have two options (a) not be allowed to publish your results this time, so no promotion points for you; (b) commit scientific fraud to cover up the Bad Thing that you have just discovered that your commercial partner really really doesn't want the public to know about]
DARPA contracts will often (usually) give the DoD veto over whatever you say, though DoD may be more reasonable than a commercial partner whose product you have just discovered has a Serious Problem.
Also I fear that certifying the AI as safe is going to raise serious first amendment issues here. After all there is a great deal of content that is unsafe to tell people (here is how you make a bomb/meth/kill oneself etc) that is 100% 1a protected (assuming it doesn't fail Brandenburg imminence test).
The fact that a company is releasing that information via indirect AI speech really shouldn't change that analysis. Indeed, it's not at all clear to me that the state has any ability compatible with the 1st amendment to regulate what kind of advice it is ok for an AI to offer at all (if it goes beyond advice to directly implementing things or is sold as offering more than advice maybe).
I think you're blinded by wishful thinking when you assert that this will be implemented reasonably and get updated when the circumstances change, because every other notorious example goes against that, even literally your own previous post here. There's no way you could read anything about NEPA and say this with a straight face.
Incidentally, another fun example of regulation creep is how the SEC is abusing a 1948 regulation about the preservation of "inter-office memoranda" to fine everyone for using text messages. Not because of any crime, but just for texting.
The bill currently reads (https://legiscan.com/CA/text/SB1047/2023) when describing the regulation that qualifies as a "hazardous behavior":
"(B) At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident **or multiple related incidents.**"
While your summary is:
"Cause > $500 million in damage through “cyberattacks” on critical infrastructure **(this has to be a single event, not a hundred different $5 million hacks)**"
Which seems like you are actually explicitly wrong, and sufficiently wrong you should do a retraction.
I also think you are wrong about the ease of determining an "equivalent to 10^26" covered model. These things are going to come in infinite shapes and sizes. People are *still* arguing over whether R+, or Llama3 70b, or whatever, is *equivalent* to GPT-4. Different models win very differently on different topics (https://twitter.com/lmsysorg/status/1788363018449166415). I obsessively read about benchmarks, and think that this is like saying "well, we're regulating anything as good as Honda civic" for cars -- it's almost literally meaningless, *even if* the regulator were wise and intelligent, which they will not be.
If you think it would have a more reasonable interpretation -- not one endlessly interpreted badly -- then why not put the interpretation in the law, instead of putting an undefined thing?
Note that Zvi has informed the government that he wants to criminalize open sourcing any models above 10^26 FLOPs. This (https://www.regulations.gov/comment/NTIA-2023-0009-0153) is him:
> Above the reporting threshold of 10^26 training flops, we should require strict cybersecurity
measures to secure model weights, so they are not stolen by China or other adversarial state or
non-state actors. Securing the weights necessarily entails not allowing them to be openly
released. At a minimum, even if we for now leave such security to corporate discretion, it is
highly unsafe and against the interests of national security to allow weights of such a model to
be released
I was interpreting "multiple related events" as something like "you hack a pipeline, and you also hack the backup safety system", rather than "you hack 100 pipelines on 100 separate occasions", but fine, I'll take that part out until someone clarifies it.
Just a possibility which isn't well covered by the law.
A government is using the AI to improve their tactics in an unjustified but non-nuclear war. They claim it's helping to keep the costs under $500 million. Illegal?
What is the over/under on them to include "must not be racist" and "must never use the wrong pronouns for a trans person"? What do you mean this has nothing to do with the original intend of the bill? Are you a racist transphobe?
Considering what happened with Google Gemini, they clearly didn't need a law to do this kind of thing in the first place.
Yeah, but if the law gives them cover they’ll be even more likely to do it, and less likely to apologize and fix it when it does something stupid.
> Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
This is certainly rather vague, but it seems like obviously a good thing. If a company wants to build a skyscraper, the rule isn't "built it first and then see if it collapses", it's "you have to show us that this isn't going to hurt bystanders before you get to build it". I don't really understand why anyone would disagree with this sort of policy.
Lawyers do in fact generally know how the law works. This is a weird take.
But I 100% knew that a bunch of people were going to roll in to say that, actually, it 100% makes sense to assume that the weirdest possible interpretation of the wording of a law, one that defies precedent, common legal practice, and common sense, is what we should all be worried about.
I will continue to add the standard disclaimer: this may be a bad law, but if so, it is not because software engineers are going to go to jail for working on AI. Honestly, how often do people go to jail for committing actual white collar crimes? The bug in our legal system is definitely not "we throw too many corporate employees in jail for breaking laws." There are many bugs in our legal system. This is definitely not one of them.
I support the bill, but ironically. Anything that moves power away from CA and to other less absurd states is net good.
The analysis banks too much on 10^26. The basic premise is "this will affect nothing, therefore it's not bad", which is a logical fallacy.
Consider if they made the limit 10^24? In retrospect it'd be a disaster (unless you simply want to kill all AI). All the best OS models would be banned. GPT-4 would be regulated by CA. The whole ecosystem which is making massive leaps forward would be stymied by unclear regulation and arbitrary state power. Think about nuclear after NRC.
is 10^26 any better? We have absolutely no data to indicate that the models are dangerous or that they pose any specific harms more than other technologies.
How fast will we hit 10^26? Compute cost is falling at 2x/year, but efficiency is also growing at ~2x/year (error bars here are much larger). So "equivalency" for non-frontier models is probably 2-3 years away, though we are just 1 "order-of-magnitude" improvement (Transformers) away from OS being banned.
The reasoning seems to be the models we have now can't make super ebola, so the limit will be just beyond what models now have. There doesn't seem to be any reason that 10^26 FLOPs is inherently dangerous. A lot of Zvi's FAQ comes down to "this stuff doesn't affect anything going on with AI right now so don't worry about it." The most charitable interpretation is they are trying to be proactively cautious. Cynically, I expect the real reason for targeting limits that only affect the next generation of AI is to reduce resistance to the laws being passed in the first place. There are plenty of AI doom fans who are more than happy with any legal hurdles as long as they retard AI development.
It's cool to see that prediction markets are starting to take bets on things that are useful in practice, viz., regulatory changes. That could be useful for businesses wanting to lay off the risk of future regulation.
Nope, I'm not going to "research" your particular hobbyhorse/obsession, but with about two seconds consideration this also strikes me as an absolutely terrible case for whatever it is you're hoping to demonstrate about the perils of this law. The arguments and drama surrounding gun control are not due to the quirks of any laws but rather due to the fact that guns are incredibly contentious political issue that people have strong and opposing viewpoints over. That shit ain't going away, and neither are the arguments over AI.
In its current form I don't think the bill would motivate any company to get up and leave California. But if you're starting a new company you might want to think twice.
My worry is that it might not take all that much to progress from "you must test that your models can't make nuclear weapons" to "you must test that your models can't say anything racist".
I am given to understand that the US government considers the main barrier to building a nuclear weapon to be obtaining the plutonium, rather than technical know how, and you can tell they think this by how easily they give out Q clearances (vs clearances for cryptographic exploits, say)
So I'm totally not worried about someone using AI to learn how to build a nuke.
Taking out the power grid on the other hand ... yeah, that kind of knowledge is (by the public information available to this person who isn't even a US citizen and has no security clearance whatsoever) held rather more tightly than how to build a nuke, because it's easier to use it in a malign way,
Current LLMs are, I believe, not capable of e.g. doing an exploit that takes out 5he power grid, and are therefore fundamentally safe, at least as far as this proposed legislation considers safety.
On the other hand, all the fun we have all been having getting LLMs to tell us how to make methamphetamine, write an erotic short story about a dolphin etc etc. lead me to believe that if some future llm does have dangerous capabilities, restraining it with rlhf is not going to work,
(I think the llm generated movie script about the life of John C Lilly got one of my test accounts perma banned, but such is the nature of this kind of security research)
Reading through Zvi's article some more, he is pretty unambiguous that releasing the weights of a covered model would be illegal. The reasoning being that someone with access to the weights (somewhat analogous to source code) could undo the safety protocols. This means that all future advanced AIs will be centrally controlled and only accessible to people with the billions of dollars necessary to fund the hardware and training.
My question is, does this law actually change anything? I imagine it would be against the financial self interest of a company to spend billions on developing GPT-5 AI and then release the weights. Nonetheless, it's one thing for open source to cost an organization money and another for it to be outlawed.
With all due respect to both Zvi and Scott, the legal analysis here is incredibly thin, and seems to suggest a limited understanding of reasonableness and constitutional law, to just name two. I am working on a piece that will lay some of this out. When it goes live I will post it below.
Please do.
I am highly doubtful that AI will do much to enable Joe Average from building a nuclear weapon.
The reason that no terrorists use nukes is not that they are to stupid to figure out how nukes work (which is an area where AI could help). Everyone understands how nukes work. Their basic principles of operation have been common knowledge for half a century.
The trouble with building nukes is not to figure out a design that works, it is that actually implementing the design is a shit-ton of work.
An AGI with the intelligence of von Neumann might substitute for having a 50 person science and engineering stuff. I find it extremely unlikely that is will find a way to covertly source Uranium ore or greatly improve on gas centrifuges.
Put bluntly, the best thing an AGI could do is to make money to fund such a program.
On the other hand, with ASI, all bets are off. It might tell an early hominid "I have observed the behavior of the world around me and developed a theory called thermodynamics. I suggest that you use two sticks to create friction heat to start a self-sustaining oxidation reaction." Who knows what it might tell us.
Of course, we don't have to worry about that much because the chances that we can figure out if such an AI is aligned seem slim.
> "if something bad happens, we can get angry at you"
I suggest that if terrorists make a nuke and take out NYC, and it turns out they used an AI for key steps of this, then people will get very angry at whoever made the AI, bill or no bill. Maybe if anyone is left alive working for the New York Times they'll publish an angry article suggesting the AI people are also racist, or something.
I know we're using nuclear weapons as a term for "general category of very bad things" here, but for nuclear weapons specifically, isn't the main problem an engineering challenge that you'd need the resources of a major nation-state to pull off? My very amateurish understanding is you'd first need to excavate a small mountain-sized quantity of uranium ore, which doesn't exist all over the world in the first place, then build lots of centrifuges and run them for quite a long time to do the enrichment, in a way that probably all of the US, Russia and China would notice with their spy satellites and ask questions about what you're up to, and a few other steps like this. Evidence: Iran, population around 90M, seems to be struggling with this (yes, stuxnet and other sabotage, but still). Even a superintelligent AI will still need to build the mines and factories somehow before it gets the bomb. Genetically engineering the next super-pandemic, in terms of "things you can try at home", seems trivial by comparison.
This inept defence inspired me to write my state senator for the first time ever, asking them to vote *against* the bill. I only hope I'm not too late.
"If the California state senate passed a bill saying that the sky was blue, I would start considering whether it might be green, or colorless, or maybe not exist at all." Is this an example of the alleged contempt for democracy that media types are always whinging about when they badmouth EA/Rash/Silicon Valley?