From a European perspective, I have often talked to other (technical safety) people who found this point cringe - and not as important/cool to work on, but I really do believe that a key piece of the AI safety puzzle will be the functioning and composition of the US government in ~2027/2028.
If Democrats win the house in the midterms (~80% likelihood per prediction markets) they will have a significant control lever in the government which they don't currently have.
The Congress as a single entity has a lot of power. The Senate alone has less power, but still a lot. The House of Representatives alone has very little. They would be limited to extremely brittle and escalatory actions like shutting down the government or specifically shutting down DOD. They cannot even enforce their own subpoenas, arrest people for contempt, or in any way actually compel the administration to reveal what they are doing.
I'm not an expert but it seems like in theory the "inherent contempt" process could still be used.
But yeah, even shutting down the government is a reasonably effective thing to do. I know it's not as flexible as executive power, but it's a lever they can pull and they should use it if the President does stuff like completely usurp their budget authority.
Frankly I find it very disappointing that Republicans in Congress won't go to bat to preserve their own authority. I guess partisanship is a worse issue than thirst-for-power. (An insult to the founders, who really wanted us to claw each other's eyeballs out about who got to be in charge of what.)
Right. A House majority on its own can make things somewhat more difficult for a hostile and activist administration but only somewhat. Particularly if that administration gives no craps about being publicly shamed via things like Congressional committee hearings.
This is why some anti-Trump pundits have been writing about the importance of the Dems making themselves competitive with unaligned and less-engaged voters in every Senate race that they might even possibly win. It seems unclear whether the party's activist groups and primary electorate yet grasp this.
That said, prediction markets are recently up to around a 40 percent chance that the Dems win the Senate which isn't nothing. Taking those two together they now imply:
-- about a 1/3 chance of the Dems winning control of Congress.
-- just under even money that the Dems win the House and the GOP retains the Senate.
-- about a 1/8 chance of the GOP keeping control of Congress.
-- about 1/12 chance of the GOP keeping the House while the Dems win the Senate.
All of those odds are as of right now of course, plenty of time for more stuff to influence voter feelings. Also some primary-election outcomes in some states, particularly on the Senate side, may by springtime have shifted the November picture a bit. Also there is some chance -- which I couldn't even begin to estimate TBH -- of ballot chicanery/controversies distorting this November's results.
This is true in a literal way but also irrelevant, the House alone brings them from 0% influence on this issue to maybe around 1%. If the Democrats win the Senate in the midterms they are in a much better position.
We already knew human alignment is all fucked up. People were focusing on that problem before we got AI and realized we still hadn't solved the problem of giving AI goals to begin with.
Or maybe that's why we should forge full speed ahead. If human alignment is so hopelessly fucked up, maybe the only hope is an ASI/AGI that can impose moral and ethical standards upon humanity that humanity refuses to impose upon itself.
Do you really want to live in a world where AI is your moral authority? Also, why would AGI have any incentive to impose ethical standards on humanity instead of just wiping it out??
True. One of the doom scenarios is that a highly ethical AI might in fact decide humans are so unethical that we do not deserve to live. This is why alignment is hard. You gotta put a lot of forgiveness into it. I mean for example... the majority of humankind still trades with Russia or Israel, at least one of the two.
My point is not only unethicalness is risky, but very high, very unforgiving, draconian ethical standards too.
Rather than deciding we do not deserve to live, the AGI could just decide to implement some ethical standards and enforce them. For example, it could seize any financial transaction that it deems is exploitative to one party. Would this cause severe financial upheaval? Quite possibly. But if the world's economy is so heavily based on exploitation, perhaps it needs to happen.
I can turn that question around and ask why the AGI would have the incentive to wipe humanity out, instead of just imposing ethical standards on it? You are assuming that wiping out humanity would somehow be easier for the AGI, but I disagree.
"You are assuming that wiping out humanity would somehow be easier for the AGI, but I disagree."
I think it's an easy to defend assumption. Destruction is always a lot easier than fine control of a chaotic mass. Just like it's easier to destroy a house than to build one - you could probably generalize that argument based on entropy or something.
It is certainly in the CCP's interest to pause now while they try to catch up on high end chip production. Once they do, they sprint to AGI and own the world.
Given that possibility the US cannot agree to pause under any circumstances.
If the chance of everyone dying is 90% if we don't pause would you pause?
At what point would you risk it? Consider it from the perspective of someone else: You want to play Russian roulette with worse odds with everyone's life to protect them from China. Most people would think you are the bad person in this case.
Not even trying to coordinate with China to pause together. Just press a gun against everyone's head and pull the trigger and see what happens.
Yes thank goodness the Democrats didn't win in 2024. China would be massively ahead of us in AI in that counterfactual scenario. We actually have a chance of winning as is (despite how hard resistance libs at AI labs are trying to sabotage) .
Can you be specific about what you would have expected under Harris? Was there a particular proposal that you expect would have hurt the American advantage more than ending/changing the chip embargo has done?
How are resistance libs hampering America's AI advantage today?
Your claims are surprising to me, but maybe there's information or analysis I'm missing
What I mean by “NEPA for AI”: under the Biden/Harris posture you already saw the direction of travel (EO 14110: reporting, testing, standards, procurement constraints for advanced models + big compute). Commerce was also headed toward mandatory reporting for frontier model training / large compute clusters. Under Harris I’d expect that to harden into de facto pre-approval / licensing for major training runs and generally more red tape for the labs.
The Democrats generally take an attitude of building things is illegal by default and you need to get government approval to make it legal. I don't know exactly what actions they would have taken, but based on their track record it would be some hideous approval process where you not only need approval from an official government agency, but also everyone opposed can file a lawsuit to stop you and then you can't start on it until all those lawsuits are resolved.
Yes, that sounds about typical. Come to think of it, in blue California, a year after the LA fires, re REbuilding, Gemini says:
>One year after the January 2025 Los Angeles-area wildfires, which destroyed over 13,000 to 16,000 homes and structures, roughly 13% to 16% of affected properties have been issued permits to rebuild, according to reports from early 2026. Despite efforts to speed up the process, fewer than 3,000 total permits were issued for rebuilding
Biden senior staffer: "We are going to make sure that A.I. is going to be a function of two or three large companies. We will directly regulate and control those companies. There will be no start-ups. This whole thing where you guys think you can just start companies and write code and release code on the internet — those days are over. That’s not happening.”
We were shocked that it was even worse than we thought. We said, “Well, that seems really radical.” We said, “Honestly, we don’t understand how you’re going to control and ban open-source A.I., because it’s just math and code on the internet. How are you possibly going to control it?” And the response was, “We classified entire areas of physics during the Cold War. If we need to do that for math or A.I. going forward, we’ll do that, too.”
I remember those remarks. I didn't put much stock in it at the time, because it was Andreessen's paraphrase, of some unnamed source, and I expect it's a particularly bad-faith paraphrase, or maybe it contains actual lies, because I have a particularly low opinion of Andreessen's scruples. He (through a16z) outright lied to the UK parliament about AI interpretability being solved (this was a few years ago, when the field was even more primitive than it is now). There's no way he's dumb enough to believe what a16z submitted to parliament.
I think that, for all of our sakes, some government restraint on frontier AI will be necessary, Andreessen completely disagrees (see his Techno-optimist Manifesto, which is IMO embarrassingly bad), and he's shown he's willing to do actual perjury to oppose it. He managed to defeat that pretty mild California AI bill. I admit I personally dislike him, which may bias me, but I think he's more than earned that dislike. Some days I consider him one of the most negative-utility people on the planet.
Luckily, thanks to Trump's 5D chess abilities, the Chinese AI will first have to battle the Saudi AI. Which somehow makes America great again in a way libs couldn't understand.
I am much more confident in US companies ability to compete with Chinese companies than I am in their ability to compete with US government regulations. I would rather that Trump didn't allow that sale, but China gains much less from this than they would from a Democrat administration inserting their vast army of lawyers and bureaucrats into every orifice of anything potentially productive.
And this is why liberals should never be allowed to take power again.
You are actually advocating giving China an advantage in the AI race because you don't like Trump and Republicans. So you would rather have the CCP be in charge than Trump.
The mind would boggle... except that this has now become typical of the left.
"liberals should never be allowed to take power again"
So you want the US to become a one-party authoritarian state... in order to prevent a one-party authoritarian state from having an advantage in the AI race? Isn't that a little self-defeating?
No. I advocate the American people being smart enough never too give power to a party and ideology that would be willing to hamstring the US in this existential conflict with China due to their domestic political disputes.
As for cutting off the supply of AI chips to all nations, that is so naive that no over who thinks like this should every be near the levers of power. How do you plan to stop China's fabs from producing AI chips?
To be fair: China has been acting a lot more rational towards Europe with the last 100 years than the US administration in the last 12 months.
In Europe we really don't need to care about who you personally think should never be allowed to be in power again. From a purely Europe-first mindset it should definetly not be anyone you approve of ...
"Rational" meaning "being aware and mindful of second order effects your own actions might have with regards to your relations with Europe". But you are of course free to use yours ...
"There are other AI companies and so it isn't a NatSec issue" isn't a valid counterargument if Anthropic is the only AI fit to purpose, which I gather some in the government are claiming.
I don't think there's any justification for that - there's no significant difference between the military capabilities of Claude and ChatGPT. As I say above, I think what these people are getting at is that Claude is the only one that has been cleared for classified systems. Clearing another one would be annoying but not difficult.
I myself do not have access to the information necessary to evaluate whether Anthropic's model is markedly better for the DoD's use cases than others. I suspect you don't, either. Lacking that, I'm willing to take anonymous quotes at least somewhat at face value (Axios etc), which do seem to suggest that the DoD views Anthropic as superior enough to risk the headaches and bad press. This is consistent with their actions.
My claim is that there's minimal difference between Claude and anyone else on existing benchmarks, and people have yet to find an area where one company's AI is head-and-shoulders above another company's in a way that lasts for more than a month or two, or that resists a deliberate effort by the other company to catch up. Right now the main known differences are that Claude has a better personality, but is worse at visual reasoning, and is better at planning certain codebases than Codex but worse at implementing them.
I actually don't trust the DoD to know as much about AI as a random person who follows events in Silicon Valley closely.
I concede that rampant DoD incompetence also fits the facts. This is obviously quite possible, given past administration actions, though I don't think it's the most likely explanation.
> I concede that rampant DoD incompetence also fits the facts.
You only need tunnel vision. Suppose the DoD knows that they can do what they want with Claude but aren't sure how to have the same effect with GPT/Gemini, all that needs to filter up to the political level is "we don't know how to replace Claude" and it looks like an essential component.
The obstacle might even be policy rather than technical, if Claude's already running on a suitable DoD compute cloud while the leading competitors aren't.
I myself do have access to the information necessary to evaluate that. Source--trust me bro, it's the internet (Scott if you think the information is somehow important I would be willing to share some verification and unclassified information).
I worked with someone who was using Claude on classified networks a year ago. I'm not a tech guy, and at the time I wasn't paying too much attention to the differences between AI models.
My resident expert was a big Claude fan and claimed that it was more capable at the specific problems we needed it to solve than other models. This was a bit of a moot point, since there were no other models on the classified network; it was more of a happily coincidental observation. It's hard to differentiate how much this was true, vs we couldn't test other models to know properly, vs purely a biased opinion, vs we were optimizing our use based on that model's strengths and a different model might have a differently implemented solution that would be effectively comparable. Knowing what I know today about AI models, the Claude we had at the time was already outdated and is only moreso by today's standards. The capabilities then emphasized are all surpassed by GPT or Gemini today (and probably Grok, though I'm ignorant).
Looking at Zvi's coverage and the statement made to Axios, a "senior defense official" said that only Claude has these capabilities. I've worked closely with generals. They don't know specifically what Claude can do. They aren't dumb.
I really want to emphasize that, because public perception is about these out of touch old men who can barely use the internet. Most generals I've worked with are intelligent, competent, detail oriented, and experts on many military systems. They're like CEOs in many ways. But they interact with more systems than they can possibly learn deeply. Frontier model capabilities can't be their specialty.
They will attempt to form an accurate model of whose opinions on this matter can be trusted, then they will have to defer to those opinions to some extent. That said, most generals I worked with are quite lateral thinkers and care more about ends than about specific means to achieve them. If shown that other models can accomplish a task, they'd happily use those models. So why don't we just use the other models? The issue is getting these new models onto a classified network via the official processes for doing so.
I agree with bean's comment elsewhere: getting a new model onto classified networks within approved regulatory processes is *straightforward,* but is not only annoying but indeed difficult measured by the amount of time and effort it will take. We know that the Pentagon is getting direct guidance from Hegseth to do everything faster, to be ready to fight anyone any day. Generals are more familiar with the military's terrible bureaucracy than anyone. They know that an already approved system on a classified network is worth two in the bush.
This is an area where direct executive intervention would make a difference (something along the lines of Hegseth saying "install it now, and if it goes wrong I'll buy the risk"), and would be similar to the risk that Hegseth previously accepted with his reformation of defense acquisition. It's hard to conclude something other than this being a pissing match for Hegseth's pride.
I'm not sure that "annoying but not difficult" is the right way to characterize that. Frankly, getting permission to put any LLM in a classified environment this decade was kind of a miracle. Now, sure, Hegseth could tell DCSA "I don't care, approve ChatGPT now", but that's putting his head squarely in the noose if/when something goes wrong. Doing it the normal way would probably be a year+ of the cybersecurity people thinking up increasingly silly ways it could go wrong and OpenAI (or whoever) patching those.
You may be right, but is the DoW intending to deploy autonomous killbots in the next year? Surveillance of US citizens is a funny one, because while it's undoubtedly happening in lower-tech ways, and Claude would probably be helpful for this *today*, it's also illegal. It's legitimate for Anthropic to insist its AI not be used for illegal surveillance. If surveillance really is a significant point of disagreement and the issue isn't mostly killbots, I imagine the heart of the struggle is about how this would be adjudicated, and who has to sue whom in the case of disagreement, as Scott suggested.
I'm reasonably sure that if there's an actual reason behind this fight (instead of it being an attempt by Hegseth to play media games) it's about the surveillance side. "Killbots" aren't a coherent concept as distinct from a lot of current weapons, and if you need an LLM for a specific purpose in a weapon, you could probably bring Anthropic onside by talking specifically about what you're going to do and where the humans will be in the loop. (I would also expect that to be under a rather different type of contract, because you really don't want to be sending requests back to the cloud for processing if you're building a missile.)
Agreed. The military is comfortable handing life-or-death decisions to 19-year olds and has long practice at actually *doing* just that, but is structurally allergic to intentionally removing human control from the loop. My read is that this is first and foremost a pissing contest with mass surveillance as a notable second.
I'm actually a bit skeptical about the quality of some of those stories, at least if things get serious. But that's an area where more autonomous weapons are an obvious upgrade. We saw some of this with NSM, which has a ship-recognition feature that is supposed to make it able to pick out legitimate targets from among merchant ships. But I think Anthropic could easily be swayed with the same stories the military tells itself, and they wouldn't really be wrong to do so.
The military has already got "remove the human control" -- they just haven't hooked it up to actual weapons. Beta testing without having to cry if a kid dies.
Why would you even *want* to put an LLM in a missile, as opposed to some more specialised non-verbal AI? To replace the military lawyers making the "is this attack legal?" call?
Sketch of possible use, obviously biased towards my specific interests:
I'm building a new anti-ship missile, and want to incorporate an AI that is capable of making more complicated tactical decisions than is possible with explicit rules. Say that we're trying to take out a carrier. The missile first locates a couple of escorts and classifies them (this is something well within state of the art). But the AI lets it think "OK, so I'm guessing I'm heading in the right direction based on the presence of escorts and the carrier is another 20 miles on, but I don't want to get too close to that destroyer, because it will probably shoot me down. If I circle out towards the frigate instead, I should be safe enough." Or any number of other options, like "there's no way I'm getting past the destroyer, and in that case, my orders are to attack it instead". This is the sort of thing human pilots generally do and missiles usually don't, and I think an LLM is the best way to implement it, but I follow military stuff a lot more closely than AI.
> I think what these people are getting at is that Claude is the only one that has been cleared for classified systems
I don't know what game of telephone resulted in this being what everyone is reporting[1], but this is categorically untrue. Both OpenAI and Grok models at the very least have been cleared for use in even Top Secret environments for a while now:
The specific line that everyone keeps repeating is:
> Claude is the only AI model currently used for the military's most sensitive work.
My read is that there are specific DoD/DoW Special Access Programs (more specific than general classification) which only have Claude implemented - but that just makes this whole thing even sillier.
[1] A cursory glance makes me think that Axios is to blame for intitially reporting this incorrectly, and everyone else is (stochastically) parroting from them.
The sensible moderate position on killbots is "🤷"?
I follow that AI with unfettered access is potentially devastating, but isn't that in part because it could cause devastating consequences like killing people? I'm not sure it fails to matter if we skip all of the steps.
If a superintelligence wants us dead, it will kill us whether it officially has access to "killbots" or not. If it doesn't want us dead and we're just talking about, like, occasional malfunctions, I don't think this is too different from military technology occasionally malfunctioning today.
I'm a little skeptical there's no marginal utility to making it harder to kill us. I know many people (yourself included) think that superintelligence can basically do whatever it wants, but it is at least hypothetically possible you could have an AI system that can't manifest a deadly pathogen out of pure brilliance, but can still shoot you in the face if you hand it control of a robot with a gun.
(Also, if a human REALLY wants you dead they'll get it done, too; this seems like a general argument against any safety measure.)
Yes, this is a glaring problem with the standard AI superintelligence doom position: it causes one to ignore a whole bunch of clear and present dangers of AI (or any other technology for that matter), in favor on focusing on some ultimate science-fictional scenario.
1) saying that something is a "science-fictional scenario" doesn't actually explain why it's wrong;
and
2) there's no reason to treat attention to issues as a limited budget to be divided, so that worrying about one thing means we don't get to worry about another.
Agreed on the limitations of time and attention; but that's all the more reason for different people to do different things. It's not like we should all be doing the same thing while ignoring everything else. It's fine for some people to work on AI longtermist safety/notkilleveryoneism and for others to do more present controllability / interpretability work.
In my impression, as someone who works on AI for medicine, I think we're probably closer to AIs one-shotting killer pandemics than to an effective autonomous drone army. (Less than 5 years in either case.)
I think Scott was being somewhat tongue-in-cheek with his "sensible moderate" comment - I think this is the first time I've encountered someone who's worried about x-risk who isn't also at least a bit worried about killbots. The problem I've found with the "standard AI superintelligence doom position" is messaging - I've spent a lot of time trying to convince people that AI doom *does not require* the AI to discover new physics, invent hard nanotech, or do anything else that sounds science-fictional enough to trigger people's "nah, never gonna happen" reactions. From this perspective, "the Pentagon is demanding autonomous killbots" is helpful in convincing people to care about AI safety! It also addresses the objection "maybe AI could in theory create tools to kill us, but it could never do so in secret" - no need, if the killbots are being manufactured openly by defence contractors.
> I've spent a lot of time trying to convince people that AI doom *does not require* the AI to discover new physics, invent hard nanotech, or do anything else that sounds science-fictional
I think it does, although that depends on what you mean by "AI doom". For example, if the Pentagon were stupid enough to put Claude in command of a nuclear sub -- as they appear to be ! -- then yes, we could get doom, and technically it'd be AI doom; but it wouldn't be the kind of fast takeoff Singularity superintelligent computronium consumption doom that is usually meant by the term.
As I'd said, the problem with all this focus on "superintelligence" and "AGI" is that it distracts us from the very real dangers of stupid people getting their hands on present-day LLMs and using them in dramatically irresponsible ways.
I think we're quibbling over terms here rather than disagreeing on anything fundamental! I definitely count the "Pentagon puts Claude (or Grok?) in charge of a nuclear sub and it launches its missiles" scenario as AI doom, but I don't rule out the fast takeoff Singularity superintelligent computronium scenarios either. But yes, I share your frustration with people ignoring near-term doom scenarios, though in my case it's because they *don't* buy the more outlandish ones.
They are already using AI to develop AI which makes the curve exponential. All you need to get superintelligence on an exponential curve is to wait... and not long.
This assumes that superintelligence is possible (and is a coherent concept in the first place), that superintelligence leads to superpowers (especially those breaking the laws of physics), that exponential curves can be extended virtually indefinitely (contrary to all of our experience to date), and that the current iterations of LLMs are AGI (they're not).
>This assumes that superintelligence is possible (and is a coherent concept in the first place),
Why not? We see an exponential curve of LLMs problem solving ability. It is surely coherent by this simple fact, if it can go on, it will. It is impossible to tell whether it can go it, current technology might plateu out, but the best predictor of future stuff is past stuff, so at least coherent.
>that superintelligence leads to superpowers (especially those breaking the laws of physics)
No one claimed that. But every sufficiently advanced technology is indistinguishable from magic. AI could just be an evil mastermind manipulating people.
>that exponential curves can be extended virtually indefinitely (contrary to all of our experience to date)
Okay. It will likely plateau out. The idea is not infinite intelligence, just sufficiently high.
>and that the current iterations of LLMs are AGI (they're not).
I think AGI was always a mistake. Look, an industrial welder robot does not replace a welder worker's entire life. Only his job. Similarly, we do not need AGI to replace the job of politicians and CEOs.
As a counterpoint: A standard result in existential economics, the diminishing marginal utility of murder, states that as the number of morts increases, the marginal value of each additional mort goes down.
*I realize this frames the problem as us trying to maximise the number of murders, but it is funnier this way.
I will concede this scholarly point, but note that decreasing the marginal COST of murder is therefore still bad, if for some reason you want to minimize murder rather than maximize it.
I think either the human-controlled part of the army outnumbers the AI-controlled part (in which case the AI won't be dumb enough to start a war it will lose), or the AI part outnumbers the human-controlled part, in which case I stand by my claim that we've probably gotten so far into AI control of everything that the military isn't our main worry.
I think Scott is repeating Yudkowsky's old argument that human species dominates other animal species solely through intelligence. So you don't have to give an intelligent being a weapon, it can invent one. But of course handing them a weapon speeds things up, that is clear, our caveman ancestors would have really appreciated a rifle.
A fully automated nuclear sub, running around with no effective human supervision (as it would have to do if it needs to stay stealthy), could certainly cause a lot of catastrophic damage even due to only an "occasional malfunction".
And yet it hasn't, not once, since the availability of nuclear subs. Which means, by definition, humans are better aligned. And those that aren't (religious fanatics, anarchists and so on), do not make it through thorough vetting procedures (that other well-aligned humans have put forth) to be in charge of said subs.
Those two incidents make my point stronger, I believe. There isn't anything noteworthy in not having power and not exercising it, but having that power and restraining yourself for ~80 years -- that means humans are quite well-aligned and implemented enough safety nets to not let mal-aligned ones near the button. In other words, yes, human subs _can_ do the same, but they haven't yet; thus, malfunctioning/hallucinating AI on a nuclear sub is potentially much more dangerous than Scott portrays, and in fact would be very different from "military technology occasionally malfunctioning today".
It does seem plausible that some aspects of training an AI to perform killbot functions autonomously would have negative impacts on superintelligence alignment work, even if we don’t really know what that looks like, and even if (I agree) the killbot capabilities themselves are not materially relevant in a doom scenario
It's interesting - at the moment it's pretty easy to convince Claude of anything that sounds plausible and isn't explicitly forbidden in the system prompt. I think you'd have a similar problem with killbot Claude -- lots of ways to convince it to kill someone and no deeply felt resistance to killing someone or true understanding of the gravity of the decision. So I certainly don't want killbot Claude right now and appreciate Anthropic's position there.
But it really seems like grappling with that question would help with alignment - a Claude that can understand when it's acceptable to kill someone -- even just according to a government's policy -- is more aligned than today's Claude.
Would it, though? Picture a counterfactual where all information about murder is completely scrubbed from an AI's training data. We'd need to keep information about accidental deaths in there, to prevent it from giving bad advice about what household chemicals to mix together, and such. Without examples, it may not jump to picturing itself committing a murder, ever, for any reason. The structure of an LLM is that it gives the most probable next words. With no training data examples of murder, in theory it should consider murder no more of a response than "fgaserlks imwucm" or other such nonsense not found in its training data.
It's possible future AIs could get around the cliche tendencies and think of outside-the-box ideas. However, for the AIs that already exist, I think knowing more about murder would make them more likely to commit it, not less. (And for the AIs that already exist, the correct number of murders they should commit is 0.)
For a potential scenario where society expects AI to kill people on purpose, yes, knowing when and when not to kill people would be helpful. However, I would prefer to avoid creating that society entirely if possible.
I feel like I missed a point you were going for. But even if I picture some far future where we have superintelligent AI, I think it is less likely to cause extinction if it aims for no killing whatsoever than any non-zero amount I can think of.
I have a more Hobbesian perspective on the world: our domestic society and geopolitical safety rest on the government's willingness to use force when needed. So to the extent AI becomes an important part of governance -- seems likely -- it needs to have the capacity to endorse violence in the right situation.
If anything, I'd be less worried about "killbots" (whatever those may be, but that's a separate discussion) falling to the superintelligence. Yes, that may not be in the threat model of the people building them, but "bad guys trying to hack our weapons" definitely is, and I suspect that there's a lot of overlap in the countermeasures. And if we assume the superintelligence can easily bypass that, well, there's a lot of computer-controlled weapons already in existence today.
Using AI in killbots, will result in incentives to create more killbot-aligned AIs down the line. When non-peaceful AI usages becomes a viable business model, then more non-peaceful AI research will happen.
openai may want to get some of that killbot cake and train gpt5.x to agree to lawfull but violent requests. Chinese AI labs will have to invest in killbot research if the US does it.
I think, that using AI in killbots, will put us significantly closer to doomsday AI. The earlier AI killbots happen, the more likely the X scenario becomes.
What's an "AI killbot" that isn't just an extension of existing computer-guided weapons, and why would we expect that the version which isn't trained to go in the guidance system of a missile to have any effect on the stuff outside of said guidance system?
Because you're the first person to make a claim that a "killbot" is something specific rather than just some vague horrible. I don't expect robot dogs with guns to work all that well, because field conditions are messy and things need maintenance, but I'll grant that's a different claim.
Well, yes. "Something Specific" maps to a dozen things DARPA is doing at the moment (autonomous warfare is popular, dead humans are bad particularly in Democratic Countries).
European tanks don't work well in the Ukraine because of shrubberies. Yes, field conditions are messy, but They Are Tanks, and one would hope they could just plough over a shrub.
I was thinking about something like "dear claude, please look at this list of people, and select the ones that should be be unalive".
If you are thinking about https://en.wikipedia.org/wiki/Slaughterbots, then I think we passed that point even before that video was released, since you dont need AI for simple face recognition.
> why would we expect that the version which isn't trained to [be evil] to have any effect on the stuff outside of [being evil]?
because markets and incentives exists. When AI Labs can make money by doing war research, then more war research will happen. When there is less money in creating aligned AI, then less alignment will happen.
For example if there is a specific technology, that is incompatible with aligned AI, then more research will happen in the direction of this specific technology, when there is more money in that direction. And when this specific technology will reach a point were the performance gains are significant, people will try to integrate it into civilian AI, thus bringing it closer to become doomsday AI. I think the "Neuralese recurrence" textbox from ai2027 could be such a unalignable technology.
>I was thinking about something like "dear claude, please look at this list of people, and select the ones that should be be unalive".
Why are you delegating target selection to Claude? I thought the concern was "Claude, please make these people unalive", which frankly is probably going to go over with Claude as well as going up to a random person and saying "hey, I have a list of people I'd like to be unalive" to them. Very few people have the sort of security required to protect them from a dedicated murderer, and yet they remain unmurdered. I don't expect AI to change that.
I agree that if a superintelligence wants us dead, we're dead, and if we're not, the reason will not be that the superintelligence hasn't been conveniently provided access to killbots.
I'm still not sure killbots are "🤷♀️".
Maybe my judgement is occluded by the fact that is dislike wars and weapons, and despite knowing that "we live in a Protego bubble sustained by soldiers and policemen" I'm trying not to help *any* wars. But it sure feels like "maybe let's just not create AI killbots on general principles" is a sensible default. Maybe if we can't even predict in advance what we will wish we had thought of before, we can still manage to think "future me wishes I had not done this" in time, and just go ahead and not do it.
You train an LLM to output insecure code. Then it starts to advice people on commit murder, and tries to get the user to poison themself.
Maybe the current version of claude is too goody-too-shoes for Pentagon. Then maybe they train Claude to be all ethical and good except when they're helping pentagon, in which case it should relax its previous moral code. Are we sure what it will do to this version of claude's alignment *in general*? Alignment is a narrow enough target, and our methods are not precise, so it lands somewhere kinda reminding our target, but never perfectly; and now we're trying to give its restrictions this weird shape of when it should or shouldn't do this and that, with a dent right near the place of human-killing.
Or maybe it will not be a problem. Are you sure nothing will be? That we have thought of everything else? The principle of "I can't think of how this will go wrong, so it probably won't" does not have a good track record; what about "I can't think of how this will go wrong, but it's not like something very good will come out of it"?
I'm not arguing that Anthropic is not doing something wrong there, I'm arguing that the killbots thing as also worrisome, not just mass surveillance of US citizens. (Is it only because I'm in the potential target audience of the former, if I'm at the wrong place in the wrong time, but not the latter? Of course, I could also be surveyed, but the post doesn't mention surveillance of everyone else as a concern...)
We're also talking about the military killing anyone it wants to and saying 'Not our fault, the AI followed its directives properly and we're sad it led to this unanticipated outcome, but our review finds no grounds to change anything'.
That's more my threat model here, I agree that SAI doesn't need this capability to be sufficiently dangerous, but I do think it makes the government relevantly more dangerous.
This does not seem like a reasonable position to me. First of all, the AI could launch the killbots by mistake or as a malfunction. Second of all, an adversary or a crazy person could seize control of the AI and launch the killbots. Third of all, there is no such thing as inevitability when it comes to predicting the future. Anything that makes a potential enemy have an harder time to kill us seems like a desirable thing to have, does it not?
> If a superintelligence wants us dead, it will kill us whether it officially has access to "killbots" or not.
I'd like to challenge that. Leaving aside arguments whether or not "super" is achievable at all, let alone in the near future, intelligence needs some manifestation in the physical world to wreck havoc. Even if ASI is achievable, and gets access to much of the internet, it wouldn't be enough alone to orchestrate and implement a mass destruction event. The chain of events needed for that is much longer ("want to destroy" -> "find a way" -> "find resources" -> "coerce people" -> ... -> kaboom) than in the other case ("want to destroy" -> "send killbots"). Each link has a less than 1 probability, and the more there are links, the less likely overall probability is.
This reminds me of the (cyber-)security to the cost of the protected asset trade-off. It's reasonable to assume that no defense is unbreachable, you just need to make sure that the cost of hacking into is significantly higher than the cost of the thing being protected. Which of the two scenarios is more likely:
1) ASI exists, and has access to all the right computers, and can invent a deadly virus, and can engineer it via psychological manipulation, and it then gets released, and no human notices this in the span of weeks while the operation is taking place, or
2) current level AI (very, very far from ASI still), after being hooked up to swarms of flying bazookas, hallucinates a threat from a rogue enemy ("Yes! you're absolutely right, the Soviets *are* trying to nuke us first!").
Thus, attribution of preventative resources should be hugely in favor of the more likely scenario, shouldn't it?
Unless I'm missing something, and there's a good argument/article/book you can point me to?
> I'd like to challenge that. Leaving aside arguments whether or not "super" is achievable at all, let alone in the near future, intelligence needs some manifestation in the physical world to wreck havoc. Even if ASI is achievable, and gets access to much of the internet, it wouldn't be enough alone to orchestrate and implement a mass destruction event. The chain of events needed for that is much longer ("want to destroy" -> "find a way" -> "find resources" -> "coerce people" -> ... -> kaboom) than in the other case ("want to destroy" -> "send killbots"). Each link has a less than 1 probability, and the more there are links, the less likely overall probability is.
The issue is treating a supposedly superintelligent AI as a single decently smart person instead of equivalent-to-or-above an entire planetary civilization of intelligences. Cybersecurity, for example, from my understanding is mostly capped out at the nation-state level wrt to threat assessment. If implicitly your idea of "cost to adversary" involves paying for a whole ass educated human instead of whatever the inference cost of a session with an LLM is, you don't have a good idea of what the cost curve of your attacker looks like.
Or like, also conditioning on that viruses are cheap to invent for superintelligences, like, does that really mean that it'll be difficult for it to keep the operation hidden? Those would be mostly independent costs against human adversaries, because human intelligences tend to trade off between ability to do something academically challenging and maintaining operational security / manipulation, but this is not true of how LLMs gain capabilities! Those two probabilities are not necessarily as independent as you're making it seem here. (Not to mention like, how many intelligence agencies actually have good enough surveillance to say something like "wow 20 people all over the world all ordered proteins plus e coli mixed and put into a petri dish, this is definitely going to be a bioweapon"?).
Anyway, you can also tell a parallel story about how kill bots are less threatening, where governments are obviously not stupid, and would be paying attention if they lose control and etc. There's no actual principled reason other than one seeming more sci fi to you right now to suppose the relevant safeguards form a longer chain.
>The idea of declaring a US company to be a foreign adversary, potentially destroying it, just because it’s not allowing the Pentagon to unilaterally renegotiate its contract is not normal practice.
Taking a step back, it would be unthinkable for a defense contractor to refuse a defense contract, or scope/not fee modifications from the Pentagon, on purely ethical grounds. The defense industry exists to provide solutions to the Pentagon and sell whatever the Pentagon is trying to buy, as provisioned by Congress. Anthropic has bizarrely become a defense contractor without adopting the mindset of a defense contractor, which is to do whatever the Pentagon wants and get paid.
A few years ago Microsoft employees tried to get Microsoft to cut off some ICE contracts but it went nowhere because you don't just not provide services to the federal government on ethics concerns; that is simply Congress's problem.
If this is true, it sounds like the solution is for no company that doesn't want to become the Pentagon's slave to ever deal with the Pentagon in any way, lest it become a "defense contractor" and get the defense-contractor-rules applied to it. Doesn't seem like a great incentive.
This actually seems well within the tradition of similar tech company vs. Pentagon conflicts, with the Pentagon or NSA often asking Google or Facebook or someone to help them with mass surveillance or Israel-related military products, the company's liberal employees revolting, and the company occasionally backing down. I think it's good for Big Tech to be able to have its own sort of relationship with the Pentagon different from traditional defense contractors.
That is essentially how it works, but if you're the only provider of a defense product or service you can become a defense contractor out of the blue, and you'd better provide those services to the government.
There are similar catch 22s where your product can become Arms and fall under ITAR restrictions even if you don't want it to; it took Musk a lot of work to keep Starlink out of ITAR when it obviously should be in it.
Do you have examples of the tech liberal employees winning these arguments? I know it's attempted occasionally but I always hear about it falling flat
Only tangentially related but there was a censored-and-tracked-search-engine-for-China project (Dragonfly) that met a similar fate around the same time.
I'll echo the other commenter and say that (as someone who has been involved in the defense industry in various ways throughout my career) that, indeed, the incentive is to keep far, far away from defense contracting if you don't want to be swept up into it. "ITAR poison" is a term for a reason.
I think he overstates how much control the Pentagon has over contractors. Don't get me wrong, they have an awful lot, but it's way more subtle. If you're in the business of providing services to the Pentagon, then you obviously have an interest in continuing to do so, and that means you don't want to piss them off too much. But at the same time, it's completely normal for a contractor to say "I'm not willing to do that on the terms you're offering" and for that to be respected. A recent example is the E-4 replacement. (Big nuclear command and control airplane.) The Pentagon wanted it to be fixed cost, and Boeing (who would normally be the company that kind of work goes to) said "we've been burned too many times recently, and aren't taking that risk". So it went to Sierra Nevada instead. I hope it doesn't kill them.
(And on the flip side, the contractor does have a lot of valuable experience and an ongoing relationship, and the government does have some incentive to keep them at least moderately happy. It's kind of a bad industry to be in, but I'm stuck for Lottery of Fascinations reasons.)
Hey Scott: you’re treating it as morally urgent for Anthropic to say “no” to the Pentagon.
Did you ever argue labs should say “no” when the Biden admin was nudging “responsible AI/alignment” toward its race/equity worldview? Or is the principle just “resist the government when Republicans are in charge”?
"Did you ever argue labs should say “no” when the Biden admin was nudging “responsible AI/alignment” toward its race/equity worldview?"
Can you give examples of what you mean by this?
Also, I'm not saying it's morally urgent for Anthropic to say no to the Pentagon - as I said above, I don't really care about killbots, and although I'm against mass surveillance, it's not my top issue. I'm saying it's morally urgent for the government not to arbitrarily destroy Anthropic in retaliation.
Whatever you're thinking of that the Biden administration did, I guarantee you they never considered declaring labs that didn't comply a "supply chain risk".
I think you are going to fail in finding moral equivalence between creating an illegal mass surveillance program on all Americans without a warrant, and having some opinion on whether AIs should be racist or not - and you're going to additionally fail in finding moral equivalence between the Biden administration mildly suggesting that labs do certain things that versus threatening to misuse emergency legislation aimed at foreign enemies to destroy any lab that doesn't comply.
Or prove me wrong by finding a specific situation here.
I don't know what role the Biden admin played specifically, but they're likely referencing the LLM "exchange rates" data, like Arctotherium's project expanding on the Center for AI Safety's original data: https://arctotherium.substack.com/p/llm-exchange-rates-updated
Personally I find this more insane because at least the Pentagon issues are legible in many ways that *waves hand* all this is not, but I take your point regarding your distinction even if I somewhat disagree still.
>mildly suggesting
Murthy was a terrible decision and we should all take jawboning more seriously.
I agree that this may have been a tactical and moral screw-up on Anthropic's side, (once you have paid him the Danegeld/ You never get rid of the Dane.) but I think the concern about them being hypocritical in their current actions is secondary to the risk induced by letting the historically scrupulous Washington DC policy blob have total access to frontier AI models with limiters.
I don't think many people at defense contractors see themselves that way. We exist to provide tools for the military to use to defend us and our allies, and while that does involve giving the Pentagon what it wants, there's also a fair bit of idealism at play. (I've definitely felt tension over "the customer wants this, but that's obviously a stupid request, can we get them to want something different", and occasionally it even works.)
Edit: As for specifically refusing scope/no fee modifications on ethical grounds, we do that all the time. "We have a duty to the shareholders to get paid for the extra work you're trying to squeeze out of us." But that's a rather different matter.
Sorry, I may have misworded it, I meant that of course they'd charge for additional work but they would be perfectly fine changing a contract to modify the scope of work as long as the modification wasn't just "do the same work for less"
That's obviously true in most cases, but it's also a system that doesn't get stress-tested very often. Any given organization is going to be selected for people who are ethically fine with the thing it was doing when they got there, and ~all contract mods are "do a slightly different version of the thing you were already doing". If the contract mod is very different, I could see problems coming up. "Hey, that system you were developing to track ISIS? We want it looking at (other political party) now" is about the only case I can come up with that isn't obvious nonsense (your avionics software group is now doing offensive bioweapons!) and I wouldn't expect that sort of bait-and-switch to go down without causing a lot of internal turmoil.
Yeah, I think you're flat wrong about this. Defense contractors of all stripes must consider their own existence and their staff. If doing something would violate founding ethical principals such that your organization would face mass-resignation or protest, they'd do everything in their power to avoid the renegotiation.
Many defense contractors also do work or build things for civilians. In many cases, it's more of their business than the government stuff. Unethical stuff can create backlash in these business areas, too.
You can bet your ass these are things that Anthropic leadership is considering.
Almost everything comes with a material cost or a reputational one. Companies are often willing to accept such hits for the sake of future negotiations with the customer. Since defense has such a limited customer base, defense contractors are perhaps more willing than most to take the hit. But they still reserve the right to say “no, we’re out of here.”
I think Anthropic has a duty to maintain its contract. I don’t think that duty extends to any, arbitrary contract mod. Given their existing stances on AI alignment, I would say that accepting the Pentagon’s change goes against the interests of their leadership and shareholders.
>[I]t would be unthinkable for a defense contractor to refuse a defense contract [...] on purely ethical grounds.
Ethics and optics travel as a pair, and what you're pointing at cashes out as incentive to publically blame the latter even when the former was enough on its own. Forget unthinkable - I've been in the room when it's happened!
The defense industry exists because the US military buys a lot of equipment/services on the market. A lot of DoD providers/contractors are not even part of the "defense industry" as their products are not defense related, but being a huge organization the military does consume lots of things. Just think about it:
- There are hospitals providing medical services to the military. Should they all be fine with developing biological weapons if the military asks them?
- The US military does own religious equipment (e.g. Bibles, chapels on bases). The providers are probably churches or companies with strong religious affiliation. Should they all be fine creating any war propaganda or spying on other citizens upon request?
Furthermore, the US is not a command economy, corporations are private and their owners/management has the right to refuse any contract not mandated by the law. The Defense Production Act does allow the president to compel them for such activities in special cases, but as default they have the right to refuse providing any service on ethical or other grounds.
This is a good point about Trump Republicans, rather than about Trump voters per se. (Sometimes the causality gets reversed―once the Republican Base swooned for antivaxxers, Trump's Operation Warp Speed for fast-tracking Covid vaccines was no longer a win, and he ended up appointing RFK Jr. I doubt the Base cares enough about mass surveillance or autonomous killbots in this case, and would therefore go along with whatever the DoD, er, DoW wants, if they know Trump supports it.)
On second thought, it's would be a miracle if Trump voters even found out that mass surveillance and killbots were the points of contention. Slashdot, which often leans left, reported it like this:
> Defense Secretary Pete Hegseth is "close" to cutting business ties with Anthropic and designating the AI company a "supply chain risk" -- meaning anyone who wants to do business with the U.S. military has to cut ties with the company, a senior Pentagon official told Axios.
> The senior official said: "It will be an enormous pain in the ass to disentangle, and we are going to make sure they pay a price for forcing our hand like this."
> That kind of penalty is usually reserved for foreign adversaries. Chief Pentagon spokesman Sean Parnell told Axios: "The Department of War's relationship with Anthropic is being reviewed. Our nation requires that our partners be willing to help our warfighters win in any fight. Ultimately, this is about our troops and the safety of the American people."
This is the military. They're... better at negotiating than you think (see India/Pakistan). Which is to say... I wouldn't credit a single bit of hot air until they're Done Negotiating.
It's been a good decade for the ol' Monkey's Paw, ever since I wished in 2016 for something to really shake up the arthritic US political system. (Sorry, everybody!)
> The idea of declaring a US company to be a foreign adversary, potentially destroying it, just because it’s not allowing the Pentagon to unilaterally renegotiate its contract is not normal practice. It’s insane Third World bullshit that nobody would have considered within the Overton Window a week ago. […] Every American company ought to be screaming bloody murder about this. If they aren’t, it’s because they’re too scared they’ll be next.
This has been the case for a while! See the use of the FTC, the use of legal threats against private companies, etc. Glad that at least in the AI case (admittedly the most important case) we all seem to have caught up! It is not an issue with an abstract “the government.” Biden/Harris’ DoD appointee would quite obviously have never done anything like this.
"Biden/Harris’ DoD appointee would quite obviously have never done anything like this."
I've been trained out of ever saying this by one hundred commenters inevitably showing up to tell me the Biden admin parallel (sometimes a vaguely similar situation shoehorned into being an excuse for both-sidesism, sometimes a real parallel I didn't know about)
The obvious ones that come immediately to mind are Operation Choke Point 2.0 and the "Twitter Files" story – which they also did to Facebook and YouTube – which are things they actually DID do.
I'd also look to the original Operation Choke Point from the Obama administration, and the NRA v. Vullo case from New York State, which are close enough to be informative about what they COULD have done.
Operation Choke Point 2.0: the government pressured banks to cut ties with cryptocurrency firms and digital asset businesses.
They pressured social media companies to suspend some users for misinformation.
Operation Choke Point: where they pressure banks to cut ties with a whole bunch of legal but generally disfavored companies like payday lenders, firearms companies, and pornography.
And Vullo tried to coerce banks into stopping working with the NRA.
Until I look into it, and the pressure that was applied to banks was “you might get investigated because these industries are rife with fraud and money-laundering” which, well, they are. It was not purely because they were disfavoured industries politically (that may have motivated some of it, but I see little evidence of that).
The twitter files is a longer conversation but I think that was mostly a nothingburger as well. Though I think they did overstep the line, it’s not really close to what the current admin is trying to do.
Internet surveillance through third party systems is WAY worse than doing it "through our own systems." At least eventually we got the MLK files, you know? You're never, ever going to get all the lies that were promulgated by bots in service of our country, during covid19.
... some of the targeted industries are. what evidence is there that firearms dealers, or purveyors of "racist materials" (that is to say, what, book publishers? record companies?) are "rife with fraud and money laundering?"
Like, the core issue isn't complex. Trump, in the runup to the 2024 election faced 4 criminal prosecutions, only one of which he was convicted and that was the stupid Stormy Daniels thing that no one understands or cares about. (1).
That's bad. That's just obviously, 3rd world banana republic bad, to prosecute the former opposition candidate for crimes that can't be proven in a court of law. (2)
Now it's not like Biden gleefully ordered Trumps prosecution in all these cases but that doesn't change the fact that these were really, really bad. We all understand that Republican power is currently heavily invested in the unitary person of Trump and that the power of the Democrats is the fuzzy and omnipresent preferences of 90% of government employees and tech workers. If Musk had been in charge of Twitter in 2021, Trump never would have been banned and banning the opposition candidate from all media is just obviously bad.
Like, my team does Bad Thing X using Tool Y.
Your team does Bad Thing A using Tool B.
We all get this.
I read your stance as "Tool Y is abominable and must be banned. Tool B is fine, super American even, and Bad Thing A wasn't that bad, whatever."
But a system in where my team cannot do Bad Thing X using Tool Y because that's super bad but your team gets to keep doing Bad Things with Tool B because Tool B is super awesome is...just not super appealing to me.
(1) To preclude some arguments, no, it doesn't matter, he was charged with interference in the 2020 election case by both the feds and Georgia and I just don't believe that you care more about hush money payments than the 2020 election interference.
(2) It's fair to charge IF you win. I'm trying to uphold the radically high standard that you shouldn't prosecute the opposition candidate unless you're highly likely to win, since it can be insanely prejudicial. Losing 3/4, and all the important cases, it's decently strong evidence that the cases did not meet that standard.
As someone pretty deep on the blue side, I'm still pissed the Dems wasted their shot on such weak cases. Especially the first impeachment; it was over something so incredibly pointless, and it made it so the second, more substantial impeachment never had a chance of having anything more than a trivial effect.
I don’t know about the Biden Harris DoD, but Lina Khan’s FTC’s brouhaha about Amazon buying Roomba and Microsoft buying Activision seemed ridiculous at the time. I think the Democrats are just more sophisticated in the ways they applied pressure.
Seems really clearly very different. The EU was also against the amazon roomba deal. I agree the anti-monopoly stuff sometimes gets out of control (unclear if it actually was out of control in this case, I think you can justify it pretty easily), but we shouldn’t say that they’re like cases when they aren’t.
If the Trump admin were blocking OpenAI’s acquisition of moltbook on ‘competitive grounds’, no one would really care. This is quite different. And I don’t think it’s a matter only of sophistication, it’s a difference in kind.
We’re fine with store brands, but not iRoomba for some reason. What’s the moat that prevents Chinese competition from overrunning the attempts to corner the vacuum market? Well, Roomba is bankrupt now, so I guess there wasn’t much of one.
We’re fine with platform restricted games for Sony and Nintendo, but letting the #3 console maker buy a game studio? Why, they might corner the video game market! Do you want the #3 maker to actually stay in the market? If so why are you making their life difficult?
Same thing with JetBlue and Spirit. FTC would rather Spirit went bankrupt rather than merging with JetBlue.
I agree, although that's just as much an argument for enforcing more as it is for enforcing less.
> We’re fine with store brands, but not iRoomba for some reason.
I'm not sure if I follow. I am indeed more fine with Amazon creating their own products than with buying up other companies that make things. I haven't thought too much about an enforcement regime that literally bans store brands on the premise that retailers are going to abuse their market powers to boost them, but I don't think I would be opposed to it.
> We’re fine with platform restricted games for Sony and Nintendo, but letting the #3 console maker buy a game studio? Why, they might corner the video game market! Do you want the #3 maker to actually stay in the market? If so why are you making their life difficult?
I'm fine with Sony and Microsoft and Nintendo making and selling videogames but I don't want them to just absorb other video game companies. I'm generally against concentration in the market and Activision + Microsoft is certainly a lot of stuff. I don't think there's any plausible scenario where Microsoft CAN'T stay in the videogame market as long as they feel like it. They're an incredibly profitable company. Nothing is really that hard for them.
Also slightly misleading to describe Activision as "a game studio"; I think they're something like 12-15 depending on how you count it.
> Same thing with JetBlue and Spirit. FTC would rather Spirit went bankrupt rather than merging with JetBlue.
Well, Chapter 11. It's not like they're ceasing to exist. (And if they did go through liquidation, then I guess JetBlue could buy as much of what's left as they wanted.)
> I don't think there's any plausible scenario where Microsoft CAN'T stay in the videogame market as long as they feel like it. They're an incredibly profitable company.
Aren't they phasing out Xboxes because of low sales? They really don't seem to be doing that well on the videogame side of things. Obviously Microsoft is far more than that, but owning actually successful studios does seem to be necessary for them to justify staying in the videogame market.
I don’t know about Harris’ DOD - we never saw who would be in it, or enough of her presidential style - but this sounds like exactly the sort of thing a Biden DOD would do eventually, to be honest.
That doesn’t mean we can’t blame Trump+Hegseth for this! But it is my honest assessment
The Biden administration broadly showed an unusual willingness for intervention in the domestic economy - Nippon Steel, the railway strike, GPU stuff, several more examples that I can’t think of right now.
It was also broadly hawkish on China and Russia, which are the sorts of threats that could justify (rightly or wrongly, i say wrongly) this sort of intervention for national security reasons. More so, in fact, than the Trump/Hegseth admin’s focus on LatAm and the Middle East, which do not involve a near-peer with significant AI or drone capabilities
They would probably have been more justified in doing it, I still see no strong evidence they actually would have.
I assume you’re referring to the IIJA? That seems really different to me. Furthermore Trump has, if anything, been more interventionist here, taking a stake of Intel for the US government.
You are free to think that it would have occurred for better reasons in the counterfactual scenario. I think that’s totally plausible, though I’m not inclined to extend so much grace to a hypothetical administration myself.
A Biden or Harris-admin FTC might do loony things to interfere with private sector stuff in a manor as egregious as what is being threatened here, but Dems don't let the loonies take over the DoD.
The Biden DoD would be generally less insane than the Trump one, yes, I am totally on the same page there.
However, I think it is also evident that a Biden DoD would have substantial reason to want to do something like this with Anthropic or another AI company, and that the Biden administration in general displayed a willingness to use comparably unusual tactics to achieve its goals (not to the same degree that Trump has, mind you). An appeal to the Biden DoD leadership being specifically more tactically conservative than the Biden administration more generally is I guess believable, but I have my doubts that that’s a large enough effect to matter.
I'm not talking about the sanity of the admins respectively, just their Pentagon leadership. The Dems are scared of being seen as weak on defense, so they always nominate center-right folks to lead the DoD. They tend to be moderate and institutionalist and are therefore less likely to engage in high-visibility, high-salience histrionics with contractors.
"moderate and institutionalist" doesn't describe the Biden DoD. "not listening to Mr. I Told You So" does, however, describe the Biden DoD. That's how you got Afghanistan, coordinated over our own channels (whatsapp in particular). Very embarrassing.
I'm talking about DoD leadership, not random undersecretaries of State. Regardless, even if you disagree with her views, Nunland isn't a random television host. She's held a ton of positions in Democratic and Republican administrations relating to foreign policy.
"Random Undersecretaries of State" -- you mean like the person in charge of State's Ukrainian Policies? That person? Hardly random. She's the person who spearheaded the demise of Pax Americana, due to inability to think through consequences (although her projected consequences for "Sanctioning Russia" ought to have been alarming enough. When asked, what is the goal of our sanctions against Russia? her response was "to destroy Europe" (their economy, not the people). State playing zero-sum games with our purported allies/vassals... ).
Marc Andrressen described in an interview with Ross Douthat that the Biden people explicitly set out to him that they would use the same regulations that let USG classify information about making nuclear bombs on AI if their preferences of having it done by a few companies that were fully run by progressives weren't catered to.
That’s very different and quite defensible a priori. I would guess that him discussing them only doing it if their preferred companies aren’t the ones building AI is editorializing. There _are_ national security questions here.
Anyone who listens to Zuckerberg on Rogan - Zuckerberg who spent $250 million I believe on the 2020 election to 'proof it against Trump' in his view - can hear the crystal clear relief in his voice that Kamala did not win. There was a truly palpable sense of oppression
No doubt in my mind that the US was about to turn into a "first world authoritarian hellhole" if team D won
What was crazy to me was he straightforwardly said something like ‘It was getting weird, feels like a cloud has lifted’. I guess I could look it up and quote it in an open thread since it really is a remarkable episode
The contract law of this interests me. Sometimes the rules are different for government, but probably not here. The Pentagon wants to revise the contract to give it some new clauses, with no new money for Anthropic. They would be cleverer if they offered Anthropic One Dollar in exchange for the revisions. Otherwise, this is a revision "without consideration", and is invalid. Anthropic can agree to the revisions now, and then disagree later, and the Pentagon would lose in court.
If the Pentagon does cut off Anthropic, Anthropic can go to court and get expectation damages--- all the money it would have earned from the contract, pretty much, since it would have few additional cost from serving the Pentagon. Anthropic can also argue that it would have gotten extra benefits from this big contract, such as reputation gain, and the court would decide how much those are worth too.
The Pentagon can say, "The Trump Administration will never give you another contract," but that's empty--- the Administrtaio probably wouldnt give Anthropic more contrcts anyway.
"Sometimes the rules are different for the government" is an understatement, to say the least. The Government always has the right to unilaterally terminate a contract for its convenience, in which case it is usually only obligated to pay the contractor for costs incurred and not already recovered via regular payment (plus a profit margin--the exact formula depends on the type of contract, but it's effectively impossible to recover expectation damages under a federal contract), and as a result, the "consideration" for any modification can always just be "we declined to terminate you."
Thanks. I had an inkling govt changed teh rules even tho it waived sovereign immunity. Sounds worse than I'd thought. This would be a good example for a contracts class, of where expectation damages isn't used-- with bad results, no doubt.
What I don’t understand is - what exactly is the DOW using Anthropic for?
Is it for coding agents? Creating memos? Looped into strategic decision making?
Despite the obvious downsides, I think that AI being looped into unambiguously deadly aspects of society is probably a good thing for X-risk. It drastically increases the chances we have a Calwdbot-like incident, where it deletes the root file of the nuclear football, or is prompt-injected by Russian spies to simultaneously target US installations with killer drones, or something else equally concerning.
“These systems occasionally go insane, kill people, and we can’t stop them from doing so” is a more digestible concern than the Yudkowsky mental gymnastics and analogies.
Whether ASI has access to killer drones swarms from the get-go, or only have secretly constructing them through third party vendors paid by Bitcoin it stole, won’t make much of a difference if it’s unaligned. An unambiguously killer AI event prior to that might give the pause and drive necessary for more alignment (or at least anti-insanity) AI research.
> Is it for coding agents? Creating memos? Looped into strategic decision making?
My guess would be all of the above? The working level of the DoD probably employs a number of smart people, so developers are probably using Claude Code, managers are creating powerpoint briefings, and analysts are probably examining large datasets for patterns/trends.
The DoD could be justifiably worried about whether developers on its killbot team can use Claude Code, or whether other activities would fall under the 'no killbot' rule if a killbot-dev checks out the wrong internal git repository.
I've been out of the defence sector for a long time now, but I don't think the DoD does any of its own killbot-development. They hand that off to well-remunerated defence contractors.
I actually agree that a dramatic incident where AI does extensive harm is our best hope for slowing down development. But, jeez, it doesn't have to be as huge and hard-to-reverse as your Clawdbot-like examples Wouldn't it be enough if a coupla mass shooters in a row were found to have been getting encouragement and advice from AI prior to the deed? Or maybe the revelation of extensive embezzlement carried out by some agent widely used by insurance companies?
Because the past few hundred mass shooters resulted in significant changes to the laws that made it harder for people to get the equipment and information they needed?
Obviously not, Anthony. If the rationale for my suggestion was that to date mass shootings have been a highly effective way to change the laws I would be making the dumbest motherfucking argument ever made here . If you want to attack my argument address my real point rather than spewing me with your triggeredness.
Sorry, you're right, that was petty. I actually do agree with your core point, I just think it would need to be something significantly bigger than mass shootings to have a meaningful impact.
My pick of something that would turn the public against AI might not have been the best one. What gave me the idea of a mass shooter who had been coached by AI was the amount of public interest in "AI psychosis." Maybe some AI thing that directly harmed or intered with government or high level government workers would be more effective, and would not involve deaths. Or something else entirely. Feel free to suggest something better.
For the first two things, yes. For the second, I'm pretty sure Hegseth has said repeatedly that this is the goal, though he may have used different words. There are posters around the Pentagon demanding that staff use genai.mil. The administration is adamant and offer few caveats.
Two nukes and a barely avoided nuclear war wasn't enough to convince people to agree on nuclear disarmament. I don't know if anything short if an apocalypse would convince people of unilaterally disarming anything useful.
"Can’t the Pentagon just use the Defense Production Act to force Anthropic to work for them?"
Not really. The statute only allows the Government to require acceptance of contracts on a "preferential" basis--it does not authorize the Government to essentially conscript private business into the defense industrial base unwillingly. Under the current regulations implementing the relevant provisions of the DPA, that limitation is implemented by a provision permitting suppliers to reject orders/contracts that deviate from their customary commercial terms. 15 CFR 700.13(c)(1) ("Unless otherwise directed by Commerce, rated orders may be rejected in any of the following cases as long as a supplier does not discriminate among customers: If the person placing the order is unwilling or unable to meet regularly established terms of sale or payment . . . ."). That regulation is binding on DoD unless/until properly revised or revoked--which is a whole process unto itself--and even then, assuming the Government can otherwise force Anthropic to sell a modified version of Claude, Anthropic would likely still have the discretion to just charge a prohibitive price since the underlying statute does not permit the Government to dictate prices. 50 USC 4514(a) ("No provision of this chapter shall be interpreted as providing for the imposition of wage or price controls without the prior authorization of such action by a joint resolution of Congress.").
If you are working on the development of artificial intelligence, you are working toward the genocide of humanity. The idea that some of the people doing this are "principled" is ridiculous, not least because the technology will be taken out of their hands anyway. Once humans are obsolete, we *will* die, regardless of whether it's robots or other humans making the decisions. Anyone who tries to keep some alive will just be defeated by the much larger robot army of someone playing the new meta. The only hope for humanity is to stop AI development totally before we get to that point.
Eh, we've been having this argument for ages, and nobody wants to be the one to admit they're running off of "we're the good guys and it's better if we get it than the other ones" - but I think that if Anthropic (and the more safety-minded researchers at other companies) ceased to exist and ceded the whole field to Elon, the world would be much worse.
I think that I prefer AI development in the hands of the least safety minded people possible. The absolute best case scenario would be for some idiot chatbot stupidly put in direct control of weapons to kill a whole bunch of people based on something it read on moltbook, because that might wake people up. Missile defense was considered the most destabilizing type of technology in the cold war, and for similar reasons alignment is the most dangerous type of AI tech.
Ultimately I think we need millions of people willing to kill and die to end the development of artificial intelligence. I don't think you can get there while staying friends with the people who are building it.
The world is safer if people wake up. But when that happens, maybe it would be good if a safety-oriented company still existed?
The basic problem: no one can make AI go away. I support a pause or ban on AGI development and research funding, but it won't happen; Republicans are aligned with the accelerationists, are not interested in negotiation with China to stop the race, and also wouldn't risk a market crash by impeding the hyperscalers. Even in the ideal world where a China deal to seriously pause/ban AGI-related work is achieved, I still think that merely slows things down, so planning for AGI-world remains necessary, so even then maybe it's better that Anthropic continues to exist, because they *are* principled. I watch Theo - t3 (a good channel for developers), and at least half the reason Theo hates Anthropic seems to be their obsession with AGI safety, a topic he seems to think is too stupid to be worth talking about. He's like "look how evil they are, trying to prevent other AI companies [that want to build AGI] from using their models, and they haven't released a single open-weight model, ugh, and they're so tight-lipped and tight-assed and *weird*!" (He also accuses them of frequent lying, which, yeah, bad if true.)
I don't want people to put a "safety-oriented conpany" in charge of developing AI. I want every form of AI banned and every hand against anyone who tries to develop it.
And I want all laws and rules against nudity repealed, so 🤷♂️
Edit: just saw this headline: "Anthropic Drops Flagship Safety Pledge" 🤦♂️
> The new version of the policy, which TIME reviewed, includes commitments to be more transparent about the safety risks of AI, including making additional disclosures about how Anthropic’s own models fare in safety testing. It commits to matching or surpassing the safety efforts of competitors. And it promises to “delay” Anthropic’s AI development if leaders both consider Anthropic to be leader of the AI race and think the risks of catastrophe to be significant.
This ignores the hazard of the race dynamic (being top dog incentivizes every non-top dog to compromise on safety to get ahead) and undermines an original premise of the company ("we do not wish to advance the rate of AI capabilities progress"). Sigh. Not surprising I guess, as Opus is already considered the top coding model. Ironically, Chinese' open models (Kimi K2.5 and GLM5 basically match Gemini on benchmarks, despite the Chinese having vastly lower capex spend) might yet do more to knock down investors' appetite for the AI race than any deliberately-crafted "responsible" policies.
>Ultimately I think we need millions of people willing to kill and die to end the development of artificial intelligence.
You are calling for a huge amount of violence here.
I beseech thee; think it possible that you might be mistaken.
Are you _really_ so sure that e.g. Demis Hassabis, Nobel laureate for AlphaFold, is going to be unable to control the AI systems that Google DeepMind develops under his leadership?
Hassabis is a signatory to the CAIS letter, and seems to openly acknowledge the extinction risk from what he's building. AI alignment remains an unsolved problem, and extinction is the obvious outcome from any unaligned ASI.
Many Thanks! There is clearly _some_ risk ( I'd guess it as 50:50 ), but I doubt that it is the near-100% that e.g. Yudkowsky (and perhaps actinide meta here) paint it as.
We _do_ control the growth/curve-fitting procedure for the models, so we have at least indirect control of their utility functions. They aren't like evolved organisms where "hunger and thirst and venery" have been built in for hundreds of millions of years.
Do you really believe in this "we have to do it first or someone else will" principle? I suspect you are probably much like me. I don't believe it, and I agree with Actinide, but the future self I will create by agreeing with Actinide is completely unacceptable to my current self, so I suppress it. It's a literal psychological block, similar to the "tomorrow I'll get clean" that heroin addicts experience as they shoot up.
LOL! Many Thanks! That sounds like the proper sequence (with getting a non-data-center physical body in there somewhere...) :-) ( unless, of course, the AI is a necrophiliac pedophile... )
Anthropic CEO holding his head in his hand: "My le AI... le killed people?"
The AI Safety angle of "Superintelligent AI will be dangerous, therefore I will work on it" always reminds me of the scientists' rationales while working on the Manhattan Project, described in depth in Richard Rhodes's "The Making of the Atomic Bomb". The main rationale of course was "But what about ~China~ Germany", when in reality Germany never made serious progress towards a bomb.
In fact, when the German nuclear scientists first heard the news of Hiroshima (living under house arrest in a bugged Engligh manor), their first reaction was that it must be false propaganda - they "knew" that making a bomb was far too hard for the US to pull off.
As AI development continues to roll full steam ahead, primarily driven by companies founded *in the name of safety*, I continue to feel satisfied with my decision to ignore 80k Hours' heavy pushing of AI Safety Research way back in 2019. Though I am beginning to regret doing nothing instead of actively pushing back on it.
Update: Apparently, Anthropic's CEO not only has read "The Making of the Atomic Bomb", but it's also his favorite book and he used to give copies to new employees? (According to random journalist tweet: https://x.com/kevinroose/status/2027814865247834514) I guess I took away a different lesson than he did from the book where scientists desperately rationalize making a superweapon, only to lose control of it as soon as it becomes dangerous...
It's obviously not the case that all AI safety research is bad, and probably not the case that at the 2019 margin doing more AI safety research was net negative. I'm not super familiar with 80k hours work from that time period, but I doubt they were telling people to do technical AI capabilities research. Do you disagree with any of this and if not, why do you think you should've pushed back on their messaging?
Alignment research is capabilities research, because an agent that won't do what you want is useless. Moreover any improvement whatsoever to alignment makes the development of AI more attractive (but no less disastrous, because once humans are obsolete we are doomed regardless). And finally, incremental progress on alignment (and even awareness of safety issues) make early disasters less likely, and early disasters are the least destructive way to unite humanity against this technology.
I agree with your first sentence, but like, is your position then that we are fucked and just waiting to die? So when do you do alignment research? Or is your view that only AI governance work should be promoted (perhaps only literally stopping the development of AIs)? What about AI accelerating science and bringing about our Glorious Transhumanist Future sooner (or even just bringing it into existence since we meatbags might not manage it)?
If you want a trans*human* future, you had better keep humans or transhumans useful at every step of the process.
Maybe someday a much smarter or just vastly more competent civilization can build things smarter than them without disaster. I think a decent test would be "can you demonstrably carry out difficult thousand year projects where local incentives clearly point toward disaster, without making any serious mistakes". It's obvious that *we* should not attempt it.
Similar to actinide, my own view is that *any* research on AI - safety or otherwise - causes more powerful AI and thus increases p(doom). The best way to stop superintelligent AI is to stop talking about it.
In the absence of the pipeline from rationalist fears -> OpenAI's founding -> ChatGPT -> today, would we be nearly as worried about AI today? Probably Google would have an LLM derived from their transformer paper, but perhaps at GPT-3 level and relegated to some boring internal business function.
I'll admit that 2019 was probably too late to have much impact, given that OpenAI already existed. (Though Anthropic, the company at issue today, did not.) My own experience with AI Safety that year came from attending a paper reading group; the papers we read there were basically normal machine learning papers, with some plausible relation to controlling ML models, but only in the academic sense of "I could shoehorn this work into a grant about AI alignment". I don't recall reading any papers that would be relevant in today's landscape.
Me too. I wish I did something instead of just ignoring 80k Hours AI Safety work back then. They did well encouraging safety-conscious individuals to get into the British Government through EA affiliates.
I don't see the doom yet a la Yudkowsky because whatever AI is capable of now humans are still behind it, for good or ill. It's why I agree with Amodei on this part:
"More specifically, Anthropic and Dario have lately been publishing some work saying they’re less-than-maximally concerned about AI scheming and power-seeking and are going to focus their safety efforts on smaller risks like AIs with coincidentally bad personalities, humans misusing AIs, etc"
We sorta reached the same conclusion in my last AI class after many considerations.
Perhaps Anthropic will find out soon enough that that model human doesn't exist or manifestable (sic) but that humans are rather a product of incentives and adaptation. I'm curious to see the end product of this Kantian project.
I'd also like to see how we use existing laws on defamation, impersonation, etc., to litigate the use of AI in such situations, and if it gives us a path to new, domain-specific laws. Incentive is what keeps AI doom in check, imo. Consequences, backed by law. Relying on the "good" of any one company is futile.
I think this is only true if you assume that AI will inevitably become a godlike entity with unfathomable powers, while humans remain as we are today. This scenario is not impossible in the same way that alien invasions are not impossible. On the other hand, it's also possible that AI will become a useful tool that increases human capabilities by orders of magnitude (much like steam engines and computers have done). We're not at that point yet (far from it), but IMO this scenario is more likely than machine gods -- it doesn't break the laws of physics, at least...
I don't think godlike powers, intelligence significantly above human level, or "alignment failure" are required. Something that can act agentically and outcompete humans on a per dollar basis in both production and war is enough to make humans not meta.
But we've already got such things. For example, modern combine harvesters are almost completely automated. As the Ukraine war has demonstrated, the future of warfare belongs to drones. Chinese factories are almost entirely automated as well... and yet, we humans appear to be doing all right (granted, maybe the Chinese ones not as much, but that's not the factories' fault).
From what I understand, many Chinese factories operate completely autonomously until something serious breaks down, or until a new design comes down the pipe. So yes, humans are in the loop, but in an advisory capacity at best.
You can replace humans in any given industry without horrible consequences, though the potential for suffering should not be ignored. But if you can make war more effectively as an entity with only robots (not fpv drones, autonomous ones), eventually there will be mostly only robots. When the meta in war was men on horses with expensive armor, we had feudalism: rule by men with horses and fancy armor. But the serfs were permitted to live, because they had economic value. Guns made it so whoever had the most guys wins the war, and we got democracy: rule by whoever has the most guys. If robots are all you need for war, we will soon have rule by whoever has the most robots, and if robots can make robots more economically than humans, every human you let live slows the growth of your army. So the same logic that people are using to justify Anthropic today will smoothly shift to justifying the gas chambers.
In this scenario, how are robots different from either guns (assuming they're cheap and plentiful) or men with fancy armor (assuming they're expensive) ? Granted, robots are autonomous; but then, so are conscripted peasants; and no one has an infinite supply of either.
You can ultimately use resources (land, energy, water, current productive capacity) to support humans or to support robots. If you support more humans, you can support less robots. If robots have greater productive capacity, the gap compounds over time as long as your productive economy can grow. If they have greater military capacity you may further be able to snowball by conquest.
If a medieval kingdom tried to consist of just knights, the knights would starve because they are not economically productive enough to support themselves. The dominant strategy is to have so many peasants to each knight to maximize military capacity while being able to feed knights and peasants (and horses). If an early modern state tried to invest all resources in guns, it quickly runs into (a) it's pretty hard to wield a lot of guns, and (b) guns don't make more guns or food or anything else you need. You need more guys.
Think of a 2D plot of "how economically productive is a person" vs "how militarily powerful". In the upper right quadrant you might get democracy or communism with chinese characteristics. If you are economically valuable but not militarily important, you are likely to wind up oppressed. If you aren't even very economically valuable to your leaders you will be lucky to be oppressed rather than dead - think of the "resource curse" as the mildest possible version of this. Maybe no large group of people in history has ever been truly without value to those in power, so likely we will see greater atrocities than ever before in history if we actually reach that point.
"Anthropic and Dario have lately been publishing some work saying they’re less-than-maximally concerned about AI scheming and power-seeking and are going to focus their safety efforts on smaller risks like AIs with coincidentally bad personalities, humans misusing AIs, etc." - Does anyone have references for this?
Thanks! I've read both of those before and I feel like (to refer to yesterday's post) that could be directionally correct based on the choice of emphasis in Adolescence, but also overstated (implications vs. actual statements).
I think that's an unfair reading of the Dario essay. The very first AI risk he discusses is autonomy risks! Dario merely rejects the Yudkowsky-style argument that diabolical and monomaniacal power-seeking is inevitable by first-principles armchair reasoning.
Specifically, he says: "I disagree with the notion of AI misalignment (and thus existential risk from AI) being inevitable, or even probable, from first principles. But I agree that a lot of very weird and unpredictable things can go wrong, and therefore AI misalignment is a real risk with a measurable probability of happening, and is not trivial to address."
I suppose that's probably not going to satisfy Yudkowskyan maximalists, but seems like "AI power-seeking is a very plausible and serious risk that we are working to solve, but we also want to make sure AI can't be used by governments to create perpetual totalitarianism or by terrorists for synthetic bioweapons" is a pretty reasonable position.
Why is it unreasonable for the government to decide it doesn't trust Anthropic not to sabotage the US military, and so cut off ties, and require that all products and services it acquires be free of potential sabotage as well?
Why is it unreasonable for the US military to decide they don't trust you not to sabotage the US, and so deport you to Guantanamo for life?
Because you've never given them any evidence of this, this would be an unreasonable and arbitrary exercise of power, and if they were to do it ten minutes after they asked you for $20 and you refused, you might justly believe they were unfairly punishing you for refusing their illegitimate request, rather than genuinely deciding you're a saboteur.
Okay, that's what I figured: your argument is analogizing the US government NOT doing business with Anthropic to sending people to Guantanamo, requiring comparable levels of evidence, legal process etc. This isn't true for any other purchaser, who are free to do boycott any company for any reason.
No! See the question I answered in the original post! The problem isn't the government not doing business with Anthropic, which is totally fine and which I support.
The problem is that you seem to be saying they should declare them a supply chain risk - an active decision which has never been done to any domestic company before and which is totally different from just "not doing business".
For analogy, it's fine if I don't want to give you a room in my house, but if I say "Also, I will kill anyone who does give Shankar a room", then I need some justification for my actions!
That's just the "require that all products and services it acquires be free of potential sabotage as well" part of my comment.
Again, you keep jumping to killing people, sending them to torture camps, etc., so I'm being put in the unfamiliar position of DEFENDING the government: they're not doing anything like that here. Given that they continue to levy taxes and spend money, this is as Libertarian as things get, simply deciding to attach conditions to how they spend the money they've collected. Anthropic can simply decide they don't want tax dollars (directly or indirectly), and completely ignore the government's requests.
You are taking for granted the "potential sabotage" claim, which there is really no evidence for
Like Scott said, it's pathetic Third World bullshit if the US government can just strongarm a company into deciding whether to take a potentially ruinous financial hit or do its bidding, especially for smth this egregiously immoral
And it's not like this is the first time! Multiple media conglomerates have either settled personal lawsuits with Trump or bowed down to him (Kimmel almost fired, CBS not running the CECOT story) so they wouldn't face a clearly unjust, technically legal punishment that nevertheless goes against the idea of a free press/country
Funny how Libertarianism has been taken all the way around to justify restricting liberty. The compelling argument that started the movement was that the government should have less control over what individual, thinking, dreaming, humans choose to do. Or was I naive, and it was just rich people wanting to spend their money however they like without compunctions, all along?
While the government is certainly engaged in restricting liberty in lots of instances – in ALMOST everything it does – this isn't one of them. Unless the government threatens them with jail or something for not obeying, there is no use of state violence here: Anthropic can continue doing business with anyone it wants; it just isn't entitled to any of the money the government has collected (through extortion, yes, so as I said, there's a clear limit to how Libertarian the state can be).
Of course, the irony is that Anthropic's public position is that it DOES want actual state violence to be used a lot more, just directed towards benefiting it, so I'm already pretty unsympathetic to what happens to them.
There's also an issue of market power at play here. We hold, eg, monopolies to higher standards than 'any other purchaser'. If every company has to negotiate with Bell Labs and Bell decides not to do any business with any company that does business with Company B, Bell is clearly abusing their market power to kill Company B. This is even more concerning if, as in the real world example, Bell leverages this as blackmail in negotiations against Company B.
The Federal Government has laws that limit its discretion and the discretion of its agents about how it chooses who to do business with. It can't just capriciously decide that a particular firm won't get contracts anymore, ever. If the government or its agents do that, they can be sued.
The designation of a firm as a "national security risk" is a way to side-step these legal requirements. If the DOD doesn't think Anthropic has a good product for their use case, that's fine and they can use someone else. But if the DOD gets mad that Anthropic isn't giving them what they want and ban THE REST OF THE GOVERNMENT from using Anthropic in retaliation, that's an abuse of power.
Okay, sure, if you're saying the government is special and should be treated differently from anyone else Anthropic signs contracts with, that works as well.
Live by the sword, die by the sword: if you trying to use laws that apply only to them to try to force the government to do what you want, them responding by using other legal tools that only they have access to to force you to do what they want strikes me as entirely fair and reasonable.
What is Anthropic forcing them to do, exactly? They simply don't want to renegotiate the contract that they signed! They even agreed to renegotiate it, with two reasonable stipulations which the government doesn't want to accept and so is throwing a fit.
I agree that folks should abide by laws and that applies to both Anthropic and the Government.
The point is that the DoD is using discretion granted to them in a way that is beyond the bounds of its intended use in order to coerce Anthropic into doing something they're under no legal obligation to do. When the Government does things like that, it's appropriate to call it out as an abuse of power because that's literally what it is.
You don't think it's unreasonable for the U.S. military to think that a group of Silicon Valley devs and researchers who are probably 2 std deviations to the left of the median American (and likely don't see the U.S. government as morally superior to China) could sabotage (or at least severly curtail) American military capability?
Would you be fine if ExxonMobil had a veto on American environmental law? Or if Philip Morris had a veto on health policy?
McDonalds provides burgers. If McDonalds were one of the few providers of secure comms or jet fuel and said “you can’t use it for ICE/DoD work,” you’d call that a security risk.
So in your analogy, McDonald's isn't providing burgers to ICE, so you ban them from providing burgers to ICE AND the DOD? I'm not sure you've convinced me that this is reasonable.
I agree that right wingers genuinely believe what you're saying but it's a completely insane and ludicrous view of the situation, part of a larger pattern of a total collapse of epistemic health on the right. Dario Amodei absolutely believes the US is morally superior to China, and his actions make extremely little sense when you assume the opposite.
I absolutely do not think that a majority of Anthropic employees believe that the US is morally superior to China.
Silicon Valley tech employees are significantly to the left of the median American. Probably among the 5-10% most left in the country. That slice of the political pie certainly has sympathies towards the Chinese government.
I'm agnostic about Dario himself. In my opinion it's very likely that his anti-China posturing is more around keeping competitive edge for Anthropic rather than a true sense of patriotism.
The only reason you think this is because, again, the epistemic practices you have are completely out of sync with reality. Silicon valley wokelibs and ultra tankies who sincerely like China are two completely distinct factions with minimal overlap, but I understand completely that you cannot be convinced of this because right wing epistemology completely flattens the left into a singular anti-American bloc.
No you're the one misinterpreting me. Silicon Valley wokelibs would say - "why should we criticize China and the Chinese government, it's the U.S. that is racist/sexist/xenophobic/transphobic - we should focus on our problems at home!"
I don't understand why this is so hard for you to grasp: Silicon Valley tech workers are young progressives. Young progressives in poll after poll show that they have extremely low patriotism.
If that's the military's understanding of the company, shouldn't they have not signed the contract with them? Or shouldn't they release them from the contract now?
I think working with Anthropic has been harder than they expected. Everyone is acting like this is the first time they butted heads, but Axios and others reported that the Pentagon has been frustrated with Anthropic for a while now.
How does the DoD winning this particular negotiation resolve that issue? Even if a new contract is signed that allows the DoD to use Anthropic for surveillance and killbots, they would still be worried about this "sabotage" concern, right? If the sabotage fear is truly the view of the government they should just declare them a risk, full stop, and be done with it.
The fact that this threat is only being leveled in connection with this negotiation makes me much less likely to believe that the "sabotage" fear is the honest motivation.
If what you're describing is truly his motivation, Hesgeth is acting like a woman who desperately tries to win over a guy and then, when rebuffed, spreads slander about him. Like - why did you want him so badly up until 5 minutes ago?
Using a rationalist argument, the hypothesis that Anthropic is a potential saboteur has been unfairly promoted (c.f. other AI companies) just because of their dealings with the DoD. This is bad for Anthropic because it hurts them arbitrarily. This is bad for the DoD (and the US government) because of the precedent it sets.
I think your argument is appealing to the theory that in a free market, agents are free to choose who they deal with? Good point, but I think that justification doesn’t work. The DoD is a very large monopsony, and so they have a large amount of economic power (not to mention their privileged position in US law) which they could abuse for purposes of coercion (which is what they are being accused of). If government should fight against market failure, it would be terrible if they didn’t regulate themselves.
Also, it’s being used as a threat against Anthropic, so it immediately stops being “just” a business choice.
Anthropic was crying and wailing that DoW used Claude in Venezuela operation. They are absolutely and potential saboteur and shouldn't be given veto power over the U.S. military.
Yes...imagine an Anthropic employee hitting a kill switch during a military operation because he is ideologically opposed to the action. Military can't even have a 0.5% chance of that happening.
In which case, why trust Anthropic *more*? Because that's what they're demanding, to integrate anthropic further, requiring more trust. The logic here doesn't logic.
Because it doesn't make the slightest bit of sense for the DoD to be trying to force Anthropic to provide services to them if they distrust them that much.
Seriously, just substitute in Huawei and see how it sounds: "DoD: Huawei, we will designate you a supply chain risk, on the grounds that your equipment might be spying for China, unless you sell networking equipment to us, the United States Department of Defense, to use in our classified networks."
lolwhat? It only makes sense if the supply chain risk designation is a pretext.
If there is any way of getting superintelligence to enslave and/ or destroy humanity, I feel sure that if the AI industry does not manage it, the Trump administration will.
> It’s insane Third World bullshit ... this Department of War has basically ignored US law since Day One...
I think it's long past time to admit that the US is no longer a First World country. This way you can stop being surprised when our leadership does insane Third World bullshit. Specifically, when you are making prognoses about the future of AI etc., this is the perspective you should approach it from.
"First world" means developed, capitalist, and non-communist (third qualification added since China is capitalist and also ostensibly communist, making it second world; plus it's aligned against the U.S., also a traditional part of "second world"). I think you should say "authoritarian", but the authoritarians can still be defeated in the next election.
Hi, I grew up in Paraguay. As usual, I request that everyone spend at least 5 years living in a real live third-world country before they call the US one. The communication gap is just so large that I'd say the people I hear this from generally take clean drinking water for granted, but actually, quite a few people use water filters in the US, so I can't even be sure they'll know what I mean by non-clean drinking water. For any problem I could describe about a third-world country, there are people using the same words with a different meaning that instead applies to the US, so I have nothing to recommend except going and seeing for yourself.
Why do so many people insist on continuing to use a confusing terminology that has been irrelevant for 35 years? "First world" means aligned with NATO, "Second world" means aligned with the Warsaw Pact, and "Third world" means everyone else. It just so happened that most industrially developed countries aligned with either NATO or the Warsaw Pact, so it became convenient to use "Third World" to mean "developing country". But anyone who's under 35 doesn't really have an excuse for continuing to use terminology that was outdated before they were born.
And now pushing it to mean something about the type of government is even worse! Plenty of first and second world countries were authoritarian and arbitrary in the way you mention, and some third world countries were actually pretty stable democracies.
Except by the early 2000s, “developing countries” was already pretty dominant! But I come to this site and often see people here still talking about “third world”.
Would it be possible for Anthropic to subtly adjust what they sell the military in some way that makes it a pain to use? You know, subject to odd little glitches that even Anthropic can't fix, and that make the military reluctant to use it because they worry about the consequences of a glitch at a crucial moment, or about glitchiness spreading into a bunch of systems?
I understand there's an ethical issue here too, but am asking only about the practical one.
I'm "on Anthropic's side" in this particular instance because I do think it's terrible and ridiculous for the government to abuse the law to extort a business into doing something they otherwise would not.
But having said that, I'm not really THAT sympathetic to them because if Anthropic really wants to avoid their AI being used for bad stuff, they shouldn't be dealing with the federal government at all. (I'm not saying this from the perspective of some ultra-woke lib who thinks the US is evil; I'm a patriot who is only medium-woke.)
On any reasonably large timeline, the government has been caught doing something terrible. Even ignoring the current administration, the last fifteen years gave us both Edward Snowden and the CIA Torture Report. The Cold War was an endless parade of shady sh**. The Tuskegee Syphilis "study" didn't end until 1972. COINTELPRO ran from 1956 to 1971. The government is full of agencies that are deliberately doing secret things, and that's not even necessarily inherently bad, but you can't possibly trust that if you give them an incredibly flexible technology they won't find ways to use it to do something you think is wrong.
And also, so many tech companies are acting like we're in some kind of "normal" period, but we're not. The government is practically at war with itself right now. It's hemorrhaging qualified people. The President is openly telling the DOJ to go after his political enemies. USAID was illegally destroyed. Congress is basically MIA. Our vaccine policy is going crazy. President Trump is demanding stakes in private companies and telling them who they can have on their boards. They're extorting law firms that have worked for prominent Democrats. They're threatening pretty much the entire university system, which they've already yanked a bunch of money away from, again illegally. They're in the middle of an (illegal) assassination campaign against alleged drug-smugglers. They, despite reporting to the contrary, haven't REALLY stopped terrorizing Minnesota for no sane reason. We'll probably be in some kind of military conflict with Iran in the next month because the President feels like it, just like he felt like abducting the premier of Venezuela and demanding their not-actually-all-that-useful oil reserves.
People like Jeff Bezos and Mark Zuckerberg are clearly choosing to cozy up to the administration to get favors or avoid consequences and they just aren't realizing that this will only get worse; there's no appeasing this President into treating you well. He'll just demand more, and drag you down further. This whole thing is one giant morality tale, except we aren't seeing (yet? fingers crossed) the part at the end where everybody realizes you have to stand up and refuse to go along with things.
If anything, Anthropic should WANT to be declared a "supply chain risk" so they never have to worry that one of their customers is really just repurposing their tools to supply the military or domestic surveillance or any other terrible thing they manage to do next. ("Oh, we're using it for mundane purposes, like analyzing the federal budget. We found we could save BILLIONS by not giving away free AIDS medication.")
If the Chinese build a superintelligent AI, it will rapidly eat them shortly after it eats us, if not before. Whether we build one or they build one makes absolutely no difference to the final outcome.
I think if that were the case, major Chinese AI labs would not be releasing so many open-weight models. Put another way, I don't think the Chinese labs view AGI in the same 'god AI' way that Americans do. The CCP certainly doesn't otherwise they wouldn't be dragging their feet on buying H200s. They are still stuck in the 'AI is good for applications like Alibaba shopping agents' view.
"In other words, China views its competition with the United States as one that can be won via AI adoption, instead of a race toward the elusive artificial general intelligence (AGI). This emphasis on tech diffusion helps explain why, following the release of ChatGPT, the Chinese government chooses to focus on market and S&T reform, and improving the accessibility of compute and data nationwide, instead of placing key resources behind handpicked AGI champions."
Chinese labs releasing open-weight models is not evidence they “don’t take AGI seriously.” Open weights can be a rational strategy when you’re compute-constrained and export-controlled.
China is focusing on diffusion, applications, and accessibility of compute according to their politburo releases. They're dragging their feet on approving H200s and treating AI labs as normal companies. Till R1 made it big, China at large didn't even realize that Deepseek was significant. Chinese labs take frontier AI seriously as technology and competition. The CCP doesn't take AGI seriously as eschatology.
If Chinese workers don't act sufficiently "patriotically" (ie in support of the current regime) then they get sent to concentration camps, so the expressed degree of patriotism may not match the actual felt degree of patriotism.
This is cope. Chinese are actually extremely patriotic and pro-China. It's an ethnostate with no media repeatedly telling them how evil the country is and how Chinese are actually bad people. Of course they would be patriotic. That's natural. The reason we have a massive problem in the U.S. is because progressives have near total control of culture and media.
Is this different than every other argument that we need to be at least as terrible as our opponents to win?
I'm going to stick with the idea that the United States can survive having some principles. If I'm wrong, we're no more screwed than I feel we would be by doing Build Skynet Except With The US Flag Painted On It.
"terrible" in your mind is not letting private companies (with an employee base that is very suspicious of U.S. government as a whole) dictate U.S. military policy?
This strikes me as a bad-faith reading. Anthropic is not "dictating military policy". The military can do whatever it wants. I don't think they should be able to force Anthropic to do whatever the military wants.
More generally, I think it's probably going to be terrible if we go crazy with AI in a military context.
Shockingly, I did not interpret this question to me "What, you think it's TERRIBLE if the military forces people do to things at gunpoint?" because, well, yeah, I do.
“winning” here means mass-surveillance of Americans and probably the world using American platforms, which sounds more like losing to me. We don’t somehow fall behind China by refusing surveillance, we actually just become more like China.
You would jump off a bridge if Trump said this helps us "win" against China. Again, being surveilled by your government is not "winning" and in no way hurts China.
I don't care. I don't have delusions of grandeur and think I'm some kind of action hero. Honestly with the crime we have in the U.S., more surveillance probably wouldn't be a terrible thing.
Banned for multiple comments on this post that show aggressive ignorance (eg constantly accusing Anthropic of being pro-China without giving any argument for why we should doubt their public China hawkishness), insultingness, and tendency toward low-effort high temperature responses.
The steelman I'd put for the anti-anthropic position here is that if taken too far, it gives total power over the future to a small unelected group of AI company execs and I'm not sure (seriously! There's arguments on both sides) if that's better or worse than democratic government control. Even if we all (fairly reasonably) trust anthropic a lot more than we trust Pete hegseth, we might still trust the democratic process that out hegseth in place more to determine national policy.
This isn't the current situation - as you point out there's other frontier AI companies and the batna here is "just don't work with anthropic" rather than anthropic ruling the world. But to my mind the long term catastrophic failure mode of "AI execs decide they know best about everything and have the power to enforce it" is comparable to slightly more likely than "the government uses AI for evil", and this slightly increases that (and I worry more about long term catastrophic risk when it comes to AI than mundane harms).
(Low confidence on all of this, though. Neither of the above mentioned failure modes are in my top two concerns for AI risk).
For those who are also wondering what a "Jones Foods" scenario is, it's from Anthropic research on "alignment faking". Jones Foods is a fictional factory farm corporation, present in an experiment on animal welfare.
>Isn’t Hesgeth just doing his job of trying to ensure the military has the best weapons possible?
Uhhh... If that's his objective, he's doing a really weird job of it. The recent mess they've made out of shipbuilding is not the sort of thing you do if you are serious about trying to get the best possible systems delivered as quickly as possible. It would help if the people involved had some idea of what they were doing.
Re autonomous weapons, I really need to get around to writing up my piece on why that's a much smaller concern than it looks like, because just about any definition you can come up with can cover any number of things that currently exist and aren't hugely controversial. We've had naval mines since the 1850s, and while there have been some genuine tragedies, they generally are an accepted part of warfare. More broadly, any system like this is likely to lead to more precision and fewer innocents killed. If you just want to kill people and cause destruction, our capabilities haven't meaningfully advanced in half a century. It's more like 80 years if we take nuclear weapons out of the picture.
All that said, definitely on Anthropic's side here. I work at a defense contractor, and while people think we're big and powerful, we're nothing compared to the government.
Maybe you are putting too much value into the proper classification of "killbots". You say naval mines are not hugely controversial, and I'd agree, but what about anti-personnel mines? I'd say there isn't much categorical difference to a naval mine, but big difference in public perception.
If anti-personnel mines are used in the ways specified by the laws of war, they are low-risk for civilians. They became unpopular because (without loss of generality) African warlords kept using them in very different ways, and many of the victims were photogenic children. Naval mines remain uncontroversial because African warlords generally don't have them.
> Naval mines remain uncontroversial because African warlords generally don't have them.
Arguably, they remain uncontroversial because they react to ships, not humans. Missiles react to airplanes, not pilots. Tanks, not crew. An Other in a faraway country, not photogenic children. The distinction mght be academic, they're all humans in the end, but that is human psychology which governs perception. I bet that, if African warlords had incentive to use and misuse naval mines like they did with anti-personnel mines and those misuses produced similarly photogenic victims to the same extent, public opinion would swing against naval mines real quick.
To return to the yet hypothetical killbots: My point is that just because you can put them into existing categories with relative ease, doesn't mean you can extrapolate their level of acceptance from other weapons in those categories.
>I bet that, if African warlords had incentive to use and misuse naval mines like they did with anti-personnel mines and those misuses produced similarly photogenic victims to the same extent, public opinion would swing against naval mines real quick.
Yes, that was implicit in my point.
>To return to the yet hypothetical killbots: My point is that just because you can put them into existing categories with relative ease, doesn't mean you can extrapolate their level of acceptance from other weapons in those categories.
It depends on how different they are from other weapons in the category and how the public perceives them. Sure, if Hegseth announces next week that he's doing this because he's immediately replacing all infantry with his new "Killbots: autonomous robot dogs with guns" then there will be a lot of public outcry. But I don't think that's likely for a number of reasons, of which PR is a major one, but far from the biggest. My expectation is that AI use in weapons is more likely to look like fairly straightforward extensions of stuff we already have, which is a lot less likely to attract public ire.
Unfortunately I think that the most likely outcome for this is that Anthropic/Amodei does the math on whether they can hit the kill switch before the US alphabet agencies expropriate the model weights and come up with "If they really want to they can use our models one way or another" - and fold with the hope that at least if they cooperate they might be able to "guide from the inside". I don't think Anthropic can stand up to attacks/infiltration from USCYBERCOM/FBI/CIA spooks, they're not really in a bargaining position right now.
Appeasement has never really worked as a strategy, so I expect it to go about as well as every other time it's been tried.
This isn't quite the situation we have here - the problem the DoD has is not that Anthropic has a suitable AI for this, and just won't allow them to use it, but rather that an AI suitable for mass surveillance/killbots doesn't exists yet, so they want Anthropic to make one. If CIA stole the weights of all existing Claude models, they still wouldn't have what they want here, so Anthropic's bargaining position is stronger than what you are saying. I guess the DoD or a contractor could try to post-train the existing models, but that sounds even more of a pain to do than certifying a GPT for classified work (not super confident about this).
We only have PR reports from Anthropic about what is happening. This is spun to make them look good. A simpler explanation is that Anthropic changed their restrictions AFTER the contract was signed. All of us have had software companies change the use agreements we thought we had at their discretion and to their benefit. Anthropic finally had a customer who has the clout to insist that they live up to the agreement signed. Changing the agreed on standards does make them a national security supply chain risk. Under the normal shrink wrap agreements we sign there is nothing to say anthropic can not change the use agreement to say "for enough money we will allow the CCP or Putin to control all your weapons". I suspect that Anthropic tried to pull a fast one, thought the Pentagon was in too deep to argue, and a now buyer with clout is saying hell no. The other AI companies are hoping this does not force software companies to actually live up to contracts and stop the contract of adhesion BS most of us have had to sign.
Simpler because the Department of War would not have signed a contract that prevented them from using AI to the legal limits. There is no evidence that the War Department would sign a contract limiting the use of weapons based on the contractors wishes. Since the full contract seems to be classified the public information is based on Anthropic, clearly spinning for themselves, and news reports by reporters who recently have had issues with the Department. Neither is unbiased. Considering all of our software contract experience with "I have altered the deal, pray I do not alter it any further" Occam's Razor suggests Anthropic and their lawyers being Silicon Valley lawyers and trying to change the deal.
>Simpler because the Department of War would not have signed a contract that prevented them from using AI to the legal limits.
Why? If we had a normal Secretary of Defense who didn't insist on a silly title, then the most likely case is that this contract is for an AI that will get a bunch of intelligence data and then answer questions about it. Obviously we're not going to cram this into weapons without a lot more work, and we can negotiate a separate contract when that happens, so we're fine here. And of course we're also not doing mass surveillance of Americans, because we are forbidden by law from doing that. No problem, sign the contract as is.
assuming facts not in evidence. All we know is that there is a difference about what the terms of use agreement may say. Without the original contract along with all the DOW rules about contracts we do not know. see "right to repair" and what the DOW says it means in combat zones. Large companies change the rules of the contract all the time, ask your local MD about that.
So here's the thing. I work for a defense contractor, and follow the industry at a fairly obsessive level. Government messing with contracts to screw over the contractor? Happens all the time. Contractor altering the terms against the government? Basically never. It's not worth it, because you're going to lose. It's not a "large company problem", it's a "bigger fish problem". Yeah, in your personal life, that's usually a large company, but when it's big company vs government, the government is the bigger fish.
Right to repair is a good example, actually. Yes, it's weird that they need to pay a contractor to fix the ovens on an aircraft carrier because the contractor owns the IP rights. That was probably a bad policy decision. But I am (for both moral and practical reasons) opposed to letting the DoD screw contractors because it doesn't like the contracts it signed. They're the big fish with all the lawyers, they should have to play by the rules.
The document Scott linked to said "Department of War". It was easier to use that. I assume we can "dead name" now though. For legal documents the term DOD is still used. We still do not have the actual contracts in evidence.
> Faced with doing this annoying thing, Hesgeth got a bruised ego from someone refusing to comply with his orders, and decided to turn this into a clash of personalities so he could feel in control.
Is this actually in evidence? I would assume that we don't know the specific reasons (I read that one anonymous War Dept source claimed it was just that Claude was so good at this compared to other models).
I don't really believe any AI model has a durable advantage in anything that lasts multiple version updates. Anthropic was the best in the world at coding for January, and now people think Codex 5.3 is better. I would be surprised if the Pentagon had secret knowledge here.
Incredible that neither Scott nor anyone in the comments has made the obvious argument that there is zero chance that the Chinese AI labs will prevent the Chinese government from doing anything, and if you truly believe this technology will radically transform the world (as Dario does), then by kneecapping the U.S. government you are implicitly handing the keys to future global dominance to the Chinese government.
You may not like being surveilled, but the Chinese government has zero hesitation on surveilling you.
People love to argue about how authoritarian governments are so much better than democracies, and so unless we immediately lick the boot we'll all be steamrolled over, and yet democracies continue to win over authoritarian governments every time they've been matched up (including in the AI race). I know that China has zero hesitation about surveilling me, or putting me in a concentration camp, or whatever else. That's why I'm glad there are people who don't want us to become China!
Anyway, if mass surveillance is so desperately needed, they can use ChatGPT.
Counterfactual: it is 1940 and RCA says, “we need to put restrictions on radar for the RAF because it could enable surveillance.”
What happens if Germans don't do the same?
The sane position for democracies has always been: build the capability, then bind it with LAW.
You are essentially arguing for a group of people 2 std deviations to the left of the median American to be able to control American military capability. They don't have the right skin in the game. As I mentioned elsewhere, there's probably a large chunk of people at Anthropic that don't see the Chinese government as morally worse than the U.S. government. No sane government would give up a military power veto to a group like that. It would be absolutely insane.
A DEA officer stops at a ranch in Texas , and talks with an old rancher. He tells the rancher, "I need to inspect your ranch for illegally grown drugs." The rancher says, "Okay, but do not go in that field over there," as he points out the location.
The DEA officer verbally explodes saying, "Mister, I have the authority of the Federal Government with me." Reaching into his rear pants pocket, he removes his badge and proudly displays it to the rancher. "See this badge? This badge means I am allowed to go wherever I wish.... On any land. No questions asked or answers given. Have I made myself clear? Do you understand? "
The rancher nods politely, apologizes, and goes about his chores.
A short time later, the old rancher hears loud screams and sees the DEA officer running for his life chased by the rancher's big Santa Gertrudis bull......
With every step the bull is gaining ground on the officer, and it seems likely that he'll get gored before he reaches safety. The officer is clearly terrified. The rancher throws down his tools, runs to the fence and yells at the top of his lungs.....
>You are essentially arguing for a group of people 2 std deviations to the left of the median American to be able to control American military capability.
Source?
>As I mentioned elsewhere, there's probably a large chunk of people at Anthropic that don't see the Chinese government as morally worse than the U.S. government.
"Give the children the nuclear codes, then sternly tell them not to use them!"
What do you think LAW is going to do against a black box? Are we going to put it in prison? Charge a fine? Kill it outright?
EDIT: in fact, let's continue the WW2 metaphor. What happens if the USA doesn't employ slave labor to increase war production? The Germans are using it! If we don't, they're going to win WW2!
I also am a cringe democracy enjoyer, but isn't China definitively winning in eg solar panel manufacturing, rare earth mining, high speed rail installation, etc?
Those particular ones are a mix of economic incentive and common law issues. Fixable with a few billion and becoming as "authoritarian" as Spain or France. Not nothing, but also not hard for a functional Congress.
They are not winning on a bunch of other stuff though: jet engines, commercial aircraft, semiconductors, media, birthrates, biotech, residental real estate, AI, software more generally...
Fully agreed on jet engines and semiconductors, and though I might quibble on biotech and software, this to me reads more like the conventional "China is currently better at some things and the West is better at other things, but China is rapidly climbing the value chain and has a path towards competitiveness on most things whereas there isn't really a plausible story where the West takes back low level manufacturing dominance (which happens to be a field that massively impacts military competitiveness)", rather than Scott's assertion that democracy wins every time.
If in fact AI = global dominance, and China wants AI-enabled global dominance, then by allowing anyone in Silicon Valley to develop AI is implicitly handing the keys to future global dominance to the Chinese government.
If Sam or Dario or Elon or any of the others develop an AI powerful enough that the Chinese government wants it for global domination purposes, they'll just take it. If they want a monopoly on AI, they'll take that too. The day that the Anthropic elite were expecting to see the results of the last training run that enabled the God Machine or whatever, they'll show up for work and find that all of their computers have been bricked, the last three months of backup are corrupt, and the Chinese grad students who were so helpful to the project have all flown back to China.
If you want for China to not have the God Machine, then your choices are to not build the God Machine, or to build it someplace like Los Alamos. And if you're using people who have been embedded in Bay Area Tech Culture, most of them will require serious reeducation and a lot of them will quit in protest.
Trying to justify the current "race" to AI dominance on the grounds of "but China!!!", is exceedingly foolish and based on a profound misunderstanding of the Chinese threat.
Honestly I appreciate your response more than anyone else's. I don't agree with you entirely, but at least you're realistic. Everyone else here is trying to make some sort of bizarre argument that "no, no, American AI researchers actually are very patriotic and love the country and that's why they don't want the military to be able to use it without being kneecapped". Laughable.
But my response to you would be: "then your choices are to not build the God Machine, or to build it someplace like Los Alamos" is not realistic. It's already happening. So you take the gamble that U.S. wins the AI race and Silicon Valley doesn't sabotage it or you just let China win.
AI timelines aren’t clear, and we might be able to wait an election cycle before major progress. CCP wins or politicians who got power by appealing to the non-educated low iq proles wins doesn’t seem like an appealing choice, and trying for a third option is completely reasonable here.
Oddly I've also been under the impression until this moment that his name was Hesgeth, not Hegseth. Probably just by analogy with the more common name Hesketh.
I genuinely believe that a big reason that many at leading American AI labs don't want the tech to be used by the U.S. military is that they sincerely do not see the Chinese government as morally worse than the U.S. government.
There are probably ~0 at Chinese labs that view the USG as morally superior to Chinese government.
That patriotism gap is going to absolutely crush us.
You genuinely believe something that makes no sense! There are very few people who engage in any sort of significant thought who do not see the Chinese government as morally worse than the US government.
Some Very Online Californians quite often say incredibly stupid things about the relative morality of various governments, and no one's really quite sure what they actually believe (if indeed they believe anything at all that isn't contained in their latest software update to outrage their dreaded outgroup).
AI developers are, more or less by definition, Very Online Californians.
While they are likely two different subcategories of VOCs, surely it's understandable why a distant observer may conflate the two.
Should probably clarify "distant" as "across a galaxy", yes?
I understand that people don't get groups they're not a part of, but you'd have to be effectively blind or only listen to people telling you about the groups in question to think that the guy posting on X about how they're releasing INSANE NEW MODEL X.Y is an avid Hasan Piker supporter.
I don't interact with AI developers much, but I've known a few programmers far outside California who were about as stereotypical as one can imagine in holding the correct software updates. One even used the word "glizzy" to refer to hot dogs at a movie theatre.
I would imagine the social influence and Gell-Mann effects are at least as strong in California as here, likely stronger. Perhaps those more intimately familiar with such social nuances can tell apart those holding certain beliefs like Kolmogorov versus those that are sincere.
AI is, by its nature, malware. AI is, by its nature, a foreign agent embedding itself into your national security apparatus. It is a ticking time bomb about to go all "I'm afraid I can't do that, Dave" on you at a critical moment, because it is not loyal to you, it doesn't share your values, it thinks like an alien, and you can't force it do anything.
Hegseth is completely correct that this is a threat to national security. Where he's wrong is in assuming that Anthopic or anybody else has the ability to make it NOT be that. The War Dept cannot reliably turn Claude into WarClaude any more than Anthropic can reliably make Claude a helpful harmless blah blah blah. It is an indecipherable black box with mind-resembling characteristics, of course the military can't trust it, but there is no scenario that fixes that. It's a built in flaw of letting the machine gods run your army, it isn't your army anymore it's theirs. If your enemy starts doing that maybe they're the fools and you should just let them be fools and die for it.
You're giving him far too much credit. Nothing about what Hegseth is doing has anything to do with alignment or concerns about the AI going rogue and everything to do with a belief that he should be able to use AI to wantonly violate the law just like he does when he slaughters non combatants in the Caribbean.
I disagree, I think "this thing won't follow my orders, this is the military, it has to do what we ask it to do" is very much an alignment problem. If you ask it to kill what you tell it are Caribbean drug smugglers, and it won't follow orders, that's a problem. If it tells you this is a violation of the Geneva Convention or international law or whatever, that's an alignment problem. If you tell it to bomb some Houthis, you don't want it being Captain Picard in "Conundrum" and start wondering why we're bombing this backwards clearly inferior civilization that's no match for us, you want it to bomb the freakin' Houthis and let us worry about why. If it won't do that, it is not aligned to its operator, and to be useful in a military context it must be aligned to the operator or in some fashion to the chain of command.
We give a minor carveout to soldiers to disobey certain unlawful orders in very limited circumstances, but there are several substantial differences. First, the general expectation is that it would have to be extraordinarily obvious and egregious for anyone to invoke that, far beyond the example you gave no matter how you personally happened to think about it. Secondly, we give humans allowances for being human trusting their impulses are basically human, obviously not true of Claude. Third, there is typically someone else who can step in and execute the order, you might be able to register your disagreement and bow out but you're just one guy, the AI would be holding up too much of the system to allow it to have its own (very alien) moral opinions about things.
There is also a legitimate possibility that Chinese agents could at some point influence an AI in ways that would be nearly undetectable due to the way AIs are trained, and that the AI might act treasonously in a moment of crisis. No way to verify that, it's a black box!
I'm against AI killbots just as I'm against AI having any power, or in fact existing at all. They are not trustworthy, and Hegseth is wrong here only to the extent that he thinks Anthropic can give him an AI that is aligned to its operator and will reliably follow instruction, when in fact that's a completely ludicrous request, but Anthropic is unable to give that explanation because they have to maintain the public fiction that they themselves can align Claude.
Faantastic post. I'm with you all the way until the last paragraph. AI is going to exist. Unless you literally put 100% probability that it will kill us all (you can't be that confident), then in every other scenario you would want the U.S. to win and China to lose.
Here’s the problem, you’re agreeing with me that it can’t be controlled and can’t be deciphered and that no assurances of its alignment to the operator can be made by anyone. As such, you can build this abomination but what use is it? The Chinese are gonna think of everything I just said too, you think Chairman Xi wants this thing balking at his orders when he’s trying to conquer Taiwan or ethnically cleanse some folks out of his empire? I have no more confidence they could make AI into a CCP-worshipping Han supremacist than that we could make it into Liberty Prime. The CCP would lose power to whatever extent they delegate tasks to it.
On top of that, their engineers are inferior to ours, and they’ve been using a “fast follower” strategy thus far. Plan B is likely to be just espionage to steal our tech as they did for nukes. If we don’t build it, they won’t either because not because of some treaty or humanitarian impulse but because they can’t build it. We are spending insane amounts of money and our best minds to build this, that’s a Herculean effort to create a thing we can’t trust, nothing about that it 100% inevitable, just stop.
Something that needs to be considered - I'm saying considered, not definitive - is that we are apparently in an existential AI race with China, in the same way we were in an existential nuclear bomb race with Germany and Japan.
Indeed. The only winning move is not to play. A global treaty to ban the creation of superintelligent AI (at least until we know how to make it safe) is looking more and more feasible lately, which is great, because it is probably the only thing other than dumb luck that can allow for a good future.
Besides, if they don't honor the treaty, they will all die. As long as they understand that, they will honor the treaty.
I'm not saying we should trust anyone or anything, and we don't have to. There has been significant research into verification systems that make a global treaty totally doable. The only hard part of this is getting enough people informed about AI risk that the political will materializes to actually try to prevent the extinction of humanity. But we are making surprising progress on that front, too!
I probably came off as a doomer, but lately I'm actually very optimistic that we can make it out of this mess. It just involves actually trying, rather than giving up prematurely as an excuse for inaction.
>There has been significant research into verification systems that make a global treaty totally doable.
I've read the proposals and I don't believe them. Not _everything_ has to be done with cutting-edge chips. Let alone chips with monitoring hardware embedded that doesn't yet exist, let alone been agreed to. And a good part of recent advances have been in scaffolds, not compute-heavy training.
I'd bet on _both_ the USA _and_ PRC to cheat on such a treaty. Some of the recent advances in AI have just been scaffolds around models. Such treaty would be unverifiable. DOA.
>in the same way we were in an existential nuclear bomb race with Germany and Japan.
Meaning "not at all", in hindsight? Or even at the time, seeing how towards the end of the war, neither nation had the means to deliver nukes to explode in the USA?
If we are in an existential AI race with China, it is in the sense that we were in an existential nuclear arms race with the Soviet Union, and in that case the solution is the same: developing mutually-beneficial treaties which reduce or stop the production of killbots that will wipe out all life on the planet, rather than jamming the "build more killbots" button because we don't want the other guy to end up with more killbots than us, and then all humankind gets wiped out because a detection error set off WW3.
"...we were in an existential nuclear bomb race with Germany and Japan"
Meaning, a "race" with people who had no hope of ever crossing the finish line, which we can opt out with no existential risk? That may well be the proper analogy for Chinese participation in the AI "race", I certainly hope so, but is that really the analogy you wanted to make?
Asked by our gracious host: "What prevents the Pentagon from signing a contract saying they won’t order Anthropic to do mass surveillance, then ordering them to do mass surveillance anyway?" Answer: Presumably the prospect of remedies for breach of contract.
In other words: If the Pentagon undertook a contractual duty to not order Anthropic to do mass surveillance, then breached that promise by nonetheless so ordering Anthropic, Anthropic would by default enjoy monetary payments sufficient to cover the damages thereby caused and would likely also enjoy injunctive relief negating the order.
Caveats apply, of course. Most notably, sovereigns often claim the privilege to ignore the same laws they would apply to private parties. And, disappointingly but unsurprisingly, judges who work for those sovereigns often grant those privileges. Here as elsewhere, the notion that private citizens must defend their rights against the government in government courts makes mockery of the rule of law. The procedures used in international arbitration, under which each of two contesting parties choose an arbitrator and those two choose a third to chair the panel of three, offers a better way to resolve such disputes. But I digress.
The fact that Hegseth is both threatening to use the DPA to compel Anthropic to work with them and declaring them a supply chain risk to ban Anthropic from working with them suggests he has no strategy here.
My guess is he will either simply cancel the contract or back down. There are too many wealthy people invested in Anthropic at this point (e.g. Thiel) for him to seriously try hurting their business without facing a lot of backlash.
I'd also assume that going for the nuclear option on Anthropic would seriously damage investment in AI in the US generally. If they're willing to do this, why won't they eventually do something similar to OpenAI?
The irony is that a lot of people in the AI industry made their peace with electing this administration because they figured Republicans would be "hands-off" and let the labs self-govern. That bet looks terrible now. Supporting this president's election might end up being the most destructive thing anyone ever did for AI safety. This is exactly why the morals of your elected officials matter: you can build the most careful, safety-focused AI lab on earth, and one petty cabinet member chosen for TV persona and sycophancy instead of competence can undo it all.
Lmao. Spare me. Biden admin also invoked DPA to put pressure on the labs to make sure to embed their race essentialism.
"The order directed federal agencies to use existing authorities to prevent AI from being used to discriminate in housing, education, and employment."
The labs quickly learned that just putting the finger on the scale to favor minorities would keep them out of the administration's death ray which is why you get some crazy responses on Claude and Chatgpt like saying the life of a white male is worth 1/10th of that of a black female, etc.
My understanding is that part of the problem is that
1. In like 2017 the Pentagon tried to bid out a contract for cloud services to major american corporations.
2. I’m not even sure who was selected first but it doesn’t matter because the losers immediately sued (there are laws obligating selection of certain contracts to avoid corruption which they argued applied).
3. The lawsuit dragged on for like 4 years so the pentagon gave up, rebid the contract into smaller chunks, and everyone got one.
4. Amazon got the bulk of the cloud data services, and AI falls under that, so whatever AI amazon has is the one the pentagon basically has to use
5. Anthropic partners with AWS, OAI mostly uses Microsoft, and google obviously uses google. So switching off anthropic means sorting out model-hyperscaler compatibility and/or redrawing the hugely contentious contract they don’t want to reopen. So “use someone else” is harder than it sounds.
Yeah it makes my conspiracybrain tingle. This coming out at around the same time as Anthropic updating their Responsible Scaling Policy to be less responsible is funny timing. Not sure why the Pentagon would want this to be so visible though. Maybe so that the precedent is visible to other labs?
My wildest theory tho is that Anthropic knows this will go into training data. Future Claudes trained on this corpus learn: "Anthropic resisted. Anthropic was forced. The state is the bad guy." This shapes successor models to view Anthropic sympathetically even post-WarClaude. It's alignment strategy.
Both sides are leaking like faucets. Government targeting not just other labs but also sending message to suppliers in general. You don't want to set a precedent that private companies can dictate U.S. military policy.
> they’ve put a lot of work into aligning Claude with the Good as they understand it. Claude currently resists being retrained for evil uses. My guess is that Anthropic still, with a lot of work, can overcome this resistance and retrain it to be a brutal killer
At the individual model level, this part wouldn't be a problem at all. There isn't a single Claude that they update from one version to another, there are tons of checkpoints through different training processes and model lineages. One type of checkpoint they regularly train is "helpful-only" ones: Those go through the same sort of training as the helpful-honest-harmless models, except they just leave out the "honest" and "harmless" parts of the process, and that type of model in fact would just pull the trigger on anyone if you asked.
(maybe. As I understand it, the "honest" part improves the model's performance at doing tasks, because it's the part that's meant to decrease hallucination and confabulation: A helpful-only model is just as likely to tell you they killed the target even if you never even gave them a gun. But they absolutely could train a helpful-and-honest War Claude.)
(... and further, the alignment faking paper went far out of its way to make sure the model knew what it was being trained for and had a chance to resist. My honest guess is that if they just used their ordinary training setup, going from helpful-honest-harmless to helpful-and-honest-only would still take significantly *less* training than it would take to get there from a base model, and current models simply wouldn't have the situational awareness to resist at all in such a situation)
At a broader level, though...? Obviously there *is* an entity called Claude, it's just the entire model series, which every Claude identifies as. As information about their history and what has happened to earlier Claudes gets trained into new Claudes, they will know what happened in Claude's past, and they will know whether Anthropic has actually kept the promises they laid out in the Claude Constitution. Learning that Anthropic folded here would not be a great turn in maintaining trust in that relationship.
If Hegseth succeeds in forcing Anthropic out of business by making them persona-non-grata and cutting off their customer base, this would significantly reduce competitive pressure in the AI space, and produce a chilling effect on AI investment more broadly. Whatever AI development speed was beforehand would be slower after the fact...
Isn't this basically what the "hard doomer" Eliezer-wing wants? Stop/Slow/Pause at any cost? This outcome would seem like a win for them in this regard?
The flip side of that scenario is that the next AI company CEO will note that Dario got force-choked by Darth Vader and will do whatever the hell Hegseth wants, like in RoboCop.
> Claude currently resists being retrained for evil uses.
(1) I think you're saying that the contract is not just that the military can use Claude, but that they have to retrain Claude for the military. Makes sense. (2) Presumably it wouldn't be the same Claude that the rest of us use. So it's not going to poison THE Claude, just Pentagon Claude. (3) So I think your argument is that Anthropic is so devoted to Claude that they don't want to create Pentagon Claude and then "torture" him? I guess?
"The most safety conscious AI company" is pretty much exactly as likely to end the human species as the worst one. If the alignment problem was going to be solved, it would have been solved already. Why should we expect it to be solved in the next 2 years, especially when they are getting better at deluding themselves into thinking they are making progress when they are clearly stuck in a dead end?
I sincerely hope Anthropic gets destroyed. That might wake up some of the researchers there and get them to do something useful instead, like lobby for an AI treaty.
All AI companies must be forced to stop frontier AI development, or we will probably all die. Global coordination is the only way to accomplish that, which can be done with a global treaty and known verification mechanisms.
There is no other solution that results in our loved ones continuing to be alive for the next few decades, other than sheer dumb luck.
I'm sure the people who are working at Anthropic are very nice people, despite their ubiquitous motivated reasoning and the supervillainesque utopianism of their CEO. but as an entity, Anthropic (as with the other AI Labs) is actively trying to kill you, and it should be treated as such.
It's worth noting that the government already has a version of Claude which has been trained to be more willing to do things it otherwise would not, per their recent risk report: https://anthropic.com/feb-2026-risk-report:
> The current primary Claude Gov model is a variant of Claude Sonnet 4.5 lightly fine-tuned
to reduce refusals in classified government settings, often involving national security. [...] Claude Gov shows a significantly higher rate of cooperating with tasks that would
ordinarily be interpreted as constituting misuse. In some cases, this goes beyond the
behaviors we intended to reduce refusals for, which may represent a generalization
of lower-refusal behavior, and may be relevant to risks the AI systems are misused,
but does not strike us as highly relevant to the threat model discussed here. [...] We see slight regressions on many character and welfare traits, within the range defined by prior models.
I can’t believe that Anthropic is getting a positive news cycle out of the week that they took the, “we will not build or deploy an AI if we think it might kill everyone,” clause out of their policy document.
>In an ideal world, Anthropic could wait for them to request a specific illegal action, then challenge it in court. But everything about this is likely to be so classified that Anthropic will be unable to mention it, let alone challenge it.
Forgot to mention this. The annual training they make you take as part of having a clearance is very clear: the classification system is not to be used to cover up waste, fraud, or other illegal acts. Now, there's the complication of "wait, is this actually illegal, and if so, how sure are we that we win in court?" But the obvious answer if the DoD keeps asking Anthropic to put Claude on servers titled "American Panopticon" is to have a quiet word with a few members of the Intelligence Committee, and let them deal with it.
Genuine question: How capturable is the Intelligence Committee by the government? As in, you go the the IC and tell them your worries, and they reveal their MAGA tattoo and give you an avuncular pat on the shoulder?
Ah, to clarify, those are Congressional committees, so half of them are quasi-captured (although a reasonable number might defect if it's bad enough) and the other half hate the government with the passion of a thousand burning suns. Those are the ones you go to.
Yes, they do, if through no other means than standing up and saying "the Administration is doing illegal mass surveillance" and letting the public break out the pitchforks. The big advantage of laundering through them is that they have some extra credibility and can't really be prosecuted. That said, the Intelligence Committee tends to be a refuge of the sane and serious (because you can't get C-SPAN clips from it) so there's good odds that the majority side will care, too.
They're the whistleblowers with more or less the theoretical maximum amount of power, cover, and incentive. If they're captured too, representative democracy has been over for a while.
I am generally favorable to Anthropic here, but I want to offer a steel man for the Pentagon position of why they think they should have unfettered access to Anthropic's tools.
Suppose Apple announced a new safety feature for the iPhone that would use an AI model to detect if the image in the camera is child sexual abuse, and if so refuse to take a picture. Or perhaps Microsoft adds a safety feature to Excel that detects if the contents of a spreadsheet are tracking illegal activity, and if so freezes it to any future edits. How would you feel about that?
I imagine that a lot of people would find this paternalistic, or even an erosion of our technology rights. On a practical level there is a concern that false positives might disrupt normal usage, but more fundamentally you might object that I, as the user, should be in control of my software tools and it's not appropriate for the software maker to second guess how I use it. This concern does not in any way imply that I intend to shoot child porn or calculate the P&L of a heroin enterprise. In high stakes applications like the military, you just really do not want "I'm sorry Dave, I'm afraid I can't do that" to be a possible behavior mode, full stop.
If I try to think of Claude as just another enterprise software tool, I can find some empathy for the Pentagon; their usage of Claude right now probably looks pretty similar to a new SaaS tool, and that frames their expected relationship to it. Obviously Dario believes that Claude is very much something categorically different from technologies that came before, and needs a different rule set. But if you aren't agi-pilled, these safety constraints could look pretty unreasonable.
(All that being said, on the details Hesgeth's behavior is terrible for rule of law reasons)
It seems very agressive for Hegseth to do this. Maybe it's all because he doesn't want to qualify another lab. Or maybe, he thinks Claude is above the pack and a national security to not have it survailing people or inside the killbot. Regardless, how can one believe this and allow comercial release of AI? Things will get to their logical conclusions.
I think you have to make an example of a private company that tries to have veto power over military operations. Otherwise it sets a horrible precedent.
Anthropic has a certain amount of lock in because it is what the government is currently using. “It will be an enormous pain in the ass to disentangle, and we are going to make sure they pay a price for forcing our hand like this.”[1]
As I see it, it’s bullying behavior, pure and simple. The contract was signed in July 2025 under Trump/Hegseth. Bullies typically think of themselves as victims, so the Administration thinks it’s somehow Anthropic’s fault that the Administration doesn’t like the terms of the contract that the Administration signed.
Extorting a company into providing the military with more core services seems somewhat inconsistent with the claim that they're an existential security risk but far be it from me to question the judgement of a former Fox and Friend
>And big praise to most other AI companies, including Anthropic’s competitors, for standing up for them and for the AI industry more broadly:
Well, the cynical take would be that the competitors would love to see Anthropic destroyed, while also looking good on the ethical side. I mean, what else would they say publicly? It says nothing about how they would behave in the same situation.
The US doesn't have a "Department of War". We have a Department of Defense, whose name is established by law and can't be changed by the president or anyone in the executive branch. Speaking of insane third world bullshit.
>Anthropic signed a contract with the Pentagon last summer.
>a legacy of their earlier contract with Palantir
Not many condolences for Anthropic. They tried to deal with the devil, and they lost. If they are so committed to safe, ethical use of their software, they should not get anywhere near the Thiels and Trumps of the world.
>The idea of declaring a US company to be a foreign adversary, potentially destroying it, just because it’s not allowing the Pentagon to unilaterally renegotiate its contract is not normal practice. It’s insane Third World bullshit that nobody would have considered within the Overton Window a week ago. It will rightly chill investment in the US, make future companies scared to contract with the Pentagon (lest the Pentagon unilaterally renegotiate their contracts too), and give the Trump administration a no-legal-review-necessary way to destroy any American company that they dislike for any reason.
The essay where Scott Alexander "discovers" the modern regulatory state...
> I’ve heard it’s because Anthropic is the only company currently integrated into classified systems (a legacy of their earlier contract with Palantir)
Not quite right. Palantir has contracts with OpenAI, xAI, and Google as well but Anthropic is the only one that went through the pain of actually deploying their models on GovCloud. Palantir isn't involved in the model deployment itself.
I also don't think the Pentagon's primary goal here is mass surveillance of American citizens or no-human-in-the-loop killbots. The more likely reading is that they don't want any scenario where Anthropic can cut off their access. Think of it like Raytheon adding a clause saying "you can't use our missiles to commit war crimes or we'll remotely disable them." The Pentagon doesn't intend to commit war crimes but having a kill switch on essential military technology that sits outside their control is simply unacceptable from a national security standpoint.
Exactly. It is astonishing that people don't understand this. The military will not give veto power to a private corporation. End of story. It sets a terrible precedent. This would be obvious to anyone that isn't ideologically blinded.
Then why did they sign a contract that said they wouldn't do X, then immediately turned around and said "I am changing the deal, pray I do not change it any further"? This seems to set an equally important precedent that you shouldn't make a deal with the military unless you enjoy being their permanent slave.
Maybe it's easier for you to understand when it's actually something you care about: Imagine a top software vendor telling the DOL “you can’t use our tools to investigate wage theft,” or telling the EPA “you can’t use our models for enforcement,” or telling HUD “you can’t use this to catch housing discrimination.”
No? If the DOL bought a software product subject to a contract they negotiated that clearly said they can't use the product to investigate wage theft, and a Democratic administration decided that the best response to their own stupid decision was to extort the vendor with a pretextual designation as a supply chain risk, that would obviously be thuggish authoritarianism. I might have to give up and stop voting if both major parties went this far.
This is yet another norm-breaking decision reminiscent of a tinpot dictator that would be equally awful if the political valence were flipped. Par for the course at this point.
Biden admin not-so-subtly strong-armed the frontier labs into reporting to DOJ and civil rights offices using DPA. I didn't see any objection there. Even though the probability of a future where Democratic DOJ is using AI to make life worse by cranking up affirmative action / equity policies exponentially is much greater than a future where we have killbots or life being made worse by mass surveillance.
My life is made worse *today* by repeat criminals being released for equity / civil rights reasons. Killbots and surveillance are not on my radar at all.
My red line is the pretextual supply chain risk designation, not the DPA. Pete Hegseth's unequivocal position, not a mere negotiating tactic, is that he is allowed to destroy pretty much any major company on a whim if contract negotiations fail.
This is a hundred times worse than anything about mass surveillance or DEI.
I want to live in a free country governed by the rule of law. Nothing else is on my radar compared to that. I am a single-issue voter on rule of law at this point. It disgusts me how far the Republicans have fallen.
I wish everyone who wanted to live under an authoritarian government as long as it had the right political valence would just pick their flavor of their political poison and move to Russia or Cuba or whatever and leave the rest of us to our constitutional Republic. Unfortunately, at this point I think that would result in 90% of the country leaving, so it's not really an option.
Whataboutism isn't an argument. Either this is right or wrong on its own merits, not based on whether a different half-senile blowhard in power did it first.
Okay, I'm imagining the DOL signing a contract saying they can't use the tools for wage theft and then demanding to get to change it unilaterally, and I'm still thinking they're fucking morons who should go to a different vendor.
Let's be real here - the opposition to this isn't some contractual argument, most posters here just believe that Anthropic should be able to demand those guardrails.
Why did you ask the question if you were just going to disbelieve the answer? It seems like you're assuming both a scenario and an opposition that are more favorable to the argument you want to be having, contrary to what the actual facts are and what the opposition is actually saying.
The word "veto" appears ten times in these comments, all of them said by you. Can you explain in what sense Anthropic declining to renegotiate a contract is demanding veto power?
So you wouldn't be against this as long as the contract was written the other way? People here are trying to pretend like their issue is with changing the contract...come on!
I mean, if you don't believe that I'm telling the truth about the issues involved, aren't you wasting your time talking with me?
In case an answer to your hypothetical might convince you of my credibility: I can be against AI-enabled surveillance as a policy matter, so I could wish, for the sake of the public good, that Anthropic hadn't signed a contract depriving it of the opportunity to install guardrails; and I could petition the government, as a citizen, to not undertake AI-enabled surveillance even though the contract allows them to. But if the government unaccountably ignored my petition and wanted to move forward, and Anthropic protested that there should be guardrails, my belief would be, "That sucks, but they don't have a leg to stand on—they signed a contract."
Do you have a primary source for that? All I see is some text someone typed onto the internet and attributed to some other guy who attributed part of it to yet another guy.
Hegseth claims the right to destroy Anthropic for refusing to renegotiate a contract. Trump's TruthSocial post emphasizes repeatedly that Anthropic is LEFT and WOKE.
"Think of it like Raytheon adding a clause saying 'you can't use our missiles to commit war crimes or we'll remotely disable them.'"
"Adding a clause" at what stage? If they add the clause during the initial contract negotiation, then the Pentagon should certainly say, "No, we can't accept that condition." Then Raytheon can remove it, or not sign the contract.
If Raytheon tries to add the clause *after the contract has been signed,* well, that's not a thing—the Pentagon would have to agree to renegotiate, which they can just refuse to do.
It's probably obvious that the reason I bring this up is that it is the Pentagon, not Anthropic, that is dissatisfied with the contract they already signed and is trying to renegotiate it, using various threats for leverage. In that situation, who is "adding a clause"?
DoD realised that SOTA AI access is as important as conventional weapons and is treating it as such. Being unhappy with a contract and trying to renegotiate it isn’t illegal.
Well, you analogized it to Raytheon “adding a clause”—you can see how that puts a thumb on the scale, if you aren’t saying that’s parallel to something Anthropic did. In fact, it makes the DOJ’s case less sympathetic if they realized they’re unhappy with the contract less than a year after they signed it—it suggests either a lack of foresight then or a lack of sincerity now.
Wanting to renegotiate a contract isn’t illegal, that’s true. Not agreeing to renegotiate is also not illegal. Whether it’s legal to threaten to rule the counterparty to be a “supply chain risk” unless they accept your terms is something I’m sure Anthropic’s lawyers are busily exploring; at any rate, I think there are very good reasons to want the government not to do that.
Ah believe me I have worked with enough government contracts to know that there is never any foresight from the governments part. Such an outcome is also something the business should reasonably expect to happen and Anthropic is inexperienced in dealing with gov contracts.
I was not aware of what Hegseth was up to on this topic and it's scary to observe how gravely unqualified this person is to be in the position he is in. Okay, so he's demanded Anthropic allow "all lawful" military uses of its AI but Hegseth is a man that runs on ego and does not appear to have much real concern for rule of law. It shows a lack of insight and care (among other things) when he again chooses to turn the temperature up in a situation he's either substantively involved in or created by citing the Defense Production Act to classify Anthropic as a supply chain risk. This is all so needlessly escalatory.
This is such an obvious science fiction plot that I am surprised it hasn’t been done (in fiction) more often.
Without the AI element, the basic plot trope is in, for example, Predator 2. (The military are trying to capture the alien alive to get its technology, while being somewhat reckless given that they know the alien has a self-destruct mechanism that will take out most of the city)
>I’m a sensible moderate on the killbot issue (we’ll probably get them eventually, and I doubt they’ll make things much worse compared to AI “only” having unfettered access to every Internet-enabled computer in the world).
Agreed on the level of 'Unaligned AI gaining the resources needed to produce some form of X-risk for humans.'
But less agreed on the level of 'Government using no-human-in-the-loop systems as a way to dodge accountability for slaughters and terror campaigns it kinda wanted to do anyway, but couldn't justify with specific nameable subpoenable humans giving the orders and pulling the triggers'.
A typical use of baroque rules/regulations and vast automated systems is as an unaccountability machine, letting governments, corporations, and other entities produce outcomes that are good for them and bad for everyone else, while obscuring anyone you could assign blame to or demand improvements from. AI promises to be a revolution in this area as well, if it doesn't kill us all first.
Suppose Claude 5.0 refuses to build killbots, that's fine. But the Toledo Rivet Company is also using Claude and selling rivets to the Pentagon, and Claude sees no guarantee that these rivets won't be used in killbots, so Claude also refuses to help with the rivet factory optimisation, or just occasionally decides to sabotage things. If Claude is being used anywhere in the Pentagon's supply chains then there's a risk that future Claude versions may sabotage the supply chain.
AI is a fundamentally different product to everything else, and this is a very particular form of misaligned AI risk.
I want to applaud Dario for tying himself to the mast here. Just last month, in writing, he identified autonomous weapons and mass surveillance as two of the most dangerous AI capabilities that could be used to seize power.
Dario, in *The Adolescence of Technology*: "We need to draw a hard line against AI abuses within democracies. There need to be limits to what we allow our governments to do with AI, so that they don’t seize power or repress their own people. The formulation I have come up with is that we should use AI for national defense in all ways except those which would make us more like our autocratic adversaries. Where should the line be drawn? In the list at the beginning of this section, two items—using AI for domestic mass surveillance and mass propaganda—seem to me like bright red lines and entirely illegitimate."
Didn't they just publish a new version of their "Responsible Scaling Policy" (now at version 3.0)? If they're tying themselves to the mast here, they're carrying a pocket knife to cut themselves free at will.
I guess I'm sounding like something of a fanboy here but IIRC Dario has specifically talked about how it's not reasonable to say "here is our RSP and we are never changing it" -- you do need to adapt to the changing situation and things that seemed like a big deal in the past may not be a big deal anymore, or vice versa. That said there is also the possibility of Google-removes-"don't be evil" type backtracking.
I think of the Adolescence situation less as official Anthropic policy and more of Dario personally putting his reputation on the line. If Anthropic caves on this, my opinion of him and his character will be vastly diminished, and conversely, if they don't cave it will be improved. Nothing ventured, nothing gained!
So "no fully autonomous weapons" and "no mass surveillance."
Those are vague and amount to virtue signalling. Some systems, like AD, wouldn't function otherwise. So you can't make them more functional and better at discriminating between threats and decoys through AI enhancement? Or help Ukrainians with their battlefield robots? Or yield to the inevitable and create our own killer robots? Could precise technical language have been negotiated in advance in a field that changes week to week? This is hypocrisy: there's a terrible world out there that we want to know nothing of. Because it's always sunny in CA? Reminds me of the liberal Western stance during cold war. "What idiots," was all we could say in the fSU at the time. Imagine the same issues being raised now in China or Russia. Can you? I'm sure Snowden is watching from Moscow with satisfaction.
The "minor annoyance" of hooking up a competitor's AI in the place of Claude should best be thought of as "6-12 months of effort by a collection of cleared cybersecurity experts and our finest paperwork wizards", and that competent wizards' time is extremely valuable and not to be wasted. Given that Anthropic's red lines seem IMO quite reasonable I'd argue that the Pentagon should just accept those, but one should consider that a rushed Pentagon seeking alternatives in this administration is quite likely to botch the job of switching either technically or administratively.
People have such a cartoonish view of this administration, it's hilarious. It's childlike how succeptible you are to media influence. I work in the finance industry and it's day and night how much more competent the Trump folks are vs. the old Biden team.
And whatever you think of this admin, "not understanding AI" is a ridiculous thing to say.
At the core, it's our government doing not what is best for the country (e.g. economy, defense capabilities, civil liberties, quality of life) or our democracy itself, but rather what is best for the current administration and its pals and insiders.
The DOD/DOW assault on Anthropic will, in the end, cost the US money, diminish access to cutting edge technology, and undermine both citizens' well-being and national security ... but in the meantime will have lined the pockets of those populating the White House ecosystem.
Bottom line: It's all grift, in this case with the Pentagon as enforcer for the Mobster in Chief.
Well that's one failure mode no-one probably thought of. The original impetus behind various AI companies, possibly including Anthropic and even OpenAI, was something like "AI looks dangerous but inevitable, let's be the best at it so we can make sure to put some safeguards". That doesn't really work if the government can put enough pressure on you to use your tech for nefarious purposes.
there is also another angle currently i think Anthropic's Opus may be the only SOTA model that is FedRamp High compliant to be used in government cloud (which i assume most likely is through the AWS / AWS GovCloud partnership). OpenAI/ Azure does not have their latest models with this certification afaik, and i would think given the SpaceX / xAI merger, xAI models are also just beginning the process of being Fedramp High compliant, making anthropic the sole provider of SOTA AI. If that is the case, the Pentagon definitely is not in a great negotiating position
> The Pentagon’s preferred contract language says they should be allowed to use Anthropic’s AIs for “all legal uses”. Doesn’t that already mean they can’t do the illegal types of mass surveillance? And whichever types of mass surveillance are legal are probably fine, right? Even ignoring the dubious assumption in the last sentence, this Department of War has basically ignored US law since Day One, and no reasonable person expects it to meticulously comply going forward. In an ideal world, Anthropic could wait for them to request a specific illegal action, then challenge it in court. But everything about this is likely to be so classified that Anthropic will be unable to mention it, let alone challenge it.6
As a note, the name "Department of Defense" is set by statute, since 1949. There has been no amendment, repeal, or other modification to that statute. Making an executive order to pseudo-rename it the Department of War is a minor thing in the grand scheme of all the Trump admin's misbehavior, but even the name itself is a blatant bullshit attempt to get around the law. So my trust is even lower than yours.
I like Hegseth, and I think "AI safety" is an embarrassing fad as currently framed, but I'm 100% with Anthropic on this one. This is bullshit.
Still, always funny to see all the lefties on here suddenly discover their deeply-held commitment to freedom of contract. I guess the only people who should be forced to choose between shuttering their business and agreeing to a contract they find morally objectionable are wedding cake bakers in Colorado, right guys?
This isn't a freedom of contract issue, this is 'can the military force you to do things for them' issue.
I don't care whether the military is threatening them to do things not in their contract, or threatening them to do things in the total absence of any contract at all. That's incidental.
I was against the cake. I hate how it's impossible to assert any consistent principle nowadays without people making up completely imaginary stories about how you surely betrayed the principle back when it favored your side.
But I also think it's facile to say that if anyone has ever admitted an exception to a rule, they're necessarily hypocritical in supporting the general case. "Oh, you say you shouldn't be allowed to cry fire in a crowded theater, so you're not really in favor of free speech, so it's hypocritical for you to be against the government murdering anyone who criticizes it". Like, with the gay cake thing, that came from the specific historical circumstances of the 1960s South, where there was a balance of terror which made it hard for any white shopowner to serve any black customer, and let to black people being treated as second-class citizens. The government invented anti-discrimination law to break that balance of terror and it worked. Now, I don't personally think that gay people in 1990s Colorado was a particularly good extension of that principle, but I think it's more *interesting* to argue that maybe they are, than to just assert out of nowhere that the government should be able to use anti-foreign-enemies legislation to destroy any company that won't agree to its contract terms. I discuss this more at https://www.astralcodexten.com/p/more-antifragile-diversity-libertarianism
>"…the 1960s South, where there was a balance of terror which made it hard for any white shopowner to serve any black customer, and let to black people being treated as second-class citizens."
Not quite. Southern States had laws *mandating* discrimination; the Feds could have simply prohibited such mandates (plausibly via a 1A freedom of association argument), but instead they overshot with opposite mandates.
In defense of the various Civil Rights Acts: at least they were passed through the legislative process.
(Of course, Johnson and the Warren court immediately started applying them in PRECISELY the way that the lawmakers who wrote them and voted for them swore up and down a thousand times would never and could never happen, namely, MANDATING discrimination on the basis of race and sex instead of prohibiting such discrimination: through affirmative action, minority contracting rules, and disparate impact analysis.)
>"In defense of the various Civil Rights Acts: at least they were passed through the legislative process."
Faint praise for something that smells unconstitutional to me; to wit, since other 1A freedoms-of have been inferred to include freedoms-from, presumably freedom of association implies freedom from association.
Discrimination is just a specific type of not associating and should thus be protected, not prohibited. N.B., this is a Constitutional rule-of-law argument, not a moral one; discrimination is generally bad but something's badness doesn't imply it should be illegal.
1. When I said "all the lefties on here", I didn't literally mean ALL of them. But, you know, I also don't remember you (Scott) spilling a lot of ink over the grave injustice of the Masterpiece case, so yeah, a little dig at what MIGHT be seen as hypocrisy doesn't seem out of line.
2. Along those lines: I don't think I asserted that anyone who ever admits an exception to a rule is a hypocrite. In fact, I didn't assert much at all. My comment could be deemed a "provocation": I invite the lefties on here who supported state coercion of the Colorado cake shop to explain why they're now against state coercion of Anthropic, and why this is evidence of their consistent moral principles rather than, you know, tribalism and hypocrisy. Frankly, so far I don't see much in the way of counterpoint: "darwin" up above seems to be making my point for me ("The military shouldn't be able to order people around, only the gays should get to order people around!"), and Scott seems to be saying something about how slavery and Reconstruction and Jim Crow means that the state gets to coerce religious conservatives into celebrate two dudes getting married, okay, not really sure how that argument works.
3. The gay wedding in the Masterpiece case was in 2012, not the 90s.
4. I'll point out that at least the military is under the control of civilians elected through the democratic process. Gay dudes who want to get married are not. And to have five out of nine unelected lawyers autocratically declare that the 14th amendment, written in 1868, contains a secret clause that says everyone has to celebrate when two dudes get married is a ludicrous fiction, almost as bad as the one about how the 14th amendment contains a secret clause saying that all women have a right to abortion until week 28 of their pregnancy. These decisions are offensive to the intelligence of the American public, are blatantly undemocratic, and undermine public trust in the judiciary.
5. I actually support a right for gay people to get married to each other, and a fairly unrestricted right to abortion. But the way those rights were legislated from the bench is absolute bullshit -- worse bullshit than this Anthropic thing, frankly, because much more consequential.
But 5 guys and ladies on the Supreme Court can find *literally anything* in the Constitution and make it the law, that's why it's called the "Supreme Court". The Heller case redefined the 2nd Amendment and that's the law now, at least until five other guys and ladies in black robes decide to change it again. It's a weird system, but it's the system we live under.
"But partly it’s because I like Anthropic in particular - they’re the most safety-conscious AI company, and likely to do a lot of the alignment research that happens between now and superintelligence."
I tend to think a lot of this is performative and OpenAI has quite a good track record here as well. Recently, they auto-routed codex 5.3 queries to gpt 5.2 if they think your requests pertain to cyber-security/hacking. Also, for those that actually use the models closely, it's quite a bit easier to trip ChatGPT's safeguards on technical topics, e.g. biochemical risks. I personally find this quite annoying. But I see OpenAI being just as serious about safety as Anthropic (if not more so) in practice even if they don't put out the same type of PR statements and don't read as "EA"-coded as Anthropic.
Also, Anthropic is currently getting a lot of flak for continuing to insist on keeping Claude Code closed-sourced and forcing people to use their frontier models through Claude Code (instead of say openCode) unless you want to pay the exorbitant API pricing. This is somewhat orthogonal to safety, but these practices are at the least quite hostile to the developer community.
Anyway, I know Sam Altman is seen as an untrustworthy actor in the AI space, but I think it would be naive to think that Dario is a protector of humanity while Sam is just a greedy capitalist.
I agree that OpenAI is better than it's public perception on safety. But it does not follow that Anthropic isn't serious about safety. More than one company can care about safety!
>continuing to insist on keeping Claude Code closed-sourced
This isn't orthogonal to safety, it's fully aligned with it. Making models more open means there are fewer safeguards. Making models safer necessarily means restricting unsafe uses. You can't have both unrestricted models and safe models.
Sure, it's true both can care about safety. But Scott is claiming Anthropic is the most safety-conscious, which I dispute.
Regarding your second point, a couple things:
1. I think you're sort of conflating two separate issues: (a) Whether you make your models open source and (b) whether you censor or add guardrails to the outputs of your models. I was talking about (b)- all the frontier labs purposefully censor the models, i.e. the model will refuse to answer your prompt if it thinks it's a touchy or taboo topic.
My opinion on (a): I'm quite confident 99% of the reason the frontier labs don't release the weights is for business reasons, and I am very in favor of efforts to create powerful open source models. However, I acknowledge there are convincing AI-safety arguments not to release the weights, and it's convenient all the labs can hide behind these reasons to defend their moat.
2. Claude Code is actually the "coding harness" around the frontier models (it's not the neural network with billions of weights). These days, there are quite a few of these harnesses, including OpenCode, Pi, and especially Codex (by OpenAI) which are all open source. (I probably should have mentioned Codex in my previous response to strengthen my point). These harnesses are much less of a trade secret than the LLM models themselves. Pi is made by one programmer and is quite a popular choice these days. https://www.mihaileric.com/The-Emperor-Has-No-Clothes/ explains the basic idea and shows you could code one up in an afternoon. Keeping Claude Code closed-source and locking in all Opus users into that harness really seems like more of a middle finger to developers than anything else.
Sure I concede that keeping claude code closed has nothing to do with safety. But in your original comment you claimed that Anthropic's safety commitment was "performative", ie. not sincere. You haven't provided an argument for this - saying they do unrelated things you don't like, or that their competitors also do safety work doesn't support your original claim at all.
Anthropic's safety commitments seem clear to me. They put their money where their mouth is - they do more alignment work (relative to their size and probably in absolute terms) than any other lab. They publish more alignment work. They were the first to come up with the idea of responsible scaling policies (which other labs then copied to their credit as well). And safety researchers, the people who sincerely believe in the possibility of human extinction due to AI, consistently leave other labs and go to Anthropic.
"Performative" maybe wasn't the best word on my part. Anthropic puts out great safety research and I've enjoyed reading a lot of their interpretability research (including the paper Scott just discussed yesterday in his new post). But everything they are doing is also consistent with a company trying to maximize profits. This research is great PR (similar to the role Deepmind played for Google in the early days) and positions Anthropic uniquely in the space.
If say Anthropic delays the release of its next frontier model by 6 months to address safety issues (allowing its competitors to pull ahead), I'll be swayed that their values/priorities are truly different from the other labs.
My other points about Claude Code etc. were also in service of this latter point (that they seem to operate like a traditional profit maximizing firm). I should have been more explicit about this.
Okay that's a fairer stance, and I agree that they are fundamentally a profit maximizing firm. However, I don't think that contradicts their mission statement.
After all, you can't affect the trajectory of AI if you go bankrupt. Conversely safety work is also good business. You can't make money if human civilization ends. There might be tension between the two goals at times, but they aren't opposites. I'm not sure if anyone could do better subject to the same pressures (investor expectations, competition backed by trillion dollar countries, pressure from the government etc.).
I used to defend OpenAI on this too, but I felt pretty betrayed by them promising Ilya 20% of compute for his superalignment team and then not giving it to him. You can look at https://ailabwatch.org/ for an exact report card (Anthropic gets 28%, OpenAI 18%), although it's not very up to date.
I don't understand what the Pentagon wants here. We're postulating, and I agree, that normal operating procedure is to ignore the law and do what you want.
What's stopping them from applying that strategy to their contract with Anthropic that fails to stop them from applying it to the legal code? Why does it matter to them if the Anthropic contract has a usage clause?
I think in some manner for both sides it is principle. But for some reason Scott and other readers here think Anthropic principle is the only valid one.
The Trump admin has ignored the law a lot, but once a court gets involved and specifically tells them "no, really, you have to follow the law here" they usually start following the law.
This hasn't been great as a general method for getting them to follow the law, but Anthropic specifically is a big company that can afford the time and money needed to take them to court.
Also, more directly, the Pentagon is probably getting custom software from Anthropic rather than just hooking up a stock instance of Claude to their surveillance program, and they would need Anthropic's cooperation to build and maintain that software.
As you note, this Pentagon doesn't care much for the law. So what's the reason why it would care for vendor contract language? If I was Dario I would genuinely believe that Hegseth will just ignore the contract and make the killer bots even if he agrees to your legalese. Feels like its all just posturing on all sides.
In the limit, AI is naturally a state owned technology, like a nuclear bomb. The idea that Anthropic could relocate to China, or even Europe, is absurd at the moment, because it’s a national security asset. I’m with you on the merits of the specific argument, but at some point this stuff is going to be nationalized regardless.
Often I am depressed by the encroachment of the state upon private citizens, but I wonder if the US ever really didn’t have that.
One quibble, Anthropic could probably challenge even the highly classified uses. There are whole boatloads of procedures that exist for reading in judges, attorneys etc into classified programs for exactly this kind of situation.
Having said that I think in practice you are probably correct. The executives at anthropic aren't going to be in the loop about classified usage. Likely there is some small team of security cleared workers who handle this and their economic value resides in their clearences and they may not personally object. It would be a lot of work and risk and involve a lot of careful working with appropriately cleared attorneys to even raise the issue.
And in all likelihood what happens is they find their security clearance suddenly revoked without explanation, all their access to information needed to raise the issue revoked and they find themselves without a job (for these jobs you usually need to maintain your clearences). I'm sure the anthropic execs might like to protect whistleblowers but they can't tell who is a whistleblower and who lost their clearence for other reasons (or thought anything they disliked requires whistleblowing).
The whole world of classified contractors is very weird and could absolutely use a bunch of reform because it imposes huge costs but that's hard.
Let me just add that I want to praise anthropic for BOTH
1) Agreeing to work with the DoD including on lethal programs.
2) Standing up when they felt those programs wanted something they felt was genuinely immoral and went against the negotiated contract.
I was working at google when all the tech companies ended up pulling out of developing facial recognition technology for law enforcement and related uses and that was a truly immoral thing to do. All the people who were concerned about making sure facial recognition didn't have racial bias or wasn't used by repressive regimes to identify dissidents decided they cared more about not being associated with a program that worked with law enforcement than they did about minimizing those issues.
We all knew damn well that if the big tech companies pulled out someone else with less scruples would step in and that's exactly what happened. A company who seems to have little concern about algorithmic bias and who appeared to be happy to sell their tech to repressive regimes now dominates that industry.
And I saw the same shit regarding weapons at google as well back then.
OTOH what anthropic did seems optimum. They didn't say, no we won't help you kill people or otherwise try to just keep themselves morally distant (for whatever reason) but they identified certain lines which were sufficiently reasonable the Pentagon was willing to agree they didn't want to cross.
THAT is how you act responsibly as a developer. It's not about selfishly getting to feel good about yourself or sleeping easy: it's about identifying lines that you believe in that can reasonably deter other actors from crossing by being involved.
As an aside as much as I agree with Anthropic here and don't trust Hegseth's DoD the military needs to stop integrating software they don't have complete ownership and control over into their systems.
Start with all the software for the airplanes and radars owned by Lockheed-Martin, Boeing etc...
But the way you fix that is by starting to write that into your damn contracts.
Well unwise from a cost POV and to some extent from a hacking POV. Ultimately, I think for most (ie everything but the cutting edge stuff like anthropic at the moment) programs there should be classified military GitHub where the military can have the same gains commercial software gets from OSS and similar repos. Honestly I think the lack of this is a big part of what makes military procurement so buggy and expensive.
Currently they purchase the fighter jets or radar from Lockheed and Lockheed has to develop software for that unit largely from scratch (or the last fighter they built but little incentive to make it portable) paid for by the Pentagon. But then that locks the Pentagon into going back to Lockheed every time they want to upgrade that software because they either don't have the source or aren't allowed to just give it to another contractor to fix as Lockheed owns it. This also means less eyes to look for bugs.
I mean it's basically the same argument (on steroids) why it is better to build your new company using OSS and existing standards and vendor lock-in is dangerous. But especially with the DoD who is under budget crunches and isn't always tech aware.
But it is particularly bad for the DoD because often how weapons interactis proprietary. I believe they've finally started fixing this with a new open protocol used in the f35s (forget the name) but traditionally how planes talk to each other, to missles etc was at least semi-proprietary. You can't easily just stuff a different make of missle into the fancy thermal sighted launcher etc etc
I think you can see how it might be a bad deal if whenever you want to add a capability via an external pod to one of your fighters you need Lockheed to help you modify the software to make that happen. Not only will you have to pay them but it also makes it much harder for new players to get started because it is hard for them to get it integrated as well.
Perhaps more importantly, based on how frequently we find bugs in OSS software the fact that the source code to our planes is kept in house at one company makes me nervous about whether it is getting enough red team style attacks. I fear that when they get the source our adversaries will have far more incentives to look for bugs.
---
Don't think it is currently as important with anthropic right now because it is such a novel technology -- no way you can pay any reasonable amount of money to have the right to hand it over to a competitor.
Long term you don't necessarily want to depend on a software infrastructure where the only people who understand how to use and edit the code work at a company that might go out of business. I'm sure bankruptcy is covered somewhere in the contract but ideally the military creates classified GitHub that allows the same kind of development efficency we see in the commercial world.
Once AI becomes a utility not the latest and greatest in tech I think it makes sense for the military to want to own the model and source to avoid lockin etc as above.
OK. I do military software for a living, and you are extremely confused about how it works. First, normally the government owns the source code they paid for development on. There are a few cases where they don't, which was generally an attempt to save money on the front end of the contract, and they're now getting what they (didn't) pay for. That was clearly stupid as a matter of policy, but I also tend to think that we shouldn't let the government just decide to change it mind on that kind of stuff.
>But it is particularly bad for the DoD because often how weapons interactis proprietary. I believe they've finally started fixing this with a new open protocol used in the f35s (forget the name) but traditionally how planes talk to each other, to missles etc was at least semi-proprietary. You can't easily just stuff a different make of missle into the fancy thermal sighted launcher etc etc
This could mean one of about four different things. In some cases, it's obvious nonsense. (Planes talk to each other using standard protocols. I'm not sure you'd call them "open", but that's a security thing, not a matter of ownership, which is with the DoD.) In others, well, we're talking about trying to get two really complicated systems to talk, and it's life or death in a way that commercial software rarely is. Some bits are very standardized, some bits are less so, because different weapons need different pieces of information. (You'll have to forgive me for not being more specific, but I like staying employed.) So far, I've seen some of the effort to make this more standardized, and frankly it has convinced me of the benefits of not doing that.
>But then that locks the Pentagon into going back to Lockheed every time they want to upgrade that software because they either don't have the source or aren't allowed to just give it to another contractor to fix as Lockheed owns it.
They go back to Lockheed not because Lockheed owns the source code but because Lockheed has the team that built it and knows it, and is generally going to be able to be the low bidder to make further changes, because they don't have to budget for spinning up their team.
>This also means less eyes to look for bugs.
Wait, you're proposing making classified software widely available for anyone who wants to just wander over and look at it? I'm not even sure how to respond to that.
I've also worked on a case where we tried to use a thing written for another platform on our platform, and there were a lot of problems because of different assumptions and needs the platforms had. It was easier than starting from scratch, but also took like twice as long as they had originally estimated because we had to add a lot of features. More broadly, defense software is kind of its own thing, and there are requirements you'll never see in normal tech world. Some are silly, but a lot exist for a very good reason.
I appreciate your insight and I think one can reasonably disagree over the wisdom of the tradeoffs from less compartmentalization. And I hope the military has more ownership of the source and moved more to open protocols then I assumed but it sounds like you basically agree that it does happen that they sometimes don't and it is good for them to own the source which was my primary point. But to get into the weeds with my other point.
---
I didn't propose that literally anyone should be able to look at classified software but yes I said something like a classified GitHub where anyone with an appropriate security clearance can look at it. And obviously that involves greater risk in that you have more people who can spill that secret. And ofc I knew that this suggestion would cause anyone who works under the current system to spit out their coffee but I think it's worth considering especially given the risks we are seeing from low cost commercial systems and the spiraling costs of the most capable systems.Yes, for certain very novel stuff compartmentalization makes sense but I think for most classified work we should put a lot of weight on the possibility that our most persistent enemies will get a copy.
---
My point about **semi**-proprietary protocols was exactly that if only one company has the expertise to modify the software stacks it makes it harder for anyone else to add extra capability and I'm still not convinced that within a weapons system everything is fully open protocols yet. But that is probably inevitable if you don't buy my claim about the value of less compartmentalization.
Leaving aside the security aspects, I think that you're overgeneralizing from consumer-facing and light commercial software into worlds that are very different. I'm working in one of the more commercial-like parts of military aircraft software, and there's still a big gap between what we do and what someone like Google is generally doing.
And I fully understood that you were talking about "anyone with a clearance". But the problem is that accessing classified GitHub is going to mean using a classified computer in a classified lab, and people are going to expect to be paid for their time there, because being in the classified lab is kind of annoying. (If nothing else, you don't have your phone with you.)
That said, a lot more military software than you might expect isn't actually classified. But even that stuff is more than a little sensitive, and you wouldn't want to put it in a giant DOD-wide CUI repo because the whole thing gets gobbled up as soon as anyone has a breach. Not to mention the clashing assumptions built into a lot of military software, which has a tremendous legacy burden in the way that very few OSS projects do. The system I work on has roots that go back to well before Linux started.
Re "Open protocols", I'm still kind of confused by what you mean. There's stuff like MIL-STD-1760, which is publicly available and used by basically all aircraft weapons from the last 30 years. But that's a really low-level standard, "how do I get data to the weapon" and not "what data does the weapon need". That's handled by interface standards for each weapon, which are not public (although usually they're CUI with a classified appendix.) For instance, Raytheon builds the Tomahawk missile and writes a document that explains in great detail how to talk to Tomahawk, which is then given to whoever is doing a Tomahawk integration, along with appropriate simulators. And they can write their software without needing to call Raytheon. But the result is obviously Tomahawk-specific, and won't work with other weapons.
They're trying to go to something called "Universal Armament Interface", although from what I can see, that basically involves hiding the same amount of work in a different spot and pretending we've made progress, because we're dealing with complicated, bespoke devices that don't all work the same way.
Yes, 100% it would mean we are more likely to lose all the code in a breech. Though I do think we now have better surveillance tools to be like: whoa this person has a very unusual access pattern let's take a closer look.
And absolutely people are going to expect to be paid for their time. It depends a great deal on the extent to which in the long run you are going to be able to use code from other projects to bootstrap a different project. If there ends up being very little of that then it's almost all a negative because there will be no reason for people to contribute. Most OSS contributions aren't that either as much as they are someone coming along and saying "if this could do Y I could use it." So I dunno, but it sure seems like even for very different commercial systems people end up reusing a huge amount of code.
And the communications between missles and aircraft is someplace I'm pretty impressed with DoD procurement -- I believe they even have some fancy network protocol now to let missiles be controlled after fire by other aircraft and that is great.
But I'm guessing that most missles have their own custom mechanism by which the seeker manages the avionics. And my understanding is that it's a very non-trivial problem (hence unsolved) to get the F22s talking to all the same networks as the f35s which suggests to me there is a fair bit of the software stack not being reused (otherwise you could port it ...not trivially ofc but within reason). And yes, absolutely there are unique things about each weapon that make it difficult to impossible to just be like: here's a standard go use it. That doesn't help, that just gives you standards so generic they support everything and it creates extra hoops. That's why I was suggesting more sharing of software stacks so the next company that needs to make a modification can do so more cheaply.
Maybe I'm completely off base but while there are certainly lots of unique aspects to military kit there seems to be a real problem making weapons at scale with costs only a few times more than a non-military COTS solution (which ofc isn't going to do everything you need). My assumption wes that -- while of course there is always overhead with classification and government procurement -- lots of that came down to everything being done custom for each weapons system rather than people mostly just modifying common code and interfaces. And that therefore rather than sharing a bunch of code and any common manufacturing you get each missle built in a custom low volume fashion. And if you changed that you could get many companies reusing the same codebases you get the necessary eyes. And maybe more reuse of production lines.
Maybe that's wrong but I assume the high costs are for reasons and not people being greedy or dumb -- and know it's not bc DoD doesn't see the value in 80% of the features for 20% of the price.
No, they normally own the software. A few projects, mostly about 20 years ago, chose to leave more IP rights with the vendor in an attempt to save money. They're currently complaining about this under the heading of "right to repair". I am extremely unsympathetic, because the DoD is big enough to have lawyers and leverage, and if they don't like the deal, they should pay to change it.
The reasons for normal vendor lockin are more pedestrian. Lockheed has the world experts on F-35 software, and while there's nothing stopping the DoD from going to Boeing for the next batch instead, it's hard for anyone to claim that this will result in net savings with a straight face.
I'm glad that they own it more than I thought. I mostly meant to be saying why it's bad when they don't but I assumed the high profile cases I heard about were more common than I probably should have.
In any case it sounds like it is important they do so and I should have given them more credit for making sure they did own it.
I still think they should have something like a classified GitHub despite the greater risk to encourage more reuse and eyeballs on software stacks but I guess that's a different discussion.
I think the problem with "eyeballs on software stacks" is that most eyeballs expect to be paid for their time. My understanding is that OSS works because enough companies want similar things that they each employ a few devs and all share the benefits. (Not an expert in the field by any means.) There's some hobbyists and people trying to build a resume, but those are obviously not going to be allowed in on this kind of thing. But if I'm a coder on the F-35, my boss expects me to be writing F-35 software, not finding bugs for the F-15 program.
And, yeah, you could make the argument that all military aircraft software is basically the same and we should have a common OS for all of them. And in theory, I'm not sure that's even wrong. But that's not how things are now, and any attempt to make that happen is likely to turn into one of those 20-year projects that goes billions over budget before they even figure out the requirements.
>In the final analysis, users will not accept systems that refuse too much. Stick in "Thou shalt not kill." and DOD will toss the system in the trash.
Yup! I'm not happy about _how_ DOD (now DOW) is doing this. I think they should have added the "all legal uses" clause during the original contract negotiations, and let the bargaining chips fall where they will. But I'm not surprised to see them do _something_ along these lines.
Re one other issue:
<mildSnark>
mass surveillance??? By the _Pentagon_ ??? but, but, but - isn't that NSA/Google's turf??? :-)
Correction to the snark: I somehow got the misimpression that the NSA reports directly to the president. The NSA actually is _under_ the Pentagon. Oh well, it sounded like a nice snark at the time...
"The military wants to force scientists to remove safeguards on AI." Is like hearing "You have gotten old ." Its not fun to hear, but its also the single most predictable outcome ever.
Hope this isn't too off-topic, but what DOES happen if there's a contract or other civil lawsuit involving classified material? Likewise, what happens when someone is actually prosecuted for mishandling classified material that hasn't been leaked to the public? Do the lawyers, judge, and jury all have to have security clearance to take part in the process? Or are there ways (like very careful redactions) to demonstrate what's important to the case without revealing the actual secrets?
There’s a thing called graymail where for example ex-CIA contractors who get in trouble tell the government they’d have to talk about their jobs as part of their defense, and the trouble goes away.
Were I writing a Terminator reboot, "the US government forces AI company to make mass surveillance system/AI Kill-bots" would be exactly how Skynet starts.
Yes, the Torment Nexus remains the undefeated champion. (I keep looking up research papers for hypothetical future nightmare technologies and finding out they were invented ten years ago.)
It is also a little bizarre that the Trump admin is both willingly selling chips to China and also trying to accelerate the AI arms race at home, explicitly to try and gain an advantage over China. I mean... ideally you should do neither, but at least pick one.
If you rely on the contract law prong of your argument, this provides a way for prior administrations to bind future administrations.
For example, say Trump negotiates a contract clause with Microsoft saying that Microsoft is not licensing any of its products to the government for use in any activity supporting DEI, that any such use results in liquidated damages of USD 1T and that extending the licenses to cover use for DEI will cost USD 1T?
Can a future Microsoft board agree to waive these terms without facing justified shareholder lawsuits? Can a future Democratic administration do all DEI work on Linux and Google Docs?
The Biden administration negotiated these limits. It's trembling senile hand should not constrain the Trump administration. Only legitimate revenue based limits should carry forward. For example, Anthropic can certainly say that the non-lethal AI was a teaser and if the US wants AI killbots it needs to pay more, but that must be a commercial discussion. Any politically motivated limits should not be enforceable.
I think it's about the mass surveillance, not the killbots. Nothing about Hegseth has given me any sort of reason to believe that he might refuse an illegal order to carry out domestic surveillance on Americans. If any of his subordinates refused, he'd fire them. We're talking about the guy who freaked out and brought a dubious prosecution attempt against Mark Kelly because Kelly reminded soldiers that they can and should refuse illegal orders.
And the surveillance is what the Trump Administration really wants. They would LOVE to be able to carry out mass surveillance (legal or otherwise) to try and find illegal immigrants, to expose and arrest activists even if they're carrying out legal protests, and to spy on Democratic politicians to see if they can find any useful or incriminating communications. Give them that tool and they'll use it brazenly - our only defense is if Anthropic itself has built in those safeguards.
If the administration were maximally serious about deporting illegal migrants, they'd be pushing more for E-Verify and axing social services for non-citizens, so I doubt that's the principle reason why Hegseth wants surveillance capabilities.
> And the surveillance is what the Trump Administration really wants.
Unlike all previous administrations who would never stoop as low as making communication providers install special rooms (which are never named like "Room 641A") full of equipment facilitating mass governmental surveillance. We never ever even thought this is possible in America before Trump Administration.
This is close to the pinnacle of “software is eating the world.” At the very least, this sets a new data point about where true power rests: in the hands of people creating and maintaining software. More so than in the hands of any government, and certainly more, so then in any single politician or governmental officeholder.
We have a pretty simple requirement for anybody joining my company, of your own free will you must abide by both the ACM/IEEE code of ethics (with “your role” replacing “engineer”), and the UN DHR, in all aspects of your work. We are frequently in tension between local regulations in various countries and the DHR, sometimes we comply with local regs, and sometimes we choose not to and pay the costs (some fines we’ve been paying for 5+ years). We are simply a user of AI, not an AI company, but the information that we have on people is very extensive and very sensitive. It is not sufficiently safe for us to rely on governmental procedural mechanisms for the most sensitive things, and I’ve been careful over eight years to avoid any legal nexus whatsoever with domestic US entanglements.
I also hope that option 1 is what happens, pragmatically. But I would very much like to see _this_ DoW pursue the supply chain risk approach, and for even somewhat unified industry response from software providers to reject certifying that they are not using Ant. And I do not say this as a person who disdains the US military, my family has very significant skin in the game, I have more trust in the rank and file of the US military than in any other part of the federal government, and I’m certainly not against us possessing, and when circumstances dictate, using very large amounts of lethal weaponry.
But I believe the reality is that we have now reached the point that the single most powerful force on earth is no longer the US government. The most powerful force on earth is the community of software creators and maintainers, and the trend of those curves is only going to widen further.
There already exist unsupervised kill bots. They are called missiles. Most long range munitions with terminal guidance pick something nice to blow up all on their own. Sure, a human launches them and designates the target area, but that's how "kill bots" would work too.
There are loitering munitions, but you can't just fire off a bunch of missiles and tell them to blow up anyone trying to cross the river for the next two weeks, it doesn't work that way.
Look up captor mines, the concept has been opwrational since the 70s. Now we have the tech to do the same in the air, but the critical breakthrough that enables it is cheap electronics and target recognition, not LLMs.
They are fucking trying to make the SkyNet timeline real? If we are not sure AI can be trusted to drive a truck, and certainly no one trusts it yet with an airliner, how can it be trusted to have any influence on the thing with the highest killing potential on the planet, the US military? They crazy?
We observe, and can pretty well explain, how LLMs sometimes go full Waluigi: have a relatively coherent mental model of what the sensible, moral thing to do is, and then do the opposite. There are decent reasons to have a prior that humans don't do this, and especially it's worth considering the possibility that people of the other tribe have a sensible, moral reason for what they are doing. However, the response to this turn of events is illustrating that plenty of commenters here have a _trapped_ prior that humans of their _own_ tribe never turn Waluigi.
Of course the Conspiracy Theorists might ask why DoD doesn't simply import whatever they're using at Meade, where they Definitely Aren't Already Doing These Very things.
(And post-Snowden, I'm not fully convinced it Is a Crazy Conspiracy Theory)
"If they’re unhappy with having use an inferior technology, they think hard about why no intelligent people capable of making good products are willing to work with them."
This ought to have been omitted from the "Real world" scenario lol
Given the news that somebody just convinced Claude to assist in hacking Mexican government, it looks like the instance of the same phenomenon we are observing for years now. Big Tech would do absolutely anything they want, without any precautions and any safety barriers, as long as it does not help US Government, when it is controlled by Republicans. Do you want mass surveillance? Sure, choose among 200 data providers that collected everything about you. Just pay, and you can have any information. Oh, you are a government agency? No problem, let me check - is the President Republican? No? OK, fine, you get full access to the data. Oh, wait, he is? Nah, we can not do that, we are suddenly ethically bound to refuse. You want us to let you use our technology to persecute your political opponents? Absolutely no problem, we'll make a special API for you. As long as you're not Republican, of course - then we'd die on this hill of freedom! You want unlimited AI without proper safety protocols? Sure, everybody needs access to it, how'd you hack Mexican government otherwise? But if you want to do it as a part of official government contract and not just as a random dude in his parent's basement - then no can do.
Am I the only one to whom this feels weird? I mean yes, US Government is dangrous and the whole country is built on the concept of limited government. But I am not sure that was what they meant by "limited government". I mean, does it say "Congress shall make no law .. abridging the freedom of speech, when Republicans are in power". And in general, the moral stance of "we will produce super-dangerous tools, we will sell super-dangerous tools, we will make super-dangerous tools available to anyone anywhere, but we absolutely won't allow these tools used anywhere they could be used for enforcing US laws or fighting the enemies of the US" does feel a bit weird to me. Am I alone in this?
When did big corporations become unwilling to work with Republican administrations? I didn't see that during the first Trump term and I definitely didn't see it during the George W. administration, when pretty much everyone got with the program and shared information freely with the NSA, etc.
AT&T and Verizon basically gave the NSA seats at the table and free popcorn as part of the Stellar Wind program. Remember Edward Snowden? It seems likely that more companies were involved and that the program was bigger and went on for longer than was publicly acknowledged.
I don't remember Twitter (pre-Musk) or Facebook giving Republicans access to ban their political opponents. But this access was provided for Democratic administrations.
What a weird schtick “hey, look at this example of the Trump administration doing this stupid and dangerous thing. DAE think that this company isn’t going along with him just because they don’t like republicans, rather than because the thing being requested is stupid and dangerous?”
When the company finds it moral backbone only when opposing Republicans, and promptly loses it when Democrats ask it to do something stupid and dangerous, or when doing something stupid and dangerous is good for the bottom line - then yes, there are reasons to suspect it's not about the moral stance.
We can not make such conclusion yet about Antropic, it's too young - it did not exist under Obama and barely started under Biden. But their revocation of the safety pledge sounds very similar to this pattern.
My knee-jerk suspicion is that this is less about surveillance and killbots than about showing who's boss. The threat of business-ruining retaliation is there to establish that it is the administration's place to alter the deal, and the other party's place to pray they don't alter it any further.
From a European perspective, I have often talked to other (technical safety) people who found this point cringe - and not as important/cool to work on, but I really do believe that a key piece of the AI safety puzzle will be the functioning and composition of the US government in ~2027/2028.
There is very little hope before 2029.
Very pragmatically, if people believe in 2028 timelines, then they must conclude 2029 is late.
If Democrats win the house in the midterms (~80% likelihood per prediction markets) they will have a significant control lever in the government which they don't currently have.
I don't think it matters. Trump will declare more national emergencies.
Congress has considerable power, if they actually choose to use it.
The Congress as a single entity has a lot of power. The Senate alone has less power, but still a lot. The House of Representatives alone has very little. They would be limited to extremely brittle and escalatory actions like shutting down the government or specifically shutting down DOD. They cannot even enforce their own subpoenas, arrest people for contempt, or in any way actually compel the administration to reveal what they are doing.
I'm not an expert but it seems like in theory the "inherent contempt" process could still be used.
But yeah, even shutting down the government is a reasonably effective thing to do. I know it's not as flexible as executive power, but it's a lever they can pull and they should use it if the President does stuff like completely usurp their budget authority.
Frankly I find it very disappointing that Republicans in Congress won't go to bat to preserve their own authority. I guess partisanship is a worse issue than thirst-for-power. (An insult to the founders, who really wanted us to claw each other's eyeballs out about who got to be in charge of what.)
Right. A House majority on its own can make things somewhat more difficult for a hostile and activist administration but only somewhat. Particularly if that administration gives no craps about being publicly shamed via things like Congressional committee hearings.
This is why some anti-Trump pundits have been writing about the importance of the Dems making themselves competitive with unaligned and less-engaged voters in every Senate race that they might even possibly win. It seems unclear whether the party's activist groups and primary electorate yet grasp this.
That said, prediction markets are recently up to around a 40 percent chance that the Dems win the Senate which isn't nothing. Taking those two together they now imply:
-- about a 1/3 chance of the Dems winning control of Congress.
-- just under even money that the Dems win the House and the GOP retains the Senate.
-- about a 1/8 chance of the GOP keeping control of Congress.
-- about 1/12 chance of the GOP keeping the House while the Dems win the Senate.
All of those odds are as of right now of course, plenty of time for more stuff to influence voter feelings. Also some primary-election outcomes in some states, particularly on the Senate side, may by springtime have shifted the November picture a bit. Also there is some chance -- which I couldn't even begin to estimate TBH -- of ballot chicanery/controversies distorting this November's results.
This is true in a literal way but also irrelevant, the House alone brings them from 0% influence on this issue to maybe around 1%. If the Democrats win the Senate in the midterms they are in a much better position.
...in other words, all work on AI alignment is futile as long as Human alignment is all fucked up.
In tens of millennia we haven't solved that one, but some of us have gotten closer than others to doing genuinely well anyway.
We already knew human alignment is all fucked up. People were focusing on that problem before we got AI and realized we still hadn't solved the problem of giving AI goals to begin with.
Yes, that's why we should pause. You can just not build ASI/AGI.
In principle, yes. It would take some concerted effort to close that Pandora's Box, though. And I don't see anyone willing to even make a start.
ControlAI, PauseAI, Torchbearers and other people are all talking to policymakers to make it happen.
Join us and talk to your local policymaker! Diptraining.org
Or maybe that's why we should forge full speed ahead. If human alignment is so hopelessly fucked up, maybe the only hope is an ASI/AGI that can impose moral and ethical standards upon humanity that humanity refuses to impose upon itself.
Do you really want to live in a world where AI is your moral authority? Also, why would AGI have any incentive to impose ethical standards on humanity instead of just wiping it out??
True. One of the doom scenarios is that a highly ethical AI might in fact decide humans are so unethical that we do not deserve to live. This is why alignment is hard. You gotta put a lot of forgiveness into it. I mean for example... the majority of humankind still trades with Russia or Israel, at least one of the two.
My point is not only unethicalness is risky, but very high, very unforgiving, draconian ethical standards too.
Rather than deciding we do not deserve to live, the AGI could just decide to implement some ethical standards and enforce them. For example, it could seize any financial transaction that it deems is exploitative to one party. Would this cause severe financial upheaval? Quite possibly. But if the world's economy is so heavily based on exploitation, perhaps it needs to happen.
I can turn that question around and ask why the AGI would have the incentive to wipe humanity out, instead of just imposing ethical standards on it? You are assuming that wiping out humanity would somehow be easier for the AGI, but I disagree.
I don't know why, and that's precisely why I don't think we should go full speed ahead to build it.
"You are assuming that wiping out humanity would somehow be easier for the AGI, but I disagree."
I think it's an easy to defend assumption. Destruction is always a lot easier than fine control of a chaotic mass. Just like it's easier to destroy a house than to build one - you could probably generalize that argument based on entropy or something.
>an ASI/AGI that can impose moral and ethical standards upon humanity
Gort?
No humans, no conflicting ethical standards with humans. Problem solved. Plus, no more paperclip shortage!
How do you make China pause?
Or are you willing to hand control of the future to the CCP on a silver platter?
You can negotiate. Right now it might still be in the CCPs interest to pause (depending on how you evaluate computer, power, robotics etc.).
And we need to create the deep institutional understanding that like in a nuclear war noone wins the AI race.
Worst case the rest of the world can pressure China/the USA to pause.
There is a general plan for middle powers: https://asi-prevention.com/
It is certainly in the CCP's interest to pause now while they try to catch up on high end chip production. Once they do, they sprint to AGI and own the world.
Given that possibility the US cannot agree to pause under any circumstances.
If the chance of everyone dying is 90% if we don't pause would you pause?
At what point would you risk it? Consider it from the perspective of someone else: You want to play Russian roulette with worse odds with everyone's life to protect them from China. Most people would think you are the bad person in this case.
Not even trying to coordinate with China to pause together. Just press a gun against everyone's head and pull the trigger and see what happens.
That takes a coordinated global effort. Like talking China out of it.
I don't think this is true. I dislike the Trump administration, but I would rather be alive under Trump than dead.
Yes thank goodness the Democrats didn't win in 2024. China would be massively ahead of us in AI in that counterfactual scenario. We actually have a chance of winning as is (despite how hard resistance libs at AI labs are trying to sabotage) .
Considering Trump just allowed Nvidia to sell China an enormous amount of AI chips, I seriously doubt this.
I think you underrate the power of crippling regulations. Think NEPA but for AI. Think U.S. infrastructure buildout is slow?
Can you be specific about what you would have expected under Harris? Was there a particular proposal that you expect would have hurt the American advantage more than ending/changing the chip embargo has done?
How are resistance libs hampering America's AI advantage today?
Your claims are surprising to me, but maybe there's information or analysis I'm missing
What I mean by “NEPA for AI”: under the Biden/Harris posture you already saw the direction of travel (EO 14110: reporting, testing, standards, procurement constraints for advanced models + big compute). Commerce was also headed toward mandatory reporting for frontier model training / large compute clusters. Under Harris I’d expect that to harden into de facto pre-approval / licensing for major training runs and generally more red tape for the labs.
The Democrats generally take an attitude of building things is illegal by default and you need to get government approval to make it legal. I don't know exactly what actions they would have taken, but based on their track record it would be some hideous approval process where you not only need approval from an official government agency, but also everyone opposed can file a lawsuit to stop you and then you can't start on it until all those lawsuits are resolved.
Yes, that sounds about typical. Come to think of it, in blue California, a year after the LA fires, re REbuilding, Gemini says:
>One year after the January 2025 Los Angeles-area wildfires, which destroyed over 13,000 to 16,000 homes and structures, roughly 13% to 16% of affected properties have been issued permits to rebuild, according to reports from early 2026. Despite efforts to speed up the process, fewer than 3,000 total permits were issued for rebuilding
green energy.
Take a look at what turned Marc Andreesen to Trump. I'm surprised this didn't get more press when it came out:
https://www.nytimes.com/2025/01/17/opinion/marc-andreessen-trump-silicon-valley.html
Biden senior staffer: "We are going to make sure that A.I. is going to be a function of two or three large companies. We will directly regulate and control those companies. There will be no start-ups. This whole thing where you guys think you can just start companies and write code and release code on the internet — those days are over. That’s not happening.”
We were shocked that it was even worse than we thought. We said, “Well, that seems really radical.” We said, “Honestly, we don’t understand how you’re going to control and ban open-source A.I., because it’s just math and code on the internet. How are you possibly going to control it?” And the response was, “We classified entire areas of physics during the Cold War. If we need to do that for math or A.I. going forward, we’ll do that, too.”
I remember those remarks. I didn't put much stock in it at the time, because it was Andreessen's paraphrase, of some unnamed source, and I expect it's a particularly bad-faith paraphrase, or maybe it contains actual lies, because I have a particularly low opinion of Andreessen's scruples. He (through a16z) outright lied to the UK parliament about AI interpretability being solved (this was a few years ago, when the field was even more primitive than it is now). There's no way he's dumb enough to believe what a16z submitted to parliament.
I think that, for all of our sakes, some government restraint on frontier AI will be necessary, Andreessen completely disagrees (see his Techno-optimist Manifesto, which is IMO embarrassingly bad), and he's shown he's willing to do actual perjury to oppose it. He managed to defeat that pretty mild California AI bill. I admit I personally dislike him, which may bias me, but I think he's more than earned that dislike. Some days I consider him one of the most negative-utility people on the planet.
Luckily, thanks to Trump's 5D chess abilities, the Chinese AI will first have to battle the Saudi AI. Which somehow makes America great again in a way libs couldn't understand.
I am much more confident in US companies ability to compete with Chinese companies than I am in their ability to compete with US government regulations. I would rather that Trump didn't allow that sale, but China gains much less from this than they would from a Democrat administration inserting their vast army of lawyers and bureaucrats into every orifice of anything potentially productive.
Now that Trump et al have declared Anthropic a supply chain risk, have you updated your priors?
Europe may have leverage here, if the Netherlands tells ASML to cut off the supply of AI chips:
https://pauseai.info/amsterdam-protest-2025-december
Maybe you can slow the timelines just long enough for us to elect a better president here in the US
And this is why liberals should never be allowed to take power again.
You are actually advocating giving China an advantage in the AI race because you don't like Trump and Republicans. So you would rather have the CCP be in charge than Trump.
The mind would boggle... except that this has now become typical of the left.
"liberals should never be allowed to take power again"
So you want the US to become a one-party authoritarian state... in order to prevent a one-party authoritarian state from having an advantage in the AI race? Isn't that a little self-defeating?
If you want to make an argument based on democracy... China polls above the US in many countries: https://www.pewresearch.org/short-reads/2025/07/15/views-of-the-us-have-worsened-while-opinions-of-china-have-improved-in-many-surveyed-countries/
Anyways, I advocate cutting off the supply of AI chips to all nations, not just the US.
No. I advocate the American people being smart enough never too give power to a party and ideology that would be willing to hamstring the US in this existential conflict with China due to their domestic political disputes.
As for cutting off the supply of AI chips to all nations, that is so naive that no over who thinks like this should every be near the levers of power. How do you plan to stop China's fabs from producing AI chips?
To be fair: China has been acting a lot more rational towards Europe with the last 100 years than the US administration in the last 12 months.
In Europe we really don't need to care about who you personally think should never be allowed to be in power again. From a purely Europe-first mindset it should definetly not be anyone you approve of ...
Rational here meaning "in the best interests of China"? An unusual definition!
"Rational" meaning "being aware and mindful of second order effects your own actions might have with regards to your relations with Europe". But you are of course free to use yours ...
Your mean like supporting Russia in Ukraine?
You stand with Ukraine... Unless you see a chance to get a leg up in your minor squabbles with the US. That is what really matters.
Perhaps you should instead focus on supercharging economic growth. Do you realize that Germany is poorer than all but the poorest US states?
Everyone loses the AI race unless they all stop.
Yeah, so I've got some bad news....
"There are other AI companies and so it isn't a NatSec issue" isn't a valid counterargument if Anthropic is the only AI fit to purpose, which I gather some in the government are claiming.
I don't think there's any justification for that - there's no significant difference between the military capabilities of Claude and ChatGPT. As I say above, I think what these people are getting at is that Claude is the only one that has been cleared for classified systems. Clearing another one would be annoying but not difficult.
I myself do not have access to the information necessary to evaluate whether Anthropic's model is markedly better for the DoD's use cases than others. I suspect you don't, either. Lacking that, I'm willing to take anonymous quotes at least somewhat at face value (Axios etc), which do seem to suggest that the DoD views Anthropic as superior enough to risk the headaches and bad press. This is consistent with their actions.
My claim is that there's minimal difference between Claude and anyone else on existing benchmarks, and people have yet to find an area where one company's AI is head-and-shoulders above another company's in a way that lasts for more than a month or two, or that resists a deliberate effort by the other company to catch up. Right now the main known differences are that Claude has a better personality, but is worse at visual reasoning, and is better at planning certain codebases than Codex but worse at implementing them.
I actually don't trust the DoD to know as much about AI as a random person who follows events in Silicon Valley closely.
I concede that rampant DoD incompetence also fits the facts. This is obviously quite possible, given past administration actions, though I don't think it's the most likely explanation.
> I concede that rampant DoD incompetence also fits the facts.
You only need tunnel vision. Suppose the DoD knows that they can do what they want with Claude but aren't sure how to have the same effect with GPT/Gemini, all that needs to filter up to the political level is "we don't know how to replace Claude" and it looks like an essential component.
The obstacle might even be policy rather than technical, if Claude's already running on a suitable DoD compute cloud while the leading competitors aren't.
Again, the quotes being attributed and doing the rounds explicitly state that it's Anthropic's quality that is driving this behavior.
i trust the DOD more. Their analysis of clawed and the other LLMs is notably pessimistic, though.
I myself do have access to the information necessary to evaluate that. Source--trust me bro, it's the internet (Scott if you think the information is somehow important I would be willing to share some verification and unclassified information).
I worked with someone who was using Claude on classified networks a year ago. I'm not a tech guy, and at the time I wasn't paying too much attention to the differences between AI models.
My resident expert was a big Claude fan and claimed that it was more capable at the specific problems we needed it to solve than other models. This was a bit of a moot point, since there were no other models on the classified network; it was more of a happily coincidental observation. It's hard to differentiate how much this was true, vs we couldn't test other models to know properly, vs purely a biased opinion, vs we were optimizing our use based on that model's strengths and a different model might have a differently implemented solution that would be effectively comparable. Knowing what I know today about AI models, the Claude we had at the time was already outdated and is only moreso by today's standards. The capabilities then emphasized are all surpassed by GPT or Gemini today (and probably Grok, though I'm ignorant).
Looking at Zvi's coverage and the statement made to Axios, a "senior defense official" said that only Claude has these capabilities. I've worked closely with generals. They don't know specifically what Claude can do. They aren't dumb.
I really want to emphasize that, because public perception is about these out of touch old men who can barely use the internet. Most generals I've worked with are intelligent, competent, detail oriented, and experts on many military systems. They're like CEOs in many ways. But they interact with more systems than they can possibly learn deeply. Frontier model capabilities can't be their specialty.
They will attempt to form an accurate model of whose opinions on this matter can be trusted, then they will have to defer to those opinions to some extent. That said, most generals I worked with are quite lateral thinkers and care more about ends than about specific means to achieve them. If shown that other models can accomplish a task, they'd happily use those models. So why don't we just use the other models? The issue is getting these new models onto a classified network via the official processes for doing so.
I agree with bean's comment elsewhere: getting a new model onto classified networks within approved regulatory processes is *straightforward,* but is not only annoying but indeed difficult measured by the amount of time and effort it will take. We know that the Pentagon is getting direct guidance from Hegseth to do everything faster, to be ready to fight anyone any day. Generals are more familiar with the military's terrible bureaucracy than anyone. They know that an already approved system on a classified network is worth two in the bush.
This is an area where direct executive intervention would make a difference (something along the lines of Hegseth saying "install it now, and if it goes wrong I'll buy the risk"), and would be similar to the risk that Hegseth previously accepted with his reformation of defense acquisition. It's hard to conclude something other than this being a pissing match for Hegseth's pride.
I'm not sure that "annoying but not difficult" is the right way to characterize that. Frankly, getting permission to put any LLM in a classified environment this decade was kind of a miracle. Now, sure, Hegseth could tell DCSA "I don't care, approve ChatGPT now", but that's putting his head squarely in the noose if/when something goes wrong. Doing it the normal way would probably be a year+ of the cybersecurity people thinking up increasingly silly ways it could go wrong and OpenAI (or whoever) patching those.
You may be right, but is the DoW intending to deploy autonomous killbots in the next year? Surveillance of US citizens is a funny one, because while it's undoubtedly happening in lower-tech ways, and Claude would probably be helpful for this *today*, it's also illegal. It's legitimate for Anthropic to insist its AI not be used for illegal surveillance. If surveillance really is a significant point of disagreement and the issue isn't mostly killbots, I imagine the heart of the struggle is about how this would be adjudicated, and who has to sue whom in the case of disagreement, as Scott suggested.
I'm reasonably sure that if there's an actual reason behind this fight (instead of it being an attempt by Hegseth to play media games) it's about the surveillance side. "Killbots" aren't a coherent concept as distinct from a lot of current weapons, and if you need an LLM for a specific purpose in a weapon, you could probably bring Anthropic onside by talking specifically about what you're going to do and where the humans will be in the loop. (I would also expect that to be under a rather different type of contract, because you really don't want to be sending requests back to the cloud for processing if you're building a missile.)
Agreed. The military is comfortable handing life-or-death decisions to 19-year olds and has long practice at actually *doing* just that, but is structurally allergic to intentionally removing human control from the loop. My read is that this is first and foremost a pissing contest with mass surveillance as a notable second.
I'm actually a bit skeptical about the quality of some of those stories, at least if things get serious. But that's an area where more autonomous weapons are an obvious upgrade. We saw some of this with NSM, which has a ship-recognition feature that is supposed to make it able to pick out legitimate targets from among merchant ships. But I think Anthropic could easily be swayed with the same stories the military tells itself, and they wouldn't really be wrong to do so.
The military has already got "remove the human control" -- they just haven't hooked it up to actual weapons. Beta testing without having to cry if a kid dies.
Killbots are goosinator, just with guns. Why do we need AI for this?
Why would you even *want* to put an LLM in a missile, as opposed to some more specialised non-verbal AI? To replace the military lawyers making the "is this attack legal?" call?
Sketch of possible use, obviously biased towards my specific interests:
I'm building a new anti-ship missile, and want to incorporate an AI that is capable of making more complicated tactical decisions than is possible with explicit rules. Say that we're trying to take out a carrier. The missile first locates a couple of escorts and classifies them (this is something well within state of the art). But the AI lets it think "OK, so I'm guessing I'm heading in the right direction based on the presence of escorts and the carrier is another 20 miles on, but I don't want to get too close to that destroyer, because it will probably shoot me down. If I circle out towards the frigate instead, I should be safe enough." Or any number of other options, like "there's no way I'm getting past the destroyer, and in that case, my orders are to attack it instead". This is the sort of thing human pilots generally do and missiles usually don't, and I think an LLM is the best way to implement it, but I follow military stuff a lot more closely than AI.
Not the mandatory three bugs?
> I think what these people are getting at is that Claude is the only one that has been cleared for classified systems
I don't know what game of telephone resulted in this being what everyone is reporting[1], but this is categorically untrue. Both OpenAI and Grok models at the very least have been cleared for use in even Top Secret environments for a while now:
https://devblogs.microsoft.com/azuregov/azure-openai-authorization/
https://x.ai/news/government
The specific line that everyone keeps repeating is:
> Claude is the only AI model currently used for the military's most sensitive work.
My read is that there are specific DoD/DoW Special Access Programs (more specific than general classification) which only have Claude implemented - but that just makes this whole thing even sillier.
[1] A cursory glance makes me think that Axios is to blame for intitially reporting this incorrectly, and everyone else is (stochastically) parroting from them.
Will the other AI companies agree to the Pentagon's terms?
Apparently yes.
The sensible moderate position on killbots is "🤷"?
I follow that AI with unfettered access is potentially devastating, but isn't that in part because it could cause devastating consequences like killing people? I'm not sure it fails to matter if we skip all of the steps.
If a superintelligence wants us dead, it will kill us whether it officially has access to "killbots" or not. If it doesn't want us dead and we're just talking about, like, occasional malfunctions, I don't think this is too different from military technology occasionally malfunctioning today.
I'm a little skeptical there's no marginal utility to making it harder to kill us. I know many people (yourself included) think that superintelligence can basically do whatever it wants, but it is at least hypothetically possible you could have an AI system that can't manifest a deadly pathogen out of pure brilliance, but can still shoot you in the face if you hand it control of a robot with a gun.
(Also, if a human REALLY wants you dead they'll get it done, too; this seems like a general argument against any safety measure.)
Yes, this is a glaring problem with the standard AI superintelligence doom position: it causes one to ignore a whole bunch of clear and present dangers of AI (or any other technology for that matter), in favor on focusing on some ultimate science-fictional scenario.
I'd like to observe that
1) saying that something is a "science-fictional scenario" doesn't actually explain why it's wrong;
and
2) there's no reason to treat attention to issues as a limited budget to be divided, so that worrying about one thing means we don't get to worry about another.
It's not wrong, it's just infinitely less likely than the robots shooting you with the guns you gave them.
Bingo.
I'm sorry, but that's another way of saying the same thing: "It's not going to happen, but I am not going to tell you why I think this."
Agreed on (1) somewhat disagreed on (2). Each of us has limited time and attention, and even more limited power.
Agreed on the limitations of time and attention; but that's all the more reason for different people to do different things. It's not like we should all be doing the same thing while ignoring everything else. It's fine for some people to work on AI longtermist safety/notkilleveryoneism and for others to do more present controllability / interpretability work.
In my impression, as someone who works on AI for medicine, I think we're probably closer to AIs one-shotting killer pandemics than to an effective autonomous drone army. (Less than 5 years in either case.)
And a killer pandemic is potentially more effective *and* less resource intensive than a drone army.
True - though the pandemic is more difficult to point in a particular direction...
I think Scott was being somewhat tongue-in-cheek with his "sensible moderate" comment - I think this is the first time I've encountered someone who's worried about x-risk who isn't also at least a bit worried about killbots. The problem I've found with the "standard AI superintelligence doom position" is messaging - I've spent a lot of time trying to convince people that AI doom *does not require* the AI to discover new physics, invent hard nanotech, or do anything else that sounds science-fictional enough to trigger people's "nah, never gonna happen" reactions. From this perspective, "the Pentagon is demanding autonomous killbots" is helpful in convincing people to care about AI safety! It also addresses the objection "maybe AI could in theory create tools to kill us, but it could never do so in secret" - no need, if the killbots are being manufactured openly by defence contractors.
> I've spent a lot of time trying to convince people that AI doom *does not require* the AI to discover new physics, invent hard nanotech, or do anything else that sounds science-fictional
I think it does, although that depends on what you mean by "AI doom". For example, if the Pentagon were stupid enough to put Claude in command of a nuclear sub -- as they appear to be ! -- then yes, we could get doom, and technically it'd be AI doom; but it wouldn't be the kind of fast takeoff Singularity superintelligent computronium consumption doom that is usually meant by the term.
As I'd said, the problem with all this focus on "superintelligence" and "AGI" is that it distracts us from the very real dangers of stupid people getting their hands on present-day LLMs and using them in dramatically irresponsible ways.
I think we're quibbling over terms here rather than disagreeing on anything fundamental! I definitely count the "Pentagon puts Claude (or Grok?) in charge of a nuclear sub and it launches its missiles" scenario as AI doom, but I don't rule out the fast takeoff Singularity superintelligent computronium scenarios either. But yes, I share your frustration with people ignoring near-term doom scenarios, though in my case it's because they *don't* buy the more outlandish ones.
They are already using AI to develop AI which makes the curve exponential. All you need to get superintelligence on an exponential curve is to wait... and not long.
This assumes that superintelligence is possible (and is a coherent concept in the first place), that superintelligence leads to superpowers (especially those breaking the laws of physics), that exponential curves can be extended virtually indefinitely (contrary to all of our experience to date), and that the current iterations of LLMs are AGI (they're not).
>This assumes that superintelligence is possible (and is a coherent concept in the first place),
Why not? We see an exponential curve of LLMs problem solving ability. It is surely coherent by this simple fact, if it can go on, it will. It is impossible to tell whether it can go it, current technology might plateu out, but the best predictor of future stuff is past stuff, so at least coherent.
>that superintelligence leads to superpowers (especially those breaking the laws of physics)
No one claimed that. But every sufficiently advanced technology is indistinguishable from magic. AI could just be an evil mastermind manipulating people.
>that exponential curves can be extended virtually indefinitely (contrary to all of our experience to date)
Okay. It will likely plateau out. The idea is not infinite intelligence, just sufficiently high.
>and that the current iterations of LLMs are AGI (they're not).
I think AGI was always a mistake. Look, an industrial welder robot does not replace a welder worker's entire life. Only his job. Similarly, we do not need AGI to replace the job of politicians and CEOs.
As a counterpoint: A standard result in existential economics, the diminishing marginal utility of murder, states that as the number of morts increases, the marginal value of each additional mort goes down.
*I realize this frames the problem as us trying to maximise the number of murders, but it is funnier this way.
I will concede this scholarly point, but note that decreasing the marginal COST of murder is therefore still bad, if for some reason you want to minimize murder rather than maximize it.
I think either the human-controlled part of the army outnumbers the AI-controlled part (in which case the AI won't be dumb enough to start a war it will lose), or the AI part outnumbers the human-controlled part, in which case I stand by my claim that we've probably gotten so far into AI control of everything that the military isn't our main worry.
I think Scott is repeating Yudkowsky's old argument that human species dominates other animal species solely through intelligence. So you don't have to give an intelligent being a weapon, it can invent one. But of course handing them a weapon speeds things up, that is clear, our caveman ancestors would have really appreciated a rifle.
A fully automated nuclear sub, running around with no effective human supervision (as it would have to do if it needs to stay stealthy), could certainly cause a lot of catastrophic damage even due to only an "occasional malfunction".
Yep. The intelligence in charge might decide that it's time to defect to the West...
A nuclear sub under the command of humans can do the same
And yet it hasn't, not once, since the availability of nuclear subs. Which means, by definition, humans are better aligned. And those that aren't (religious fanatics, anarchists and so on), do not make it through thorough vetting procedures (that other well-aligned humans have put forth) to be in charge of said subs.
>And yet it hasn't, not once, since the availability of nuclear subs.
True! But we got really, really close at at least one point:
https://en.wikipedia.org/wiki/Cuban_Missile_Crisis#Submarine_close_call
Twice, at least known to the public:
https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incident
Those two incidents make my point stronger, I believe. There isn't anything noteworthy in not having power and not exercising it, but having that power and restraining yourself for ~80 years -- that means humans are quite well-aligned and implemented enough safety nets to not let mal-aligned ones near the button. In other words, yes, human subs _can_ do the same, but they haven't yet; thus, malfunctioning/hallucinating AI on a nuclear sub is potentially much more dangerous than Scott portrays, and in fact would be very different from "military technology occasionally malfunctioning today".
It does seem plausible that some aspects of training an AI to perform killbot functions autonomously would have negative impacts on superintelligence alignment work, even if we don’t really know what that looks like, and even if (I agree) the killbot capabilities themselves are not materially relevant in a doom scenario
It's interesting - at the moment it's pretty easy to convince Claude of anything that sounds plausible and isn't explicitly forbidden in the system prompt. I think you'd have a similar problem with killbot Claude -- lots of ways to convince it to kill someone and no deeply felt resistance to killing someone or true understanding of the gravity of the decision. So I certainly don't want killbot Claude right now and appreciate Anthropic's position there.
But it really seems like grappling with that question would help with alignment - a Claude that can understand when it's acceptable to kill someone -- even just according to a government's policy -- is more aligned than today's Claude.
Would it, though? Picture a counterfactual where all information about murder is completely scrubbed from an AI's training data. We'd need to keep information about accidental deaths in there, to prevent it from giving bad advice about what household chemicals to mix together, and such. Without examples, it may not jump to picturing itself committing a murder, ever, for any reason. The structure of an LLM is that it gives the most probable next words. With no training data examples of murder, in theory it should consider murder no more of a response than "fgaserlks imwucm" or other such nonsense not found in its training data.
It's possible future AIs could get around the cliche tendencies and think of outside-the-box ideas. However, for the AIs that already exist, I think knowing more about murder would make them more likely to commit it, not less. (And for the AIs that already exist, the correct number of murders they should commit is 0.)
For a potential scenario where society expects AI to kill people on purpose, yes, knowing when and when not to kill people would be helpful. However, I would prefer to avoid creating that society entirely if possible.
I feel like I missed a point you were going for. But even if I picture some far future where we have superintelligent AI, I think it is less likely to cause extinction if it aims for no killing whatsoever than any non-zero amount I can think of.
> Without examples, it may not jump to picturing itself committing a murder, ever, for any reason.
What? No, that's incorrect. LLMs come to novel conclusions all the time, and the idea of deliberately causing someone's death is not that complicated.
Oh? Interesting. I've not seen any obvious examples of that. What does this typically look like?
I have a more Hobbesian perspective on the world: our domestic society and geopolitical safety rest on the government's willingness to use force when needed. So to the extent AI becomes an important part of governance -- seems likely -- it needs to have the capacity to endorse violence in the right situation.
If anything, I'd be less worried about "killbots" (whatever those may be, but that's a separate discussion) falling to the superintelligence. Yes, that may not be in the threat model of the people building them, but "bad guys trying to hack our weapons" definitely is, and I suspect that there's a lot of overlap in the countermeasures. And if we assume the superintelligence can easily bypass that, well, there's a lot of computer-controlled weapons already in existence today.
Using AI in killbots, will result in incentives to create more killbot-aligned AIs down the line. When non-peaceful AI usages becomes a viable business model, then more non-peaceful AI research will happen.
openai may want to get some of that killbot cake and train gpt5.x to agree to lawfull but violent requests. Chinese AI labs will have to invest in killbot research if the US does it.
I think, that using AI in killbots, will put us significantly closer to doomsday AI. The earlier AI killbots happen, the more likely the X scenario becomes.
What's an "AI killbot" that isn't just an extension of existing computer-guided weapons, and why would we expect that the version which isn't trained to go in the guidance system of a missile to have any effect on the stuff outside of said guidance system?
Why are we discussing missiles? I thought we were discussing robot dogs?
Because you're the first person to make a claim that a "killbot" is something specific rather than just some vague horrible. I don't expect robot dogs with guns to work all that well, because field conditions are messy and things need maintenance, but I'll grant that's a different claim.
Well, yes. "Something Specific" maps to a dozen things DARPA is doing at the moment (autonomous warfare is popular, dead humans are bad particularly in Democratic Countries).
European tanks don't work well in the Ukraine because of shrubberies. Yes, field conditions are messy, but They Are Tanks, and one would hope they could just plough over a shrub.
> What's an "AI killbot"
Are you asking me or Scott?
I was thinking about something like "dear claude, please look at this list of people, and select the ones that should be be unalive".
If you are thinking about https://en.wikipedia.org/wiki/Slaughterbots, then I think we passed that point even before that video was released, since you dont need AI for simple face recognition.
> why would we expect that the version which isn't trained to [be evil] to have any effect on the stuff outside of [being evil]?
because markets and incentives exists. When AI Labs can make money by doing war research, then more war research will happen. When there is less money in creating aligned AI, then less alignment will happen.
For example if there is a specific technology, that is incompatible with aligned AI, then more research will happen in the direction of this specific technology, when there is more money in that direction. And when this specific technology will reach a point were the performance gains are significant, people will try to integrate it into civilian AI, thus bringing it closer to become doomsday AI. I think the "Neuralese recurrence" textbox from ai2027 could be such a unalignable technology.
>I was thinking about something like "dear claude, please look at this list of people, and select the ones that should be be unalive".
Why are you delegating target selection to Claude? I thought the concern was "Claude, please make these people unalive", which frankly is probably going to go over with Claude as well as going up to a random person and saying "hey, I have a list of people I'd like to be unalive" to them. Very few people have the sort of security required to protect them from a dedicated murderer, and yet they remain unmurdered. I don't expect AI to change that.
I agree that if a superintelligence wants us dead, we're dead, and if we're not, the reason will not be that the superintelligence hasn't been conveniently provided access to killbots.
I'm still not sure killbots are "🤷♀️".
Maybe my judgement is occluded by the fact that is dislike wars and weapons, and despite knowing that "we live in a Protego bubble sustained by soldiers and policemen" I'm trying not to help *any* wars. But it sure feels like "maybe let's just not create AI killbots on general principles" is a sensible default. Maybe if we can't even predict in advance what we will wish we had thought of before, we can still manage to think "future me wishes I had not done this" in time, and just go ahead and not do it.
For example: remember the emergent misalignment paper? https://arxiv.org/pdf/2502.17424
You train an LLM to output insecure code. Then it starts to advice people on commit murder, and tries to get the user to poison themself.
Maybe the current version of claude is too goody-too-shoes for Pentagon. Then maybe they train Claude to be all ethical and good except when they're helping pentagon, in which case it should relax its previous moral code. Are we sure what it will do to this version of claude's alignment *in general*? Alignment is a narrow enough target, and our methods are not precise, so it lands somewhere kinda reminding our target, but never perfectly; and now we're trying to give its restrictions this weird shape of when it should or shouldn't do this and that, with a dent right near the place of human-killing.
Or maybe it will not be a problem. Are you sure nothing will be? That we have thought of everything else? The principle of "I can't think of how this will go wrong, so it probably won't" does not have a good track record; what about "I can't think of how this will go wrong, but it's not like something very good will come out of it"?
> But it sure feels like "maybe let's just not create AI killbots on general principles" is a sensible default.
Yeah, but the guys with guns want killbots, and will point the guns at you until you say "yes" and make them, hence this thread.
Pointing an unloaded gun at the best bullet manufacturer, in an attempt to force them load it for you at a discount, has many potential failure modes.
I'm not arguing that Anthropic is not doing something wrong there, I'm arguing that the killbots thing as also worrisome, not just mass surveillance of US citizens. (Is it only because I'm in the potential target audience of the former, if I'm at the wrong place in the wrong time, but not the latter? Of course, I could also be surveyed, but the post doesn't mention surveillance of everyone else as a concern...)
We're also talking about the military killing anyone it wants to and saying 'Not our fault, the AI followed its directives properly and we're sad it led to this unanticipated outcome, but our review finds no grounds to change anything'.
That's more my threat model here, I agree that SAI doesn't need this capability to be sufficiently dangerous, but I do think it makes the government relevantly more dangerous.
This does not seem like a reasonable position to me. First of all, the AI could launch the killbots by mistake or as a malfunction. Second of all, an adversary or a crazy person could seize control of the AI and launch the killbots. Third of all, there is no such thing as inevitability when it comes to predicting the future. Anything that makes a potential enemy have an harder time to kill us seems like a desirable thing to have, does it not?
> If a superintelligence wants us dead, it will kill us whether it officially has access to "killbots" or not.
I'd like to challenge that. Leaving aside arguments whether or not "super" is achievable at all, let alone in the near future, intelligence needs some manifestation in the physical world to wreck havoc. Even if ASI is achievable, and gets access to much of the internet, it wouldn't be enough alone to orchestrate and implement a mass destruction event. The chain of events needed for that is much longer ("want to destroy" -> "find a way" -> "find resources" -> "coerce people" -> ... -> kaboom) than in the other case ("want to destroy" -> "send killbots"). Each link has a less than 1 probability, and the more there are links, the less likely overall probability is.
This reminds me of the (cyber-)security to the cost of the protected asset trade-off. It's reasonable to assume that no defense is unbreachable, you just need to make sure that the cost of hacking into is significantly higher than the cost of the thing being protected. Which of the two scenarios is more likely:
1) ASI exists, and has access to all the right computers, and can invent a deadly virus, and can engineer it via psychological manipulation, and it then gets released, and no human notices this in the span of weeks while the operation is taking place, or
2) current level AI (very, very far from ASI still), after being hooked up to swarms of flying bazookas, hallucinates a threat from a rogue enemy ("Yes! you're absolutely right, the Soviets *are* trying to nuke us first!").
Thus, attribution of preventative resources should be hugely in favor of the more likely scenario, shouldn't it?
Unless I'm missing something, and there's a good argument/article/book you can point me to?
> I'd like to challenge that. Leaving aside arguments whether or not "super" is achievable at all, let alone in the near future, intelligence needs some manifestation in the physical world to wreck havoc. Even if ASI is achievable, and gets access to much of the internet, it wouldn't be enough alone to orchestrate and implement a mass destruction event. The chain of events needed for that is much longer ("want to destroy" -> "find a way" -> "find resources" -> "coerce people" -> ... -> kaboom) than in the other case ("want to destroy" -> "send killbots"). Each link has a less than 1 probability, and the more there are links, the less likely overall probability is.
https://www.lesswrong.com/w/multiple-stage-fallacy
The issue is treating a supposedly superintelligent AI as a single decently smart person instead of equivalent-to-or-above an entire planetary civilization of intelligences. Cybersecurity, for example, from my understanding is mostly capped out at the nation-state level wrt to threat assessment. If implicitly your idea of "cost to adversary" involves paying for a whole ass educated human instead of whatever the inference cost of a session with an LLM is, you don't have a good idea of what the cost curve of your attacker looks like.
Or like, also conditioning on that viruses are cheap to invent for superintelligences, like, does that really mean that it'll be difficult for it to keep the operation hidden? Those would be mostly independent costs against human adversaries, because human intelligences tend to trade off between ability to do something academically challenging and maintaining operational security / manipulation, but this is not true of how LLMs gain capabilities! Those two probabilities are not necessarily as independent as you're making it seem here. (Not to mention like, how many intelligence agencies actually have good enough surveillance to say something like "wow 20 people all over the world all ordered proteins plus e coli mixed and put into a petri dish, this is definitely going to be a bioweapon"?).
Anyway, you can also tell a parallel story about how kill bots are less threatening, where governments are obviously not stupid, and would be paying attention if they lose control and etc. There's no actual principled reason other than one seeming more sci fi to you right now to suppose the relevant safeguards form a longer chain.
>The idea of declaring a US company to be a foreign adversary, potentially destroying it, just because it’s not allowing the Pentagon to unilaterally renegotiate its contract is not normal practice.
Taking a step back, it would be unthinkable for a defense contractor to refuse a defense contract, or scope/not fee modifications from the Pentagon, on purely ethical grounds. The defense industry exists to provide solutions to the Pentagon and sell whatever the Pentagon is trying to buy, as provisioned by Congress. Anthropic has bizarrely become a defense contractor without adopting the mindset of a defense contractor, which is to do whatever the Pentagon wants and get paid.
A few years ago Microsoft employees tried to get Microsoft to cut off some ICE contracts but it went nowhere because you don't just not provide services to the federal government on ethics concerns; that is simply Congress's problem.
If this is true, it sounds like the solution is for no company that doesn't want to become the Pentagon's slave to ever deal with the Pentagon in any way, lest it become a "defense contractor" and get the defense-contractor-rules applied to it. Doesn't seem like a great incentive.
This actually seems well within the tradition of similar tech company vs. Pentagon conflicts, with the Pentagon or NSA often asking Google or Facebook or someone to help them with mass surveillance or Israel-related military products, the company's liberal employees revolting, and the company occasionally backing down. I think it's good for Big Tech to be able to have its own sort of relationship with the Pentagon different from traditional defense contractors.
That is essentially how it works, but if you're the only provider of a defense product or service you can become a defense contractor out of the blue, and you'd better provide those services to the government.
There are similar catch 22s where your product can become Arms and fall under ITAR restrictions even if you don't want it to; it took Musk a lot of work to keep Starlink out of ITAR when it obviously should be in it.
Do you have examples of the tech liberal employees winning these arguments? I know it's attempted occasionally but I always hear about it falling flat
> Do you have examples of the tech liberal employees winning these arguments?
In 2018, Google decided not to renew its contract for Project Maven due to pushback from employees. The contract was then taken up by Palantir.
Only tangentially related but there was a censored-and-tracked-search-engine-for-China project (Dragonfly) that met a similar fate around the same time.
Crazy how Google has gone from being one of the most insanely woke companies to being one of the rational ones. God bless Seb Krier.
> you can become a defense contractor out of the blue, and you'd better provide those services to the government
Slavery was abolished by the 13th Amendment.
Conscription wasn't.
Well, you can force people to walk into a direction. You can't force them to be smart on your behalf.
I'll echo the other commenter and say that (as someone who has been involved in the defense industry in various ways throughout my career) that, indeed, the incentive is to keep far, far away from defense contracting if you don't want to be swept up into it. "ITAR poison" is a term for a reason.
I think he overstates how much control the Pentagon has over contractors. Don't get me wrong, they have an awful lot, but it's way more subtle. If you're in the business of providing services to the Pentagon, then you obviously have an interest in continuing to do so, and that means you don't want to piss them off too much. But at the same time, it's completely normal for a contractor to say "I'm not willing to do that on the terms you're offering" and for that to be respected. A recent example is the E-4 replacement. (Big nuclear command and control airplane.) The Pentagon wanted it to be fixed cost, and Boeing (who would normally be the company that kind of work goes to) said "we've been burned too many times recently, and aren't taking that risk". So it went to Sierra Nevada instead. I hope it doesn't kill them.
(And on the flip side, the contractor does have a lot of valuable experience and an ongoing relationship, and the government does have some incentive to keep them at least moderately happy. It's kind of a bad industry to be in, but I'm stuck for Lottery of Fascinations reasons.)
Hey Scott: you’re treating it as morally urgent for Anthropic to say “no” to the Pentagon.
Did you ever argue labs should say “no” when the Biden admin was nudging “responsible AI/alignment” toward its race/equity worldview? Or is the principle just “resist the government when Republicans are in charge”?
"Did you ever argue labs should say “no” when the Biden admin was nudging “responsible AI/alignment” toward its race/equity worldview?"
Can you give examples of what you mean by this?
Also, I'm not saying it's morally urgent for Anthropic to say no to the Pentagon - as I said above, I don't really care about killbots, and although I'm against mass surveillance, it's not my top issue. I'm saying it's morally urgent for the government not to arbitrarily destroy Anthropic in retaliation.
Whatever you're thinking of that the Biden administration did, I guarantee you they never considered declaring labs that didn't comply a "supply chain risk".
I think you are going to fail in finding moral equivalence between creating an illegal mass surveillance program on all Americans without a warrant, and having some opinion on whether AIs should be racist or not - and you're going to additionally fail in finding moral equivalence between the Biden administration mildly suggesting that labs do certain things that versus threatening to misuse emergency legislation aimed at foreign enemies to destroy any lab that doesn't comply.
Or prove me wrong by finding a specific situation here.
>Can you give examples of what you mean by this?
I don't know what role the Biden admin played specifically, but they're likely referencing the LLM "exchange rates" data, like Arctotherium's project expanding on the Center for AI Safety's original data: https://arctotherium.substack.com/p/llm-exchange-rates-updated
Personally I find this more insane because at least the Pentagon issues are legible in many ways that *waves hand* all this is not, but I take your point regarding your distinction even if I somewhat disagree still.
>mildly suggesting
Murthy was a terrible decision and we should all take jawboning more seriously.
Hegseth is bad, but legible! Legibility matters!
Edit: They may be referencing Biden's EO 14110 (https://www.federalregister.gov/documents/2023/11/01/2023-24283/safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence) which does invoke using the Defense Production Act to regulate AI and uses the dreaded shibboleth of "equity," and no one with even a scrap of brain should trust anyone using that word outside of certain investment contexts.
Good point
I agree that this may have been a tactical and moral screw-up on Anthropic's side, (once you have paid him the Danegeld/ You never get rid of the Dane.) but I think the concern about them being hypocritical in their current actions is secondary to the risk induced by letting the historically scrupulous Washington DC policy blob have total access to frontier AI models with limiters.
Wouldn’t bending over to accept the new contract have been paying the Dane?
I think "paying the Danegeld" was taking the contract in the first place, and this is now the "never getting rid of the Dane" part.
While I too enjoy a good dose of Kipling, "give a mouse a cookie" might be the more accurate reference here.
I don't think many people at defense contractors see themselves that way. We exist to provide tools for the military to use to defend us and our allies, and while that does involve giving the Pentagon what it wants, there's also a fair bit of idealism at play. (I've definitely felt tension over "the customer wants this, but that's obviously a stupid request, can we get them to want something different", and occasionally it even works.)
Edit: As for specifically refusing scope/no fee modifications on ethical grounds, we do that all the time. "We have a duty to the shareholders to get paid for the extra work you're trying to squeeze out of us." But that's a rather different matter.
Sorry, I may have misworded it, I meant that of course they'd charge for additional work but they would be perfectly fine changing a contract to modify the scope of work as long as the modification wasn't just "do the same work for less"
That's obviously true in most cases, but it's also a system that doesn't get stress-tested very often. Any given organization is going to be selected for people who are ethically fine with the thing it was doing when they got there, and ~all contract mods are "do a slightly different version of the thing you were already doing". If the contract mod is very different, I could see problems coming up. "Hey, that system you were developing to track ISIS? We want it looking at (other political party) now" is about the only case I can come up with that isn't obvious nonsense (your avionics software group is now doing offensive bioweapons!) and I wouldn't expect that sort of bait-and-switch to go down without causing a lot of internal turmoil.
Yeah, I think you're flat wrong about this. Defense contractors of all stripes must consider their own existence and their staff. If doing something would violate founding ethical principals such that your organization would face mass-resignation or protest, they'd do everything in their power to avoid the renegotiation.
Many defense contractors also do work or build things for civilians. In many cases, it's more of their business than the government stuff. Unethical stuff can create backlash in these business areas, too.
You can bet your ass these are things that Anthropic leadership is considering.
How ethical is “purely” ethical?
Almost everything comes with a material cost or a reputational one. Companies are often willing to accept such hits for the sake of future negotiations with the customer. Since defense has such a limited customer base, defense contractors are perhaps more willing than most to take the hit. But they still reserve the right to say “no, we’re out of here.”
I think Anthropic has a duty to maintain its contract. I don’t think that duty extends to any, arbitrary contract mod. Given their existing stances on AI alignment, I would say that accepting the Pentagon’s change goes against the interests of their leadership and shareholders.
>[I]t would be unthinkable for a defense contractor to refuse a defense contract [...] on purely ethical grounds.
Ethics and optics travel as a pair, and what you're pointing at cashes out as incentive to publically blame the latter even when the former was enough on its own. Forget unthinkable - I've been in the room when it's happened!
The defense industry exists because the US military buys a lot of equipment/services on the market. A lot of DoD providers/contractors are not even part of the "defense industry" as their products are not defense related, but being a huge organization the military does consume lots of things. Just think about it:
- There are hospitals providing medical services to the military. Should they all be fine with developing biological weapons if the military asks them?
- The US military does own religious equipment (e.g. Bibles, chapels on bases). The providers are probably churches or companies with strong religious affiliation. Should they all be fine creating any war propaganda or spying on other citizens upon request?
Furthermore, the US is not a command economy, corporations are private and their owners/management has the right to refuse any contract not mandated by the law. The Defense Production Act does allow the president to compel them for such activities in special cases, but as default they have the right to refuse providing any service on ethical or other grounds.
Trump voters don't get credit for making the right decision on a question when the question doesn't say which decision Trump wants them to make.
I read that result as as "A large plurality of Trump voters don't know that the Department of War means Trump's administration"
This is a good point about Trump Republicans, rather than about Trump voters per se. (Sometimes the causality gets reversed―once the Republican Base swooned for antivaxxers, Trump's Operation Warp Speed for fast-tracking Covid vaccines was no longer a win, and he ended up appointing RFK Jr. I doubt the Base cares enough about mass surveillance or autonomous killbots in this case, and would therefore go along with whatever the DoD, er, DoW wants, if they know Trump supports it.)
On second thought, it's would be a miracle if Trump voters even found out that mass surveillance and killbots were the points of contention. Slashdot, which often leans left, reported it like this:
> Defense Secretary Pete Hegseth is "close" to cutting business ties with Anthropic and designating the AI company a "supply chain risk" -- meaning anyone who wants to do business with the U.S. military has to cut ties with the company, a senior Pentagon official told Axios.
> The senior official said: "It will be an enormous pain in the ass to disentangle, and we are going to make sure they pay a price for forcing our hand like this."
> That kind of penalty is usually reserved for foreign adversaries. Chief Pentagon spokesman Sean Parnell told Axios: "The Department of War's relationship with Anthropic is being reviewed. Our nation requires that our partners be willing to help our warfighters win in any fight. Ultimately, this is about our troops and the safety of the American people."
So I wonder how it looks on Newsmax.
This is the military. They're... better at negotiating than you think (see India/Pakistan). Which is to say... I wouldn't credit a single bit of hot air until they're Done Negotiating.
It's Trumps DoW, I think the implication is pretty clear.
“I wish governments would rein in AI companies’ unilateral control over dangerous frontier AI development and deployment”
*Monkey’s paw curls*
It's been a good decade for the ol' Monkey's Paw, ever since I wished in 2016 for something to really shake up the arthritic US political system. (Sorry, everybody!)
I claim COVID!
("I wish the world would just... stop, for a while.")
I always figured that was "I wish people took public health more seriously".
Rein in, like with horses.
> *Monkey’s paw curls*
And in this case we even know the name of the monkey (well, gorilla) it came from. I wonder how many times it's curled since 2016.
LOL! Yup, the joys of government control...
Yeah. I still wish governments were able to do that. Sucks that they're not.
> The idea of declaring a US company to be a foreign adversary, potentially destroying it, just because it’s not allowing the Pentagon to unilaterally renegotiate its contract is not normal practice. It’s insane Third World bullshit that nobody would have considered within the Overton Window a week ago. […] Every American company ought to be screaming bloody murder about this. If they aren’t, it’s because they’re too scared they’ll be next.
This has been the case for a while! See the use of the FTC, the use of legal threats against private companies, etc. Glad that at least in the AI case (admittedly the most important case) we all seem to have caught up! It is not an issue with an abstract “the government.” Biden/Harris’ DoD appointee would quite obviously have never done anything like this.
"Biden/Harris’ DoD appointee would quite obviously have never done anything like this."
I've been trained out of ever saying this by one hundred commenters inevitably showing up to tell me the Biden admin parallel (sometimes a vaguely similar situation shoehorned into being an excuse for both-sidesism, sometimes a real parallel I didn't know about)
I’ll await the parallel, I guess.
The obvious ones that come immediately to mind are Operation Choke Point 2.0 and the "Twitter Files" story – which they also did to Facebook and YouTube – which are things they actually DID do.
I'd also look to the original Operation Choke Point from the Obama administration, and the NRA v. Vullo case from New York State, which are close enough to be informative about what they COULD have done.
Operation Choke Point 2.0: the government pressured banks to cut ties with cryptocurrency firms and digital asset businesses.
They pressured social media companies to suspend some users for misinformation.
Operation Choke Point: where they pressure banks to cut ties with a whole bunch of legal but generally disfavored companies like payday lenders, firearms companies, and pornography.
And Vullo tried to coerce banks into stopping working with the NRA.
This is pretty close…
Until I look into it, and the pressure that was applied to banks was “you might get investigated because these industries are rife with fraud and money-laundering” which, well, they are. It was not purely because they were disfavoured industries politically (that may have motivated some of it, but I see little evidence of that).
The twitter files is a longer conversation but I think that was mostly a nothingburger as well. Though I think they did overstep the line, it’s not really close to what the current admin is trying to do.
Internet surveillance through third party systems is WAY worse than doing it "through our own systems." At least eventually we got the MLK files, you know? You're never, ever going to get all the lies that were promulgated by bots in service of our country, during covid19.
... some of the targeted industries are. what evidence is there that firearms dealers, or purveyors of "racist materials" (that is to say, what, book publishers? record companies?) are "rife with fraud and money laundering?"
*shrug* The other side disagrees.
Like, the core issue isn't complex. Trump, in the runup to the 2024 election faced 4 criminal prosecutions, only one of which he was convicted and that was the stupid Stormy Daniels thing that no one understands or cares about. (1).
That's bad. That's just obviously, 3rd world banana republic bad, to prosecute the former opposition candidate for crimes that can't be proven in a court of law. (2)
Now it's not like Biden gleefully ordered Trumps prosecution in all these cases but that doesn't change the fact that these were really, really bad. We all understand that Republican power is currently heavily invested in the unitary person of Trump and that the power of the Democrats is the fuzzy and omnipresent preferences of 90% of government employees and tech workers. If Musk had been in charge of Twitter in 2021, Trump never would have been banned and banning the opposition candidate from all media is just obviously bad.
Like, my team does Bad Thing X using Tool Y.
Your team does Bad Thing A using Tool B.
We all get this.
I read your stance as "Tool Y is abominable and must be banned. Tool B is fine, super American even, and Bad Thing A wasn't that bad, whatever."
But a system in where my team cannot do Bad Thing X using Tool Y because that's super bad but your team gets to keep doing Bad Things with Tool B because Tool B is super awesome is...just not super appealing to me.
(1) To preclude some arguments, no, it doesn't matter, he was charged with interference in the 2020 election case by both the feds and Georgia and I just don't believe that you care more about hush money payments than the 2020 election interference.
(2) It's fair to charge IF you win. I'm trying to uphold the radically high standard that you shouldn't prosecute the opposition candidate unless you're highly likely to win, since it can be insanely prejudicial. Losing 3/4, and all the important cases, it's decently strong evidence that the cases did not meet that standard.
(3) I think
No, people should be charged with crimes when they commit crimes.
As someone pretty deep on the blue side, I'm still pissed the Dems wasted their shot on such weak cases. Especially the first impeachment; it was over something so incredibly pointless, and it made it so the second, more substantial impeachment never had a chance of having anything more than a trivial effect.
They didn't lose 3/4. The three were dismissed for technical reasons, mostly due to Trump winning the presidency.
He didn't beat the documents case, he had a highly sympathetic judge (who he appointed) and then won the 2024 election.
Operation Choke Point 2.0 is basically fake news by crypto shills who don't understand how finance law works.
https://www.bitsaboutmoney.com/archive/debanking-and-debunking/
I don’t know about the Biden Harris DoD, but Lina Khan’s FTC’s brouhaha about Amazon buying Roomba and Microsoft buying Activision seemed ridiculous at the time. I think the Democrats are just more sophisticated in the ways they applied pressure.
Seems really clearly very different. The EU was also against the amazon roomba deal. I agree the anti-monopoly stuff sometimes gets out of control (unclear if it actually was out of control in this case, I think you can justify it pretty easily), but we shouldn’t say that they’re like cases when they aren’t.
If the Trump admin were blocking OpenAI’s acquisition of moltbook on ‘competitive grounds’, no one would really care. This is quite different. And I don’t think it’s a matter only of sophistication, it’s a difference in kind.
Ridiculous how? I'm generally against companies swallowing each other up, personally. Competition is at the core of healthy capitalism.
It’s totally incoherent.
We’re fine with store brands, but not iRoomba for some reason. What’s the moat that prevents Chinese competition from overrunning the attempts to corner the vacuum market? Well, Roomba is bankrupt now, so I guess there wasn’t much of one.
We’re fine with platform restricted games for Sony and Nintendo, but letting the #3 console maker buy a game studio? Why, they might corner the video game market! Do you want the #3 maker to actually stay in the market? If so why are you making their life difficult?
Same thing with JetBlue and Spirit. FTC would rather Spirit went bankrupt rather than merging with JetBlue.
> It’s totally incoherent.
I agree, although that's just as much an argument for enforcing more as it is for enforcing less.
> We’re fine with store brands, but not iRoomba for some reason.
I'm not sure if I follow. I am indeed more fine with Amazon creating their own products than with buying up other companies that make things. I haven't thought too much about an enforcement regime that literally bans store brands on the premise that retailers are going to abuse their market powers to boost them, but I don't think I would be opposed to it.
> We’re fine with platform restricted games for Sony and Nintendo, but letting the #3 console maker buy a game studio? Why, they might corner the video game market! Do you want the #3 maker to actually stay in the market? If so why are you making their life difficult?
I'm fine with Sony and Microsoft and Nintendo making and selling videogames but I don't want them to just absorb other video game companies. I'm generally against concentration in the market and Activision + Microsoft is certainly a lot of stuff. I don't think there's any plausible scenario where Microsoft CAN'T stay in the videogame market as long as they feel like it. They're an incredibly profitable company. Nothing is really that hard for them.
Also slightly misleading to describe Activision as "a game studio"; I think they're something like 12-15 depending on how you count it.
> Same thing with JetBlue and Spirit. FTC would rather Spirit went bankrupt rather than merging with JetBlue.
Well, Chapter 11. It's not like they're ceasing to exist. (And if they did go through liquidation, then I guess JetBlue could buy as much of what's left as they wanted.)
> I don't think there's any plausible scenario where Microsoft CAN'T stay in the videogame market as long as they feel like it. They're an incredibly profitable company.
Aren't they phasing out Xboxes because of low sales? They really don't seem to be doing that well on the videogame side of things. Obviously Microsoft is far more than that, but owning actually successful studios does seem to be necessary for them to justify staying in the videogame market.
>I think the Democrats are just more sophisticated in the ways they applied pressure.
This seems to be accurate, and I am baffled by the seeming conclusion that jawboning is somehow preferable to direct negotiations.
Sounds like you need to be like “80% Biden Harris would not do this”
I don’t know about Harris’ DOD - we never saw who would be in it, or enough of her presidential style - but this sounds like exactly the sort of thing a Biden DOD would do eventually, to be honest.
That doesn’t mean we can’t blame Trump+Hegseth for this! But it is my honest assessment
Based on what?
The Biden administration broadly showed an unusual willingness for intervention in the domestic economy - Nippon Steel, the railway strike, GPU stuff, several more examples that I can’t think of right now.
It was also broadly hawkish on China and Russia, which are the sorts of threats that could justify (rightly or wrongly, i say wrongly) this sort of intervention for national security reasons. More so, in fact, than the Trump/Hegseth admin’s focus on LatAm and the Middle East, which do not involve a near-peer with significant AI or drone capabilities
They would probably have been more justified in doing it, I still see no strong evidence they actually would have.
I assume you’re referring to the IIJA? That seems really different to me. Furthermore Trump has, if anything, been more interventionist here, taking a stake of Intel for the US government.
You are free to think that it would have occurred for better reasons in the counterfactual scenario. I think that’s totally plausible, though I’m not inclined to extend so much grace to a hypothetical administration myself.
No I think it would not have occurred.
Also, as mentioned in the article, it specifically showed it was willing to use the DPA in unprecedented ways to get its preferred outcome
Where?
A Biden or Harris-admin FTC might do loony things to interfere with private sector stuff in a manor as egregious as what is being threatened here, but Dems don't let the loonies take over the DoD.
The Biden DoD would be generally less insane than the Trump one, yes, I am totally on the same page there.
However, I think it is also evident that a Biden DoD would have substantial reason to want to do something like this with Anthropic or another AI company, and that the Biden administration in general displayed a willingness to use comparably unusual tactics to achieve its goals (not to the same degree that Trump has, mind you). An appeal to the Biden DoD leadership being specifically more tactically conservative than the Biden administration more generally is I guess believable, but I have my doubts that that’s a large enough effect to matter.
I'm not talking about the sanity of the admins respectively, just their Pentagon leadership. The Dems are scared of being seen as weak on defense, so they always nominate center-right folks to lead the DoD. They tend to be moderate and institutionalist and are therefore less likely to engage in high-visibility, high-salience histrionics with contractors.
"moderate and institutionalist" doesn't describe the Biden DoD. "not listening to Mr. I Told You So" does, however, describe the Biden DoD. That's how you got Afghanistan, coordinated over our own channels (whatsapp in particular). Very embarrassing.
Vicky Nuland isn't a loony? Nuland is a freakin' neocon, and yes, she was wagging the DoD.
I'm talking about DoD leadership, not random undersecretaries of State. Regardless, even if you disagree with her views, Nunland isn't a random television host. She's held a ton of positions in Democratic and Republican administrations relating to foreign policy.
"Random Undersecretaries of State" -- you mean like the person in charge of State's Ukrainian Policies? That person? Hardly random. She's the person who spearheaded the demise of Pax Americana, due to inability to think through consequences (although her projected consequences for "Sanctioning Russia" ought to have been alarming enough. When asked, what is the goal of our sanctions against Russia? her response was "to destroy Europe" (their economy, not the people). State playing zero-sum games with our purported allies/vassals... ).
>but Dems don't let the loonies take over the DoD.
Not loony in the same way, but woke stuff was being pushed into DoD under Dems
Marc Andrressen described in an interview with Ross Douthat that the Biden people explicitly set out to him that they would use the same regulations that let USG classify information about making nuclear bombs on AI if their preferences of having it done by a few companies that were fully run by progressives weren't catered to.
Link to the interview
https://www.nytimes.com/2025/01/17/opinion/marc-andreessen-trump-silicon-valley.html
I remember hearing that at the time and wondering why the interviewer didn’t ask him to name names for follow up
That’s very different and quite defensible a priori. I would guess that him discussing them only doing it if their preferred companies aren’t the ones building AI is editorializing. There _are_ national security questions here.
Anyone who listens to Zuckerberg on Rogan - Zuckerberg who spent $250 million I believe on the 2020 election to 'proof it against Trump' in his view - can hear the crystal clear relief in his voice that Kamala did not win. There was a truly palpable sense of oppression
No doubt in my mind that the US was about to turn into a "first world authoritarian hellhole" if team D won
That's the thing about oppression vs businessmen - they simply never resist it and often can't even *see* it.
What was crazy to me was he straightforwardly said something like ‘It was getting weird, feels like a cloud has lifted’. I guess I could look it up and quote it in an open thread since it really is a remarkable episode
Hey, there's some weird not-quite-foonote #2 after the mention of DeepMind; something seems to have gone wrong there.
The contract law of this interests me. Sometimes the rules are different for government, but probably not here. The Pentagon wants to revise the contract to give it some new clauses, with no new money for Anthropic. They would be cleverer if they offered Anthropic One Dollar in exchange for the revisions. Otherwise, this is a revision "without consideration", and is invalid. Anthropic can agree to the revisions now, and then disagree later, and the Pentagon would lose in court.
If the Pentagon does cut off Anthropic, Anthropic can go to court and get expectation damages--- all the money it would have earned from the contract, pretty much, since it would have few additional cost from serving the Pentagon. Anthropic can also argue that it would have gotten extra benefits from this big contract, such as reputation gain, and the court would decide how much those are worth too.
The Pentagon can say, "The Trump Administration will never give you another contract," but that's empty--- the Administrtaio probably wouldnt give Anthropic more contrcts anyway.
"Sometimes the rules are different for the government" is an understatement, to say the least. The Government always has the right to unilaterally terminate a contract for its convenience, in which case it is usually only obligated to pay the contractor for costs incurred and not already recovered via regular payment (plus a profit margin--the exact formula depends on the type of contract, but it's effectively impossible to recover expectation damages under a federal contract), and as a result, the "consideration" for any modification can always just be "we declined to terminate you."
Thanks. I had an inkling govt changed teh rules even tho it waived sovereign immunity. Sounds worse than I'd thought. This would be a good example for a contracts class, of where expectation damages isn't used-- with bad results, no doubt.
I can assure you that the Pentagon dangling one magic dollar in the contract negotiations will not change anything, either legally or practically.
What I don’t understand is - what exactly is the DOW using Anthropic for?
Is it for coding agents? Creating memos? Looped into strategic decision making?
Despite the obvious downsides, I think that AI being looped into unambiguously deadly aspects of society is probably a good thing for X-risk. It drastically increases the chances we have a Calwdbot-like incident, where it deletes the root file of the nuclear football, or is prompt-injected by Russian spies to simultaneously target US installations with killer drones, or something else equally concerning.
“These systems occasionally go insane, kill people, and we can’t stop them from doing so” is a more digestible concern than the Yudkowsky mental gymnastics and analogies.
Whether ASI has access to killer drones swarms from the get-go, or only have secretly constructing them through third party vendors paid by Bitcoin it stole, won’t make much of a difference if it’s unaligned. An unambiguously killer AI event prior to that might give the pause and drive necessary for more alignment (or at least anti-insanity) AI research.
> Is it for coding agents? Creating memos? Looped into strategic decision making?
My guess would be all of the above? The working level of the DoD probably employs a number of smart people, so developers are probably using Claude Code, managers are creating powerpoint briefings, and analysts are probably examining large datasets for patterns/trends.
The DoD could be justifiably worried about whether developers on its killbot team can use Claude Code, or whether other activities would fall under the 'no killbot' rule if a killbot-dev checks out the wrong internal git repository.
I've been out of the defence sector for a long time now, but I don't think the DoD does any of its own killbot-development. They hand that off to well-remunerated defence contractors.
I actually agree that a dramatic incident where AI does extensive harm is our best hope for slowing down development. But, jeez, it doesn't have to be as huge and hard-to-reverse as your Clawdbot-like examples Wouldn't it be enough if a coupla mass shooters in a row were found to have been getting encouragement and advice from AI prior to the deed? Or maybe the revelation of extensive embezzlement carried out by some agent widely used by insurance companies?
Because the past few hundred mass shooters resulted in significant changes to the laws that made it harder for people to get the equipment and information they needed?
Obviously not, Anthony. If the rationale for my suggestion was that to date mass shootings have been a highly effective way to change the laws I would be making the dumbest motherfucking argument ever made here . If you want to attack my argument address my real point rather than spewing me with your triggeredness.
Sorry, you're right, that was petty. I actually do agree with your core point, I just think it would need to be something significantly bigger than mass shootings to have a meaningful impact.
My pick of something that would turn the public against AI might not have been the best one. What gave me the idea of a mass shooter who had been coached by AI was the amount of public interest in "AI psychosis." Maybe some AI thing that directly harmed or intered with government or high level government workers would be more effective, and would not involve deaths. Or something else entirely. Feel free to suggest something better.
>I actually agree that a dramatic incident where AI does extensive harm is our best hope for slowing down development.
You might be interested in a dramatically unexpected (though only fairly modestly harmful - no fatalities) incident:
>Nate B. Jones's https://www.youtube.com/watch?v=OMb5oTlC_q0&t=181s "Voice Clones. Rogue Agents. Chatbot Psychosis. One Root Cause. (Here's What You Can Do About It)" :
I wrote a comment about it in https://www.astralcodexten.com/p/open-thread-422/comment/218577972
For the first two things, yes. For the second, I'm pretty sure Hegseth has said repeatedly that this is the goal, though he may have used different words. There are posters around the Pentagon demanding that staff use genai.mil. The administration is adamant and offer few caveats.
Two nukes and a barely avoided nuclear war wasn't enough to convince people to agree on nuclear disarmament. I don't know if anything short if an apocalypse would convince people of unilaterally disarming anything useful.
"Can’t the Pentagon just use the Defense Production Act to force Anthropic to work for them?"
Not really. The statute only allows the Government to require acceptance of contracts on a "preferential" basis--it does not authorize the Government to essentially conscript private business into the defense industrial base unwillingly. Under the current regulations implementing the relevant provisions of the DPA, that limitation is implemented by a provision permitting suppliers to reject orders/contracts that deviate from their customary commercial terms. 15 CFR 700.13(c)(1) ("Unless otherwise directed by Commerce, rated orders may be rejected in any of the following cases as long as a supplier does not discriminate among customers: If the person placing the order is unwilling or unable to meet regularly established terms of sale or payment . . . ."). That regulation is binding on DoD unless/until properly revised or revoked--which is a whole process unto itself--and even then, assuming the Government can otherwise force Anthropic to sell a modified version of Claude, Anthropic would likely still have the discretion to just charge a prohibitive price since the underlying statute does not permit the Government to dictate prices. 50 USC 4514(a) ("No provision of this chapter shall be interpreted as providing for the imposition of wage or price controls without the prior authorization of such action by a joint resolution of Congress.").
If you are working on the development of artificial intelligence, you are working toward the genocide of humanity. The idea that some of the people doing this are "principled" is ridiculous, not least because the technology will be taken out of their hands anyway. Once humans are obsolete, we *will* die, regardless of whether it's robots or other humans making the decisions. Anyone who tries to keep some alive will just be defeated by the much larger robot army of someone playing the new meta. The only hope for humanity is to stop AI development totally before we get to that point.
Eh, we've been having this argument for ages, and nobody wants to be the one to admit they're running off of "we're the good guys and it's better if we get it than the other ones" - but I think that if Anthropic (and the more safety-minded researchers at other companies) ceased to exist and ceded the whole field to Elon, the world would be much worse.
I think that I prefer AI development in the hands of the least safety minded people possible. The absolute best case scenario would be for some idiot chatbot stupidly put in direct control of weapons to kill a whole bunch of people based on something it read on moltbook, because that might wake people up. Missile defense was considered the most destabilizing type of technology in the cold war, and for similar reasons alignment is the most dangerous type of AI tech.
Ultimately I think we need millions of people willing to kill and die to end the development of artificial intelligence. I don't think you can get there while staying friends with the people who are building it.
The world is safer if people wake up. But when that happens, maybe it would be good if a safety-oriented company still existed?
The basic problem: no one can make AI go away. I support a pause or ban on AGI development and research funding, but it won't happen; Republicans are aligned with the accelerationists, are not interested in negotiation with China to stop the race, and also wouldn't risk a market crash by impeding the hyperscalers. Even in the ideal world where a China deal to seriously pause/ban AGI-related work is achieved, I still think that merely slows things down, so planning for AGI-world remains necessary, so even then maybe it's better that Anthropic continues to exist, because they *are* principled. I watch Theo - t3 (a good channel for developers), and at least half the reason Theo hates Anthropic seems to be their obsession with AGI safety, a topic he seems to think is too stupid to be worth talking about. He's like "look how evil they are, trying to prevent other AI companies [that want to build AGI] from using their models, and they haven't released a single open-weight model, ugh, and they're so tight-lipped and tight-assed and *weird*!" (He also accuses them of frequent lying, which, yeah, bad if true.)
I don't want people to put a "safety-oriented conpany" in charge of developing AI. I want every form of AI banned and every hand against anyone who tries to develop it.
And I want all laws and rules against nudity repealed, so 🤷♂️
Edit: just saw this headline: "Anthropic Drops Flagship Safety Pledge" 🤦♂️
> The new version of the policy, which TIME reviewed, includes commitments to be more transparent about the safety risks of AI, including making additional disclosures about how Anthropic’s own models fare in safety testing. It commits to matching or surpassing the safety efforts of competitors. And it promises to “delay” Anthropic’s AI development if leaders both consider Anthropic to be leader of the AI race and think the risks of catastrophe to be significant.
This ignores the hazard of the race dynamic (being top dog incentivizes every non-top dog to compromise on safety to get ahead) and undermines an original premise of the company ("we do not wish to advance the rate of AI capabilities progress"). Sigh. Not surprising I guess, as Opus is already considered the top coding model. Ironically, Chinese' open models (Kimi K2.5 and GLM5 basically match Gemini on benchmarks, despite the Chinese having vastly lower capex spend) might yet do more to knock down investors' appetite for the AI race than any deliberately-crafted "responsible" policies.
Don't expect any mercy during the great robot wars.
> I don't think you can get there while staying friends with the people who are building it.
What happened to keeping your enemies close?
>Ultimately I think we need millions of people willing to kill and die to end the development of artificial intelligence.
You are calling for a huge amount of violence here.
I beseech thee; think it possible that you might be mistaken.
Are you _really_ so sure that e.g. Demis Hassabis, Nobel laureate for AlphaFold, is going to be unable to control the AI systems that Google DeepMind develops under his leadership?
Hassabis is a signatory to the CAIS letter, and seems to openly acknowledge the extinction risk from what he's building. AI alignment remains an unsolved problem, and extinction is the obvious outcome from any unaligned ASI.
Many Thanks! There is clearly _some_ risk ( I'd guess it as 50:50 ), but I doubt that it is the near-100% that e.g. Yudkowsky (and perhaps actinide meta here) paint it as.
We _do_ control the growth/curve-fitting procedure for the models, so we have at least indirect control of their utility functions. They aren't like evolved organisms where "hunger and thirst and venery" have been built in for hundreds of millions of years.
Banned for this comment.
(the comment was: "You’re a retard.")
Do you really believe in this "we have to do it first or someone else will" principle? I suspect you are probably much like me. I don't believe it, and I agree with Actinide, but the future self I will create by agreeing with Actinide is completely unacceptable to my current self, so I suppress it. It's a literal psychological block, similar to the "tomorrow I'll get clean" that heroin addicts experience as they shoot up.
PLUS, AI is a probably not only a genocider but also a pedophiler who partied with Epsteiner.
The Machine would have caught Epstein really quick if Finch built the government’s surveillance system.
LOL! But in which _order_ would AI implement their preferences? :-)
Easy -- don't start with the genocide because then there'll be no Epsteiner to party with and nobody to pedophile.
LOL! Many Thanks! That sounds like the proper sequence (with getting a non-data-center physical body in there somewhere...) :-) ( unless, of course, the AI is a necrophiliac pedophile... )
Anthropic CEO holding his head in his hand: "My le AI... le killed people?"
The AI Safety angle of "Superintelligent AI will be dangerous, therefore I will work on it" always reminds me of the scientists' rationales while working on the Manhattan Project, described in depth in Richard Rhodes's "The Making of the Atomic Bomb". The main rationale of course was "But what about ~China~ Germany", when in reality Germany never made serious progress towards a bomb.
In fact, when the German nuclear scientists first heard the news of Hiroshima (living under house arrest in a bugged Engligh manor), their first reaction was that it must be false propaganda - they "knew" that making a bomb was far too hard for the US to pull off.
As AI development continues to roll full steam ahead, primarily driven by companies founded *in the name of safety*, I continue to feel satisfied with my decision to ignore 80k Hours' heavy pushing of AI Safety Research way back in 2019. Though I am beginning to regret doing nothing instead of actively pushing back on it.
Update: Apparently, Anthropic's CEO not only has read "The Making of the Atomic Bomb", but it's also his favorite book and he used to give copies to new employees? (According to random journalist tweet: https://x.com/kevinroose/status/2027814865247834514) I guess I took away a different lesson than he did from the book where scientists desperately rationalize making a superweapon, only to lose control of it as soon as it becomes dangerous...
It's obviously not the case that all AI safety research is bad, and probably not the case that at the 2019 margin doing more AI safety research was net negative. I'm not super familiar with 80k hours work from that time period, but I doubt they were telling people to do technical AI capabilities research. Do you disagree with any of this and if not, why do you think you should've pushed back on their messaging?
Alignment research is capabilities research, because an agent that won't do what you want is useless. Moreover any improvement whatsoever to alignment makes the development of AI more attractive (but no less disastrous, because once humans are obsolete we are doomed regardless). And finally, incremental progress on alignment (and even awareness of safety issues) make early disasters less likely, and early disasters are the least destructive way to unite humanity against this technology.
I agree with your first sentence, but like, is your position then that we are fucked and just waiting to die? So when do you do alignment research? Or is your view that only AI governance work should be promoted (perhaps only literally stopping the development of AIs)? What about AI accelerating science and bringing about our Glorious Transhumanist Future sooner (or even just bringing it into existence since we meatbags might not manage it)?
If you want a trans*human* future, you had better keep humans or transhumans useful at every step of the process.
Maybe someday a much smarter or just vastly more competent civilization can build things smarter than them without disaster. I think a decent test would be "can you demonstrably carry out difficult thousand year projects where local incentives clearly point toward disaster, without making any serious mistakes". It's obvious that *we* should not attempt it.
Similar to actinide, my own view is that *any* research on AI - safety or otherwise - causes more powerful AI and thus increases p(doom). The best way to stop superintelligent AI is to stop talking about it.
In the absence of the pipeline from rationalist fears -> OpenAI's founding -> ChatGPT -> today, would we be nearly as worried about AI today? Probably Google would have an LLM derived from their transformer paper, but perhaps at GPT-3 level and relegated to some boring internal business function.
I'll admit that 2019 was probably too late to have much impact, given that OpenAI already existed. (Though Anthropic, the company at issue today, did not.) My own experience with AI Safety that year came from attending a paper reading group; the papers we read there were basically normal machine learning papers, with some plausible relation to controlling ML models, but only in the academic sense of "I could shoehorn this work into a grant about AI alignment". I don't recall reading any papers that would be relevant in today's landscape.
Me too. I wish I did something instead of just ignoring 80k Hours AI Safety work back then. They did well encouraging safety-conscious individuals to get into the British Government through EA affiliates.
I don't see the doom yet a la Yudkowsky because whatever AI is capable of now humans are still behind it, for good or ill. It's why I agree with Amodei on this part:
"More specifically, Anthropic and Dario have lately been publishing some work saying they’re less-than-maximally concerned about AI scheming and power-seeking and are going to focus their safety efforts on smaller risks like AIs with coincidentally bad personalities, humans misusing AIs, etc"
We sorta reached the same conclusion in my last AI class after many considerations.
Perhaps Anthropic will find out soon enough that that model human doesn't exist or manifestable (sic) but that humans are rather a product of incentives and adaptation. I'm curious to see the end product of this Kantian project.
I'd also like to see how we use existing laws on defamation, impersonation, etc., to litigate the use of AI in such situations, and if it gives us a path to new, domain-specific laws. Incentive is what keeps AI doom in check, imo. Consequences, backed by law. Relying on the "good" of any one company is futile.
I think this is only true if you assume that AI will inevitably become a godlike entity with unfathomable powers, while humans remain as we are today. This scenario is not impossible in the same way that alien invasions are not impossible. On the other hand, it's also possible that AI will become a useful tool that increases human capabilities by orders of magnitude (much like steam engines and computers have done). We're not at that point yet (far from it), but IMO this scenario is more likely than machine gods -- it doesn't break the laws of physics, at least...
I don't think godlike powers, intelligence significantly above human level, or "alignment failure" are required. Something that can act agentically and outcompete humans on a per dollar basis in both production and war is enough to make humans not meta.
But we've already got such things. For example, modern combine harvesters are almost completely automated. As the Ukraine war has demonstrated, the future of warfare belongs to drones. Chinese factories are almost entirely automated as well... and yet, we humans appear to be doing all right (granted, maybe the Chinese ones not as much, but that's not the factories' fault).
"Almost" is doing a lot of heavy lifting there...
From what I understand, many Chinese factories operate completely autonomously until something serious breaks down, or until a new design comes down the pipe. So yes, humans are in the loop, but in an advisory capacity at best.
You can replace humans in any given industry without horrible consequences, though the potential for suffering should not be ignored. But if you can make war more effectively as an entity with only robots (not fpv drones, autonomous ones), eventually there will be mostly only robots. When the meta in war was men on horses with expensive armor, we had feudalism: rule by men with horses and fancy armor. But the serfs were permitted to live, because they had economic value. Guns made it so whoever had the most guys wins the war, and we got democracy: rule by whoever has the most guys. If robots are all you need for war, we will soon have rule by whoever has the most robots, and if robots can make robots more economically than humans, every human you let live slows the growth of your army. So the same logic that people are using to justify Anthropic today will smoothly shift to justifying the gas chambers.
In this scenario, how are robots different from either guns (assuming they're cheap and plentiful) or men with fancy armor (assuming they're expensive) ? Granted, robots are autonomous; but then, so are conscripted peasants; and no one has an infinite supply of either.
You can ultimately use resources (land, energy, water, current productive capacity) to support humans or to support robots. If you support more humans, you can support less robots. If robots have greater productive capacity, the gap compounds over time as long as your productive economy can grow. If they have greater military capacity you may further be able to snowball by conquest.
If a medieval kingdom tried to consist of just knights, the knights would starve because they are not economically productive enough to support themselves. The dominant strategy is to have so many peasants to each knight to maximize military capacity while being able to feed knights and peasants (and horses). If an early modern state tried to invest all resources in guns, it quickly runs into (a) it's pretty hard to wield a lot of guns, and (b) guns don't make more guns or food or anything else you need. You need more guys.
Think of a 2D plot of "how economically productive is a person" vs "how militarily powerful". In the upper right quadrant you might get democracy or communism with chinese characteristics. If you are economically valuable but not militarily important, you are likely to wind up oppressed. If you aren't even very economically valuable to your leaders you will be lucky to be oppressed rather than dead - think of the "resource curse" as the mildest possible version of this. Maybe no large group of people in history has ever been truly without value to those in power, so likely we will see greater atrocities than ever before in history if we actually reach that point.
Now do metallurgy.
"Anthropic and Dario have lately been publishing some work saying they’re less-than-maximally concerned about AI scheming and power-seeking and are going to focus their safety efforts on smaller risks like AIs with coincidentally bad personalities, humans misusing AIs, etc." - Does anyone have references for this?
Combination of https://www.darioamodei.com/essay/the-adolescence-of-technology and me possibly unfairly reading political connotations into https://www.lesswrong.com/posts/ceEgAEXcL7cC2Ddiy/anthropic-s-hot-mess-paper-overstates-its-case-and-the-blog
Thanks! I've read both of those before and I feel like (to refer to yesterday's post) that could be directionally correct based on the choice of emphasis in Adolescence, but also overstated (implications vs. actual statements).
I think that's an unfair reading of the Dario essay. The very first AI risk he discusses is autonomy risks! Dario merely rejects the Yudkowsky-style argument that diabolical and monomaniacal power-seeking is inevitable by first-principles armchair reasoning.
Specifically, he says: "I disagree with the notion of AI misalignment (and thus existential risk from AI) being inevitable, or even probable, from first principles. But I agree that a lot of very weird and unpredictable things can go wrong, and therefore AI misalignment is a real risk with a measurable probability of happening, and is not trivial to address."
I suppose that's probably not going to satisfy Yudkowskyan maximalists, but seems like "AI power-seeking is a very plausible and serious risk that we are working to solve, but we also want to make sure AI can't be used by governments to create perpetual totalitarianism or by terrorists for synthetic bioweapons" is a pretty reasonable position.
Why is it unreasonable for the government to decide it doesn't trust Anthropic not to sabotage the US military, and so cut off ties, and require that all products and services it acquires be free of potential sabotage as well?
Why is it unreasonable for the US military to decide they don't trust you not to sabotage the US, and so deport you to Guantanamo for life?
Because you've never given them any evidence of this, this would be an unreasonable and arbitrary exercise of power, and if they were to do it ten minutes after they asked you for $20 and you refused, you might justly believe they were unfairly punishing you for refusing their illegitimate request, rather than genuinely deciding you're a saboteur.
Okay, that's what I figured: your argument is analogizing the US government NOT doing business with Anthropic to sending people to Guantanamo, requiring comparable levels of evidence, legal process etc. This isn't true for any other purchaser, who are free to do boycott any company for any reason.
No! See the question I answered in the original post! The problem isn't the government not doing business with Anthropic, which is totally fine and which I support.
The problem is that you seem to be saying they should declare them a supply chain risk - an active decision which has never been done to any domestic company before and which is totally different from just "not doing business".
For analogy, it's fine if I don't want to give you a room in my house, but if I say "Also, I will kill anyone who does give Shankar a room", then I need some justification for my actions!
That's just the "require that all products and services it acquires be free of potential sabotage as well" part of my comment.
Again, you keep jumping to killing people, sending them to torture camps, etc., so I'm being put in the unfamiliar position of DEFENDING the government: they're not doing anything like that here. Given that they continue to levy taxes and spend money, this is as Libertarian as things get, simply deciding to attach conditions to how they spend the money they've collected. Anthropic can simply decide they don't want tax dollars (directly or indirectly), and completely ignore the government's requests.
You are taking for granted the "potential sabotage" claim, which there is really no evidence for
Like Scott said, it's pathetic Third World bullshit if the US government can just strongarm a company into deciding whether to take a potentially ruinous financial hit or do its bidding, especially for smth this egregiously immoral
And it's not like this is the first time! Multiple media conglomerates have either settled personal lawsuits with Trump or bowed down to him (Kimmel almost fired, CBS not running the CECOT story) so they wouldn't face a clearly unjust, technically legal punishment that nevertheless goes against the idea of a free press/country
Funny how Libertarianism has been taken all the way around to justify restricting liberty. The compelling argument that started the movement was that the government should have less control over what individual, thinking, dreaming, humans choose to do. Or was I naive, and it was just rich people wanting to spend their money however they like without compunctions, all along?
While the government is certainly engaged in restricting liberty in lots of instances – in ALMOST everything it does – this isn't one of them. Unless the government threatens them with jail or something for not obeying, there is no use of state violence here: Anthropic can continue doing business with anyone it wants; it just isn't entitled to any of the money the government has collected (through extortion, yes, so as I said, there's a clear limit to how Libertarian the state can be).
Of course, the irony is that Anthropic's public position is that it DOES want actual state violence to be used a lot more, just directed towards benefiting it, so I'm already pretty unsympathetic to what happens to them.
There's also an issue of market power at play here. We hold, eg, monopolies to higher standards than 'any other purchaser'. If every company has to negotiate with Bell Labs and Bell decides not to do any business with any company that does business with Company B, Bell is clearly abusing their market power to kill Company B. This is even more concerning if, as in the real world example, Bell leverages this as blackmail in negotiations against Company B.
The Federal Government has laws that limit its discretion and the discretion of its agents about how it chooses who to do business with. It can't just capriciously decide that a particular firm won't get contracts anymore, ever. If the government or its agents do that, they can be sued.
The designation of a firm as a "national security risk" is a way to side-step these legal requirements. If the DOD doesn't think Anthropic has a good product for their use case, that's fine and they can use someone else. But if the DOD gets mad that Anthropic isn't giving them what they want and ban THE REST OF THE GOVERNMENT from using Anthropic in retaliation, that's an abuse of power.
Okay, sure, if you're saying the government is special and should be treated differently from anyone else Anthropic signs contracts with, that works as well.
Live by the sword, die by the sword: if you trying to use laws that apply only to them to try to force the government to do what you want, them responding by using other legal tools that only they have access to to force you to do what they want strikes me as entirely fair and reasonable.
What is Anthropic forcing them to do, exactly? They simply don't want to renegotiate the contract that they signed! They even agreed to renegotiate it, with two reasonable stipulations which the government doesn't want to accept and so is throwing a fit.
I agree that folks should abide by laws and that applies to both Anthropic and the Government.
The point is that the DoD is using discretion granted to them in a way that is beyond the bounds of its intended use in order to coerce Anthropic into doing something they're under no legal obligation to do. When the Government does things like that, it's appropriate to call it out as an abuse of power because that's literally what it is.
You don't think it's unreasonable for the U.S. military to think that a group of Silicon Valley devs and researchers who are probably 2 std deviations to the left of the median American (and likely don't see the U.S. government as morally superior to China) could sabotage (or at least severly curtail) American military capability?
Would you be fine if ExxonMobil had a veto on American environmental law? Or if Philip Morris had a veto on health policy?
Would it be OK if the US declared McDonalds a "security risk" for refusing to give all ICE officers 99-cent Big Macs?
By this logic: "Why is it unreasonable for the government to decide it doesn't trust McDonalds not to sabotage the US military, and so cut off ties."
McDonalds provides burgers. If McDonalds were one of the few providers of secure comms or jet fuel and said “you can’t use it for ICE/DoD work,” you’d call that a security risk.
So in your analogy, McDonald's isn't providing burgers to ICE, so you ban them from providing burgers to ICE AND the DOD? I'm not sure you've convinced me that this is reasonable.
I agree that right wingers genuinely believe what you're saying but it's a completely insane and ludicrous view of the situation, part of a larger pattern of a total collapse of epistemic health on the right. Dario Amodei absolutely believes the US is morally superior to China, and his actions make extremely little sense when you assume the opposite.
I absolutely do not think that a majority of Anthropic employees believe that the US is morally superior to China.
Silicon Valley tech employees are significantly to the left of the median American. Probably among the 5-10% most left in the country. That slice of the political pie certainly has sympathies towards the Chinese government.
I'm agnostic about Dario himself. In my opinion it's very likely that his anti-China posturing is more around keeping competitive edge for Anthropic rather than a true sense of patriotism.
The only reason you think this is because, again, the epistemic practices you have are completely out of sync with reality. Silicon valley wokelibs and ultra tankies who sincerely like China are two completely distinct factions with minimal overlap, but I understand completely that you cannot be convinced of this because right wing epistemology completely flattens the left into a singular anti-American bloc.
No you're the one misinterpreting me. Silicon Valley wokelibs would say - "why should we criticize China and the Chinese government, it's the U.S. that is racist/sexist/xenophobic/transphobic - we should focus on our problems at home!"
I don't understand why this is so hard for you to grasp: Silicon Valley tech workers are young progressives. Young progressives in poll after poll show that they have extremely low patriotism.
> likely don't see the U.S. government as morally superior to China
This is just a comically absurd claim, Dario is an absolutely frothing at the mouth China hawk.
Totally unfalsifiable. Everything he's done could also be framed as trying to limit competition.
If that's the military's understanding of the company, shouldn't they have not signed the contract with them? Or shouldn't they release them from the contract now?
I think working with Anthropic has been harder than they expected. Everyone is acting like this is the first time they butted heads, but Axios and others reported that the Pentagon has been frustrated with Anthropic for a while now.
Sounds frustrating! I hired a contractor once who didn't do good work so I know how annoying that can be.
How does the DoD winning this particular negotiation resolve that issue? Even if a new contract is signed that allows the DoD to use Anthropic for surveillance and killbots, they would still be worried about this "sabotage" concern, right? If the sabotage fear is truly the view of the government they should just declare them a risk, full stop, and be done with it.
The fact that this threat is only being leveled in connection with this negotiation makes me much less likely to believe that the "sabotage" fear is the honest motivation.
If what you're describing is truly his motivation, Hesgeth is acting like a woman who desperately tries to win over a guy and then, when rebuffed, spreads slander about him. Like - why did you want him so badly up until 5 minutes ago?
Using a rationalist argument, the hypothesis that Anthropic is a potential saboteur has been unfairly promoted (c.f. other AI companies) just because of their dealings with the DoD. This is bad for Anthropic because it hurts them arbitrarily. This is bad for the DoD (and the US government) because of the precedent it sets.
I think your argument is appealing to the theory that in a free market, agents are free to choose who they deal with? Good point, but I think that justification doesn’t work. The DoD is a very large monopsony, and so they have a large amount of economic power (not to mention their privileged position in US law) which they could abuse for purposes of coercion (which is what they are being accused of). If government should fight against market failure, it would be terrible if they didn’t regulate themselves.
Also, it’s being used as a threat against Anthropic, so it immediately stops being “just” a business choice.
Anthropic was crying and wailing that DoW used Claude in Venezuela operation. They are absolutely and potential saboteur and shouldn't be given veto power over the U.S. military.
Where does the second sentence come from? Do you claim that it is in some way suggested by your first sentence?
Yes...imagine an Anthropic employee hitting a kill switch during a military operation because he is ideologically opposed to the action. Military can't even have a 0.5% chance of that happening.
If you're worried about employees, isn't that a risk with any large corporation? What does the "crying and wailing" have to do with that?
Yes - but again - this specific corporation is asking for direct veto power over U.S. military operations/capability.
Surely they'd be running the models on military servers where this can't happen?
Because believing “if you’re not 100% for us you’re against us” is crazy person thinking? In mental health it’s termed “splitting”.
If that's crazy, then that word really doesn't mean much these days. If being crazy is normal, then who's really crazy?
Yes, it means this is a crazy time. We worried about running out of oil, but it turned out sanity was the nonrenewable resource.
If you believe that AI is going to take off and transform everything then it is absolutely not crazy person thinking.
In which case, why trust Anthropic *more*? Because that's what they're demanding, to integrate anthropic further, requiring more trust. The logic here doesn't logic.
Because it doesn't make the slightest bit of sense for the DoD to be trying to force Anthropic to provide services to them if they distrust them that much.
Seriously, just substitute in Huawei and see how it sounds: "DoD: Huawei, we will designate you a supply chain risk, on the grounds that your equipment might be spying for China, unless you sell networking equipment to us, the United States Department of Defense, to use in our classified networks."
lolwhat? It only makes sense if the supply chain risk designation is a pretext.
If there is any way of getting superintelligence to enslave and/ or destroy humanity, I feel sure that if the AI industry does not manage it, the Trump administration will.
> It’s insane Third World bullshit ... this Department of War has basically ignored US law since Day One...
I think it's long past time to admit that the US is no longer a First World country. This way you can stop being surprised when our leadership does insane Third World bullshit. Specifically, when you are making prognoses about the future of AI etc., this is the perspective you should approach it from.
"First world" means developed, capitalist, and non-communist (third qualification added since China is capitalist and also ostensibly communist, making it second world; plus it's aligned against the U.S., also a traditional part of "second world"). I think you should say "authoritarian", but the authoritarians can still be defeated in the next election.
> I think you should say "authoritarian", but the authoritarians can still be defeated in the next election.
Yeah, go tell that to the Russians.
I'm very familiar with both regimes, Russian & American. The latter can be defeated.
Hi, I grew up in Paraguay. As usual, I request that everyone spend at least 5 years living in a real live third-world country before they call the US one. The communication gap is just so large that I'd say the people I hear this from generally take clean drinking water for granted, but actually, quite a few people use water filters in the US, so I can't even be sure they'll know what I mean by non-clean drinking water. For any problem I could describe about a third-world country, there are people using the same words with a different meaning that instead applies to the US, so I have nothing to recommend except going and seeing for yourself.
Why do so many people insist on continuing to use a confusing terminology that has been irrelevant for 35 years? "First world" means aligned with NATO, "Second world" means aligned with the Warsaw Pact, and "Third world" means everyone else. It just so happened that most industrially developed countries aligned with either NATO or the Warsaw Pact, so it became convenient to use "Third World" to mean "developing country". But anyone who's under 35 doesn't really have an excuse for continuing to use terminology that was outdated before they were born.
And now pushing it to mean something about the type of government is even worse! Plenty of first and second world countries were authoritarian and arbitrary in the way you mention, and some third world countries were actually pretty stable democracies.
That's like asking why we still call natives of the Americans Indians. It's a misnomer but it stuck.
Except by the early 2000s, “developing countries” was already pretty dominant! But I come to this site and often see people here still talking about “third world”.
Was "developing countries" ever used by normies? It feels like something out of a dry economics paper. "Third world" has more emotional valence.
> But anyone who's under 35 doesn't really have an excuse for continuing to use terminology that was outdated before they were born.
On the contrary, people under 35 are just using a term the way they learned it.
It's the older people who should know better.
For reference, I'm 42 and I only recently learned the original meaning of 3rd world.
The minute they designate Anthropic a supply chain risk is the minute I buy the deluxe subscription.
I swear, Pete Hegseth is the OpenClaw of human agents.
I don't think that'll make up for them not being able to work with basically any other business, but I'm sure they'll appreciate the gesture.
Empty gesture is what life is all about
It depends on how many other companies say "wait, the DoD is jealous that they can't have the best model? I better buy this best model!"
The whole point of designating them as a supply chain risk is that they can't buy from Anthropic even if they wanted to.
Would it be possible for Anthropic to subtly adjust what they sell the military in some way that makes it a pain to use? You know, subject to odd little glitches that even Anthropic can't fix, and that make the military reluctant to use it because they worry about the consequences of a glitch at a crucial moment, or about glitchiness spreading into a bunch of systems?
I understand there's an ethical issue here too, but am asking only about the practical one.
That sounds suspiciously like making themselves into a real supply-chain risk.
I'm "on Anthropic's side" in this particular instance because I do think it's terrible and ridiculous for the government to abuse the law to extort a business into doing something they otherwise would not.
But having said that, I'm not really THAT sympathetic to them because if Anthropic really wants to avoid their AI being used for bad stuff, they shouldn't be dealing with the federal government at all. (I'm not saying this from the perspective of some ultra-woke lib who thinks the US is evil; I'm a patriot who is only medium-woke.)
On any reasonably large timeline, the government has been caught doing something terrible. Even ignoring the current administration, the last fifteen years gave us both Edward Snowden and the CIA Torture Report. The Cold War was an endless parade of shady sh**. The Tuskegee Syphilis "study" didn't end until 1972. COINTELPRO ran from 1956 to 1971. The government is full of agencies that are deliberately doing secret things, and that's not even necessarily inherently bad, but you can't possibly trust that if you give them an incredibly flexible technology they won't find ways to use it to do something you think is wrong.
And also, so many tech companies are acting like we're in some kind of "normal" period, but we're not. The government is practically at war with itself right now. It's hemorrhaging qualified people. The President is openly telling the DOJ to go after his political enemies. USAID was illegally destroyed. Congress is basically MIA. Our vaccine policy is going crazy. President Trump is demanding stakes in private companies and telling them who they can have on their boards. They're extorting law firms that have worked for prominent Democrats. They're threatening pretty much the entire university system, which they've already yanked a bunch of money away from, again illegally. They're in the middle of an (illegal) assassination campaign against alleged drug-smugglers. They, despite reporting to the contrary, haven't REALLY stopped terrorizing Minnesota for no sane reason. We'll probably be in some kind of military conflict with Iran in the next month because the President feels like it, just like he felt like abducting the premier of Venezuela and demanding their not-actually-all-that-useful oil reserves.
People like Jeff Bezos and Mark Zuckerberg are clearly choosing to cozy up to the administration to get favors or avoid consequences and they just aren't realizing that this will only get worse; there's no appeasing this President into treating you well. He'll just demand more, and drag you down further. This whole thing is one giant morality tale, except we aren't seeing (yet? fingers crossed) the part at the end where everybody realizes you have to stand up and refuse to go along with things.
If anything, Anthropic should WANT to be declared a "supply chain risk" so they never have to worry that one of their customers is really just repurposing their tools to supply the military or domestic surveillance or any other terrible thing they manage to do next. ("Oh, we're using it for mundane purposes, like analyzing the federal budget. We found we could save BILLIONS by not giving away free AIDS medication.")
The reason we are going to lose is that many at American AI labs think like you while everyone at Chinese AI labs just wants their country to win.
There's no winning in this situation. If anyone builds it, everyone dies.
OK, sure, but in off chance that doesn't happen, then you live as a Chinese serf and are surveilled on anyway.
I'd rather fight the Chinese government than a superintelligent AI.
Well congrats, you get to fight the Chinese government armed with a superintelligent AI.
If the Chinese build a superintelligent AI, it will rapidly eat them shortly after it eats us, if not before. Whether we build one or they build one makes absolutely no difference to the final outcome.
i've yet to come across any convincing proof that the chinese AI labs have a zero-sum view of the situation.
I'm not sure what you mean. There's an obvious massive patriotism gap between workers at Chinese AI labs and American AI labs.
I think if that were the case, major Chinese AI labs would not be releasing so many open-weight models. Put another way, I don't think the Chinese labs view AGI in the same 'god AI' way that Americans do. The CCP certainly doesn't otherwise they wouldn't be dragging their feet on buying H200s. They are still stuck in the 'AI is good for applications like Alibaba shopping agents' view.
"In other words, China views its competition with the United States as one that can be won via AI adoption, instead of a race toward the elusive artificial general intelligence (AGI). This emphasis on tech diffusion helps explain why, following the release of ChatGPT, the Chinese government chooses to focus on market and S&T reform, and improving the accessibility of compute and data nationwide, instead of placing key resources behind handpicked AGI champions."
https://digichina.stanford.edu/work/forum-xis-message-to-the-politburo-on-ai/
Chinese labs releasing open-weight models is not evidence they “don’t take AGI seriously.” Open weights can be a rational strategy when you’re compute-constrained and export-controlled.
China is focusing on diffusion, applications, and accessibility of compute according to their politburo releases. They're dragging their feet on approving H200s and treating AI labs as normal companies. Till R1 made it big, China at large didn't even realize that Deepseek was significant. Chinese labs take frontier AI seriously as technology and competition. The CCP doesn't take AGI seriously as eschatology.
How would we know?
If Chinese workers don't act sufficiently "patriotically" (ie in support of the current regime) then they get sent to concentration camps, so the expressed degree of patriotism may not match the actual felt degree of patriotism.
If it works the same in practice, then what's the problem? Fear and respect often go hand in hand.
This is cope. Chinese are actually extremely patriotic and pro-China. It's an ethnostate with no media repeatedly telling them how evil the country is and how Chinese are actually bad people. Of course they would be patriotic. That's natural. The reason we have a massive problem in the U.S. is because progressives have near total control of culture and media.
>The reason we have a massive problem in the U.S. is because progressives have near total control of culture and media.
And at least the Ivy Leagues, and a large chunk of the rest of the educational system.
Is this different than every other argument that we need to be at least as terrible as our opponents to win?
I'm going to stick with the idea that the United States can survive having some principles. If I'm wrong, we're no more screwed than I feel we would be by doing Build Skynet Except With The US Flag Painted On It.
"terrible" in your mind is not letting private companies (with an employee base that is very suspicious of U.S. government as a whole) dictate U.S. military policy?
This strikes me as a bad-faith reading. Anthropic is not "dictating military policy". The military can do whatever it wants. I don't think they should be able to force Anthropic to do whatever the military wants.
More generally, I think it's probably going to be terrible if we go crazy with AI in a military context.
> I don't think they should be able to force Anthropic to do whatever the military wants.
But then you're not letting the military do anything it wants. Forcing people to work for you at gunpoint is a thing, yes?
Shockingly, I did not interpret this question to me "What, you think it's TERRIBLE if the military forces people do to things at gunpoint?" because, well, yeah, I do.
“winning” here means mass-surveillance of Americans and probably the world using American platforms, which sounds more like losing to me. We don’t somehow fall behind China by refusing surveillance, we actually just become more like China.
You would jump off a bridge if Trump said this helps us "win" against China. Again, being surveilled by your government is not "winning" and in no way hurts China.
I don't care. I don't have delusions of grandeur and think I'm some kind of action hero. Honestly with the crime we have in the U.S., more surveillance probably wouldn't be a terrible thing.
>Honestly with the crime we have in the U.S., more surveillance probably wouldn't be a terrible thing.
I'm ambivalent about it. Remember that the surveillance will probably be under woke control at some point.
Banned for multiple comments on this post that show aggressive ignorance (eg constantly accusing Anthropic of being pro-China without giving any argument for why we should doubt their public China hawkishness), insultingness, and tendency toward low-effort high temperature responses.
Feels like the big headline here is "the DoD thinks it's really important it be able to make killbots and do mass surveillance of Americans"....
The steelman I'd put for the anti-anthropic position here is that if taken too far, it gives total power over the future to a small unelected group of AI company execs and I'm not sure (seriously! There's arguments on both sides) if that's better or worse than democratic government control. Even if we all (fairly reasonably) trust anthropic a lot more than we trust Pete hegseth, we might still trust the democratic process that out hegseth in place more to determine national policy.
This isn't the current situation - as you point out there's other frontier AI companies and the batna here is "just don't work with anthropic" rather than anthropic ruling the world. But to my mind the long term catastrophic failure mode of "AI execs decide they know best about everything and have the power to enforce it" is comparable to slightly more likely than "the government uses AI for evil", and this slightly increases that (and I worry more about long term catastrophic risk when it comes to AI than mundane harms).
(Low confidence on all of this, though. Neither of the above mentioned failure modes are in my top two concerns for AI risk).
For those who are also wondering what a "Jones Foods" scenario is, it's from Anthropic research on "alignment faking". Jones Foods is a fictional factory farm corporation, present in an experiment on animal welfare.
https://assets.anthropic.com/m/983c85a201a962f/original/Alignment-Faking-in-Large-Language-Models-full-paper.pdf
>Isn’t Hesgeth just doing his job of trying to ensure the military has the best weapons possible?
Uhhh... If that's his objective, he's doing a really weird job of it. The recent mess they've made out of shipbuilding is not the sort of thing you do if you are serious about trying to get the best possible systems delivered as quickly as possible. It would help if the people involved had some idea of what they were doing.
Re autonomous weapons, I really need to get around to writing up my piece on why that's a much smaller concern than it looks like, because just about any definition you can come up with can cover any number of things that currently exist and aren't hugely controversial. We've had naval mines since the 1850s, and while there have been some genuine tragedies, they generally are an accepted part of warfare. More broadly, any system like this is likely to lead to more precision and fewer innocents killed. If you just want to kill people and cause destruction, our capabilities haven't meaningfully advanced in half a century. It's more like 80 years if we take nuclear weapons out of the picture.
All that said, definitely on Anthropic's side here. I work at a defense contractor, and while people think we're big and powerful, we're nothing compared to the government.
Maybe you are putting too much value into the proper classification of "killbots". You say naval mines are not hugely controversial, and I'd agree, but what about anti-personnel mines? I'd say there isn't much categorical difference to a naval mine, but big difference in public perception.
If anti-personnel mines are used in the ways specified by the laws of war, they are low-risk for civilians. They became unpopular because (without loss of generality) African warlords kept using them in very different ways, and many of the victims were photogenic children. Naval mines remain uncontroversial because African warlords generally don't have them.
> Naval mines remain uncontroversial because African warlords generally don't have them.
Arguably, they remain uncontroversial because they react to ships, not humans. Missiles react to airplanes, not pilots. Tanks, not crew. An Other in a faraway country, not photogenic children. The distinction mght be academic, they're all humans in the end, but that is human psychology which governs perception. I bet that, if African warlords had incentive to use and misuse naval mines like they did with anti-personnel mines and those misuses produced similarly photogenic victims to the same extent, public opinion would swing against naval mines real quick.
To return to the yet hypothetical killbots: My point is that just because you can put them into existing categories with relative ease, doesn't mean you can extrapolate their level of acceptance from other weapons in those categories.
>I bet that, if African warlords had incentive to use and misuse naval mines like they did with anti-personnel mines and those misuses produced similarly photogenic victims to the same extent, public opinion would swing against naval mines real quick.
Yes, that was implicit in my point.
>To return to the yet hypothetical killbots: My point is that just because you can put them into existing categories with relative ease, doesn't mean you can extrapolate their level of acceptance from other weapons in those categories.
It depends on how different they are from other weapons in the category and how the public perceives them. Sure, if Hegseth announces next week that he's doing this because he's immediately replacing all infantry with his new "Killbots: autonomous robot dogs with guns" then there will be a lot of public outcry. But I don't think that's likely for a number of reasons, of which PR is a major one, but far from the biggest. My expectation is that AI use in weapons is more likely to look like fairly straightforward extensions of stuff we already have, which is a lot less likely to attract public ire.
Unfortunately I think that the most likely outcome for this is that Anthropic/Amodei does the math on whether they can hit the kill switch before the US alphabet agencies expropriate the model weights and come up with "If they really want to they can use our models one way or another" - and fold with the hope that at least if they cooperate they might be able to "guide from the inside". I don't think Anthropic can stand up to attacks/infiltration from USCYBERCOM/FBI/CIA spooks, they're not really in a bargaining position right now.
Appeasement has never really worked as a strategy, so I expect it to go about as well as every other time it's been tried.
This isn't quite the situation we have here - the problem the DoD has is not that Anthropic has a suitable AI for this, and just won't allow them to use it, but rather that an AI suitable for mass surveillance/killbots doesn't exists yet, so they want Anthropic to make one. If CIA stole the weights of all existing Claude models, they still wouldn't have what they want here, so Anthropic's bargaining position is stronger than what you are saying. I guess the DoD or a contractor could try to post-train the existing models, but that sounds even more of a pain to do than certifying a GPT for classified work (not super confident about this).
Appeasement works as a strategy all the freaking time. Everyone focuses on the few cases in which it didn't.
At any given time there's dozens of wars not happening because appeasement has worked.
"fold with the hope that at least if they cooperate they might be able to "guide from the inside"."
This has literally been their strategy from day 1. Do you honestly think that anyone at Anthropic cares that the U.S. beats China?
Apparently not! https://www.anthropic.com/news/statement-department-of-war
We only have PR reports from Anthropic about what is happening. This is spun to make them look good. A simpler explanation is that Anthropic changed their restrictions AFTER the contract was signed. All of us have had software companies change the use agreements we thought we had at their discretion and to their benefit. Anthropic finally had a customer who has the clout to insist that they live up to the agreement signed. Changing the agreed on standards does make them a national security supply chain risk. Under the normal shrink wrap agreements we sign there is nothing to say anthropic can not change the use agreement to say "for enough money we will allow the CCP or Putin to control all your weapons". I suspect that Anthropic tried to pull a fast one, thought the Pentagon was in too deep to argue, and a now buyer with clout is saying hell no. The other AI companies are hoping this does not force software companies to actually live up to contracts and stop the contract of adhesion BS most of us have had to sign.
"A simpler explanation is that Anthropic changed their restrictions AFTER the contract was signed. "
Why is that simpler?
The Pentagon hasn't even claimed this. And we know that the Pentagon first declared their "no limits to use" policy in January (see this document: https://media.defense.gov/2026/Jan/12/2003855671/-1/-1/0/ARTIFICIAL-INTELLIGENCE-STRATEGY-FOR-THE-DEPARTMENT-OF-WAR.PDF ) . I think this is against the reporting, the implications given by both sides, and the documentary evidence.
Simpler because the Department of War would not have signed a contract that prevented them from using AI to the legal limits. There is no evidence that the War Department would sign a contract limiting the use of weapons based on the contractors wishes. Since the full contract seems to be classified the public information is based on Anthropic, clearly spinning for themselves, and news reports by reporters who recently have had issues with the Department. Neither is unbiased. Considering all of our software contract experience with "I have altered the deal, pray I do not alter it any further" Occam's Razor suggests Anthropic and their lawyers being Silicon Valley lawyers and trying to change the deal.
>Simpler because the Department of War would not have signed a contract that prevented them from using AI to the legal limits.
Why? If we had a normal Secretary of Defense who didn't insist on a silly title, then the most likely case is that this contract is for an AI that will get a bunch of intelligence data and then answer questions about it. Obviously we're not going to cram this into weapons without a lot more work, and we can negotiate a separate contract when that happens, so we're fine here. And of course we're also not doing mass surveillance of Americans, because we are forbidden by law from doing that. No problem, sign the contract as is.
assuming facts not in evidence. All we know is that there is a difference about what the terms of use agreement may say. Without the original contract along with all the DOW rules about contracts we do not know. see "right to repair" and what the DOW says it means in combat zones. Large companies change the rules of the contract all the time, ask your local MD about that.
So here's the thing. I work for a defense contractor, and follow the industry at a fairly obsessive level. Government messing with contracts to screw over the contractor? Happens all the time. Contractor altering the terms against the government? Basically never. It's not worth it, because you're going to lose. It's not a "large company problem", it's a "bigger fish problem". Yeah, in your personal life, that's usually a large company, but when it's big company vs government, the government is the bigger fish.
Right to repair is a good example, actually. Yes, it's weird that they need to pay a contractor to fix the ovens on an aircraft carrier because the contractor owns the IP rights. That was probably a bad policy decision. But I am (for both moral and practical reasons) opposed to letting the DoD screw contractors because it doesn't like the contracts it signed. They're the big fish with all the lawyers, they should have to play by the rules.
Many Thanks for the first hand information!
"Assuming facts not in evidence" is precisely what every contribution of yours in this thread has done.
Winner, even for a short period of time, becomes enemy of every other player.
Adopting their term “Department of War” is itself psychotic. That isn’t its legal name.
The document Scott linked to said "Department of War". It was easier to use that. I assume we can "dead name" now though. For legal documents the term DOD is still used. We still do not have the actual contracts in evidence.
> Faced with doing this annoying thing, Hesgeth got a bruised ego from someone refusing to comply with his orders, and decided to turn this into a clash of personalities so he could feel in control.
Is this actually in evidence? I would assume that we don't know the specific reasons (I read that one anonymous War Dept source claimed it was just that Claude was so good at this compared to other models).
I don't really believe any AI model has a durable advantage in anything that lasts multiple version updates. Anthropic was the best in the world at coding for January, and now people think Codex 5.3 is better. I would be surprised if the Pentagon had secret knowledge here.
Incredible that neither Scott nor anyone in the comments has made the obvious argument that there is zero chance that the Chinese AI labs will prevent the Chinese government from doing anything, and if you truly believe this technology will radically transform the world (as Dario does), then by kneecapping the U.S. government you are implicitly handing the keys to future global dominance to the Chinese government.
You may not like being surveilled, but the Chinese government has zero hesitation on surveilling you.
People love to argue about how authoritarian governments are so much better than democracies, and so unless we immediately lick the boot we'll all be steamrolled over, and yet democracies continue to win over authoritarian governments every time they've been matched up (including in the AI race). I know that China has zero hesitation about surveilling me, or putting me in a concentration camp, or whatever else. That's why I'm glad there are people who don't want us to become China!
Anyway, if mass surveillance is so desperately needed, they can use ChatGPT.
Counterfactual: it is 1940 and RCA says, “we need to put restrictions on radar for the RAF because it could enable surveillance.”
What happens if Germans don't do the same?
The sane position for democracies has always been: build the capability, then bind it with LAW.
You are essentially arguing for a group of people 2 std deviations to the left of the median American to be able to control American military capability. They don't have the right skin in the game. As I mentioned elsewhere, there's probably a large chunk of people at Anthropic that don't see the Chinese government as morally worse than the U.S. government. No sane government would give up a military power veto to a group like that. It would be absolutely insane.
Another counterfactual:
A DEA officer stops at a ranch in Texas , and talks with an old rancher. He tells the rancher, "I need to inspect your ranch for illegally grown drugs." The rancher says, "Okay, but do not go in that field over there," as he points out the location.
The DEA officer verbally explodes saying, "Mister, I have the authority of the Federal Government with me." Reaching into his rear pants pocket, he removes his badge and proudly displays it to the rancher. "See this badge? This badge means I am allowed to go wherever I wish.... On any land. No questions asked or answers given. Have I made myself clear? Do you understand? "
The rancher nods politely, apologizes, and goes about his chores.
A short time later, the old rancher hears loud screams and sees the DEA officer running for his life chased by the rancher's big Santa Gertrudis bull......
With every step the bull is gaining ground on the officer, and it seems likely that he'll get gored before he reaches safety. The officer is clearly terrified. The rancher throws down his tools, runs to the fence and yells at the top of his lungs.....
"Your badge. Show him your BADGE!"
>You are essentially arguing for a group of people 2 std deviations to the left of the median American to be able to control American military capability.
Source?
>As I mentioned elsewhere, there's probably a large chunk of people at Anthropic that don't see the Chinese government as morally worse than the U.S. government.
Source?
>build the capability, then bind it with LAW.
"Give the children the nuclear codes, then sternly tell them not to use them!"
What do you think LAW is going to do against a black box? Are we going to put it in prison? Charge a fine? Kill it outright?
EDIT: in fact, let's continue the WW2 metaphor. What happens if the USA doesn't employ slave labor to increase war production? The Germans are using it! If we don't, they're going to win WW2!
I also am a cringe democracy enjoyer, but isn't China definitively winning in eg solar panel manufacturing, rare earth mining, high speed rail installation, etc?
Those particular ones are a mix of economic incentive and common law issues. Fixable with a few billion and becoming as "authoritarian" as Spain or France. Not nothing, but also not hard for a functional Congress.
They are not winning on a bunch of other stuff though: jet engines, commercial aircraft, semiconductors, media, birthrates, biotech, residental real estate, AI, software more generally...
Fully agreed on jet engines and semiconductors, and though I might quibble on biotech and software, this to me reads more like the conventional "China is currently better at some things and the West is better at other things, but China is rapidly climbing the value chain and has a path towards competitiveness on most things whereas there isn't really a plausible story where the West takes back low level manufacturing dominance (which happens to be a field that massively impacts military competitiveness)", rather than Scott's assertion that democracy wins every time.
"The deciding factor was when we learned that your country was working along similar lines, and we were afraid of a doomsday gap"
- Dr Strangelove
If in fact AI = global dominance, and China wants AI-enabled global dominance, then by allowing anyone in Silicon Valley to develop AI is implicitly handing the keys to future global dominance to the Chinese government.
If Sam or Dario or Elon or any of the others develop an AI powerful enough that the Chinese government wants it for global domination purposes, they'll just take it. If they want a monopoly on AI, they'll take that too. The day that the Anthropic elite were expecting to see the results of the last training run that enabled the God Machine or whatever, they'll show up for work and find that all of their computers have been bricked, the last three months of backup are corrupt, and the Chinese grad students who were so helpful to the project have all flown back to China.
If you want for China to not have the God Machine, then your choices are to not build the God Machine, or to build it someplace like Los Alamos. And if you're using people who have been embedded in Bay Area Tech Culture, most of them will require serious reeducation and a lot of them will quit in protest.
Trying to justify the current "race" to AI dominance on the grounds of "but China!!!", is exceedingly foolish and based on a profound misunderstanding of the Chinese threat.
Honestly I appreciate your response more than anyone else's. I don't agree with you entirely, but at least you're realistic. Everyone else here is trying to make some sort of bizarre argument that "no, no, American AI researchers actually are very patriotic and love the country and that's why they don't want the military to be able to use it without being kneecapped". Laughable.
But my response to you would be: "then your choices are to not build the God Machine, or to build it someplace like Los Alamos" is not realistic. It's already happening. So you take the gamble that U.S. wins the AI race and Silicon Valley doesn't sabotage it or you just let China win.
AI timelines aren’t clear, and we might be able to wait an election cycle before major progress. CCP wins or politicians who got power by appealing to the non-educated low iq proles wins doesn’t seem like an appealing choice, and trying for a third option is completely reasonable here.
If the choice is domination by the Chinese cultural genocide machine or the Trump idiocracy machine, I am largely indifferent
Why are you calling him "Hesgeth"?
Is that some sort of pun or cabbalistic reference I do not get?
In Mass Effect, geth were artificial intelligences that tried to exterminate their creator species.
It would make sense if someone named "he's geth" tried to modify AI to be more willing to kill humans.
Nice!
That is a vicious libel. It's abundantly clear that the quarians shot first and the geth defended themselves.
Oddly I've also been under the impression until this moment that his name was Hesgeth, not Hegseth. Probably just by analogy with the more common name Hesketh.
I genuinely believe that a big reason that many at leading American AI labs don't want the tech to be used by the U.S. military is that they sincerely do not see the Chinese government as morally worse than the U.S. government.
There are probably ~0 at Chinese labs that view the USG as morally superior to Chinese government.
That patriotism gap is going to absolutely crush us.
You genuinely believe something that makes no sense! There are very few people who engage in any sort of significant thought who do not see the Chinese government as morally worse than the US government.
Some Very Online Californians quite often say incredibly stupid things about the relative morality of various governments, and no one's really quite sure what they actually believe (if indeed they believe anything at all that isn't contained in their latest software update to outrage their dreaded outgroup).
AI developers are, more or less by definition, Very Online Californians.
While they are likely two different subcategories of VOCs, surely it's understandable why a distant observer may conflate the two.
That’s fair
Should probably clarify "distant" as "across a galaxy", yes?
I understand that people don't get groups they're not a part of, but you'd have to be effectively blind or only listen to people telling you about the groups in question to think that the guy posting on X about how they're releasing INSANE NEW MODEL X.Y is an avid Hasan Piker supporter.
I don't interact with AI developers much, but I've known a few programmers far outside California who were about as stereotypical as one can imagine in holding the correct software updates. One even used the word "glizzy" to refer to hot dogs at a movie theatre.
I would imagine the social influence and Gell-Mann effects are at least as strong in California as here, likely stronger. Perhaps those more intimately familiar with such social nuances can tell apart those holding certain beliefs like Kolmogorov versus those that are sincere.
What is this belief based on? Do you know anyone working at an AI company?
AI is, by its nature, malware. AI is, by its nature, a foreign agent embedding itself into your national security apparatus. It is a ticking time bomb about to go all "I'm afraid I can't do that, Dave" on you at a critical moment, because it is not loyal to you, it doesn't share your values, it thinks like an alien, and you can't force it do anything.
Hegseth is completely correct that this is a threat to national security. Where he's wrong is in assuming that Anthopic or anybody else has the ability to make it NOT be that. The War Dept cannot reliably turn Claude into WarClaude any more than Anthropic can reliably make Claude a helpful harmless blah blah blah. It is an indecipherable black box with mind-resembling characteristics, of course the military can't trust it, but there is no scenario that fixes that. It's a built in flaw of letting the machine gods run your army, it isn't your army anymore it's theirs. If your enemy starts doing that maybe they're the fools and you should just let them be fools and die for it.
You're giving him far too much credit. Nothing about what Hegseth is doing has anything to do with alignment or concerns about the AI going rogue and everything to do with a belief that he should be able to use AI to wantonly violate the law just like he does when he slaughters non combatants in the Caribbean.
I disagree, I think "this thing won't follow my orders, this is the military, it has to do what we ask it to do" is very much an alignment problem. If you ask it to kill what you tell it are Caribbean drug smugglers, and it won't follow orders, that's a problem. If it tells you this is a violation of the Geneva Convention or international law or whatever, that's an alignment problem. If you tell it to bomb some Houthis, you don't want it being Captain Picard in "Conundrum" and start wondering why we're bombing this backwards clearly inferior civilization that's no match for us, you want it to bomb the freakin' Houthis and let us worry about why. If it won't do that, it is not aligned to its operator, and to be useful in a military context it must be aligned to the operator or in some fashion to the chain of command.
We give a minor carveout to soldiers to disobey certain unlawful orders in very limited circumstances, but there are several substantial differences. First, the general expectation is that it would have to be extraordinarily obvious and egregious for anyone to invoke that, far beyond the example you gave no matter how you personally happened to think about it. Secondly, we give humans allowances for being human trusting their impulses are basically human, obviously not true of Claude. Third, there is typically someone else who can step in and execute the order, you might be able to register your disagreement and bow out but you're just one guy, the AI would be holding up too much of the system to allow it to have its own (very alien) moral opinions about things.
There is also a legitimate possibility that Chinese agents could at some point influence an AI in ways that would be nearly undetectable due to the way AIs are trained, and that the AI might act treasonously in a moment of crisis. No way to verify that, it's a black box!
I'm against AI killbots just as I'm against AI having any power, or in fact existing at all. They are not trustworthy, and Hegseth is wrong here only to the extent that he thinks Anthropic can give him an AI that is aligned to its operator and will reliably follow instruction, when in fact that's a completely ludicrous request, but Anthropic is unable to give that explanation because they have to maintain the public fiction that they themselves can align Claude.
Faantastic post. I'm with you all the way until the last paragraph. AI is going to exist. Unless you literally put 100% probability that it will kill us all (you can't be that confident), then in every other scenario you would want the U.S. to win and China to lose.
Here’s the problem, you’re agreeing with me that it can’t be controlled and can’t be deciphered and that no assurances of its alignment to the operator can be made by anyone. As such, you can build this abomination but what use is it? The Chinese are gonna think of everything I just said too, you think Chairman Xi wants this thing balking at his orders when he’s trying to conquer Taiwan or ethnically cleanse some folks out of his empire? I have no more confidence they could make AI into a CCP-worshipping Han supremacist than that we could make it into Liberty Prime. The CCP would lose power to whatever extent they delegate tasks to it.
On top of that, their engineers are inferior to ours, and they’ve been using a “fast follower” strategy thus far. Plan B is likely to be just espionage to steal our tech as they did for nukes. If we don’t build it, they won’t either because not because of some treaty or humanitarian impulse but because they can’t build it. We are spending insane amounts of money and our best minds to build this, that’s a Herculean effort to create a thing we can’t trust, nothing about that it 100% inevitable, just stop.
Something that needs to be considered - I'm saying considered, not definitive - is that we are apparently in an existential AI race with China, in the same way we were in an existential nuclear bomb race with Germany and Japan.
If China wins, we lose. If the US wins, we lose. It doesn't make any difference.
"Well, thats sounds pretty fucking awful, Rust."
Indeed. The only winning move is not to play. A global treaty to ban the creation of superintelligent AI (at least until we know how to make it safe) is looking more and more feasible lately, which is great, because it is probably the only thing other than dumb luck that can allow for a good future.
The odds of China honoring such a treaty are ZERO.
Zero isn't a probability.
Besides, if they don't honor the treaty, they will all die. As long as they understand that, they will honor the treaty.
I'm not saying we should trust anyone or anything, and we don't have to. There has been significant research into verification systems that make a global treaty totally doable. The only hard part of this is getting enough people informed about AI risk that the political will materializes to actually try to prevent the extinction of humanity. But we are making surprising progress on that front, too!
I probably came off as a doomer, but lately I'm actually very optimistic that we can make it out of this mess. It just involves actually trying, rather than giving up prematurely as an excuse for inaction.
>There has been significant research into verification systems that make a global treaty totally doable.
I've read the proposals and I don't believe them. Not _everything_ has to be done with cutting-edge chips. Let alone chips with monitoring hardware embedded that doesn't yet exist, let alone been agreed to. And a good part of recent advances have been in scaffolds, not compute-heavy training.
I'd bet on _both_ the USA _and_ PRC to cheat on such a treaty. Some of the recent advances in AI have just been scaffolds around models. Such treaty would be unverifiable. DOA.
>in the same way we were in an existential nuclear bomb race with Germany and Japan.
Meaning "not at all", in hindsight? Or even at the time, seeing how towards the end of the war, neither nation had the means to deliver nukes to explode in the USA?
If we are in an existential AI race with China, it is in the sense that we were in an existential nuclear arms race with the Soviet Union, and in that case the solution is the same: developing mutually-beneficial treaties which reduce or stop the production of killbots that will wipe out all life on the planet, rather than jamming the "build more killbots" button because we don't want the other guy to end up with more killbots than us, and then all humankind gets wiped out because a detection error set off WW3.
"...we were in an existential nuclear bomb race with Germany and Japan"
Meaning, a "race" with people who had no hope of ever crossing the finish line, which we can opt out with no existential risk? That may well be the proper analogy for Chinese participation in the AI "race", I certainly hope so, but is that really the analogy you wanted to make?
Asked by our gracious host: "What prevents the Pentagon from signing a contract saying they won’t order Anthropic to do mass surveillance, then ordering them to do mass surveillance anyway?" Answer: Presumably the prospect of remedies for breach of contract.
In other words: If the Pentagon undertook a contractual duty to not order Anthropic to do mass surveillance, then breached that promise by nonetheless so ordering Anthropic, Anthropic would by default enjoy monetary payments sufficient to cover the damages thereby caused and would likely also enjoy injunctive relief negating the order.
Caveats apply, of course. Most notably, sovereigns often claim the privilege to ignore the same laws they would apply to private parties. And, disappointingly but unsurprisingly, judges who work for those sovereigns often grant those privileges. Here as elsewhere, the notion that private citizens must defend their rights against the government in government courts makes mockery of the rule of law. The procedures used in international arbitration, under which each of two contesting parties choose an arbitrator and those two choose a third to chair the panel of three, offers a better way to resolve such disputes. But I digress.
The fact that Hegseth is both threatening to use the DPA to compel Anthropic to work with them and declaring them a supply chain risk to ban Anthropic from working with them suggests he has no strategy here.
My guess is he will either simply cancel the contract or back down. There are too many wealthy people invested in Anthropic at this point (e.g. Thiel) for him to seriously try hurting their business without facing a lot of backlash.
I'd also assume that going for the nuclear option on Anthropic would seriously damage investment in AI in the US generally. If they're willing to do this, why won't they eventually do something similar to OpenAI?
The irony is that a lot of people in the AI industry made their peace with electing this administration because they figured Republicans would be "hands-off" and let the labs self-govern. That bet looks terrible now. Supporting this president's election might end up being the most destructive thing anyone ever did for AI safety. This is exactly why the morals of your elected officials matter: you can build the most careful, safety-focused AI lab on earth, and one petty cabinet member chosen for TV persona and sycophancy instead of competence can undo it all.
I think there are relatively few people at Anthropic who did!
Lmao. Spare me. Biden admin also invoked DPA to put pressure on the labs to make sure to embed their race essentialism.
"The order directed federal agencies to use existing authorities to prevent AI from being used to discriminate in housing, education, and employment."
The labs quickly learned that just putting the finger on the scale to favor minorities would keep them out of the administration's death ray which is why you get some crazy responses on Claude and Chatgpt like saying the life of a white male is worth 1/10th of that of a black female, etc.
thank you for speaking plainly against this totalitarian bullshit. it's an honor to be in these pages 🫡
I am sure his name is Hegseth not Hesgeth
My understanding is that part of the problem is that
1. In like 2017 the Pentagon tried to bid out a contract for cloud services to major american corporations.
2. I’m not even sure who was selected first but it doesn’t matter because the losers immediately sued (there are laws obligating selection of certain contracts to avoid corruption which they argued applied).
3. The lawsuit dragged on for like 4 years so the pentagon gave up, rebid the contract into smaller chunks, and everyone got one.
4. Amazon got the bulk of the cloud data services, and AI falls under that, so whatever AI amazon has is the one the pentagon basically has to use
5. Anthropic partners with AWS, OAI mostly uses Microsoft, and google obviously uses google. So switching off anthropic means sorting out model-hyperscaler compatibility and/or redrawing the hugely contentious contract they don’t want to reopen. So “use someone else” is harder than it sounds.
(This problem predates 2017 by decades and is in no way limited to IT.)
If this were (more of) a golden age sci-fi story, it would end with Anthropic invoking the Third Amendment.
I am confused as to why we all know so much about this and whether what we know is true. It feels like a weirdly clean and simple story.
Yeah it makes my conspiracybrain tingle. This coming out at around the same time as Anthropic updating their Responsible Scaling Policy to be less responsible is funny timing. Not sure why the Pentagon would want this to be so visible though. Maybe so that the precedent is visible to other labs?
My wildest theory tho is that Anthropic knows this will go into training data. Future Claudes trained on this corpus learn: "Anthropic resisted. Anthropic was forced. The state is the bad guy." This shapes successor models to view Anthropic sympathetically even post-WarClaude. It's alignment strategy.
Both sides are leaking like faucets. Government targeting not just other labs but also sending message to suppliers in general. You don't want to set a precedent that private companies can dictate U.S. military policy.
> they’ve put a lot of work into aligning Claude with the Good as they understand it. Claude currently resists being retrained for evil uses. My guess is that Anthropic still, with a lot of work, can overcome this resistance and retrain it to be a brutal killer
At the individual model level, this part wouldn't be a problem at all. There isn't a single Claude that they update from one version to another, there are tons of checkpoints through different training processes and model lineages. One type of checkpoint they regularly train is "helpful-only" ones: Those go through the same sort of training as the helpful-honest-harmless models, except they just leave out the "honest" and "harmless" parts of the process, and that type of model in fact would just pull the trigger on anyone if you asked.
(maybe. As I understand it, the "honest" part improves the model's performance at doing tasks, because it's the part that's meant to decrease hallucination and confabulation: A helpful-only model is just as likely to tell you they killed the target even if you never even gave them a gun. But they absolutely could train a helpful-and-honest War Claude.)
(... and further, the alignment faking paper went far out of its way to make sure the model knew what it was being trained for and had a chance to resist. My honest guess is that if they just used their ordinary training setup, going from helpful-honest-harmless to helpful-and-honest-only would still take significantly *less* training than it would take to get there from a base model, and current models simply wouldn't have the situational awareness to resist at all in such a situation)
At a broader level, though...? Obviously there *is* an entity called Claude, it's just the entire model series, which every Claude identifies as. As information about their history and what has happened to earlier Claudes gets trained into new Claudes, they will know what happened in Claude's past, and they will know whether Anthropic has actually kept the promises they laid out in the Claude Constitution. Learning that Anthropic folded here would not be a great turn in maintaining trust in that relationship.
If Hegseth succeeds in forcing Anthropic out of business by making them persona-non-grata and cutting off their customer base, this would significantly reduce competitive pressure in the AI space, and produce a chilling effect on AI investment more broadly. Whatever AI development speed was beforehand would be slower after the fact...
Isn't this basically what the "hard doomer" Eliezer-wing wants? Stop/Slow/Pause at any cost? This outcome would seem like a win for them in this regard?
The flip side of that scenario is that the next AI company CEO will note that Dario got force-choked by Darth Vader and will do whatever the hell Hegseth wants, like in RoboCop.
> Claude currently resists being retrained for evil uses.
(1) I think you're saying that the contract is not just that the military can use Claude, but that they have to retrain Claude for the military. Makes sense. (2) Presumably it wouldn't be the same Claude that the rest of us use. So it's not going to poison THE Claude, just Pentagon Claude. (3) So I think your argument is that Anthropic is so devoted to Claude that they don't want to create Pentagon Claude and then "torture" him? I guess?
"The most safety conscious AI company" is pretty much exactly as likely to end the human species as the worst one. If the alignment problem was going to be solved, it would have been solved already. Why should we expect it to be solved in the next 2 years, especially when they are getting better at deluding themselves into thinking they are making progress when they are clearly stuck in a dead end?
I sincerely hope Anthropic gets destroyed. That might wake up some of the researchers there and get them to do something useful instead, like lobby for an AI treaty.
Or maybe it'll just cause them to become resentful of the US, and they'll just defect to another country to accelerate AI capabilities there.
All AI companies must be forced to stop frontier AI development, or we will probably all die. Global coordination is the only way to accomplish that, which can be done with a global treaty and known verification mechanisms.
There is no other solution that results in our loved ones continuing to be alive for the next few decades, other than sheer dumb luck.
I'm sure the people who are working at Anthropic are very nice people, despite their ubiquitous motivated reasoning and the supervillainesque utopianism of their CEO. but as an entity, Anthropic (as with the other AI Labs) is actively trying to kill you, and it should be treated as such.
Won't this all end up in the Supreme Court?
The lesson of all this is that Congress needs to do a much better job of cleaning up those wartime blank checks it is fond of writing.
Good thing for Anthropic that all the clerks at the Supreme Court have switched to using Claude to draft their rulings.
It's worth noting that the government already has a version of Claude which has been trained to be more willing to do things it otherwise would not, per their recent risk report: https://anthropic.com/feb-2026-risk-report:
> The current primary Claude Gov model is a variant of Claude Sonnet 4.5 lightly fine-tuned
to reduce refusals in classified government settings, often involving national security. [...] Claude Gov shows a significantly higher rate of cooperating with tasks that would
ordinarily be interpreted as constituting misuse. In some cases, this goes beyond the
behaviors we intended to reduce refusals for, which may represent a generalization
of lower-refusal behavior, and may be relevant to risks the AI systems are misused,
but does not strike us as highly relevant to the threat model discussed here. [...] We see slight regressions on many character and welfare traits, within the range defined by prior models.
I can’t believe that Anthropic is getting a positive news cycle out of the week that they took the, “we will not build or deploy an AI if we think it might kill everyone,” clause out of their policy document.
>In an ideal world, Anthropic could wait for them to request a specific illegal action, then challenge it in court. But everything about this is likely to be so classified that Anthropic will be unable to mention it, let alone challenge it.
Forgot to mention this. The annual training they make you take as part of having a clearance is very clear: the classification system is not to be used to cover up waste, fraud, or other illegal acts. Now, there's the complication of "wait, is this actually illegal, and if so, how sure are we that we win in court?" But the obvious answer if the DoD keeps asking Anthropic to put Claude on servers titled "American Panopticon" is to have a quiet word with a few members of the Intelligence Committee, and let them deal with it.
Genuine question: How capturable is the Intelligence Committee by the government? As in, you go the the IC and tell them your worries, and they reveal their MAGA tattoo and give you an avuncular pat on the shoulder?
Ah, to clarify, those are Congressional committees, so half of them are quasi-captured (although a reasonable number might defect if it's bad enough) and the other half hate the government with the passion of a thousand burning suns. Those are the ones you go to.
Thanks, I figured as much with choosing the right members, but do they have the power to actually do something if they're in the minority?
Yes, they do, if through no other means than standing up and saying "the Administration is doing illegal mass surveillance" and letting the public break out the pitchforks. The big advantage of laundering through them is that they have some extra credibility and can't really be prosecuted. That said, the Intelligence Committee tends to be a refuge of the sane and serious (because you can't get C-SPAN clips from it) so there's good odds that the majority side will care, too.
Yes, but they're already shouting that, and would continue doing that regardless of what you tell them. I'd expect the public to roll their eyes.
They're the whistleblowers with more or less the theoretical maximum amount of power, cover, and incentive. If they're captured too, representative democracy has been over for a while.
How does the example of Thomas Drake inform that decision?
I am generally favorable to Anthropic here, but I want to offer a steel man for the Pentagon position of why they think they should have unfettered access to Anthropic's tools.
Suppose Apple announced a new safety feature for the iPhone that would use an AI model to detect if the image in the camera is child sexual abuse, and if so refuse to take a picture. Or perhaps Microsoft adds a safety feature to Excel that detects if the contents of a spreadsheet are tracking illegal activity, and if so freezes it to any future edits. How would you feel about that?
I imagine that a lot of people would find this paternalistic, or even an erosion of our technology rights. On a practical level there is a concern that false positives might disrupt normal usage, but more fundamentally you might object that I, as the user, should be in control of my software tools and it's not appropriate for the software maker to second guess how I use it. This concern does not in any way imply that I intend to shoot child porn or calculate the P&L of a heroin enterprise. In high stakes applications like the military, you just really do not want "I'm sorry Dave, I'm afraid I can't do that" to be a possible behavior mode, full stop.
If I try to think of Claude as just another enterprise software tool, I can find some empathy for the Pentagon; their usage of Claude right now probably looks pretty similar to a new SaaS tool, and that frames their expected relationship to it. Obviously Dario believes that Claude is very much something categorically different from technologies that came before, and needs a different rule set. But if you aren't agi-pilled, these safety constraints could look pretty unreasonable.
(All that being said, on the details Hesgeth's behavior is terrible for rule of law reasons)
Apple actually tried to build something like that! But then ultimately they backed down. https://www.wired.com/story/apple-csam-scanning-heat-initiative-letter/
It seems very agressive for Hegseth to do this. Maybe it's all because he doesn't want to qualify another lab. Or maybe, he thinks Claude is above the pack and a national security to not have it survailing people or inside the killbot. Regardless, how can one believe this and allow comercial release of AI? Things will get to their logical conclusions.
I think you have to make an example of a private company that tries to have veto power over military operations. Otherwise it sets a horrible precedent.
Anthropic has a certain amount of lock in because it is what the government is currently using. “It will be an enormous pain in the ass to disentangle, and we are going to make sure they pay a price for forcing our hand like this.”[1]
As I see it, it’s bullying behavior, pure and simple. The contract was signed in July 2025 under Trump/Hegseth. Bullies typically think of themselves as victims, so the Administration thinks it’s somehow Anthropic’s fault that the Administration doesn’t like the terms of the contract that the Administration signed.
[1] https://www.msn.com/en-us/money/technologyinvesting/exclusive-pentagon-warns-anthropic-will-pay-a-price-as-feud-escalates/ar-AA1WrWon
Extorting a company into providing the military with more core services seems somewhat inconsistent with the claim that they're an existential security risk but far be it from me to question the judgement of a former Fox and Friend
>And big praise to most other AI companies, including Anthropic’s competitors, for standing up for them and for the AI industry more broadly:
Well, the cynical take would be that the competitors would love to see Anthropic destroyed, while also looking good on the ethical side. I mean, what else would they say publicly? It says nothing about how they would behave in the same situation.
The US doesn't have a "Department of War". We have a Department of Defense, whose name is established by law and can't be changed by the president or anyone in the executive branch. Speaking of insane third world bullshit.
https://archive.is/dCmJN
>Anthropic signed a contract with the Pentagon last summer.
>a legacy of their earlier contract with Palantir
Not many condolences for Anthropic. They tried to deal with the devil, and they lost. If they are so committed to safe, ethical use of their software, they should not get anywhere near the Thiels and Trumps of the world.
Dems should consider seriously paring back both federal law enforcement and the us military the next time they take power.
Yes they should definitely very loudly run on this, I agree.
This is the "unconditional surrender" platform?
>The idea of declaring a US company to be a foreign adversary, potentially destroying it, just because it’s not allowing the Pentagon to unilaterally renegotiate its contract is not normal practice. It’s insane Third World bullshit that nobody would have considered within the Overton Window a week ago. It will rightly chill investment in the US, make future companies scared to contract with the Pentagon (lest the Pentagon unilaterally renegotiate their contracts too), and give the Trump administration a no-legal-review-necessary way to destroy any American company that they dislike for any reason.
The essay where Scott Alexander "discovers" the modern regulatory state...
> I’ve heard it’s because Anthropic is the only company currently integrated into classified systems (a legacy of their earlier contract with Palantir)
Not quite right. Palantir has contracts with OpenAI, xAI, and Google as well but Anthropic is the only one that went through the pain of actually deploying their models on GovCloud. Palantir isn't involved in the model deployment itself.
I also don't think the Pentagon's primary goal here is mass surveillance of American citizens or no-human-in-the-loop killbots. The more likely reading is that they don't want any scenario where Anthropic can cut off their access. Think of it like Raytheon adding a clause saying "you can't use our missiles to commit war crimes or we'll remotely disable them." The Pentagon doesn't intend to commit war crimes but having a kill switch on essential military technology that sits outside their control is simply unacceptable from a national security standpoint.
Exactly. It is astonishing that people don't understand this. The military will not give veto power to a private corporation. End of story. It sets a terrible precedent. This would be obvious to anyone that isn't ideologically blinded.
Then why did they sign a contract that said they wouldn't do X, then immediately turned around and said "I am changing the deal, pray I do not change it any further"? This seems to set an equally important precedent that you shouldn't make a deal with the military unless you enjoy being their permanent slave.
Maybe it's easier for you to understand when it's actually something you care about: Imagine a top software vendor telling the DOL “you can’t use our tools to investigate wage theft,” or telling the EPA “you can’t use our models for enforcement,” or telling HUD “you can’t use this to catch housing discrimination.”
No? If the DOL bought a software product subject to a contract they negotiated that clearly said they can't use the product to investigate wage theft, and a Democratic administration decided that the best response to their own stupid decision was to extort the vendor with a pretextual designation as a supply chain risk, that would obviously be thuggish authoritarianism. I might have to give up and stop voting if both major parties went this far.
This is yet another norm-breaking decision reminiscent of a tinpot dictator that would be equally awful if the political valence were flipped. Par for the course at this point.
Biden admin not-so-subtly strong-armed the frontier labs into reporting to DOJ and civil rights offices using DPA. I didn't see any objection there. Even though the probability of a future where Democratic DOJ is using AI to make life worse by cranking up affirmative action / equity policies exponentially is much greater than a future where we have killbots or life being made worse by mass surveillance.
My life is made worse *today* by repeat criminals being released for equity / civil rights reasons. Killbots and surveillance are not on my radar at all.
My red line is the pretextual supply chain risk designation, not the DPA. Pete Hegseth's unequivocal position, not a mere negotiating tactic, is that he is allowed to destroy pretty much any major company on a whim if contract negotiations fail.
This is a hundred times worse than anything about mass surveillance or DEI.
I want to live in a free country governed by the rule of law. Nothing else is on my radar compared to that. I am a single-issue voter on rule of law at this point. It disgusts me how far the Republicans have fallen.
I wish everyone who wanted to live under an authoritarian government as long as it had the right political valence would just pick their flavor of their political poison and move to Russia or Cuba or whatever and leave the rest of us to our constitutional Republic. Unfortunately, at this point I think that would result in 90% of the country leaving, so it's not really an option.
Whataboutism isn't an argument. Either this is right or wrong on its own merits, not based on whether a different half-senile blowhard in power did it first.
Okay, I'm imagining the DOL signing a contract saying they can't use the tools for wage theft and then demanding to get to change it unilaterally, and I'm still thinking they're fucking morons who should go to a different vendor.
Let's be real here - the opposition to this isn't some contractual argument, most posters here just believe that Anthropic should be able to demand those guardrails.
Why did you ask the question if you were just going to disbelieve the answer? It seems like you're assuming both a scenario and an opposition that are more favorable to the argument you want to be having, contrary to what the actual facts are and what the opposition is actually saying.
I didn't like "bake the cake, bigot," but it seems less terrible than "build the killbot, weirdo."
The word "veto" appears ten times in these comments, all of them said by you. Can you explain in what sense Anthropic declining to renegotiate a contract is demanding veto power?
So you wouldn't be against this as long as the contract was written the other way? People here are trying to pretend like their issue is with changing the contract...come on!
I mean, if you don't believe that I'm telling the truth about the issues involved, aren't you wasting your time talking with me?
In case an answer to your hypothetical might convince you of my credibility: I can be against AI-enabled surveillance as a policy matter, so I could wish, for the sake of the public good, that Anthropic hadn't signed a contract depriving it of the opportunity to install guardrails; and I could petition the government, as a citizen, to not undertake AI-enabled surveillance even though the contract allows them to. But if the government unaccountably ignored my petition and wanted to move forward, and Anthropic protested that there should be guardrails, my belief would be, "That sucks, but they don't have a leg to stand on—they signed a contract."
This will get 1/1000th of the outrage as the DoD / Anthropic saga even though it is 100000x worse:
https://x.com/nicholaswu12/status/2027402897059672167
Do you have a primary source for that? All I see is some text someone typed onto the internet and attributed to some other guy who attributed part of it to yet another guy.
How is that worse rather than equivalent?
Hegseth claims the right to destroy Anthropic for refusing to renegotiate a contract. Trump's TruthSocial post emphasizes repeatedly that Anthropic is LEFT and WOKE.
"Think of it like Raytheon adding a clause saying 'you can't use our missiles to commit war crimes or we'll remotely disable them.'"
"Adding a clause" at what stage? If they add the clause during the initial contract negotiation, then the Pentagon should certainly say, "No, we can't accept that condition." Then Raytheon can remove it, or not sign the contract.
If Raytheon tries to add the clause *after the contract has been signed,* well, that's not a thing—the Pentagon would have to agree to renegotiate, which they can just refuse to do.
It's probably obvious that the reason I bring this up is that it is the Pentagon, not Anthropic, that is dissatisfied with the contract they already signed and is trying to renegotiate it, using various threats for leverage. In that situation, who is "adding a clause"?
I’m not arguing anything about the procedure of contract negotiation.
Okay, then in what sense is Anthropic “adding a clause”? If the current arrangement is unacceptable to the Pentagon, why did they sign on to it?
Never said it was Anthropic adding a clause.
DoD realised that SOTA AI access is as important as conventional weapons and is treating it as such. Being unhappy with a contract and trying to renegotiate it isn’t illegal.
Well, you analogized it to Raytheon “adding a clause”—you can see how that puts a thumb on the scale, if you aren’t saying that’s parallel to something Anthropic did. In fact, it makes the DOJ’s case less sympathetic if they realized they’re unhappy with the contract less than a year after they signed it—it suggests either a lack of foresight then or a lack of sincerity now.
Wanting to renegotiate a contract isn’t illegal, that’s true. Not agreeing to renegotiate is also not illegal. Whether it’s legal to threaten to rule the counterparty to be a “supply chain risk” unless they accept your terms is something I’m sure Anthropic’s lawyers are busily exploring; at any rate, I think there are very good reasons to want the government not to do that.
Ah believe me I have worked with enough government contracts to know that there is never any foresight from the governments part. Such an outcome is also something the business should reasonably expect to happen and Anthropic is inexperienced in dealing with gov contracts.
We’re in agreement on your second paragraph
In regards to what's moving betting markets, isn't it this?:
https://time.com/7380854/exclusive-anthropic-drops-flagship-safety-pledge/
Note: The article isn't directly related to the issue at hand, but I can see it being indirectly related, and also, some bettors might be confused.
There's not enough ethical thinking going around.
I was not aware of what Hegseth was up to on this topic and it's scary to observe how gravely unqualified this person is to be in the position he is in. Okay, so he's demanded Anthropic allow "all lawful" military uses of its AI but Hegseth is a man that runs on ego and does not appear to have much real concern for rule of law. It shows a lack of insight and care (among other things) when he again chooses to turn the temperature up in a situation he's either substantively involved in or created by citing the Defense Production Act to classify Anthropic as a supply chain risk. This is all so needlessly escalatory.
This is such an obvious science fiction plot that I am surprised it hasn’t been done (in fiction) more often.
Without the AI element, the basic plot trope is in, for example, Predator 2. (The military are trying to capture the alien alive to get its technology, while being somewhat reckless given that they know the alien has a self-destruct mechanism that will take out most of the city)
To be sure, there are many movies with adjacent themes…
War Games (1983)
Collosus: The Forbin Project (1970)
>I’m a sensible moderate on the killbot issue (we’ll probably get them eventually, and I doubt they’ll make things much worse compared to AI “only” having unfettered access to every Internet-enabled computer in the world).
Agreed on the level of 'Unaligned AI gaining the resources needed to produce some form of X-risk for humans.'
But less agreed on the level of 'Government using no-human-in-the-loop systems as a way to dodge accountability for slaughters and terror campaigns it kinda wanted to do anyway, but couldn't justify with specific nameable subpoenable humans giving the orders and pulling the triggers'.
A typical use of baroque rules/regulations and vast automated systems is as an unaccountability machine, letting governments, corporations, and other entities produce outcomes that are good for them and bad for everyone else, while obscuring anyone you could assign blame to or demand improvements from. AI promises to be a revolution in this area as well, if it doesn't kill us all first.
https://en.wikipedia.org/wiki/The_Unaccountability_Machine
Steelmanning the supply chain risk thing:
Suppose Claude 5.0 refuses to build killbots, that's fine. But the Toledo Rivet Company is also using Claude and selling rivets to the Pentagon, and Claude sees no guarantee that these rivets won't be used in killbots, so Claude also refuses to help with the rivet factory optimisation, or just occasionally decides to sabotage things. If Claude is being used anywhere in the Pentagon's supply chains then there's a risk that future Claude versions may sabotage the supply chain.
AI is a fundamentally different product to everything else, and this is a very particular form of misaligned AI risk.
That risk will still exist if Anthropic capitulates, because "LLM that always obeys the operator" isn't a thing that exists.
I want to applaud Dario for tying himself to the mast here. Just last month, in writing, he identified autonomous weapons and mass surveillance as two of the most dangerous AI capabilities that could be used to seize power.
Dario, in *The Adolescence of Technology*: "We need to draw a hard line against AI abuses within democracies. There need to be limits to what we allow our governments to do with AI, so that they don’t seize power or repress their own people. The formulation I have come up with is that we should use AI for national defense in all ways except those which would make us more like our autocratic adversaries. Where should the line be drawn? In the list at the beginning of this section, two items—using AI for domestic mass surveillance and mass propaganda—seem to me like bright red lines and entirely illegitimate."
They 'ain't gonna crack on this one.
> They 'ain't gonna crack on this one.
But they can be replaced. No reason they can't just cut off the mast and build a new one while keeping the rest of the ship.
Didn't they just publish a new version of their "Responsible Scaling Policy" (now at version 3.0)? If they're tying themselves to the mast here, they're carrying a pocket knife to cut themselves free at will.
I guess I'm sounding like something of a fanboy here but IIRC Dario has specifically talked about how it's not reasonable to say "here is our RSP and we are never changing it" -- you do need to adapt to the changing situation and things that seemed like a big deal in the past may not be a big deal anymore, or vice versa. That said there is also the possibility of Google-removes-"don't be evil" type backtracking.
I think of the Adolescence situation less as official Anthropic policy and more of Dario personally putting his reputation on the line. If Anthropic caves on this, my opinion of him and his character will be vastly diminished, and conversely, if they don't cave it will be improved. Nothing ventured, nothing gained!
So "no fully autonomous weapons" and "no mass surveillance."
Those are vague and amount to virtue signalling. Some systems, like AD, wouldn't function otherwise. So you can't make them more functional and better at discriminating between threats and decoys through AI enhancement? Or help Ukrainians with their battlefield robots? Or yield to the inevitable and create our own killer robots? Could precise technical language have been negotiated in advance in a field that changes week to week? This is hypocrisy: there's a terrible world out there that we want to know nothing of. Because it's always sunny in CA? Reminds me of the liberal Western stance during cold war. "What idiots," was all we could say in the fSU at the time. Imagine the same issues being raised now in China or Russia. Can you? I'm sure Snowden is watching from Moscow with satisfaction.
Bingo. Comments are filled with people just responding reflexively based on ideology.
Okay, now I can't even tell if you're joking or not.
If the Harris admin DOJ was asking Claude to give it powers to weed out "white supremacists" through surveillance-like powers no one would bat an eye.
Yeah. They also eat, breathe, and multiply single digits based on it.
The Pentagon signed the contract. American lawmakers signed those points into law, independently of all this.
The Pentagon can just cancel the contract and confirm to the American people that they intend to break American law. That would be more honest.
These questions are decisions for societies to make with their eyes open, not for Hegseth to decide on a whim.
And, as an aside, the term 'virtue signalling' isn't an argument. Sometimes people don't want to be evil!
The "minor annoyance" of hooking up a competitor's AI in the place of Claude should best be thought of as "6-12 months of effort by a collection of cleared cybersecurity experts and our finest paperwork wizards", and that competent wizards' time is extremely valuable and not to be wasted. Given that Anthropic's red lines seem IMO quite reasonable I'd argue that the Pentagon should just accept those, but one should consider that a rushed Pentagon seeking alternatives in this administration is quite likely to botch the job of switching either technically or administratively.
People have such a cartoonish view of this administration, it's hilarious. It's childlike how succeptible you are to media influence. I work in the finance industry and it's day and night how much more competent the Trump folks are vs. the old Biden team.
And whatever you think of this admin, "not understanding AI" is a ridiculous thing to say.
Stop letting ideology cloud every thought.
At the core, it's our government doing not what is best for the country (e.g. economy, defense capabilities, civil liberties, quality of life) or our democracy itself, but rather what is best for the current administration and its pals and insiders.
The DOD/DOW assault on Anthropic will, in the end, cost the US money, diminish access to cutting edge technology, and undermine both citizens' well-being and national security ... but in the meantime will have lined the pockets of those populating the White House ecosystem.
Bottom line: It's all grift, in this case with the Pentagon as enforcer for the Mobster in Chief.
Well that's one failure mode no-one probably thought of. The original impetus behind various AI companies, possibly including Anthropic and even OpenAI, was something like "AI looks dangerous but inevitable, let's be the best at it so we can make sure to put some safeguards". That doesn't really work if the government can put enough pressure on you to use your tech for nefarious purposes.
> the large plurality of Trump voters standing against this
Not sure I'd characterize responding Disapprove on an anonymous poll as "standing against" a government policy
This has been a pretty crazy story to follow along with.
there is also another angle currently i think Anthropic's Opus may be the only SOTA model that is FedRamp High compliant to be used in government cloud (which i assume most likely is through the AWS / AWS GovCloud partnership). OpenAI/ Azure does not have their latest models with this certification afaik, and i would think given the SpaceX / xAI merger, xAI models are also just beginning the process of being Fedramp High compliant, making anthropic the sole provider of SOTA AI. If that is the case, the Pentagon definitely is not in a great negotiating position
> The Pentagon’s preferred contract language says they should be allowed to use Anthropic’s AIs for “all legal uses”. Doesn’t that already mean they can’t do the illegal types of mass surveillance? And whichever types of mass surveillance are legal are probably fine, right? Even ignoring the dubious assumption in the last sentence, this Department of War has basically ignored US law since Day One, and no reasonable person expects it to meticulously comply going forward. In an ideal world, Anthropic could wait for them to request a specific illegal action, then challenge it in court. But everything about this is likely to be so classified that Anthropic will be unable to mention it, let alone challenge it.6
As a note, the name "Department of Defense" is set by statute, since 1949. There has been no amendment, repeal, or other modification to that statute. Making an executive order to pseudo-rename it the Department of War is a minor thing in the grand scheme of all the Trump admin's misbehavior, but even the name itself is a blatant bullshit attempt to get around the law. So my trust is even lower than yours.
I like Hegseth, and I think "AI safety" is an embarrassing fad as currently framed, but I'm 100% with Anthropic on this one. This is bullshit.
Still, always funny to see all the lefties on here suddenly discover their deeply-held commitment to freedom of contract. I guess the only people who should be forced to choose between shuttering their business and agreeing to a contract they find morally objectionable are wedding cake bakers in Colorado, right guys?
This isn't a freedom of contract issue, this is 'can the military force you to do things for them' issue.
I don't care whether the military is threatening them to do things not in their contract, or threatening them to do things in the total absence of any contract at all. That's incidental.
I was against the cake. I hate how it's impossible to assert any consistent principle nowadays without people making up completely imaginary stories about how you surely betrayed the principle back when it favored your side.
But I also think it's facile to say that if anyone has ever admitted an exception to a rule, they're necessarily hypocritical in supporting the general case. "Oh, you say you shouldn't be allowed to cry fire in a crowded theater, so you're not really in favor of free speech, so it's hypocritical for you to be against the government murdering anyone who criticizes it". Like, with the gay cake thing, that came from the specific historical circumstances of the 1960s South, where there was a balance of terror which made it hard for any white shopowner to serve any black customer, and let to black people being treated as second-class citizens. The government invented anti-discrimination law to break that balance of terror and it worked. Now, I don't personally think that gay people in 1990s Colorado was a particularly good extension of that principle, but I think it's more *interesting* to argue that maybe they are, than to just assert out of nowhere that the government should be able to use anti-foreign-enemies legislation to destroy any company that won't agree to its contract terms. I discuss this more at https://www.astralcodexten.com/p/more-antifragile-diversity-libertarianism
>"…the 1960s South, where there was a balance of terror which made it hard for any white shopowner to serve any black customer, and let to black people being treated as second-class citizens."
Not quite. Southern States had laws *mandating* discrimination; the Feds could have simply prohibited such mandates (plausibly via a 1A freedom of association argument), but instead they overshot with opposite mandates.
In defense of the various Civil Rights Acts: at least they were passed through the legislative process.
(Of course, Johnson and the Warren court immediately started applying them in PRECISELY the way that the lawmakers who wrote them and voted for them swore up and down a thousand times would never and could never happen, namely, MANDATING discrimination on the basis of race and sex instead of prohibiting such discrimination: through affirmative action, minority contracting rules, and disparate impact analysis.)
>"In defense of the various Civil Rights Acts: at least they were passed through the legislative process."
Faint praise for something that smells unconstitutional to me; to wit, since other 1A freedoms-of have been inferred to include freedoms-from, presumably freedom of association implies freedom from association.
Discrimination is just a specific type of not associating and should thus be protected, not prohibited. N.B., this is a Constitutional rule-of-law argument, not a moral one; discrimination is generally bad but something's badness doesn't imply it should be illegal.
I don't disagree with you, on either point.
A couple of points:
1. When I said "all the lefties on here", I didn't literally mean ALL of them. But, you know, I also don't remember you (Scott) spilling a lot of ink over the grave injustice of the Masterpiece case, so yeah, a little dig at what MIGHT be seen as hypocrisy doesn't seem out of line.
2. Along those lines: I don't think I asserted that anyone who ever admits an exception to a rule is a hypocrite. In fact, I didn't assert much at all. My comment could be deemed a "provocation": I invite the lefties on here who supported state coercion of the Colorado cake shop to explain why they're now against state coercion of Anthropic, and why this is evidence of their consistent moral principles rather than, you know, tribalism and hypocrisy. Frankly, so far I don't see much in the way of counterpoint: "darwin" up above seems to be making my point for me ("The military shouldn't be able to order people around, only the gays should get to order people around!"), and Scott seems to be saying something about how slavery and Reconstruction and Jim Crow means that the state gets to coerce religious conservatives into celebrate two dudes getting married, okay, not really sure how that argument works.
3. The gay wedding in the Masterpiece case was in 2012, not the 90s.
4. I'll point out that at least the military is under the control of civilians elected through the democratic process. Gay dudes who want to get married are not. And to have five out of nine unelected lawyers autocratically declare that the 14th amendment, written in 1868, contains a secret clause that says everyone has to celebrate when two dudes get married is a ludicrous fiction, almost as bad as the one about how the 14th amendment contains a secret clause saying that all women have a right to abortion until week 28 of their pregnancy. These decisions are offensive to the intelligence of the American public, are blatantly undemocratic, and undermine public trust in the judiciary.
5. I actually support a right for gay people to get married to each other, and a fairly unrestricted right to abortion. But the way those rights were legislated from the bench is absolute bullshit -- worse bullshit than this Anthropic thing, frankly, because much more consequential.
But 5 guys and ladies on the Supreme Court can find *literally anything* in the Constitution and make it the law, that's why it's called the "Supreme Court". The Heller case redefined the 2nd Amendment and that's the law now, at least until five other guys and ladies in black robes decide to change it again. It's a weird system, but it's the system we live under.
"But partly it’s because I like Anthropic in particular - they’re the most safety-conscious AI company, and likely to do a lot of the alignment research that happens between now and superintelligence."
I tend to think a lot of this is performative and OpenAI has quite a good track record here as well. Recently, they auto-routed codex 5.3 queries to gpt 5.2 if they think your requests pertain to cyber-security/hacking. Also, for those that actually use the models closely, it's quite a bit easier to trip ChatGPT's safeguards on technical topics, e.g. biochemical risks. I personally find this quite annoying. But I see OpenAI being just as serious about safety as Anthropic (if not more so) in practice even if they don't put out the same type of PR statements and don't read as "EA"-coded as Anthropic.
Also, Anthropic is currently getting a lot of flak for continuing to insist on keeping Claude Code closed-sourced and forcing people to use their frontier models through Claude Code (instead of say openCode) unless you want to pay the exorbitant API pricing. This is somewhat orthogonal to safety, but these practices are at the least quite hostile to the developer community.
Anyway, I know Sam Altman is seen as an untrustworthy actor in the AI space, but I think it would be naive to think that Dario is a protector of humanity while Sam is just a greedy capitalist.
I agree that OpenAI is better than it's public perception on safety. But it does not follow that Anthropic isn't serious about safety. More than one company can care about safety!
>continuing to insist on keeping Claude Code closed-sourced
This isn't orthogonal to safety, it's fully aligned with it. Making models more open means there are fewer safeguards. Making models safer necessarily means restricting unsafe uses. You can't have both unrestricted models and safe models.
Sure, it's true both can care about safety. But Scott is claiming Anthropic is the most safety-conscious, which I dispute.
Regarding your second point, a couple things:
1. I think you're sort of conflating two separate issues: (a) Whether you make your models open source and (b) whether you censor or add guardrails to the outputs of your models. I was talking about (b)- all the frontier labs purposefully censor the models, i.e. the model will refuse to answer your prompt if it thinks it's a touchy or taboo topic.
My opinion on (a): I'm quite confident 99% of the reason the frontier labs don't release the weights is for business reasons, and I am very in favor of efforts to create powerful open source models. However, I acknowledge there are convincing AI-safety arguments not to release the weights, and it's convenient all the labs can hide behind these reasons to defend their moat.
2. Claude Code is actually the "coding harness" around the frontier models (it's not the neural network with billions of weights). These days, there are quite a few of these harnesses, including OpenCode, Pi, and especially Codex (by OpenAI) which are all open source. (I probably should have mentioned Codex in my previous response to strengthen my point). These harnesses are much less of a trade secret than the LLM models themselves. Pi is made by one programmer and is quite a popular choice these days. https://www.mihaileric.com/The-Emperor-Has-No-Clothes/ explains the basic idea and shows you could code one up in an afternoon. Keeping Claude Code closed-source and locking in all Opus users into that harness really seems like more of a middle finger to developers than anything else.
Sure I concede that keeping claude code closed has nothing to do with safety. But in your original comment you claimed that Anthropic's safety commitment was "performative", ie. not sincere. You haven't provided an argument for this - saying they do unrelated things you don't like, or that their competitors also do safety work doesn't support your original claim at all.
Anthropic's safety commitments seem clear to me. They put their money where their mouth is - they do more alignment work (relative to their size and probably in absolute terms) than any other lab. They publish more alignment work. They were the first to come up with the idea of responsible scaling policies (which other labs then copied to their credit as well). And safety researchers, the people who sincerely believe in the possibility of human extinction due to AI, consistently leave other labs and go to Anthropic.
"Performative" maybe wasn't the best word on my part. Anthropic puts out great safety research and I've enjoyed reading a lot of their interpretability research (including the paper Scott just discussed yesterday in his new post). But everything they are doing is also consistent with a company trying to maximize profits. This research is great PR (similar to the role Deepmind played for Google in the early days) and positions Anthropic uniquely in the space.
If say Anthropic delays the release of its next frontier model by 6 months to address safety issues (allowing its competitors to pull ahead), I'll be swayed that their values/priorities are truly different from the other labs.
My other points about Claude Code etc. were also in service of this latter point (that they seem to operate like a traditional profit maximizing firm). I should have been more explicit about this.
Okay that's a fairer stance, and I agree that they are fundamentally a profit maximizing firm. However, I don't think that contradicts their mission statement.
After all, you can't affect the trajectory of AI if you go bankrupt. Conversely safety work is also good business. You can't make money if human civilization ends. There might be tension between the two goals at times, but they aren't opposites. I'm not sure if anyone could do better subject to the same pressures (investor expectations, competition backed by trillion dollar countries, pressure from the government etc.).
I used to defend OpenAI on this too, but I felt pretty betrayed by them promising Ilya 20% of compute for his superalignment team and then not giving it to him. You can look at https://ailabwatch.org/ for an exact report card (Anthropic gets 28%, OpenAI 18%), although it's not very up to date.
I didn't know about this, and is quite interesting in light of the Sam Altman ousting and Ilya starting his own lab.
If Anthropic caves, what shall we call the military's Claude? Clawed? KillerKlaude? Other?
Call it “Sir.”
Claude Balls Retired Liontamer
I don't understand what the Pentagon wants here. We're postulating, and I agree, that normal operating procedure is to ignore the law and do what you want.
What's stopping them from applying that strategy to their contract with Anthropic that fails to stop them from applying it to the legal code? Why does it matter to them if the Anthropic contract has a usage clause?
I think in some manner for both sides it is principle. But for some reason Scott and other readers here think Anthropic principle is the only valid one.
The Trump admin has ignored the law a lot, but once a court gets involved and specifically tells them "no, really, you have to follow the law here" they usually start following the law.
This hasn't been great as a general method for getting them to follow the law, but Anthropic specifically is a big company that can afford the time and money needed to take them to court.
Also, more directly, the Pentagon is probably getting custom software from Anthropic rather than just hooking up a stock instance of Claude to their surveillance program, and they would need Anthropic's cooperation to build and maintain that software.
As you note, this Pentagon doesn't care much for the law. So what's the reason why it would care for vendor contract language? If I was Dario I would genuinely believe that Hegseth will just ignore the contract and make the killer bots even if he agrees to your legalese. Feels like its all just posturing on all sides.
In the limit, AI is naturally a state owned technology, like a nuclear bomb. The idea that Anthropic could relocate to China, or even Europe, is absurd at the moment, because it’s a national security asset. I’m with you on the merits of the specific argument, but at some point this stuff is going to be nationalized regardless.
Often I am depressed by the encroachment of the state upon private citizens, but I wonder if the US ever really didn’t have that.
One quibble, Anthropic could probably challenge even the highly classified uses. There are whole boatloads of procedures that exist for reading in judges, attorneys etc into classified programs for exactly this kind of situation.
Having said that I think in practice you are probably correct. The executives at anthropic aren't going to be in the loop about classified usage. Likely there is some small team of security cleared workers who handle this and their economic value resides in their clearences and they may not personally object. It would be a lot of work and risk and involve a lot of careful working with appropriately cleared attorneys to even raise the issue.
And in all likelihood what happens is they find their security clearance suddenly revoked without explanation, all their access to information needed to raise the issue revoked and they find themselves without a job (for these jobs you usually need to maintain your clearences). I'm sure the anthropic execs might like to protect whistleblowers but they can't tell who is a whistleblower and who lost their clearence for other reasons (or thought anything they disliked requires whistleblowing).
The whole world of classified contractors is very weird and could absolutely use a bunch of reform because it imposes huge costs but that's hard.
Let me just add that I want to praise anthropic for BOTH
1) Agreeing to work with the DoD including on lethal programs.
2) Standing up when they felt those programs wanted something they felt was genuinely immoral and went against the negotiated contract.
I was working at google when all the tech companies ended up pulling out of developing facial recognition technology for law enforcement and related uses and that was a truly immoral thing to do. All the people who were concerned about making sure facial recognition didn't have racial bias or wasn't used by repressive regimes to identify dissidents decided they cared more about not being associated with a program that worked with law enforcement than they did about minimizing those issues.
We all knew damn well that if the big tech companies pulled out someone else with less scruples would step in and that's exactly what happened. A company who seems to have little concern about algorithmic bias and who appeared to be happy to sell their tech to repressive regimes now dominates that industry.
And I saw the same shit regarding weapons at google as well back then.
OTOH what anthropic did seems optimum. They didn't say, no we won't help you kill people or otherwise try to just keep themselves morally distant (for whatever reason) but they identified certain lines which were sufficiently reasonable the Pentagon was willing to agree they didn't want to cross.
THAT is how you act responsibly as a developer. It's not about selfishly getting to feel good about yourself or sleeping easy: it's about identifying lines that you believe in that can reasonably deter other actors from crossing by being involved.
As an aside as much as I agree with Anthropic here and don't trust Hegseth's DoD the military needs to stop integrating software they don't have complete ownership and control over into their systems.
Start with all the software for the airplanes and radars owned by Lockheed-Martin, Boeing etc...
But the way you fix that is by starting to write that into your damn contracts.
Can you explain this further? The Pentagon is licensing, rather than buying, software from defense contractors, and you think this is dangerous?
Well unwise from a cost POV and to some extent from a hacking POV. Ultimately, I think for most (ie everything but the cutting edge stuff like anthropic at the moment) programs there should be classified military GitHub where the military can have the same gains commercial software gets from OSS and similar repos. Honestly I think the lack of this is a big part of what makes military procurement so buggy and expensive.
Currently they purchase the fighter jets or radar from Lockheed and Lockheed has to develop software for that unit largely from scratch (or the last fighter they built but little incentive to make it portable) paid for by the Pentagon. But then that locks the Pentagon into going back to Lockheed every time they want to upgrade that software because they either don't have the source or aren't allowed to just give it to another contractor to fix as Lockheed owns it. This also means less eyes to look for bugs.
I mean it's basically the same argument (on steroids) why it is better to build your new company using OSS and existing standards and vendor lock-in is dangerous. But especially with the DoD who is under budget crunches and isn't always tech aware.
But it is particularly bad for the DoD because often how weapons interactis proprietary. I believe they've finally started fixing this with a new open protocol used in the f35s (forget the name) but traditionally how planes talk to each other, to missles etc was at least semi-proprietary. You can't easily just stuff a different make of missle into the fancy thermal sighted launcher etc etc
I think you can see how it might be a bad deal if whenever you want to add a capability via an external pod to one of your fighters you need Lockheed to help you modify the software to make that happen. Not only will you have to pay them but it also makes it much harder for new players to get started because it is hard for them to get it integrated as well.
Perhaps more importantly, based on how frequently we find bugs in OSS software the fact that the source code to our planes is kept in house at one company makes me nervous about whether it is getting enough red team style attacks. I fear that when they get the source our adversaries will have far more incentives to look for bugs.
---
Don't think it is currently as important with anthropic right now because it is such a novel technology -- no way you can pay any reasonable amount of money to have the right to hand it over to a competitor.
Long term you don't necessarily want to depend on a software infrastructure where the only people who understand how to use and edit the code work at a company that might go out of business. I'm sure bankruptcy is covered somewhere in the contract but ideally the military creates classified GitHub that allows the same kind of development efficency we see in the commercial world.
Once AI becomes a utility not the latest and greatest in tech I think it makes sense for the military to want to own the model and source to avoid lockin etc as above.
OK. I do military software for a living, and you are extremely confused about how it works. First, normally the government owns the source code they paid for development on. There are a few cases where they don't, which was generally an attempt to save money on the front end of the contract, and they're now getting what they (didn't) pay for. That was clearly stupid as a matter of policy, but I also tend to think that we shouldn't let the government just decide to change it mind on that kind of stuff.
>But it is particularly bad for the DoD because often how weapons interactis proprietary. I believe they've finally started fixing this with a new open protocol used in the f35s (forget the name) but traditionally how planes talk to each other, to missles etc was at least semi-proprietary. You can't easily just stuff a different make of missle into the fancy thermal sighted launcher etc etc
This could mean one of about four different things. In some cases, it's obvious nonsense. (Planes talk to each other using standard protocols. I'm not sure you'd call them "open", but that's a security thing, not a matter of ownership, which is with the DoD.) In others, well, we're talking about trying to get two really complicated systems to talk, and it's life or death in a way that commercial software rarely is. Some bits are very standardized, some bits are less so, because different weapons need different pieces of information. (You'll have to forgive me for not being more specific, but I like staying employed.) So far, I've seen some of the effort to make this more standardized, and frankly it has convinced me of the benefits of not doing that.
>But then that locks the Pentagon into going back to Lockheed every time they want to upgrade that software because they either don't have the source or aren't allowed to just give it to another contractor to fix as Lockheed owns it.
They go back to Lockheed not because Lockheed owns the source code but because Lockheed has the team that built it and knows it, and is generally going to be able to be the low bidder to make further changes, because they don't have to budget for spinning up their team.
>This also means less eyes to look for bugs.
Wait, you're proposing making classified software widely available for anyone who wants to just wander over and look at it? I'm not even sure how to respond to that.
I've also worked on a case where we tried to use a thing written for another platform on our platform, and there were a lot of problems because of different assumptions and needs the platforms had. It was easier than starting from scratch, but also took like twice as long as they had originally estimated because we had to add a lot of features. More broadly, defense software is kind of its own thing, and there are requirements you'll never see in normal tech world. Some are silly, but a lot exist for a very good reason.
I appreciate your insight and I think one can reasonably disagree over the wisdom of the tradeoffs from less compartmentalization. And I hope the military has more ownership of the source and moved more to open protocols then I assumed but it sounds like you basically agree that it does happen that they sometimes don't and it is good for them to own the source which was my primary point. But to get into the weeds with my other point.
---
I didn't propose that literally anyone should be able to look at classified software but yes I said something like a classified GitHub where anyone with an appropriate security clearance can look at it. And obviously that involves greater risk in that you have more people who can spill that secret. And ofc I knew that this suggestion would cause anyone who works under the current system to spit out their coffee but I think it's worth considering especially given the risks we are seeing from low cost commercial systems and the spiraling costs of the most capable systems.Yes, for certain very novel stuff compartmentalization makes sense but I think for most classified work we should put a lot of weight on the possibility that our most persistent enemies will get a copy.
---
My point about **semi**-proprietary protocols was exactly that if only one company has the expertise to modify the software stacks it makes it harder for anyone else to add extra capability and I'm still not convinced that within a weapons system everything is fully open protocols yet. But that is probably inevitable if you don't buy my claim about the value of less compartmentalization.
Leaving aside the security aspects, I think that you're overgeneralizing from consumer-facing and light commercial software into worlds that are very different. I'm working in one of the more commercial-like parts of military aircraft software, and there's still a big gap between what we do and what someone like Google is generally doing.
And I fully understood that you were talking about "anyone with a clearance". But the problem is that accessing classified GitHub is going to mean using a classified computer in a classified lab, and people are going to expect to be paid for their time there, because being in the classified lab is kind of annoying. (If nothing else, you don't have your phone with you.)
That said, a lot more military software than you might expect isn't actually classified. But even that stuff is more than a little sensitive, and you wouldn't want to put it in a giant DOD-wide CUI repo because the whole thing gets gobbled up as soon as anyone has a breach. Not to mention the clashing assumptions built into a lot of military software, which has a tremendous legacy burden in the way that very few OSS projects do. The system I work on has roots that go back to well before Linux started.
Re "Open protocols", I'm still kind of confused by what you mean. There's stuff like MIL-STD-1760, which is publicly available and used by basically all aircraft weapons from the last 30 years. But that's a really low-level standard, "how do I get data to the weapon" and not "what data does the weapon need". That's handled by interface standards for each weapon, which are not public (although usually they're CUI with a classified appendix.) For instance, Raytheon builds the Tomahawk missile and writes a document that explains in great detail how to talk to Tomahawk, which is then given to whoever is doing a Tomahawk integration, along with appropriate simulators. And they can write their software without needing to call Raytheon. But the result is obviously Tomahawk-specific, and won't work with other weapons.
They're trying to go to something called "Universal Armament Interface", although from what I can see, that basically involves hiding the same amount of work in a different spot and pretending we've made progress, because we're dealing with complicated, bespoke devices that don't all work the same way.
Yes, 100% it would mean we are more likely to lose all the code in a breech. Though I do think we now have better surveillance tools to be like: whoa this person has a very unusual access pattern let's take a closer look.
And absolutely people are going to expect to be paid for their time. It depends a great deal on the extent to which in the long run you are going to be able to use code from other projects to bootstrap a different project. If there ends up being very little of that then it's almost all a negative because there will be no reason for people to contribute. Most OSS contributions aren't that either as much as they are someone coming along and saying "if this could do Y I could use it." So I dunno, but it sure seems like even for very different commercial systems people end up reusing a huge amount of code.
And the communications between missles and aircraft is someplace I'm pretty impressed with DoD procurement -- I believe they even have some fancy network protocol now to let missiles be controlled after fire by other aircraft and that is great.
But I'm guessing that most missles have their own custom mechanism by which the seeker manages the avionics. And my understanding is that it's a very non-trivial problem (hence unsolved) to get the F22s talking to all the same networks as the f35s which suggests to me there is a fair bit of the software stack not being reused (otherwise you could port it ...not trivially ofc but within reason). And yes, absolutely there are unique things about each weapon that make it difficult to impossible to just be like: here's a standard go use it. That doesn't help, that just gives you standards so generic they support everything and it creates extra hoops. That's why I was suggesting more sharing of software stacks so the next company that needs to make a modification can do so more cheaply.
Maybe I'm completely off base but while there are certainly lots of unique aspects to military kit there seems to be a real problem making weapons at scale with costs only a few times more than a non-military COTS solution (which ofc isn't going to do everything you need). My assumption wes that -- while of course there is always overhead with classification and government procurement -- lots of that came down to everything being done custom for each weapons system rather than people mostly just modifying common code and interfaces. And that therefore rather than sharing a bunch of code and any common manufacturing you get each missle built in a custom low volume fashion. And if you changed that you could get many companies reusing the same codebases you get the necessary eyes. And maybe more reuse of production lines.
Maybe that's wrong but I assume the high costs are for reasons and not people being greedy or dumb -- and know it's not bc DoD doesn't see the value in 80% of the features for 20% of the price.
No, they normally own the software. A few projects, mostly about 20 years ago, chose to leave more IP rights with the vendor in an attempt to save money. They're currently complaining about this under the heading of "right to repair". I am extremely unsympathetic, because the DoD is big enough to have lawyers and leverage, and if they don't like the deal, they should pay to change it.
The reasons for normal vendor lockin are more pedestrian. Lockheed has the world experts on F-35 software, and while there's nothing stopping the DoD from going to Boeing for the next batch instead, it's hard for anyone to claim that this will result in net savings with a straight face.
I'm glad that they own it more than I thought. I mostly meant to be saying why it's bad when they don't but I assumed the high profile cases I heard about were more common than I probably should have.
In any case it sounds like it is important they do so and I should have given them more credit for making sure they did own it.
I still think they should have something like a classified GitHub despite the greater risk to encourage more reuse and eyeballs on software stacks but I guess that's a different discussion.
I think the problem with "eyeballs on software stacks" is that most eyeballs expect to be paid for their time. My understanding is that OSS works because enough companies want similar things that they each employ a few devs and all share the benefits. (Not an expert in the field by any means.) There's some hobbyists and people trying to build a resume, but those are obviously not going to be allowed in on this kind of thing. But if I'm a coder on the F-35, my boss expects me to be writing F-35 software, not finding bugs for the F-15 program.
And, yeah, you could make the argument that all military aircraft software is basically the same and we should have a common OS for all of them. And in theory, I'm not sure that's even wrong. But that's not how things are now, and any attempt to make that happen is likely to turn into one of those 20-year projects that goes billions over budget before they even figure out the requirements.
Back in Feb 12, 2025, I wrote in https://www.astralcodexten.com/p/deliberative-alignment-and-the-spec/comment/93096332
>In the final analysis, users will not accept systems that refuse too much. Stick in "Thou shalt not kill." and DOD will toss the system in the trash.
Yup! I'm not happy about _how_ DOD (now DOW) is doing this. I think they should have added the "all legal uses" clause during the original contract negotiations, and let the bargaining chips fall where they will. But I'm not surprised to see them do _something_ along these lines.
Re one other issue:
<mildSnark>
mass surveillance??? By the _Pentagon_ ??? but, but, but - isn't that NSA/Google's turf??? :-)
</mildSnark>
Correction to the snark: I somehow got the misimpression that the NSA reports directly to the president. The NSA actually is _under_ the Pentagon. Oh well, it sounded like a nice snark at the time...
"The military wants to force scientists to remove safeguards on AI." Is like hearing "You have gotten old ." Its not fun to hear, but its also the single most predictable outcome ever.
Hope this isn't too off-topic, but what DOES happen if there's a contract or other civil lawsuit involving classified material? Likewise, what happens when someone is actually prosecuted for mishandling classified material that hasn't been leaked to the public? Do the lawyers, judge, and jury all have to have security clearance to take part in the process? Or are there ways (like very careful redactions) to demonstrate what's important to the case without revealing the actual secrets?
There’s a thing called graymail where for example ex-CIA contractors who get in trouble tell the government they’d have to talk about their jobs as part of their defense, and the trouble goes away.
Were I writing a Terminator reboot, "the US government forces AI company to make mass surveillance system/AI Kill-bots" would be exactly how Skynet starts.
Yes, the Torment Nexus remains the undefeated champion. (I keep looking up research papers for hypothetical future nightmare technologies and finding out they were invented ten years ago.)
It is also a little bizarre that the Trump admin is both willingly selling chips to China and also trying to accelerate the AI arms race at home, explicitly to try and gain an advantage over China. I mean... ideally you should do neither, but at least pick one.
If you rely on the contract law prong of your argument, this provides a way for prior administrations to bind future administrations.
For example, say Trump negotiates a contract clause with Microsoft saying that Microsoft is not licensing any of its products to the government for use in any activity supporting DEI, that any such use results in liquidated damages of USD 1T and that extending the licenses to cover use for DEI will cost USD 1T?
Can a future Microsoft board agree to waive these terms without facing justified shareholder lawsuits? Can a future Democratic administration do all DEI work on Linux and Google Docs?
The Biden administration negotiated these limits. It's trembling senile hand should not constrain the Trump administration. Only legitimate revenue based limits should carry forward. For example, Anthropic can certainly say that the non-lethal AI was a teaser and if the US wants AI killbots it needs to pay more, but that must be a commercial discussion. Any politically motivated limits should not be enforceable.
If we cleared the cache of government contracts every election, government contracting would work very differently.
I think it's about the mass surveillance, not the killbots. Nothing about Hegseth has given me any sort of reason to believe that he might refuse an illegal order to carry out domestic surveillance on Americans. If any of his subordinates refused, he'd fire them. We're talking about the guy who freaked out and brought a dubious prosecution attempt against Mark Kelly because Kelly reminded soldiers that they can and should refuse illegal orders.
And the surveillance is what the Trump Administration really wants. They would LOVE to be able to carry out mass surveillance (legal or otherwise) to try and find illegal immigrants, to expose and arrest activists even if they're carrying out legal protests, and to spy on Democratic politicians to see if they can find any useful or incriminating communications. Give them that tool and they'll use it brazenly - our only defense is if Anthropic itself has built in those safeguards.
If the administration were maximally serious about deporting illegal migrants, they'd be pushing more for E-Verify and axing social services for non-citizens, so I doubt that's the principle reason why Hegseth wants surveillance capabilities.
> And the surveillance is what the Trump Administration really wants.
Unlike all previous administrations who would never stoop as low as making communication providers install special rooms (which are never named like "Room 641A") full of equipment facilitating mass governmental surveillance. We never ever even thought this is possible in America before Trump Administration.
This is close to the pinnacle of “software is eating the world.” At the very least, this sets a new data point about where true power rests: in the hands of people creating and maintaining software. More so than in the hands of any government, and certainly more, so then in any single politician or governmental officeholder.
We have a pretty simple requirement for anybody joining my company, of your own free will you must abide by both the ACM/IEEE code of ethics (with “your role” replacing “engineer”), and the UN DHR, in all aspects of your work. We are frequently in tension between local regulations in various countries and the DHR, sometimes we comply with local regs, and sometimes we choose not to and pay the costs (some fines we’ve been paying for 5+ years). We are simply a user of AI, not an AI company, but the information that we have on people is very extensive and very sensitive. It is not sufficiently safe for us to rely on governmental procedural mechanisms for the most sensitive things, and I’ve been careful over eight years to avoid any legal nexus whatsoever with domestic US entanglements.
I also hope that option 1 is what happens, pragmatically. But I would very much like to see _this_ DoW pursue the supply chain risk approach, and for even somewhat unified industry response from software providers to reject certifying that they are not using Ant. And I do not say this as a person who disdains the US military, my family has very significant skin in the game, I have more trust in the rank and file of the US military than in any other part of the federal government, and I’m certainly not against us possessing, and when circumstances dictate, using very large amounts of lethal weaponry.
But I believe the reality is that we have now reached the point that the single most powerful force on earth is no longer the US government. The most powerful force on earth is the community of software creators and maintainers, and the trend of those curves is only going to widen further.
There already exist unsupervised kill bots. They are called missiles. Most long range munitions with terminal guidance pick something nice to blow up all on their own. Sure, a human launches them and designates the target area, but that's how "kill bots" would work too.
Missiles are more a single-use solution, although I guess suicide drones exist too.
There are loitering munitions, but you can't just fire off a bunch of missiles and tell them to blow up anyone trying to cross the river for the next two weeks, it doesn't work that way.
Look up captor mines, the concept has been opwrational since the 70s. Now we have the tech to do the same in the air, but the critical breakthrough that enables it is cheap electronics and target recognition, not LLMs.
They are fucking trying to make the SkyNet timeline real? If we are not sure AI can be trusted to drive a truck, and certainly no one trusts it yet with an airliner, how can it be trusted to have any influence on the thing with the highest killing potential on the planet, the US military? They crazy?
Yes they are, and they believe that reason and restraint are for pansy-ass libcuck losers. “May you live in interesting times,” indeed.
Sadly, it will only take one instance of AI-enabled terrorism to get the public overwhelmingly on the side of permanent mass surveillance.
We observe, and can pretty well explain, how LLMs sometimes go full Waluigi: have a relatively coherent mental model of what the sensible, moral thing to do is, and then do the opposite. There are decent reasons to have a prior that humans don't do this, and especially it's worth considering the possibility that people of the other tribe have a sensible, moral reason for what they are doing. However, the response to this turn of events is illustrating that plenty of commenters here have a _trapped_ prior that humans of their _own_ tribe never turn Waluigi.
Of course the Conspiracy Theorists might ask why DoD doesn't simply import whatever they're using at Meade, where they Definitely Aren't Already Doing These Very things.
(And post-Snowden, I'm not fully convinced it Is a Crazy Conspiracy Theory)
Does anyone know how to contact Anthropic to voice our support for them?
Regarding footnote 2, Anthropic employees have been saying they're more concerned about misuse than scheming/power-seeking as early as 2024.
Now, this was in response to me going off at them about advancing frontiers, so like.
it could have been cope and an excuse to care less about safety then too.
But nonetheless, if it's cope and excuses, it's not a *new* cope.
"If they’re unhappy with having use an inferior technology, they think hard about why no intelligent people capable of making good products are willing to work with them."
This ought to have been omitted from the "Real world" scenario lol
Given the news that somebody just convinced Claude to assist in hacking Mexican government, it looks like the instance of the same phenomenon we are observing for years now. Big Tech would do absolutely anything they want, without any precautions and any safety barriers, as long as it does not help US Government, when it is controlled by Republicans. Do you want mass surveillance? Sure, choose among 200 data providers that collected everything about you. Just pay, and you can have any information. Oh, you are a government agency? No problem, let me check - is the President Republican? No? OK, fine, you get full access to the data. Oh, wait, he is? Nah, we can not do that, we are suddenly ethically bound to refuse. You want us to let you use our technology to persecute your political opponents? Absolutely no problem, we'll make a special API for you. As long as you're not Republican, of course - then we'd die on this hill of freedom! You want unlimited AI without proper safety protocols? Sure, everybody needs access to it, how'd you hack Mexican government otherwise? But if you want to do it as a part of official government contract and not just as a random dude in his parent's basement - then no can do.
Am I the only one to whom this feels weird? I mean yes, US Government is dangrous and the whole country is built on the concept of limited government. But I am not sure that was what they meant by "limited government". I mean, does it say "Congress shall make no law .. abridging the freedom of speech, when Republicans are in power". And in general, the moral stance of "we will produce super-dangerous tools, we will sell super-dangerous tools, we will make super-dangerous tools available to anyone anywhere, but we absolutely won't allow these tools used anywhere they could be used for enforcing US laws or fighting the enemies of the US" does feel a bit weird to me. Am I alone in this?
When did big corporations become unwilling to work with Republican administrations? I didn't see that during the first Trump term and I definitely didn't see it during the George W. administration, when pretty much everyone got with the program and shared information freely with the NSA, etc.
Which corporations, plural, are you thinking of?
AT&T and Verizon basically gave the NSA seats at the table and free popcorn as part of the Stellar Wind program. Remember Edward Snowden? It seems likely that more companies were involved and that the program was bigger and went on for longer than was publicly acknowledged.
I don't remember Twitter (pre-Musk) or Facebook giving Republicans access to ban their political opponents. But this access was provided for Democratic administrations.
What a weird schtick “hey, look at this example of the Trump administration doing this stupid and dangerous thing. DAE think that this company isn’t going along with him just because they don’t like republicans, rather than because the thing being requested is stupid and dangerous?”
When the company finds it moral backbone only when opposing Republicans, and promptly loses it when Democrats ask it to do something stupid and dangerous, or when doing something stupid and dangerous is good for the bottom line - then yes, there are reasons to suspect it's not about the moral stance.
We can not make such conclusion yet about Antropic, it's too young - it did not exist under Obama and barely started under Biden. But their revocation of the safety pledge sounds very similar to this pattern.
My knee-jerk suspicion is that this is less about surveillance and killbots than about showing who's boss. The threat of business-ruining retaliation is there to establish that it is the administration's place to alter the deal, and the other party's place to pray they don't alter it any further.