>Of course, the Early Christian Strategy is to ignore all of this and do the right thing in every case, including unilateral military disarmament. But they can’t keep getting away with it, can they?
I mean, maybe not. Time will tell. It's not over yet.
The idea is interesting, but the thing about critiquing strategies based on their success so far is that we don't know when the endgame is. I don't think it's pointless to compare strategies and try to understand their advantages, but it's not the endgame yet.
Empirically, the Christians did *not* stick with unilateral military disarmament. See especially: the crusades and the inquisition. They eventually returned to a peaceful approach, though.
He was baptized on his deathbed. It was standard at the time to delay baptism as long as possible, to maximize its effects (less time for more sins to be committed after your one sin-erasing moment). The New Testament kind of clashes with this notion, but it's not like the New Testament canon was even finalized yet in Constantine's lifetime.
That he converted 25 years before that is almost universally acknowledged by historians. There may have been some lingering bits of syncretism, but you don't claim to see a burning cross in the sky that says "In this sign you will conquer" and not immediately convert to the cross-having-religion, that's just not how anything works.
Even after the Milvian Bridge Constantine continued to make sacrifices to the gods as Pontifex Maximus, use divination, and mint coins depicting the pagan gods.
We could say that Constantine first added the christian god to his polytheistic worship and he then became fully and exclusively a christian at the end of his life.
Funny enough a yt channel I follow just posted a video on "Constantine The Great's Pagan Coins"
Those aren't early. Crusades happened approximately 1,000 years after the beginning of Christianity. Inquisition happened another half a millennium after that. Scott is talking about the first century, or maybe first couple centuries, when Christianity was being persecuted by the Roman Empire.
Yes, *Scott* was talking about an earlier era. The comment I replied to said, "I mean, maybe not. Time will tell. It's not over yet." Clearly they weren't talking about Early Christians anymore.
Christian states were fielding armies long before that.
Its easy to preach unilateral disarmament when you're a small sect and you have no effect on military policy anyway. Its another thing entirely when you're the state.
The gap between Constantine converting and Augustine of Hippo's just war doctrine are very small. And Rome didn't disband its army in the interim period.
Because Christians belong to Christ, rather than Christ belonging to Christians, the only true Christians are the ones who follow Christ's commands. Everyone else only serves to blaspheme God. And this was a problem between God and people long before there were even Christians. As it says in the book of Romans, in the New Testament:
Well then, if you teach others, why don’t you teach yourself? You tell others not to steal, but do you steal? You say it is wrong to commit adultery, but do you commit adultery? You condemn idolatry, but do you use items stolen from pagan temples? You are so proud of knowing the law, but you dishonor God by breaking it. No wonder the Scriptures say, “The Gentiles blaspheme the name of God because of you.”
I'm not interested in questioning who counts as a "true" Christian. The bible also makes very clear that we're all sinners who fall short of our ideals and God's commands.
Very true, that's also in Romans. Seeing as you were making a comment on Christians and their actions, though, it wouldn't really make sense for you to not be interested in what actual Christians did vs. posers. If not, it doesn't really make sense for you to post your initial comment to begin with. Let alone to start it with the word "empirically".
Whereas I think that if the use of "Christians" as a term for a historical group makes sense at all, then IMO it only makes sense to use it if you're applying it to an identifiable group, rather than a scattered collection of individuals within such a group. In which case, I can't not apply it to Catholics from the conversion of Constantine through the Protestant Reformation. If you want to argue that many of the people in the group I'm labeling "Christian" are deeply misguided, I have no problem with that! If you want to go full Chesterton and say that Christianity has been found difficult and left untried, I have no problem with that either! But *specifically* excluding the largest group that calls itself Christian, which made up an even larger proportion in the period under consideration, I find to be an unambiguous No True Scotsman definition.
So the early Christians were peaceful and powerless, then Christianity became powerful and non peaceful, then it lost power and became peaceful again...?
Reading Lyman Stone's article sets off my Historian Bullshit Sense. In particular, the source he links for his claim that Philip the Arab was a Christian really, really does not back up his bolded claim that:
> Almost a century before Constantine, Christian ideological leaders already have the view that Christians in politics should be declaring open war on pagan worship.
I don't know this is wrong, it seems pretty plausible to me, I just think it's totally unjustified by any of the sources he claims to cite. He's doing the flimsy-chain-of-logical-deductions-each-individually-plausible thing, and most of his claims about Japan and China seem to be about as unjustified. I do not and would not trust anything he says without independent research.
It's also certainly unjustified by the story he tells in the text. There's a major difference between thinking that a Christian emperor should not *participate in pagan ceremonies* (all Christians were forbidden from doing that), and thinking that as a political matter, he should shut them down.
Both may be true -- I have absolutely no idea -- but he needs to distinguish and provide evidence for the one he needs, and he doesn't.
Ah, reading that and looking him up, my immediate reactions are:
(1) He's a Lutheran? Figures! Reflexive remnants of anti-Catholicism, because the second the name "Constantine" comes up, the Reformation trope of "he was the one who perverted Pure Early Christianity by tying it to the state" and "This was when Roman Catholicism was invented, that's not Christianity!" rears its head.
(2) Okay, now he's jumped on the hobbyhorse and is galloping furiously off. "No, Christianity did not succeed because X, Y or Z, but because My Thing!"
Christians were high fertility and *that* was their winning strategy, says guy working for pro-natalist movement. Gosh, you don't say? So them being Christian was nothing to do with it, just that they were (ahem, excuse me while I take the piss here) sexist patriarchal oppressors of women who forced their wives to be barefoot and pregnant and engaged in marital rape to keep them popping out a baby every year? Is that your message, Lyman? 🤣
"Pagans Had Low Fertility, Christians Had High-ish Fertility
...I have a huge amount of new, never-before-used evidence demonstrating that Christians really were disproportionately female, Latin-speakers (especially low-social-status Latin speakers!) really did have low fertility in the 2nd/3rd century, and Christians very likely did have higher fertility, not least because of their absorption of the highly fecund norms of Egypt."
Dude. Duuuuude. "New, never-before-used evidence" about female and low-social status Christians? Damn, I thought I lived under a rock! Then again, he qualified as a sociologist, I shouldn't be too harsh on the guy for being ignorant of basic, established, well-known historical content.
EDIT: Also, Lyman, not to throw shade on you, but it wasn't the highly fecund norms of Egypt, it was the anti-abortion stuff. They didn't kill their daughters, they didn't abandon their newborns, they even went around picking up abandoned babies off dungheaps, imagine!
As an aside, I had never heard of Philip the Arab as Emperor, but to my shame I'm not familiar with the Year of the Six Emperors and their various successors who were technically emperor for nine days before the next guy stabbed them. I've never read any claims that he was the first Christian emperor, so sorry Lyman, I think you're spurring on the hobbyhorse here.
Broke: we can't easily judge strategies outside their endgame.
Woke: we're in or blatantly running into many potential civilizational "endgames" by many metrics (used by you and/or others), from climate change and AI to national politics and (depending on your religion) Middle East prophecies.
Bespoke: you can sorta model people and groups by how "endgame mode" their strategic thinking is, a little like Burja's live actors vs dead actors. E.g. dead plus not-endgame-thinking might be "political campaigning like it's 2010 again", endgame-vs-not can cause rifts by who's a transhumanist vs not, etc etc politics politics social groupings
There's some distinction to be drawn between *actions* and *policies* (a policy describes how to act, given a context/situation, it's a decision rule). In between there are partial policies of varying flexibility. I think the armament case can perhaps be resolved by treating the rule you wish to be universalised as being (in large part) a *policy* (perhaps something like 'disarm when multilaterally possible, don't be first to escalate, seek de-escalatory stances and movement, ...').
This distinction might solve the general case too, unsure.
It's an important distinction in general for consequentialists. For example, basic sequential search theory against a known distribution suggests an optimal stopping rule--keep searching until you find something better than a threshold value. That looks like a "satisficing" action, but the policy is optimal if the stopping value is chosen appropriately.
Then you can wonder about optimal meta-policy if figuring out the optimal level of the stopping value is costly. And so on in a potentially infinite hierarchy.
Yes, but if you write it all out that way it wraps around to deontology again. Albeit a deontology even more complicated than biblically accurate Jewish law.
> every time someone tries to create the Peaceful Nonviolent Pause AI Movement Based On Peaceful Nonviolence For Peaceful Nonviolent People, it spends an inordinate amount of resources keeping out violent crazies who want to tag along.
This is untrue, and a strange thing to confidently claim a priori. It actually takes very little resources to keep the crazies out, in both absolute and relative terms. More than zero, but it's really not that bad.
There was the StopAI founder who went crazy and is missing, and the guy who attacked Sam Altman tried to join one of the PauseAI things and had to be kicked out, and the PauseAI people sure seem to spend a lot of their time loudly condemning violence and having sections on their site about how bad violence is.
Guido is still around and tweets, did some protesting last summer, old timey nonviolent stuff, and popped up after the altman house incidents to re-commit to that. Like Emile Torres, he sometimes gets sidetracked attacking LW/rationalist types and Yud in particular, over various posthumanist things Yud has said but which were conditional statements with extraordinarily low probability that will never occur in any case if AI goes badly, so he's kind of alienated a lot of possible allies for what seems like nothing. But I still like the guy, he's fine.
Sam must be the one who disappeared, I don't know the deal but he said he was "no longer affiliated" with StopAI late last year and I haven't seen any mention of him since then.
I can't speak to Stop AI so much; they have their own particular.. uh.. issues. On the loud condemnations, there's some recency bias on how many tweets are about that, I suppose, given the need to respond to those attacks, and even then it's still a fraction of a percent of the total effort involved in running an advocacy org. I don't know what would be inordinate. Maybe 5% of total resources? That would be ridiculous.
This is a thing that it is very important to get right, but it's also not a thing that consumes a great deal of resources once you have established norms. In the case of PauseAI US, we're structurally set up to significantly reduce this risk vector. We just don't let violent crazies tag along. If they try, we tell them to buzz off. If volunteers suck, we kick them out. This is not a particularly hard problem to solve.
Of course, it's attractive to imagine that this is a debilitating problem if you are looking for reasons not to do advocacy!
And if they refuse to do so? It's the open street, they have as much a right as you to walk along it, they're not part of your protest they're doing their own protest, etc.?
I think that the last issue, constantly and loudly condemning violence, is less an effort to keep crazies out, and more an effort to prevent themselves being interpreted by hostile outsiders as endorsing violence. Consider how much Eliezer in particular gets accused of endorsing violence measures to prevent AI doom, in spite of his disclaiming that very publicly. However vocal he is about disclaiming his support of violent measures, it's clearly not vocal *enough* to prevent people from reading him as being in support.
The man talks out of both sides of his mouth. He says he never said "bombing datacenters," but DID say "destroy a rogue datacenter by airstrike."
He never advocated for the use for force against enemies. But DID advocate for credibly threatening to use force so they comply to your demands and you don't need to actually follow through. Also, that doesn't count as "violence" as long as the violence is predictable. https://www.lesswrong.com/posts/5CfBDiQNg9upfipWk/only-law-can-prevent-extinction
You can see why one who isn't already favorably inclined towards him might not appreciate the subtle distinctions.
To be fair, one could interpret it as a sort of "violence" to use force to enforce treaty mandated controls on AI research, but I think it more or less *requires* already being negatively disposed towards him, or trusting people who're negatively disposed towards him to report what he means, to interpret it as such. By the same token, we could call it "advocating violence" for a person to speak in support of nuclear nonproliferation treaties. After all, actually enforcing those treaties requires the threat of violence.
I think it's more than a little disingenuous to suggest that having a legal treaty to disallow something, which is theoretically enforceable via violence, equates to "advocating violence." By the same token, advocating for essentially any law at all equates to advocating violence, because all laws are ultimately backed by government force.
Eh, I think Eliezer likes his little fictional fantasy scenarios where he is the mysterious sage figure of giant authority and wisdom, and sometimes he talks them out loud, but he doesn't really mean "call in the airstrikes", it's just more of the fictional interludes in The Sequences about the Sekrit Gnostic Training Of Leaders bit where the Ancient Sage *could* airstrike, via mysterious sekrit Gnostic techniques that led to him having political authority amongst the ostensible leadership to bring this about. Ancient Sage is not the king, but rather Richelieu or Talleyrand who manipulate kings and emperors.
Actually international treaties are an interesting case. According to Opus 4.7 “the NPT itself contains no built-in penalties. There’s no clause that says “if you violate this, X happens.” Enforcement is indirect, political, and widely considered the treaty’s weakest feature.” And DPRK violated it and suffered no violence.
This is a parallel to how we enforce the treaties and international norms against development of WMDs. We destroy a rogue nation's nuclear research facility by airstrike. One presumes we would also destroy a datacenter in the same way.
The problem with "bombing datacenters" vs "destroy via airstrike" is that the former can be said out of context to suggest the destruction is being done by rogue actors whereas "destroy via airstrike" very clearly implies it's the actions of a state acting in some capacity that is recognizable within the system of international law and norms. "Yud says we should bomb datacenters" could be (and has been) said to imply that he thinks normies like me who oppose AI should be bombing them, when he explicitly does not think that would be effective or correct.
If you object even to the use of airstrikes against a rogue nation to enforce the conditions of a treaty by destroying the site where they're researching dangerous technology, then your real problem is that AI is being analogized to nuclear/WMD control in the first place, not with the words.
Given his beliefs, it's perfectly reasonable for him to advocate for violence, sure. There are plenty of people I'd like great violence perpetrated upon as well, and out of purely personal animus without even the pretense that it's to prevent the end of the world/human civilization or whatever.
My objection is only to the duplicity in claiming he DIDN'T call for violence, when he very clearly did, just with some aesthetic preference about what it looks like.
I was a prosecutor for 15 years, I would not have described it as "calling for violence" on the several thousand occasions I requested an arrest warrant. But all state action is either violence or the threat of violence, certainly *some* of the people whose arrest I demanded did not just quietly submit. Yud's point is that it's "predictable, avoidable violence" in both the case of my arrest warrants and the enforcement of non-proliferation regimes, the standards are clearly articulated and the use of force visibly telegraphed in advance, actually having to DO the violence is the undesired but completely avoidable consequence.
You and others keep doing this thing of asserting that anyone who believes AI will destroy the species ought to be supporting "violence" writ large, because you know the public doesn't like that and it would discredit the anti-AI movement. Yud and others on our side keep repeatedly pointing out that even under a strictly consequentialist analysis, acts of violence are not going to result in avoiding the thing without state power (or something equivalent thereto) behind it, and that they don't support that, but you're very insistent on abbreviating this position to the overly broad "justifies violence" as a rhetorical move to frame opposition to AI in a way you think is easier for people to dismiss.
Though I grit my teeth to say it, advocating violence by a nation, in its war-making capacity, is distinct from advocating violence by non-national actors. And "destroy a rogue datacenter by airstrike." sounds much more like the former than like the latter.
( As someone who wants to _see_ AGI, I don't support this use of war making capacity, or, for that matter, any flavor of 'pause', though I think the testing transparency proposal like NY's RAISE act are reasonable. )
Zizians seem like a good example of how you can't keep sufficiently determined crazies away from your movement.
The connection between Zizians and the rationalist community was that (1) Ziz attended a CFAR workshop, or maybe multiple workshops, and (2) Ziz recruited among the members of the rationalist community.
How could this be prevented? In theory, CFAR could do a psychological screening of every potential workshop participant -- but that seems excessive, and as far as I know other similar organizations don't do that. Afterwards, you can't prevent a crazy person from living in the Bay Area, or contacting people in the community.
In 2019, Zizians protested *against* CFAR, and had police called on them.
And yet, when in 2022, Zizians started to murder people, Wikipedia in its infinite wisdom decided to call them "rationalists".
In other words, once a crazy person participates in your workshop, even the fact that they publicly declared that they hate you does not prevent Wikipedia to keep describing them as one of you if they do something bad a few years later.
(As an analogy, imagine if the Wikipedia page on Donald Trump started by calling him a Democrat. That would actually make more sense, given his 2001–2009 membership. Ziz only participated at a CFAR workshop, which apparently makes them a rationalist forever.)
I see you have never, ever gone to a political protest of any kind -- if you do, you will surely find a few completely out-to-lunch loonies who were never invited, but somehow show up.
It's just not that overridingly hard to deal with.
It's a necessary cost of activism but as Nathan reports it is not a particularly expensive one in time or money. You will take reputational damage sometimes. You invest some in background culture and individual responses to reduce that damage.
Context: I volunteer full-time for Pause AI but rather randomly am also an old friend of someone who went on to found Rising Up and Extinction Rebellion.
I have attended a few protests of various sizes (including a very small one I organized myself), and didn't directly run into any problematic loonies. I'm sure they show up often enough to need a plan for them, but obviously not even every time, and again, this is a manageable problem.
If you think protesting is bad or something, you should just say that, instead of inventing reasons to not want to do it.
I don't think protesting is bad, and I'm not inventing reasons not to do it. I'm just pointing out that it is in no way "easy" to avoid having loonies show up to your protest once it reaches any reasonable scale. No Kings, Women's March, March for Science, March for our Lives, BLM, etc all 100% have complete crazies (hopefully off to the side) who show up, and you do need to figure out a way to deal with that instead of just saying it's "easy" and not a problem.
Scott is equivocating between the obvious evil of working for a company that is trying to build something that is likely to kill everyone, and forming broad coalitions that include distasteful people in the shared interest of protecting humanity. He needlessly tacked on the bit about how hard it is to keep the crazies out as a way of communicating his distaste for activism.
Clown management is a real thing you have to attend to. That doesn't make it a giant unmanageable issue that should dissuade people from being involved. Buying a car means spending some time on car maintenance. It doesn't mean spending an "inordinate" amount of time on car maintenance. That kind of language (along with calling pithy and useful slogans "vapid") is meant to dissuade, not to clearly communicate what you're in for.
Most people who say these things (most definitely Scott included) simply believe that activism is a low-status activity, and they cower in fear to think of the personal social consequences of associating with ordinary people and empowering them to improve the state of the world.
That is what I am arguing against here. Snobbery and cowardice, and the undue magnification of real but manageable problems as a way of hiding that cowardice.
There's an entire social media formula where you go to any large political gathering, find the loonies, and then get them on camera doing or saying loonie things for everyone to gawk at. It works for a reason. There are lots of weirdos in any sufficiently large gathering that is open to the public - even more so when there's some kind of emotionally charged topic involved. The mass movement is necessary, but the snobbery is warranted. The type of person who likes having intellectual debates about complex topics usually finds it a bit embarrassing to stand in a mob and repeatedly chant "Go Team!". Society needs someone to change diapers at the old folks home, but that doesn't make it enjoyable or glamorous.
"It actually takes very little resources to keep the crazies out, in both absolute and relative terms."
I was just listening to the morning talk show on local radio where members of Sinn Féin were being questioned on, amongst other things, participating in the recent fuel protests here and asked about standing on platforms with guys from new, far-right parties and guys with convictions for animal abuse.
It's tougher than you make out to keep the undesirables away.
The Christian belief in a Holy Ghost based gut instinct is inseparable from vibes. Rationalists are realizing you simply cannot make the right rational calculated choice on a dime, these things take great intellectual work!
It's never clear whether you're improving the overall works of the newly founded socialist workers party or whether you neeeever should have gotten involved.
Shouldn't have to say same for AI safety lab association, one should see the general problem.
I don't know what plasma had in mind, but one might look at virtue ethics as suggesting to cultivate the good aspects of being by great practice and reflection and intellectual engagement *so that* one's vibes and acts are good.
I still think "virtue ethics" acts as some sort of uh, virtue signal, where it never grounds itself in any particular position, but gets to feel smugly superior to everyone else because they're not practicing "virtue ethics", the vague practice of doing everything right with no effort (or, if you decide effort is good, lots of effort, but it's somehow magically different from and better than everyone else's effort). I have tried to read books on virtue ethics and I still can't figure out how it isn't this.
I suspect this is a matter of sampling bias due to your calm-down-and-be-reasonable aura. Virtue ethics is the decision-theory equivalent of those IRS computers that still use COBOL, or https://thecodelesscode.com/case/234 It's optimized for providing quick answers under conditions of high stress and low working memory, so whenever consequentialism has enough swap space to give an answer at all, it'll give a better answer, and leave virtue-based responses looking half-baked. Feels like a weird edge case to you, but many people live under "no room to think" conditions more or less full-time.
As for the paradoxical amounts of effort involved, consider AI training runs vs. inference. Cultivating the proper virtues might be a laborious process over many years, involving self-examination, digesting standard datasets, and personalized feedback from peers or mentors; then, habits having been fixed, applying them in the moment needs little or no conscious attention.
(1) I've been interested in the implicit model of divine action present in the Homeric epics.
The background mythology includes some very overt acts, like turning people into stone. But most actions attributed to the gods are extremely reasonable. The gods do things like inflicting disease, guiding projectiles to strike or miss a target, and speaking to people in a manner that other nearby people don't hear/notice.
(2) There's a scene from the book Taran Wanderer that's always stayed with me.
Taran is a foundling, and he has gone off into the world to try to learn where he came from. He meets a farmer, Craddoc, who claims to be Taran's father, having given him up long ago for reasons. We, the readers, learn that Craddoc is lying, but Taran joins his household and helps operate the farm.
But he's frustrated. The farm feels like a dead end to him. He wants to be adventuring.
One day Craddoc falls down the side of a ravine and is badly injured. Taran is presented with the choice of attempting to rescue him, or not. It's dangerous.
The book tells us that Taran considers the fact that, if he simply abandons Craddoc to die in the snow, it will solve all of his problems, and no one will have a word to say against that decision. There's no sense in killing yourself trying to rescue someone from certain death. And it says: 𝘏𝘦 𝘴𝘦𝘦𝘮𝘦𝘥 𝘵𝘰 𝘩𝘦𝘢𝘳 𝘩𝘪𝘴 𝘰𝘸𝘯 𝘷𝘰𝘪𝘤𝘦 𝘴𝘱𝘦𝘢𝘬 𝘵𝘩𝘦𝘴𝘦 𝘸𝘰𝘳𝘥𝘴, 𝘢𝘯𝘥 𝘩𝘦 𝘭𝘪𝘴𝘵𝘦𝘯𝘦𝘥 𝘪𝘯 𝘴𝘩𝘢𝘮𝘦 𝘢𝘯𝘥 𝘩𝘰𝘳𝘳𝘰𝘳.
He ultimately decides to make the attempt, because he doesn't want to be the kind of man who makes the other choice.
---
Something that I wonder about in relation to this is that epic belief in gods speaking to you in a way that no one else can hear. I can imagine a culture in which the antibody against Taran making the selfish choice relies less on him acknowledging that, even though that choice has obvious benefits and no obvious drawbacks, he should nevertheless sacrifice himself... and more on Taran recognizing that those unwelcome thoughts in his head aren't 𝘩𝘪𝘴 thoughts at all, but rather the lying words of an evil god attempting to trick him.
Maybe rejecting antisocial ideas is easier if you have a scapegoat handy to blame them on.
I've only seen summaries of Julian Jaynes's "The Origin of Consciousness in the Breakdown of the Bicameral Mind", but what you describe sounds similar to what I recall of those summaries. You might find it interesting, albeit controversial.
I think* "on a dime" is an expression that means something like "trivially"?
(* Disclaimer: I'm not an American and mostly deducing this from context)
If so, Plasmarob didn't say "should be done on vibes" or "can't be calculated", but rather "*is* done on vibes" (by Christians who're following their conscience) and "can't be *trivially* calculated" (by rationalists who are trying to scratch-derive ethical principles). And I think in both of these things Plasmarob was agreeing with Scott's existing position?
I am saying there will be times where you are in a gun-to-the-head position that tests your deontologic bar and you do not have time to make a rational calculation in the way we do in online long form content hypotheticals.
Some hard problems like you describe with AI take great intellectual work but and AI can show up at any time and ask you a question! Some things have to be done on vibes and we can only rationally calculate if our decision was best later.
The proper imo early Christian strategy is by feeling, moral vibe.
three (ok 4) examples - politics, ai, and practical home emergency:
1) let's say you are a partisan invited to visit a world leader you consider doing a great evil that will cause many people to die. While visiting, to your astonishment someone attempts to slip you a weapon and says "do it". What do you do? People will be watching in less than 4 seconds. You may die or be in prison forever. Zero time to react. Real trolley problems don't give you a day to debate it.
2) AI. Once you donate money to a tech company you think is less likely to create evil, there's probably no takebacksies. How long is sufficient to wait to see about that 80% or 80% odds? 2B) let's say you are training an AI with every input you make. How much time should you spend determining the long term consequences of each prompt or each data input? The answer isn't zero but how rational and how crystallized does your model of deontologic bars need to be? (with what you won't ask an AI to do).
3) One common family encounter is medium-level injuries. Every parent will eventually have to determine whether it is worth calling 911 for an ambulance ride for their child. I've seen families bankrupted from how often they do this. If your child hits their head and blood starts coming out, how much blood is too much? I was surprised how much blood is not actually an issue, I have worked through this with a nurse nearby. You don't want to wreck your family's future with unneeded expense insurance doesn't fully cover but this is your child and their life could be at stake! How bad is enough for 911?
All deontologic bars eventually to be broken. Especially not killing, not just in war, but because if someone armed breaks into your child's bedroom window, and this scenario happens in America plenty, there is simply no calling the police to solve my moral problem.
It's a time problem, and every scenario is different. Deontologic bars are rules of thumb. Maybe some are moral principles, but "do the right thing" can vary more than we appreciate, and in the moment all you have to call down is a prayer and your own character and experience.
While oversimplified, the way *you've* described conscience is fair; the mistake is conflating it with acts of the Holy Spirit, which does not necessarily play a role in conscientious discernment.
The immediately following paragraphs in the Catechism focus on the proper formation of conscience. This foregrounded concept of formation is diametrically opposite to "automatic" and "built-in".
It really depends on which theologian you are quoting. Remember "masturbation is the sin against the Holy Ghost", but others has used it as a quite different meaning. This is what happens with an ungrounded metaphor.
The correct approach is to ignore all of this and do the right thing in every case, which absolutely does not include anything stupid like unilateral military disarmament.
I have no opinion on whether constraint utilitarianism can be coherently formulated, but it too is wrong, like (regular) utilitarianism and deontology, which it tries to hybridize.
But the whole point is that it's hard to know what the right thing is. Even if you're a deontologist, different people have different deontologies, so who's right? If your claim is that people should just follow their particular deontology, that can justify basically anything, so we're back at square one.
I agree that that knowing what the right thing is hard. My point is that neither utilitarianism, nor deontology, nor a hybrid of the two like constraint utilitarianism, gives a correct answer to that problem.
You said: "The correct approach is to ignore all of this and do the right thing in every case." My question is, How would one know what the right thing is to follow this?
Pick whichever philosophy seems most right to you, and if ever it stops seeming right, ignore it in that case and do what seems right.
Many horrific things have been done in the name of consistency, when even simple instinct, basic decency, and common sense point in the opposite direction. Morality is not a solvable problem, so don't ignore every other heuristic in favor of just one!
Many horrific things have also happened because people were just doing what seemed right. There was a time when slavery seemed so to most. It's thanks to the people who questioned those instincts that we've moved past that.
Point taken that moral innovation is important. (That said, nah, people pretty much knew slavery was bad at the time. The slaves visibly didn't like it.)
When it comes to your day-to-day life, you are mostly already just doing whatever seems right to you. If you get convinced that something else is even more right, then great! But for people who live their lives by hard-and-fast rules with no deviation, moral innovation is impossible.
>How would one know what the right thing is to follow this?
By using their entire capacity for reason. Of course, that still may not suffice, but a simple shortcut certainly can't suffice in every case. If there was such a shortcut, millennia of moral philosophical effort would have found it.
Reason alone CANNOT define right actions. You need a set of axioms to base that reasoning on. The problem is that peoples "sets of axioms" generally favor their own self interest. So organizations try to replace those sets of axioms with different ones that favor the organization's interest.
What you can a shortcut, others might call taking advantage of insight distilled from those millenia of philosophical effort. It doesn't excuse less thought but it would be suboptimal to try to do everything from scratch.
Yes, I don't endorse the original commenters formulation of "ignore all of this". Those shortcuts have their proper place, and, if nothing else, that millennia of philosophical effort proved inadequate for completely solving ethics should be a strong precaution against embracing claims of massive novel breakthroughs too readily.
Oh great, just "do the right thing". This is like saying "There's no need to learn calculus, just put down the right answer to every problem on the calculus test."
Well ... one version of "just do the right thing" is: "Morality" is just a set of quick & dirty heuristics. The hope is to find a set of simple rules that are easy to understand, that are consistent, that you can apply to all situations, and expect that it pretty much quickly guides you to behavior that you won't later regret. But all actual philosophical attempts at constructing such a code of ethics seem to wind up with inconsistencies, ambiguities, or at least an opposition philosopher can construct a disturbing hypothetical corner case where your moral code doesn't seem appealing after all.
But you can just abandon the attempt to find a simple uniform easily understood set of universal rules. Instead, just accept that each situation is unique, and it's easily possible that human preferences are not 100% consistent.
What you do instead is simple decision theory. At every decision point, look at the actions you have available, estimate the likely consequences of each action and the probability mass across the possible worlds that would result, and choose the action that maximizes your expected utility or general happiness with whatever world(s) would likely result. Stop searching for the heuristic summary of "doing the right thing", but instead just make a complete evaluation of each unique case from (your) first principles.
Maybe there's too much uncertainty to make a confident decision. Maybe you don't know what your preference is between the different sets of possible future worlds. In that case, don't worry about it! But when you clearly WOULD prefer one probability mass over another, then just choose that action in that moment.
And don't worry about whether your full set of decision choices over your lifetime, is easily described by a simple list of logically coherent rules.
Very close! But any time you try to summarize all decisions with a simple sentence, you're vulnerable to strange hypotheticals. Act Utilitarianism generally seeks the "greatest net happiness" -- but what if you refuse to aggregate, and think that the happiness of two distinct individuals is incommensurable? Or perhaps you feel that you and your family are more important (for your own actions) than the welfare of strangers? Or perhaps you're after "pleasure for the greatest number of people" ... but then you fall into the Repugnant Conclusion?
So I was actually attempting to be serious about: Don't try to summarize the morality in a small paragraph. Don't even seek global consistency. Just evaluate each scenario in isolation, and do your own decision theory calculation for that specific scenario.
Going by the virtue-ethics system from the Exalted RPG (which is, to be clear, definitely not what I use in real life, but happens to be conveniently well-defined for illustrative purposes)...
- unilateral military disarmament goes against Valor and possibly Conviction
- ...but negotiated *mutual* disarmament, or unilateral de-escalation when not in immediate existential danger, is supported by Temperance
- deliberate political use of misinformation clearly goes against Temperance
- breaking an oath goes against Temperance, and abandoning an ally in need goes against Conviction, but participating in (or up-close passively observing / tolerating) atrocities goes against Compassion, so getting a job as a concentration-camp guard, or doomsday-bomb factory worker, or otherwise entering into a situation where you'll inevitably need to either be directly complicit in horrible things or quit on short notice, is setting yourself up to fail, so don't even start on that road (harm theoretically prevented relative to someone else doing it instead isn't even considered as a potential factor)
- assassinating the leader of your own country is treason, thus solidly against both Conviction and Temperance
- ...but if Compassion and/or Valor say his cruelty and incompetence are intolerable, removal should be pursued through the proper legal channels
- assassinating the leader of a country you're currently at war with is likely supported by Conviction, plus Valor if the plan involves daring heroic risks, plus Compassion if the enemy in question is notably complicit in atrocities
Definitely not perfect, but it seems to have the basics covered.
You can give three people who have almost nothing else in common the same calculus problem and, if two of them come up with identical answers while the third disagrees, be very confident that the third has made some mistake which they'll recant as soon as it's pointed out. Is there an ethical framework anywhere near as clear and consistent?
> You can give three people who have almost nothing else in common the same calculus problem and, if two of them come up with identical answers while the third disagrees, be very confident that the third has made some mistake which they'll recant as soon as it's pointed out.
This isn't true at all. It's not difficult to set up a problem that plays into a common mistake. At that point, whether you get more agreement on the correct answer or "the" incorrect answer is fully dependent on which three people you select. You can always choose people who will make the mistake. You'll know in advance that they will.
I’ve noticed that utilitarians (and to some extent other consequentialists) seem to think that ethics is a solvable problem which could be discovered as calculus could be discovered. And therefore an isolated demand for rigour appears: find a correct way of deciding ethical decisions before taking them or justifying them. But there’s no reason ISTM to suppose that ethics is solvable in that way, and that anybody has ‘learned ethics’ in the sense that people have learned calculus. In the absence of the existence of calculus ‘just put down the right answer, having learned from the relative success or failure of those who’ve answered it before’ is a good approach.
I'm happy to work with Bannon, Sanders, NIMBYs and anyone else if it helps prevent human extinction -- because that's a more important issue than any other possible issue: whatever issues someone cares about, they are all null and void if there are no humans and therefore no human caringness in the universe.
It will certainly be necessary to work with the Chinese Communist Party, and probably further down the line Vladimir Putin, Kim Jong Un and the IRGC. And that too is clearly worth it. In a hypothetical where it was genuinely necessary to work with Pol Pot to avoid extinction, I'd do that (though in practice a regime change war would probably be preferable there).
It's funny because - this is something few people know - America did work with Pol Pot to counter the Vietnamese. Before his death he said "when I die, my only wish is that Cambodia remain Cambodia and belong to the West. It is over for communism, and I want to stress that."
Everything has to be decided on a case-by-case basis, erring towards wisdom and caution. For example, Yudkowsky has said he thinks that in extremis, he is OK with a pro-AI-safety coalition bombing the data centers of countries that refuse to comply, and I agree.
Things that humans value are only valued if there are humans. Without humans, quite possibly nothing in the universe has any value at all. This is worth minus infinity utilons, therefore (almost) anything else is better.
But, most people are against murder, so assassinating people looks bad, so it would weaken the cause and thus be a bad tactic.
Humans have an intuitive sense of physics, that is useful to them as a starting point in intuiting physics problems, even if it does not go all the way to solving them. Similarly, morality is intuitive game theory, and is useful as a starting point in intuiting co-ordination problems between (semi-)rational actors, even if on its own it cannot solve them.
>Everything has to be decided on a case-by-case basis, erring towards wisdom and caution. For example, Yudkowsky has said he thinks that in extremis, he is OK with a pro-AI-safety coalition bombing the data centers of countries that refuse to comply, and I agree.
You think bombing Chinese datacenters, and very likely starting WW3 in return, is "erring towards wisdom and caution"?
FWIW, I, personally, disagree. I value the intellectual triumphs of humanity, and I think these likely to survive ASIs taking over, while I think there are pretty high odds of losing them forever in a WW3. Naturally, I would _prefer_ something like <evidenceFromFiction> Culture Minds keeping humans as pets </evidenceFromFiction> over both these scenarios.
WWIII would almost certainly kill under half of humans, and Wikipedia would certainly survive on hard drives in EMP shields. There's nothing permanent or even particularly long-lasting about its effects on humanity.
I cannot speak for Yudkowsky, but I can speak for me. IMO, unaligned ASI would likely kill everyone and is therefore far worse than nuclear war.
There are ways you could limit the amount of destruction against the holdout nation (e.g. China). You could make sure that in your first strike you only use a small part of your missiles, that you only strike data centers and try to limit the damage to cities etc. You can make clear that if they make a retaliatory strike, their entire country will be wiped out, but that if they don’t, they will be given lots of help to rebuild their damaged infrastructure (sans AI data centers, of course).
In a way it’s a shame we’re facing this problem now and not in say the 1990s when liberal democracy was undoubtedly on top and where it would have been easier for the important countries to agree. However, we are where we are, and saying “I wouldn’t start from here” is not a plan.
So it sounds like the answer to my question is "yes": you are choosing to perpetrate global thermonuclear war, but you think that you can do so in a way that mutually assured destruction will not apply.
I understand that your plan makes sense in your worldview; but can you envision what someone who is not already 100% on board with your cause would think of this plan ?
In point of fact, I think it's probably possible to avoid both, but sure, the metaphorical end of the world sure beats the literal.
>but can you envision what someone who is not already 100% on board with your cause would think of this plan?
This doesn't really matter for someone who subscribes to the Yudkowskian view of neural nets. You cannot bargain with reality; you do what is necessary, or you die. The willingness to go to war to enforce the ban is a necessary component of making it work, and the ban working is the only way to avoid us all dying. We clearly have some work to do to create that willingness, but there don't seem to be any better options available.
> "Everything has to be decided on a case-by-case basis, erring towards wisdom and caution."
I'm saying this is the kind of act utilitarianism I don't trust, or at least it degenerates into this without some kind of very strong operationalization of "wisdom and caution", which is what I'm trying to discuss here.
I believe Yud's recent remark on this topic was, to paraphrase, something like "consequentialists are right, but the vast majority of people aren't smart enough to BE consequentialists and get lots of stuff wrong so they should just be deontologists instead". Not consequentialists with a deontological bar, but instead just plain deontologists out of humility and caution I suppose. Some guy at the RAND corporation can do the consequentialist stuff about how many megadeaths from this or that nuclear scenario, the kind of people who decide whether militaries exist can do that, but Joe Everyman shouldn't try to assassinate a candidate because it would be bad if that were common. Ethics doesn't really need to universalize.
For my part I just try to act to be the person I want to be, various ethical rules and modes of analysis are just there to be quick heuristics and guardrails but any of them are breakable in extremis. (Although some rules become part of one's identity over time.) Facing a mortal threat or a complete destruction of my dignity and self-image, I might break nearly any of them but almost certainly along a vector I already considered an acceptable part of my self.
Saying "everything has to be decided on a case-by-case basis" is obviously correct, but gives no hint to the basis of decision. It's also true of every math problem...but the basis of the decision there comes from outside context. (E.g. is it multiplication of integers or matricies?)
So "everything has to be decided on a case-by-case basis" is quite insufficient.
No one needs to have a consistent moral philosophy. You can just have preferences and strategies and do things. Yes, work with Bannon and Sanders. No, don't permit assassinations. Ta-da! Solved. If you have more confusions, you can ask more questions, until we collectively have an intuition about what is acceptable and what is unacceptable. But most of it is covered by "don't be an idiot" and "don't be terrible."
The moral philosopher Jonathan Dancy (father of Hugh, the actor) is known for his view of "moral particularism" under which there is no general rule. He uses the analogy of humor, saying there's no general rule for what's funny. https://www.youtube.com/watch?v=qHH9WiHNhv4
Well, as the other guy said, that sounds like a confusion that you can ask more questions about, until we collectively have an intuition about which side is the good guys and which side is the bad guys.
Ah, but how do we decide which questions are legitimate, and which are not legitimate? The problem is that people have an endless amount of preferences.
I feel most violence is blocked by simple consequentialism, it is rarely an effective strategy, the violent actor is rarely going to have a monopoly on violence so increasing violence even if well directed makes things worse.
I think your comment is a good illustration of why one should clamp one's expected value of projected outcomes to somewhere around continental scale, at most (and ideally much lower than that). Otherwise, you end up saying things like "because I am laboring the entire humanity/Solar system/Universe, no cause can be weightier than mine, and thus my ends justify any means". Mathematically, this is a perfectly valid conclusion. Empirically, it never, ever brings about net-positive results.
Yes, of course. And that's what everyone else says too. Everyone's got some kind of a pet cause that is the biggest problem humans have ever faced, be it AI or sin or even just Trump. And it's easy to say, "but of course all those other problems are fake over overblown, mine is the only real one". Everyone says that too. That's why you need that expected-value-clamping heurisitc.
Self evidently, human extinction is a worse problem than sin or trump: sin has been around for as long as humans have, and so have bad leaders, and neither has caused human extinction. Now, with AI, humans have the power to cause our extinction, which we didn’t have before.
> Self evidently, human extinction is a worse problem than sin or trump...
Whoa, "self evidently" to whom ? If I were a Judeo-Christian theist who believed in Hell, I'd say that sin is a way worse problem. When AI kills you, if you are sinless then you'll go to Heaven and experience eternal bliss. Humanity cannot in fact die out, because this Earthly life is just a prelude to the eternity. But if you are sinful, then you will be eternally tortured in Hell forever ! This is much worse than merely shortening your already infinitesimal stay here on Earth by a few years; it's even worse than ceasing to exist; it's a literally infinite negative utility ! Surely, you must stop wasting your time on AI, and repent !
That’s not how Christian theology works. The way it works is all are sinners but if you sincerely ask God for forgiveness, you’ll be forgiven. Being without sin is simply impossible for humans.
Ramez Naam doesn't think so, and he successfully predicted the rapidity of solar power's exponential takeoff further in advance than anyone I'm aware of, certainly far better than any of the official government-backed projections.
You jump back and forth between decisions made as individuals and decisions made as groups (political parties, Ukraine) here a few times, and there are cases where this distinction importantly matters.
For the military disbanding, there's two more arguments why it's bad. One is that "no military at all" is an unstable equilibrium - the first country to get one will conquer the world - but we have a better alternative where countries can arm up as needed (but no further) which is both more stable and converges to peace.
The other is that when you universalize a principle, you universalize it more for people similar to you. People in your culture are likely to reach the same conclusions as you (either because you convince them or because you think similarly), so you can successfully hit slightly more unstable equilibria with them. Trying to sync with people in a different country or culture is harder, and the more difference it is the less ambitious you can be. Coordinating with your clone is trivial, coordinating with your countrymen is doable, coordinating with another country is harder, coordinating with unknown aliens is basically impossible (hence AI alignment being hard!). And "no army" is the third of these.
> coordinating with unknown aliens is basically impossible
Interesting enough, Heinlein's Starship Troopers has a classroom discussion of how that could be possible as the next step after humans are all cooperating, despite the impression people get from the Verhoeven film that it's all xenophobic militarism.
Is there a difference between constraint consequentialism and rule utilitarianism? If not, why not use the standard term, which is rule utilitarianism? And why not look at the literature of rule utilitarianism and see what it has to say about what kinds of rules are good rules?
Rule utilitarianism means you should always follow rules and stop worrying about what has good consequences (outside the rules). Constraint consequentialism says you should try to do things that have good consequences, unless there happens to be a rule in the way.
Thanks, Scott. I don't think rule utilitarianism says to stop worrying about what has good consequences. That would be deontology. Rule utilitarianism tries to pick rules that will lead to maximum utility. Rule utilitarianism would in theory not worry about violating rights if the rule maximizes utility. Constraint consequentialism seems like consequentialism with a deontology wrapping. I would argue that that's just consequentialism in practice.
Deontologists say certain things are intrinsically good regardless of their effects on overall human happiness because they make God happy. Rule utilitarians say you should stop worrying about whether the rules are actually good because that makes the rules less effective towards the end of overall human happiness.
Deontology (at least the categorical imperative flavor) is functionally equivalent to rule utilitarianism; no necessity to multiply entities by introducing a deity.
Rule utilitarianism, can and does have differences from pure utilitarianism. A pure utilitarian would push the fat man to save just two people. A rule utilitarian might refuse to , to save any number of people , or require some higher threshold to break the rule. (Rule universalism probably implies non absolute deontology).
One thing that kind of confuses me is, why add non-consequentialist rules if not for assuming it somehow results in better consequences? And wouldn't that make the rules themselves consequentialist? Is this distinction a matter of "mostly easy to track consequences" vs "here's a rule and we assume it is good but there's no way we can actually know"?
OK, nevermind, I looked it up and there is a difference. Sorry, I should have done that before commenting. Rule utilitarianism seeks to select rules that would maximize utility. Constraint consequentialism is act consequentialism with non-consequentialist rules applied on top.
I would throw out the disarmament example. First, I don't accept that all wars of conquest are immoral. Second, norms have to be enforced eg by armies, I know it's a thought experiment but it's hard to even spell out the magic assumptions required for it to work.
I am just starting to read Kant, so this comment is attempting to start a conversation moreso than attempting to give a complete answer (via Cunningham's law).
> A common formulation is “act as if your maxim would become a general law”. This correctly rederives the bar against assassination, but fails even in some simple cases. For example, should the Ukraine abolish its military?
This is Kant's 'Formula of Universal Law'. I think it's meant to serve as a filter. That is, it can only rule out maxims, but not confirm that they are actually correct. It is just one of many filters that Kant has. I'm not sure which of these other filters might stop the Ukraine problem, though.
Assuming that your rule about acting as if your reasoning would become a general law is based on the categorical imperative, you are misunderstanding it for Ukraine. The general rule would be something like if I am surrounded by hostile, neighbours or neighbours, which could become hostile in the future or otherwise, likely to need a military to promote good consequences. I should maintain a military. Sure if this was a general law, you would never find yourself in that situation where you have a hostel neighbour, which could become potentially hostile, assuming you’re not the kind of person to become a potentially hostile neighbour yourself, but this doesn’t mean you have to act like that when actually surrounded by potentially hostile neighbours, you can have general rules about things that are universally followed, even if in the world where everyone follows them, that situation never comes up.
Honestly, your second proposal might not even be a real deontological bar. Couldn’t you just think of it as the cost of damaging social rules being great enough to more than cancel out any benefits from assassinations? The outside view definitely doesn’t sound like a deontological bar to me. That’s just basic epistemic determination of probabilities before making a decision. It’s not a deontological bar just because you’re not consciously thinking of maximising stuff. The fact I don’t jump off. A cliff doesn’t become a deontological bar just because I’m following a rule because this is very much a policy which maximise beneficial consequences in the situations where I follow it. Of course, this kind of rule really doesn’t handle edge cases, but that’s why you need a general moral theory. Just because you want to maximise good consequences doesn’t mean you think about that all the time that would waste so much time and effort that you would definitely not be actually the best policy for maximising good consequences. You only think about that regarding situations where the answer isn’t obvious.
Your final suggested rule makes intuitive sense, but of course the problem is that the exact edge cases are unclear since it’s not clear when exactly the other person counts as defecting against you. Does a 90% chance of defecting count,? What about 50% what counts is defecting against you, personally is lying to you through their own articles defecting against you? I also think this rule neglect many of the reasons why you should not be promoting propaganda on your blog. As an example, if you are in Germany under Hitler and your good friend, a member of the SS who is super gullible towards you comes knocking. It still seems like you can lie to him, even if he actually is a very good friend, who is trying to cooperate with you, and the German government isn’t actually defecting against you personally. I think the obvious reasons why the general rule against propaganda doesn’t apply has less to do with defection and more to do with things like lying damages, social trust, and lying to good people harm their ability to achieve their goal which are presumably good and the fact that actual people are not super gullible most of the time, and if you started promoting propaganda, people would read you less and trust you less. This is why it would be fine to lie to your friend because that’s propaganda against somebody who buy stipulation in the thought. Experiment is both evil and gullible and benefit is unusual huge compared to the Small social cost of damaging trust to the single lie instead of a consistent pattern of propaganda that will repeat in the future and will go out to multiple people instead of just one.
Now to be fair, my own approach has certain problems, for example, appears to imply that difficult edge cases are exactly when I should be thinking long and hard about the consequences and yet when I’m condemning people trying to murder Trump, it doesn’t appear to me that I am actually thinking, long and hard, but I think those kind of cases are easy explained because my internal intuition is that it’s just straightforwardly obvious that that will have bad consequences so I don’t think about it hard for the same reason. I don’t think about whether an Apple I throw will reach escape velocity. It just obviously will not.
" The general rule would be something like if I am surrounded by hostile, neighbours or neighbours, which could become hostile in the future or otherwise, likely to need a military to promote good consequences. I should maintain a military. "
I'm nervous about this - why shouldn't the rule about assassination be "don't assassinate someone unless you're in a situation where it would go well and solve all your problems"? At this level of granularity you might as well just go back to consequentialism.
I mean, I actually agree that that’s a perfectly viable rule. Although I think that’s a feature, not a bug. In the real world, the actual reason why you should not go around assassinating people definitely has a lot to do with the fact that it generally does not in fact work out and solve all your problems, and cases where it would in fact, reliably improve the situation like for example, assassinating Stalin actually seem like situations where we want to use assassinations. Of course, that doesn’t actually work like standard deontological rules where you can avoid just thinking about the consequences in trade-off, but firstly again I think that’s a feature not a bug and secondly, I think honestly the categorical imperative while many other people studying it don’t appear to realise this for some reason is basically just functional decision theory using natural language instead of mathematics. Still, it does take care of things like whether to cooperate or not. Since most people are in fact, not cooperate bought but still willing to cooperate with you for mutual benefit while having sophisticated strategies to punish you if you start defecting.
You can of course argue that this doesn’t actually solve the problem you’re looking at, but honestly a strategy of non-specific rules that would be good to be universally adopted. Sounds like a pretty terrible way to come up with the deontological bars as you appear to have noticed in the original post, so I don’t really think that’s much of a loss. The best thing you can say about it is that it encourages cooperation and is self endorsing but otherwise it’s just not a great system and the lack of specificity is a problem and if you get rid of that most moral philosophy will at least arguably pass since they’re meant to be universal. If you adopt version of the categorical imperative, that is basically just functional decision theory which you should do anyways, then you don’t get this problem of basically every possible ethical position being okay, but you should do that anyways because function decision theory is just obviously the kind of theory beneficial to you to adopt.
To be clear, I’m not actually suggesting you assassinate anyone when you really confident it will work out because of things like outside view and the damage to social rules, et cetera. But it’s not actually possible to construct ethical rules with no reference to empirical reality. Otherwise, the rule will turn out to be either really stupid or turn out so detailed that it’s obvious that you are in practice consulting empirical reality before formulating them in which case you should just try to find out the general principles that you are combining with empirical data to formulate the rules.
No, we know for a fact that after he died, his successor were all less crazy, although that partially because they all went through his rain of terror, and therefore understandably, not willing to repeat that that’s why I gave that as an example of a situation where we know it would definitely work out at least in a particular time period. I find it possible it would not work out in the 1920s and even then I suspect given what a disaster he was, it would be beneficial in expectation. To be clear, this is an extreme example, and most political figures probably should not be assassinated, especially if they’re not dictators.
Is that rule bad bc it probably wouldn't solve all your problems? If an assassination actually would solve everything and would only be a net positive what's the reason to not do it? the sub reasons for the deontological bar of no assassinations were practical ones, surely in a hypothetical where there are no practial issues it would be fine, no?
Yeah I was wondering how one decides how vague or specific the rule is, too little specificity and the rule can be misapplied, too much and your desired action becomes the only one the rule allows.
I don't think these two situations are exactly symmetrical. Most people who support slightly-less-reckless AI companies do so because they think some AI company will end up winning no matter what, so they're hoping it'll be the one who's creation is slightly less likely to kill everyone. Here, the lack of a functioning norm is a major factor because it's the whole reason why they make the calculation that one of the AI companies will end up winning and so the best way to minimize risk is to support the less reckless ones. This is similar to the reason why one might support the less evil major-party candidate in an election even if there are non-evil third party candidates to choose from or the option to sit out.
On the other hand, most people who support movements like Pause AI do so because they think the first-order benefits of mobilizing a political response against AI development outweigh the second-order risks of breaking whatever norm exists against partisanship. In this case, the lack of a norm against partisanship is not as great of a factor because they'd likely make this calculation even if such a norm did exist. It's similar to the reason why most people would support assassinating Hitler even in the presence of strong norms against political assassinations.
If a constraint is argued for & justified on consequentialist grounds, it's not really a deontological bar, is it? It's just another complication in our consequentialist calculations. So in a sense the ideal case is obvious: calculate and don't forget the unaesthetic fiddly bits about social coordination.
On the other hand, an intriguing area of investigation is what would make good deontological bars, justified only on deontological grounds. A religious person could take up this challenge effectively I think. As a secular person I don't hold my principles sacred and inviolable in the same way, so there's always imaginable consequences that would render breaking a principle a reasonable choice. But it would feel so much cooler for there to be sacred inviolable principles!
I feel like this is a piece in a general, long, slow multi-year series entitled "Scott realizes that consequentialism doesn't work, tries to salvage parts of it, those don't work either, also realizes that there just generally aren't really answers in philosophy."
What? What does that mean? I am a consequentialist and believe both of those things, but I can’t really begin to explain how I “square” them unless you explain what you think the problem is.
The problem with consequentialism is that it's like... only a thing in thought experiments. The trolley problem is a fun thing to think about. But if you actually try to make the trolley problem an actionable moral code, you end up with chest-thumping young men talking about how they reserve the right to kill arbitrary numbers of people, and when you try to get away from that, you get stuff like this post.
Outside of thought experiments, the world is too vastly complex with too many second and third and fourth and nth order complications to actually do any of the computation that consequentialism relies on. You can always redo any consequentialist argument to be either, "Yes, we absolutely should assassinate Donald Trump" or "No, we absolutely should not." It doesn't actually provide a guide to behavior.
That doesn't mean it's not a worthy thing to think about. But trying to BE a consequentialist just doesn't work.
Obviously nothing works perfectly, but you also have to answer questions (ie either do a thing, or refuse to do it because it's immoral). The people who never consider the questions and act according to instinct and prejudice aren't just "more advanced" than the people who consider them while gradually refining their views.
There's some evidence that in fact following custom and prejudice can be less risky and damaging (consequentially) than trying to consciously calculate the best thing to do. Pretty much the entire history of the 20th century (if not earlier) tells you what can happen when intellectual ideologues argue for overthrowing traditions of bourgeois morality, etc. The real question is finding the dividing line between liberal experimentation and rejection of established custom and hierarchy on the one hand versus illiberal and totalitarian departures from tradition. It's easier when your "tradition" is basically liberal, as in the United States.
But "traditions of bourgeois morality" themselves represented a massive challenge to longstanding custom and prejudice. The defenders of aristocratic morality made the same conservative arguments against the rise of the liberal bourgeoisie.
Those nightmare-spawning 20th century intellectual ideologues thought they were just the next step in the prosperity-broadening economic, scientific, and political transformations that had followed from the post-Reformation and post-Enlightenment refusals to be limited by custom and prejudice.
The fact that so many were catastrophically wrong doesn't mean that following custom (even the historically recent batch thrown up by bourgeois morality) is always optimal.
That the U.S. tradition is liberal and somewhat revolutionary compared to the aristocratic and hierarchical value system it replaced was noted in my comment. As I said, the hard part is understanding where the dividing line between liberal experimentation and destruction of essential norms comes in.
That was the essence of the discussion of the French Revolution among the Enlightenment players in the U.S. and England who were all fairly Whiggish in ideological orientation. Some saw it as an inspiring step forward while others warned it was the road to perdition (e.g. Burke in England and the Federalists in the U.S.). The latter had the better of the hindsight derby, at least in the medium term, as the devolution into dangerous mob rule and then Caesarism via Napoleon with the consequent massive wars made the likes of Paine look naive.
At this point, we know a lot more about what norms, customs, and laws preserve social and economic freedom and dynamism than they did back in 1790 or 1917 or 1963 or 1975. But many are determined to ignore these lessons.
This doesn't work like the original conservatives imagined because we found out most traditions are pretty novel in human history. A serious modern conservatism has to reexamine custom through new knowledge.
I didn't suggest that anyone was more advanced. And I do think it's worthwhile to think about philosophy. But I don't really think that thinking about philosophy yields frameworks that let you compute answers to moral question. At best perhaps it sharpens your moral instincts a little? But in an unsatisfying way that still does not lead to answers.
There's too many thousands of years of prior context to any choice that could be made in the universe, and there's no philosophy that is aware or able to consider all of it to say what is actually the right or wrong thing to do.
Morality is fake. Universalist morality was a European Christian invention. Everyone else treats their tribe and the other tribe very differently and makes no apology for it.
People focussed on trying to do good is an overall net negative. The human condition is improved by knowledge, not morality.
I really hope not, because that would be Scott going in entirely the wrong direction and getting more confused about the world.
Nothing in the above generates any significant problem for consequentialism, and “there aren’t really answers in philosophy” is nonsense. Philosophy involves lots of questions about which there is definitionally a truth of the matter, whether or not we know those answers yet.
I think Scott is entirely too intelligent (and too familiar with the basics of analytic philosophy) to think anything of the sort.
I think time dependence is an overlooked factor here.
I might be 10% less cruel than the other guards... Initially. Is that likely to be sustainable? Am I justifiably confident the experience will not have negative emotional impacts that eliminate the initial benefit?
Or to flip the military question: there would be clear benefits to Europe rebuilding major military capacity again in light of current events. That's weighed against "how likely is that to lead to WW3 in Europe?". Because even if one generation might be responsible about it, the next might not.
I don't think you need to invoke a deontological bar to justify not working for Anthropic. Accelerating race dynamics in the current environment is bad, period. (Yes, working for the other big labs is even worse, but these are all bad decisions.)
Wouldn't this imply that if nobody had ever joined Anthropic, AI would be much less advanced right now? But why should that be true? OpenAI is already neck-and-neck with Anthropic, removing Anthropic from the picture probably wouldn't have slowed them down, and there are reasons to think it could have sped them up (eg they have more compute available for their own models)
My only counterpoint here is that I think Claude code has led some of the recent push into AI acceleration, and both Google and OpenAI have spent a lot of effort building cool doohickeys (Sora, Dall-E), that it isn’t clear to me that they would’ve gotten a high powered coding platform online as quickly, and thus enjoyed as much of the resulting hype boom. It’s possible the other things (less competition would outweigh it), but I think CC is a big point towards Anthropic causing acceleration in a different way than Google or OpenAI seemed to be thinking seriously about.
I think it is straightforwardly true that creating Anthropic sped-up AI timelines.
1. Having peer competition is what is causing OpenAI to frantically spend as much as they can as fast as they can while releasing as many models as they can. There is immense commercial pressure to build and maintain market share. Less competition means less financial investment and less compute. It also means more pressure to build and release potentially unsafe models (because "if we don't, the other guys will beat us").
2. The fact that they were able to catch-up to OpenAI so quickly suggests that they have some sort of superior process. It is possible that this is entirely a function of their talent, and so Anthropic not existing means that the same process would have happened at OpenAI or Google, but I suspect that this isn't the whole story.
It's not just about how advanced AI currently is. Anthropic is taking mind-share by pretending to be safer than the other labs, performing safety-washing while still racing to superintelligence and otherwise proceeding in just as dangerous as fashion as everyone else.
They are contributing more danger to the world than OpenAI. OpenAI was wasting its time on stupid things like Sora until Anthropic surged ahead. Here and in general, more actors in the space means more competition, means more effort toward the end goal.
A world without Anthropic in it would be a marginally safer world.
I think this is too 11D chess "bad things are actually good". I would not be happier with a world where Musk, Zuck, or other frank accelerationists are in the lead, and I don't think the straight lines on graphs that predict AI capabilities growth so well would have suddenly asymptoted out if Anthropic hadn't existed.
No, it is straightforward that bad things are actually really bad. Building superintelligence (which will probably kill everyone) is bad. Most people agree that it is bad and do not want it.
Maybe I'm wrong about the actual effect Anthropic has had on the world. We can can banter back and forth about which nth order effects to consider. But the thing that they are doing is obviously extraordinarily evil and must be stopped.
Isn't it self-defeating to accelerate AI development in order to beat the AI accelerationists??? The pressure to win has already forced Anthropic to drop their unilateral safety commitments.
There are, I think, two intuitions about morality which are in conflict:
1) morality is in some sense maximalist. Depending on your style of morality, it could be doing the greatest amount of good, or avoiding the most amount of blameworthiness.
2) there is no excuse for not being able to identify the moral action. This implies that morality is knowable by everyone, otherwise they would have an excuse. (The law follows this inconsistently. On the one hand not knowing the law does not exempt you. On the other, people are required to be notified that their actions would, for example, take them across a property boundary; an absence of due notice is an excuse for trespass).
The latter intuition makes the idea of "moral strategy" seem incoherent, it should not be possible to discover that you have been acting imorally because you were not galaxy-brained enough to spot some exceedingly subtle, but powerful, moral gambit.
But AI risk does seem to require a deep level of analysis to find a good outcome (unless a good outcome is no longer possible and the best we can do is to avoid being blameworthy). Perhaps some moral intuitions are rules of thumb which don't really apply to it.
It would certainly be bold to argue that all moral evaluations are easy (although saying that not many are easy also seems quite a strong claim). However the second intuition does not require that all moral evaluations are easy, only that some tractably-evaluable moral choice is always available. In your example, a person does not have to decide if it is moral to kill that individual in order to have a moral choice open to them; if they can at least evaluate whether it would be moral not to kill them.
That is why it is in conflict with a maximalist notion of morality: the maximalist must be able to identify the maximally moral position, and identifying maxima is often intractable.
To return to the subject of the post: as well as their normative aspect, deontological rules also often exist to set out a tractably-evaluable rule where a maximum is difficult to evaluate.
Maybe it would be useful to imagine that, rather than picking a rule that literally everyone will magically follow, you're picking a rule that MANY people will follow, but there will also be other people who defect against the rule unless the first group can enforce it. So the rule will apply just to the cooperators in a mixed group of cooperators and defectors. This seems like a good way to derive e.g. police and courts.
Looking at your military example more specifically, it seems significant that having a military doesn't necessarily harm anyone, if you don't do anything bad with it. It increases the amount of harm you COULD do, and that creates some undesirable dynamics, but it still feels like a different category than imprisoning someone or conning them out of money, which are intrinsically harmful rather than potentially harmful.
Also, even if you could magically guarantee that there will never be human-on-human fighting, having SOME capacity for violence still seems good for dealing with wild animals and certain kinds of natural disasters (e.g. deflecting an incoming meteor). Also, many tools that are useful for other tasks can double as weapons in a pinch (lumber axes, cars, mining drills, etc.). So I don't think you'd want literally zero "military" power even if you could actually universalize the rule. (Though you'd probably want less than Ukraine has, so maybe this doesn't help in practice.)
You need violence or force even to keep people from non-violently squatting in your living room, or ultimately to get them to pay what they owe you if they refuse, etc.
You can only shun people who respond to shunning. If a group of homeless people decided to squat in some Amish buildings, the strategy probably wouldn't work.
Every time I read something by an ethicist or that tries to rigorously define ethics, I'm struck by how much it's just trying to find a question that gives the answers we want. Strongly suspect that ethical theories are more strategies than rules and that you should pick the strategy that is most likely to make you do the things you've a priori decided are right.
This is a feature, not a bug - it's how philosophy (and reasoning!) work. If you're coming up with a theory of gravity, table stakes is that it should say things fall down at the rate we know they fall down at. Once it perfectly predicts the obvious cases, then we can investigate what it has to say about black holes / the early universe / etc, and use it to solve the non-obvious ones.
That's an assumption on your side. Consider the changes and refinements and quantification of our understanding of nature; while not accurate to the same degree, an prescriptive ethical theory is much more refined than "go with your guts", similar to how "stuff falls" is very different from a good understanding of gravity.
I don't think this is true. If I punched you in the face, I think you would have the right to ask why. If I said "new rule, I'm allowed to punch you in the face", you could justly appeal to some kind of shared norm of nonviolence. If I said "nah, I've decided to treat everything on a case by case basis", then I've retreated into basically random action.
Case by case is not random. We also internalize fuzzy heuristics that compress those cases. We tend to communicate those heuristics as deontological rules, but because there is no such thing as a deontological rule in real life, we have to bear in mind that those fuzzy edges are always with us and act accordingly.
I think that (most) humans ultimately are only capable of caring about themselves and their extended family members. The entire history of moral philosophy is basically an attempt to work around this biological fact, but ultimately the problem is intractable.
Basically agreed. We evolved to deal with groups around Dunbar's number, including many close relatives. I suspect that _actual_ empathy is limited to people we know at least approximately as well as we would have known tribe members in our environment of evolutionary adaptation.
When someone does moral philosophizing, they are doing _something_ , but I don't think what they are doing has much in common with empathy for relatives or close associates. Maybe intellectual one-upsmanship? Maybe purity spirals? It often looks pretty pathological (notably all the woke stuff over the last 20 years - e.g. people talking themselves into supporting terrorists...).
That appeal would imply that norms are the authority of morality, which seems to contradict what you say elsewhere.
Also, norms are generally recognized due to enforcement, social or otherwise. So unless you’ve changed the norm to fit your case determination, you can probably expect to receive some violence yourself. At least, I would expect your case to be determined with that result.
You once wrote a piece, I can't find it right now, where you wrote something to the effect of "At that point, you're just one labcoat removed from being an indigenous farmer yourself", about how scientists try to understand the process of growing and preparing one particular edible plant and have to go into ever greater detail of soil composition and microclimates.
Humans are much the same way. Human values change with place and time, often in mutually exclusive ways, unlike a ball falling from a tower which will behave predictably if you know a few parameters. This ties in nicely with the impossibility of defining "universal human values" for AI alignment, doesn't it?
Does it make sense to evaluate deontological bars on consequentialist terms? I feel like, if you're really operating that way, then they are axiomatic. I think what you're demonstrating is that this is really a purely consequentialist discussion.
Pure or not, it's still consequentialist thinking. It looks to me like straightforward rule consequentialism. But maybe I'm bleaching the term of its meaning, since I don't know of any way of analyzing ethics without getting into the consequences of following one ethical program or another.
The fault lies in myself more than in the tool, as I missed the first O from "Deontological" and instead asked for a made-up word that Firefly seems to have, not-unreasonably, interpreted as having something to do with dentistry.
- If AI takeoff is impossible for humans to control, the only thing that can control it are other AIs, especially if natural chokepoints exist which allow a response, again even if only by other AIs
- Hardware limits are always at their greatest in the present, absent drastic global degredation, so advances in AI software will be safer now than later
- An ideal approach to AI would be with great caution, but that ship has sailed for the time being. When ChatGPT first become an obviously remarkable thing I thought "amazing, I'm sure they will never connect this thing to the internet", and it immediately became a search engine
So, we should advance AI with the idea of having something go dramatically wrong sooner, with the least capable hardware. This doesn't preclude warning that something could go dramatically wrong as if it happens you are then the most credible group to craft a response. 10% safer AI companies are more likely to be allies in this case, so there is a case for them both for producing good AIs if possible and for having powerful allies if the moment for great restriction comes
My thing is that norm-protection is doomed without moral boundaries underneath it. Norms don’t preserve themselves. They need people who still feel that certain things cannot be done, even when everyone else is doing them, and even when defection can be recast as realism.
Take the concentration-camp-guard example. The problem isn’t only that one is worsening consequences or eroding a norm. It’s that one is becoming part of something. That seems like a different category.
Same with misinformation. If a writer knowingly manipulates his readers, the issue isn’t just whether other bloggers are defecting, or whether he is losing enough subscribers to justify doing the same. It’s that he has violated a particular relation of trust.
Yes, I think that’s the strongest case for being on the inside. But the guard is still a guard. He is still helping the institution function, even if he is mitigating some of its effects.
So even if good can be done from within, the “within” part has its own moral meaning. It isn’t just a neutral place from which one produces better or worse consequences.
If something is the right thing to do, it doesn't stop being right just because Steve Bannon, or Bernie Sanders, also supports it.
I suppose what you are really saying is that Bannon's support or Sanders's support is sufficient to make someone doubt his own judgement that it's the right thing to do. If that's the case, I wonder if that person should be doing philosophy at all.
I don’t understand how ‘deontological bars’ in consequentialism can make sense outside of being shorthand for 'policies that create a coordinated Nash equilibrium' (so, basically, the social contract).
If that isn’t self explanatory, the longer explanation is as follows:
<explanation>
Let us imagine that I am a special version of a constraint utilitarian. I think it’s moral for me to assassinate dictators. In a similar way (though immoral) I’d also cause great harm to many for the sake of my child.
If I had the power to act on these impulses, I might apply the sub-explanations in this post to stay my hand: maybe I’m not as brilliant a forecaster as I think I am.
But even if I was convinced I was as brilliant as forecaster as I think I am, I may still recognize that others are not. This is concerning, because those idiots might feel licensed to assassinate the good leaders or harm my [precious] child for the sake of their [irrelevant] one.
So, I conclude that it is moral for me to repress these idiotic people and take away their capacity to assassinate and harm.
When I try and do this, I run into two problems. First, I lack the power to do so unilaterally. Second, people are willing to help me with this, but they seem to want *me* to agree to not assassinate or harm either!
I weigh my options. A world in which I assassinate with impunity but so do they, vs. a world in which I bind myself to bind others. Meanwhile, 7 billion others weigh the same option.
We all come to the same consequentialist conclusion and bind ourselves to mutual policy. We loudly proclaim our ‘deontological bar’: to do no harm. In our heart of hearts, we retain our secret morality: “rules for thee, not for me”. But no good comes from speaking that aloud. All it will do is confuse the mission and give license to our enemies. So we keep it inside and say “no, I don’t think it is moral for anyone to assassinate dictators”, a statement that becomes true and not true at the same time.
</explanation>
I think this is just rule-utilitarianism. But if a constraint utilitarian starts to tell me about ‘deontological bars’ that don’t have social-contract justifications (or if they take umbrage with an Oracle committing a perfect secret assassination), I’m not going to say “I would be better positioned to navigate this debate if I knew what deontological bars were or where they came from.” I would simply lump them in with the rest of the deontologists as having weird and arbitrary justifications/carve-outs in their broadly consequentialist philosophy.
The social contract justification addresses all the examples in the article. Regarding Ukraine: you should contract with your enemies for an enforceable policy of peace (unless you don’t need to because you can crush your enemies and you KNOW you’re right), but if they’re unwilling or it’s impossible, then build up your army.
It turns the maxim “Don’t be the first person to defect from a generally functioning norm” into the equivalent “If people haven’t realized they can defect, don’t point it out by being the first one to defect” (which, in itself, is a Nash equilibrium!).
The concentration camp/“least-irresponsible AI company” example is tricky because it rubs against our intuition that there shouldn’t be a camp/company in the first place. Some might feel that by participating in the camp, you legitimize the building of/participation in future camps. For example, mass participation in atrocity might weaken people’s moral fiber to oppose the next atrocity. The people building the next atrocity might take heart knowing that if they are only 90% bad, they will find supporters for their mission, as opposed to being worried that badness will leave them without any support. The people who don’t mind participating in some amount of bad will feel emboldened to support the 90% bad things of the future because you won’t have a leg to stand on if you try and censure them. If you do not care about the shape of the next bad thing (for example, if you believe that AI companies are the worst bad thing or the final bad thing), and you feel that any norms of “don’t support an AI company” are thoroughly broken, then as a consequentialist, go ahead and work for the least irresponsible one.
If norms are only 50% broken and you’re deciding if that’s enough for you to follow the herd, then you have to apply more complicated versions of the logic above to determine your decision.
Seems like you’re having trouble with two theories of ethics! You should solve it by picking your deontological bars via virtue ethics. That way, you actually don’t end up like a huge jerk and try to be honorable and act with integrity in your deontology to lead to good outcomes. In practice, this decides the pauseAI example by saying don’t start lying to win because lying isn’t virtuous, you can work at an AI lab but might get fired when you speak up about practices you don’t like due to your commitments to acting according to your principles, and you don’t sign up as a prison guard because gassing innocents is wrong.
This essay frames the questions as "Should I do this thing or not?" - questions in isolation, disconnected to what _else_ we might be doing.
It'll be very rare that your _only_ choices are, for example, to become a concentration camp guard or an SS trooper. (It does happen sometimes. Ukrainians conscripted by Russians face horrible choices like this.) It's more common that that _looks like_ the only choice, perhaps because a bad person has applied various manipulation tactics.
So a useful slant on deontological bars might be, "Do I really believe that the world is so bad that my best choice is to ______? Why do I believe that? What's the likelihood that I'm wrong?"
Breaking out of a thinking-box may require an act of faith, which looks a lot like an act of morals. "No, no matter how it looks to me now, I will not become a concentration camp guard. The world simply is not that bad, and I won't do it."
This is not a personal moral position. It's a partly grounded belief about external reality, which is personally preferable to the more pessimistic alternative belief that would constrain you to the undesirable behavior.
"Do I really believe the world is so bad that my best choice is to become a murderer and forever distort my country's politics?" ... No, probably not. And as a further filter, "If I do believe that, am I likely to be insane and should not act?" There's no deontological bar there, but the result is similar.
I'm not Scott to think through the implications of this in exhaustive detail, but it seems worth doing - it seems to solve at least some of the conundrums that Scott's article leaves unsolved.
It would seem that in the short term moral calculation is doomed to failure because we lack access to relevant data and lack the capacity to do the calculations fast enough to be relevant. Alas we are also doomed in the long term because our capacity to calculate increases linearly while relevant variables and relations increase exponentially.
It would seem we either need an omnipresencent being or a being that can bring exponentially increasing computational power to moral calculation if want moral calculation to be anything more than an intuition pump.
A religious consequencelist can do there best and rely on God to make all things work towards good. On the other hand it would seem the best hope for a rigorous secular cosequentalism is to be found in some form of technolgical singularity.
There is an important distinction between working with the companies and pause-AI regulations positions. First giving you much more personal gain, immediate or potential. Since i know personal gain makes people lower their personal deontological bars significantly so i cannot trusty their notation of common sense and have more sympathy towards, for example, nimbys then them
A surprising number of people are advocating moral intuitionism. They are assuming that we already know what the right thing to do is, and I must say, ethics and all normative reasoning becomes mush easier if you can start with the assumption that we intuitively know and fundamentally agree on what is right.
That probably works well for most of the day-to-day moral concerns people have. It in no way helps for questions that affect large scale populations or anything approaching an edge case.
The lesson we should all take from trolley problems is that moral intuitions are nonsense. Most people agree that trolley problems have obvious answers, they just disagree about which answer is obviously correct. All the variations on trolley problems exist to show that our intuitions break down with the slightest pressure. The ways to run through a trolley problem in a philosophy class are (1) get two people who think opposing answers are obvious and let them fumble through an argument and (2) take the most confident person and keep tweaking the problem slightly until their intuition fails.
And moral intuition always fails for any reasonable person because our moral intuitions do not scale to the size of society we have now. It might have worked for a sense of fairness in a village of 150 people. You cannot intuitively know what the third-order effects of instituting a minimum wage are. We have people *in this thread* with opposing moral intuitions about whether assassination is appropriate in some cases, and "do not kill" is about the most fundamental moral rule we have. But our moral intuitions also suggest a host of exceptions, which again vary between people because it comes down to vibes.
It does not save moral intuitionism to add a disclaimer that some people are bad and wrong, so we can ignore their intuitions. At least a third of the country, and probably more of the world as a whole, will disagree with you on important moral questions. You, yes you personally, are in the minority on multiple moral questions. Your moral theory cannot work if it says that many or most people are moral mutants, and it certainly does not work in your favor if most of the world counts you as one of the moral mutants.
This is why it's impossible to have a morality that's entirely based on the subject's private considerations. Intuitionism is basically the height of this questionable approach. You have to leave at least some rule for taking into account objective factors (other people's thoughts & feelings, customs, institutions, etc). No individual can know the best possible normative action through solitary design, or at least, rarely and exceptionally.
To be more accurate, Bennet describes it in terms of a cycle of increasing language skills leading to more gossip and punishment of defectors promoting more altruism to non-kin:
***
If someone lied or freeloaded in a group that tended to gossip, everyone would quickly learn about it: "Did you hear that Billy stole from Jill?" If groups imposed costs on cheaters by punishing them, either by withholding altruism or by directly harming them, then gossip would enable a stable system of reciprocal altruism among a large group of individuals.
Gossip also enables more effective rewarding of altruistic behaviors: "Did you hear that Smita jumped in front of the lion to save Ben?" If these heroic acts are heralded and become ways to climb the social ladder, this further accelerates the selection for altruistic behaviors.
***
(Quick aside: I enjoyed most of "A Brief History of Intelligence", though I thought it got weak toward the end when Bennet was discussing AI. “Amazingly” he writes, Chat Gpt-4 can correctly answer questions that Chat Gpt-3 got wrong... because Chat Gpt-4 was trained on those questions. :facepalm: And it's a just nit, but there are a few places where his grammar checker chose phrasing that grated on me enough to remember.)
Jumping in front of a lion is a much higher bar than my experience of internet communities, which might be described as "being polite while talking past one another". Twice in my life, I have been in a situation where I really felt I was putting myself on the line for someone who was neither family nor a close friend. Both situations took soul searching and mustering courage, and I doubt that I'd be strong enough to do it again. To me, those situations were a very different thing than an acquaintance of an internet acquaintance is in a bad place so I'll toss some money into their GoFundme.
So I think of internet communities as generally being on the level of having an AI girlfriend. The one online community I'm in that's beyond that got that way because we met in person, with different subsets of people holding gatherings and some people flying across the US to attend them.
It happens that another book I'm reading is "The Different Drum: Community Making and Peace". The author, M. Scott Peck, says that creating true community is really hard and thus they are vanishingly rare. This includes places like churches that are supposedly all about community. (As Josh Hickman commented about in the Early Christians post.)
Another diversion: M. Scott Peck's famous book was "The Road Less Travelled" which talks about his experience as a psychiatrist doing psychotherapy. Now ACX Scott has expressed that he prefers to prescribe than to act as a therapist. Similarly, Peck writes: "During my first ten months of training I worked with highly disturbed inpatients who seemed to benefit much more from pills or shock treatments or good nursing care than they did from me, but I learned the traditional magic words and techniques of interaction." I can't find an indication that ACX Scott has read "The Road Less Travelled", but maybe it would totally not be his thing.
In the book, Peck likens raising children to a long-term therapeutic relationship. "So it is in good parenting as well as in good psychotherapy." That got me wondering if ACX Scott is doing psychotherapy with his kids.
The big breakthrough of Christianity was promising people everlasting life and a very nice one if you followed the rules. Not just the ruling class but everyone.
Yeah, that changes the rules of the game. Kind of like a carrot version of altering Prisoner's Dilemma so that there is an X% chance the mob boss will scrag a criminal who turns stool pigeon.
In short: it's unlikely that either of your examples would be ruled out by a deontological bar unless they're already impermissible on consequentialist grounds - but I'm unsure whether contemporary deontology is actually what you had in mind.
It's notable that the decision-theory-adjacent conception of deontology that you suggest, which is something like "deontology as a hypothetical coordination mechanism", isn't necessarily deontological. E.g. that reasoning is a major motivation for rule utilitarianism against act utilitarianism.
Contemporary deontology tends to be pretty heterogeneous (but then again, so does every philosophical tendency). Its proponents often use the term "non-consequentialism" to describe the tendency as it's easier to see what they're united against than anything they're united for.
So what *is* contemporary non-consequentialism if not a hypothetical coordination mechanism? Non-consequentialism tends to formulate particular moral devices which stick closely to our intuitions about particular cases - it's much more intuition-heavy and "bottom-up" than most consequentialisms. Examples include:
* A means principle - a benefit's ability to justify causing a harm is significantly reduced if the harm isn't an unlucky by-product of the benefit but part of the causal mechanism which achieves the benefit. This is why perhaps even a majority of contemporary deontologists would flip the lever in the original trolley problem, but very few would endorse killing and harvesting 1 person's organs to save 5.
* Adjustment for responsibility - a person's entitlements, e.g. to not suffer punishment, can be changed by their history of meritorious or blameworthy actions, even without any changes in expected future consequences.
* Excludable claims - we might want to exclude interests below a certain weight from consideration when much weightier interests are at stake - e.g. if a life is at stake, it doesn't matter that half the universe will get a paper cut; that simply shouldn't factor into our considerations.
Non-consequentialists do offer theoretical explanations which attempt to integrate particular devices into a wider philosophy, improving their theory's parsimony. Some pretty common theoretical moves are to suggest that the absence of that device cannot be be justified to a relevant constituency - the modern heir to the old social contract tradition - or to argue that it is needed to respect a special status held by moral patients.
Why do I think that non-consequentialists wouldn't have much to add for the two cases you consider? 4 reasons:
1. Non-consequentialists care a lot about causing and preventing harm. Most of the complaints about each side, even on non-consequentialism grounds, is whetjer each side prevents more harm than they threaten to cause - and that's the bigger, pre-existing controversy.
2. Most non-consequentialists think that side-constraints can be overridden when the stakes are high enough - and both sides believe that the stakes are *immense*.
3. In the case of people working on safety for AI companies, complicity is a real concern - but the attention of most non-consequentialists would be drawn to the difference between "is it permissible to intentionally harm someone if I deliberately harm them less than they would otherwise be harmed?", as in the case of the would-be concentration camp guard, and "is it permissible to attempt to prevent harm, at the expense of increasing the danger that that harm occurs earlier and by a different mechanism, while working to enrich a grossly unjust company?" The latter situation is troubling but easier to justify than the former.
4. That kind of mass politics is inevitable in a democracy, and non-consequentialists tend to be fond of democracies. Perhaps there's a stronger case if there's a major risk that the mass politics will overreach and cause significant injustice to non-culpable third parties? Perhaps a concern could be that it must involve deceiving people by deliberately misleading them about the reasons for you position? These both seem very hypothetical.
Nope. Consequentialism is a theory of what makes actions right or wrong - I.e. what we should or shouldn't choose. Right and wrong in that sense =/= what the laws should permit or prohibit. E.g. imagine what consequences would come of, say, 10,000 consequentialists suddenly supporting changing the laws to make maximising utility the only legal option.
I believe the correct approach is deontological consequentialism, wherein there exist a set of duties, and we must maximize our adherence to them.
We have a duty treat our fellow man with dignity. We also have a duty to minimize suffering. Becoming a concentration camp guard would clearly violate the first duty, and only maybe vaguely adhere to the second duty. In our pursuit of duty maximization, we would therefore not become a concentration camp guard.
Assuming we can unilaterally disband Ukraine’s military, we would need to weigh the duty not to kill people, the duty to minimize suffering, a duty to protect the independence/freedom/sovereignty of others, and a duty not to deprive others of the ability to fulfill their own duties. A wealth of historical knowledge makes us believe we probably should not disband their military, as likely doing so would end up creating more suffering in aggregate, a greater loss of freedom, and possibly more death. However…. we don’t know for sure. What we are sure of is it would deprive people of their ability to perform the duty of defending their nation, and thus it is almost certainly wrong to disband Ukraine’s military.
If you accept this lens, the next question is trying to understand what duties are impotent. The good news is most people already have pretty good instincts around this.
I am not familiar enough with the situation to be able to give a good analysis of how to resolve/prioritize the conflicting duties.
My brief google search indicated most of the conflict was around the dogs mauling the children, and it seemed those incidents were what you were referencing.
The framework I laid out in brief is for understanding human morality. I am not claiming I can convince anyone to do anything.
They are all unconvincing because they are all functionally nonsense. There is no such thing as a perfect moral framework in a world that depends on physical needs and realities. I think this whole inquiry boils down to economics on its most rudimentary level. We rage against it because we don't like it but also because it's immutable. If we want to eat, we have to kill something else. People who painted in caves in France 40,000 years ago understood this and it disturbed them.
Regarding a way to harmonize "act as if your maxim became general law" with "Ukraine can have an army", IMO the right way to think about it is whether following your maxim brings you marginally closer to the world that you're supposed to get with the general law.
For the Ukraine example, them disbanding their army won't bring us closer to world peace, it will just empower people who keep their army, perhaps even making them *more* likely to do so. But I do kinda think that not-spreading-misinformation brings us closer to a no-misinformation world.
IMO the relevant game here isn't the prisoner's dilemma but the stag hunt (in particular with n > 2 people). And the right move in that game depends on what you expect other people to do, i.e., on what norms exist.
Also - I don't think this maxim is needed to explain the assassination thing. The way I think about it is, in considering the consequences of your actions, prioritize immediate, obvious things and have some humility over your ability to predict or weight more remote effects.
I.e. if I came up with a model of society and predicted that if I went and killed a random guy it would lead to world peace in 20 years, I wouldn't do it, because I don't actually have faith in my model. If someone said "OK but what if you *knew* that killing the guy..." my answer is "no fuck you, I'm not gonna kill a guy out of dedication to not fighting the hypo".
This is also my answer to "should you kill a guy to save trillions and trillions of people from a very tiny bit of passing discomfort".
"Not being the first to defect from a well-functioning norm" also doesn't work, in my opinion, because it risks giving society too much inertia.
How local is the decision? Because most norms don't apply globally or through time. Should soldiers working in concentration camps in Nazi Germany have killed and tortured jewish people? Most of them did, so, localized in time and space to Nazi Germany, this was a norm. Of course, we would call it non-functioning, and it breaks the "don't commit murder" norm we have globally. But it's unclear how you would design a system that automatically selects for the correct norm, or who gets to decide what is well-functioning.
Maybe we should just always pick the most global? But even this gives too much inertia in my opinion. Factory farming is widespread, and the vast majority of people support it (at least economically). Undoubtedly a normalized practice worldwide. And if you ask the average person wether they like the way food is produced, so long as you don't specify too much, they'll probably say yes. Normalized practice, so a norm, and approved by most people, so well-functioning (?). But of course, few people who care about ethics would actually argue in favor of factory farming.
You could counter that most people don't actually support the specific practices present in factory farming (see OWID's recent article on the topic). But their actions don't reflect what they say. So again, who gets to decide which criteria matters more? Is a norm "well-functioning" if most people say it is, or if most people act as if it is?
Factory farming _of animals_, should've specified, apologies. The discussion here is on ethics. If you look into the ethics literature of the treatment of animals, it's pretty clear that animal factory farms are morally indefensible the moment you grant any credence to the idea that their suffering matters. It has also long been a point of interest in rationalist and EA circles (e.g. Scott's text from 2015 on how meat eaters can still help reduce extreme animal suffering https://slatestarcodex.com/2015/09/23/vegetarianism-for-meat-eaters/ ) Pretty much the only moral defense of factory farms is precisely the fact that they are widespread. Which is exactly my criticism: if our moral system allows for things to be considered moral simply because they are common, then that's a pretty weak system.
Sorry for the delay in response, I've been quite busy. I totally understand what you're saying, and I must say I really did use to think the same thing before. But as I said, once you look into it, then it becomes difficult to justify factory farming. Check out, for example, this article from Oxford ( https://www.ox.ac.uk/news/2021-11-11-sustainable-eating-cheaper-and-healthier-oxford-study ). Many such articles and studies back this finding: vegan diets are typically much cheaper than ones with meat or animal products. Now, most of the research is based in the west, but I don't think it's an unreasonable assumption to say that factory farming is unlikely to help the poor, because it would be more cost effective to feed the world less meat anyway.
You could reasonably complain about health, and the literature there is a little less decisive. But most findings from independent researchers do seem to show that a vegan diet has more positive effects than negative (see this meta-analysis : https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=vegan+health+meta+analysis&oq=vegan+health+metaan#d=gs_qabs&t=1777961896288&u=%23p%3DDyzIiZ2FnksJ ). That's not to say that we should impose vegan diets on anyone, but rather that if helping the poor was our goal, factory farming of animals is not a way to do it. It would be more cost effective to produce plant-based products, and would have little to no health drawback.
In regards to cage-free eggs, I think it's pretty well known that cage-free is nothing more than a shitty marketing term, so of course I'd have to agree with you there. You can look up what "open air" or "cage free" actually looks like in factory farms, and you'll see that it's still miles away from a traditional farm. Yes, the broken bones of chickens while stuck in a factory farm 10000% counts, and is again many of the unjustifiable acts of cruelty present in our current food system. As you said, a farm who lets their birds go out and actually act like chickens would be way, way better.
Finally for the cows, I was unable to find any authoritative source claiming whether cows do or don't like being milked. But consider that cows, like any mammal, don't produce milk unless they've given birth. So what they do is forcefully inseminate them by restraining them to a giant machine, and then having a technician pierce into their vagina and inject them from inside. Typically without any anesthetics. And then they spent their lives inside a factory-farm where they can't move more than a few centimeters in any direction. Again, unimaginable cruelty for little benefit. And impossibly far away from the ideal of a 'traditional farm' that most people imagine when they think of farmed animals.
I agree with Scott that these particular deontological bars, both somewhat about collusion, are affecting what folk do.
I think AI safety en masse is too shy about both of them in these two particular cases.
I volunteer for Pause AI and endorse protests where we share space with folk who care about different AI risks and have some positions I would not endorse. Open-hearted discussions about those disagreements even occurs in the ground. Activism can be net good.
I also don't, myself, treat anyone working in a frontier lab as an adversary. A subset are folk to talk to and work with. Different folk make different trade-offs based on different values and assessments. I will stand outside Anthropic and protest, but would also welcome chatting over coffee.
Our combination of best moves to make extinction from superintelligence less likely almost certainly form an impure portfolio. This is OK.
Tangential: the thing about becoming a concentration camp guard to make things easier on the prisoners, is that the guy who wanted the job but didn't get it is still going to be somewhere, and will presumably be as bad for the people around them as they would have as a guard. So it's not just "will it improve the camp," it's "will it improve the camp more than the other guy will degrade their new environment." Which is almost certainly No.
On the main point: Consequentialism can't have Morals Despite The Consequences. You either need external factors like "God said to do it this way,", or internal factors like "I'll be able to sleep with this decision".
The idea of a deontological bar seems unnecessary. Regular consequentialism + uncertainty works fine for all the examples I can think of. And, in terms of coarser societal norms, something like: "bad stuff should have a proportional cost" covers everything. The question of “defining and quantifying badness” is still open, but having a deontological bar seems to say “we should draw the line at some things that are infinite badness, like assassination”, but this is obviously folly.
With the assassination question - consequentialism works, and assassination already has a massive cost. I don't commit political assassinations partly because I think it would be bad for my movement, and would have negative utility more generally. But also because I don't want to go to prison!
Do I need a deontological bar on top of this? No, we don't, and it's pretty good that we don't! There was a well-publicised attempted assassination 3 days ago and I've heard no attempts to condemn it, because most people think assassination is okay in some contexts.
With the "don't work with an AI company", "don't work with Steven Bannon", "don't join a protest with terrible people" if we say: "doing bad stuff should have a proportional cost", and we acknowledge that these things are bad, then doing these things should have a cost. And they do (albeit not always proportional)! If the PauseAI team worked closely with Steve Bannon, the CCP, or Antifa, it would have a personal, and political cost to the team, which they would have to trade off against the benefits.
When EAs work closely with those creating superintelligence, like Anthropic, it should have a reputational cost, which they have to trade off against the benefits. This is more difficult, of course, but I don’t see any place that a deontological bar makes anything clearer or easier.
Regarding shifting norms on presidential assassinations: there have always been crazy people writing threats or even planning killing presidents. Skimming wikipedia, it seems that under Obama the number of such threats escalated. Many of these people were arrested and jailed for a few years or killed during the arrest. However, if we look at serious assassination attempts by domestic actors - where the assailant actually had intent and means to kill a president and was executing a remotely realistic plan to do that - there were very between the attempt at Reagan in 1981 and the attempts at Trump in 2024-2026. Perhaps Clinton 1994 should count as one, and the ricin letters to Obama in 2013 and Trump in 2020, probably gun grab on Trump in 2016 too. There were three such attempts on Trump in the last 2 years, plus two more in his first term.
So if we have a Poisson model for such attempts, we count 5 events in 1976-2023, two of them on Trump. And three events in 2024-2026. Annual frequency changes from 0.1 to 1.2 per year. This is statistically significant at ~0.2% level. Trump vs non-Trump gives 0.09 per year vs 0.72 per year, which is also significant at ~0.1% level. So yeah, the norm has definitely changed.
Alternative hypothesis would be the norm itself remaining fairly consistent as something like "don't assassinate remotely decent presidents, only insane tyrants who march armies against a legitimate American government." That covers broad popular approval for traveling back in time to kill Hitler, or John Wilkes Booth (insofar as his complaint about Lincoln was the fall of the Confederacy), or Trump after his coup attempt, and is even arguably consistent with some statements by the Founding Fathers.
Doesn't cover Charles J. Guiteau or John Hinkley Jr. so well, but consensus seems to be they were out of touch with reality, so deviation from social norms is to be expected. And the Kennedy thing is such a mess I'm not even gonna try.
1960s-1970s are also statistically different from both preceding and subsequent periods, with two attempts on JFK (one of them successfull), RFK killed while campaigning, and two attempts at Gerald Ford. Given short terms for both JFK and Ford, these are higher incident frequencies than for Trump. So for your hypothesis to be plausible one would need to brand JFK and Ford insane tyrants as well :)
There was definitely some controversy about the whole Catholicism thing.
As for frequency relative to Trump... how does that difference compare to historical trends in the overall homicide rate? Pretty sure it's gone way down, and a smaller pool of people willing to commit murder at all would naturally imply proportional reductions in any given specialized subtype.
I will be eternally amused if in a few years there are, I don't know, ten deontological bars that involve things like murder, covetousness, respect for parents, and graven images
I think focusing on "how well-functioning is the current norm?" is slightly missing what is motivating people's considerations in many of these cases.
I think that what people imagine is that perhaps, right now, (resisting) becoming a concentration camp guard or (resisting) working for an AI company doesn't immediately change anyone's actions, but that resisting is slightly increasing the future probability of others resisting and shifting the general norms in the resistance direction, whereas capitulating is slightly shifting norms in the wrong direction.
It might be that the probability of anyone else resisting is very low <1% and the increase in probability from your resistance is even lower, so this might seem like a consideration barely worth taking into account. But I think people also often think ~ the only path we have to changing things is slowly convincing everyone that concentration camps / AI companies are bad, so it's vital we resist even if success is very low probability and the effect is minimal.
I consider myself a meta-utilitarian deontologist. 1st level utilitarianism is not practically possible, as evidenced by all these stupid thought experiments that utilitarians get wrong all the time. The utility of moral choices is probably poisson distributed. Deontologism prevents mistakes that will outweigh any net gain all the clever utilitarian thoughts would have had
For those bars, is deontology the only correct framing?
Consequentially - for the one for pause activists, isn't there an argument that you never get a confident p(doom) reduction from marginal dollars given to labs, due to e.g. safetywashing, dual use safety research, or motivated reasoning coloring risk perceptions?
For the one for lab cooperators, I think the modal concern is backfire risk - that activism could increase p(doom), either by making it an issue subject to political cycles, or a fear that a properly AGI-pilled government and populace would take counterproductive actions? There's also, I think, a concern that the concrete asks necessary to reduce p(doom) would inevitably get watered down into something that wouldn't work.
You could also look at it from an identity (virtue?) angle. Pause activists are generally either people who've gotten fed up with the labs, due to them lying, or breaking safety-related promises, or being shut out by them when they argue for a pause, or they are people who never would have the opportunity and/or inclination to do "inside game" work. So their identity settles into being "people who don't like labs."
Lab cooperators tend to come from an elite pipeline; their whole adult lives have been "inside game," and do not see themselves as populists. They are used to seeing the world from an engineer's or researcher's POV - STEM isn't supposed to be political! - or from the POV of someone working at a 501(c)(3), where you are legally and culturally forbidden from being overly political. The only thing politics means to them is unpleasant things they see on social media. So, they see activists, and activism, as somewhat suspect.
Re the Ukraine example, I think there’s a significant distinction between “having a military” and “using your military to invade other countries.” On the individual level, human beings have the capacity to commit all sorts of terrible immoral crimes, rape, murder, and so on. But original sin notwithstanding, we do make a moral distinction between having the capacity to do something terrible, and actually doing it. You can theoretically have a large standing army which just sits around waiting in case it’s needed, and while you might say that’s still morally wrong, I think it’s a different category of wrong from launching an invasion of your neighbour. We can also view violence in self-defence as a morally distinct category from unprovoked violence. That also distinguishes between the Ukrainian army case and concentration camp guard example: if it was possible to take the concentration camp guard job, draw your salary, and actually just sit at home, not turn up to work, and therefore not do anything morally wrong in the course of your employment, maybe that actually would be justifiable, particularly if it prevented the Nazis from hiring a more diligent concentration camp guard. But in practice of course the Nazis aren’t going to let you have that cushy sinecure, and even the 90% less cruel concentration camp guard is going to have to actually do terrible things in the course of their employment.
So maybe the multilateral principle works in cases of positive action, but not in cases about whether it’s acceptable to even have the capacity to perform a certain positive action.
This also seems relevant to the difference of the AI case, because part of the debate is about who is the actual moral agent whose decision matters here. If AI is just a powerful weapon, then you can make the same moral distinction between building it and using it. But the argument of the AI safety people is it’s more like a nuclear weapon that could just decide to launch itself. So having the AI companies or the government bound by laws and norms about what it can be used for, is about as useful for AI safety as my solemn vow that I will under no circumstances launch a nuclear first strike is a contribution to nuclear deterrence.
"I don’t know how to get around this and think it might require common sense."
Smackdab inside thoughful reasoning, the messy real world requires the blurry "common sense". But only as a when no logic can wrap the problem. Given how often "common sense" works, common sense should be prominent in reasoning, less fallback more checksum.
"For example, you shouldn’t assassinate democratically-elected leaders, even very bad ones."
You'd think this would be obvious, but I'm seeing social media posting (which is probably more about virtue signalling rather than actually asking for it to happen) that it's such a shame the latest assassination attempt failed, or that eventually someone will succeed.
And I can't stop myself yelling at the screen "You idiots, don't you realise this holds for your side, too? If you're okay with Trump being shot, then you're also calling for Kamala Harris to be shot, or AOC, or your Fave Trans Nonbinary DSA-splinter offshoot Activist Just Elected As State Representative In Goosehollow to be shot!"
Because if you call up that which you cannot put down, it will happen to *you* also. Oh, but Trump is bad! He's uniquely bad! That isn't murder, it's revolution! Yes, and do you think there is no-one with a gun out there who doesn't think Harris/AOC/Dangerhair isn't uniquely bad and it's revolution not murder to kill them?
I consider myself left leaning but, I think one of the biggest failures of the modern left/liberalism is forgetting or failing to imagine how much worse it can get. The word fascism gets thrown around a lot but normalizing assassination is the kind of thing that *actually* leads to us sliding into some fascist security state.
> your Fave Trans Nonbinary DSA-splinter offshoot Activist Just Elected As State Representative In Goosehollow
There aren't so many of those. Lot of left-leaning Americans look at what the Democratic party establishment has been doing... seemingly far more focused on keeping left-leaning candidates from getting elected, and pandering to anyone on the right who'll listen, than in properly solving what their own fundraising emails breathlessly describe as existential threats... the Epstein stuff... even just the sheer gerontocracy of it all... and they think to themselves "yeah, if all the big career politicians, on both sides, simultaneously died, and we had to pick a new set completely at random, odds are it would be an improvement."
"I would be better positioned to navigate this debate if I knew what deontological bars were or where they came from."
A deontological bar is the cure for what ales you.
"So I’m not sure what’s left of this one under the “don’t break unbroken norms” hypothesis - although I notice this would also justify taking the slightly-less-brutal concentration camp guard job.
(in order to turn down the concentration camp guard job, I think you have to strongly commit to acting as if everyone would follow your rule, even though you know it’s not true - but then how do we handle the military disarmament example?)"
Sometimes it is worth attempting to establish new norms, maybe by refusing such a job and encouraging others to do likewise.
Others have pointed out the individual vs. group action distinction, but I think I have an answer for the concentration camp guard question:
Going from 100% to 90% brutality will remove an amount of badness from the world equal to 10% of the badness of a single concentration camp guard (over replacement--being a factory worker or something instead).
It therefore only makes sense to take the job if I think 0.1 marginal brutal guards is the most good I could reasonably hope to do (= the evil I could reasonably hope to remove). I think 0.1 MBGs is a pretty low amount, since it's not as though any one guard could hope to stop the gas chambers or the machine feeding them; I'm not convinced a few beatings one way or the other really makes much difference in comparison to the Holocaust.
Again, unless the system is sufficiently loose that one person can make a lasting impact, it's almost certainly possible to do more good by working elsewhere. Help refugees fleeing the country so they don't end up in the camps in the first place, things like that. I think even just working a civilian job to try and keep society running so people don't starve is going to be more of a net positive than giving out slightly fewer unnecessary beatings in Auschwitz.
For your section about Ukraine abolishing their military, the key is that just because there is a law against something doesn’t mean everyone will follow it. We have laws against murder but people still murder each other, that doesn’t mean we should abandon the law. As a society we generally agree that political assassination is bad but there are still political assassins, if our favored politician is killed the vast majority of people don’t think that the appropriate response is to kill the opposing politician in revenge.
So countries can’t come together and agree to get rid of their armies because there’s always a chance of someone going rogue and invading their neighbors anyways, so the cost is too high. But what they can do is enter into alliances with each other to cut down on the number of people they are worried about. They can sign treaties like the Geneva Convention limiting what types of weapons and tactics will be used. Opposing nuclear powers can agree to decrease their nuclear stockpiles, each one allowing the other to verify that they are upholding their end of the deal. These are all ways of countries trying to find the equilibrium between being as peaceful as possible while still keeping themselves safe and prosperous.
" If one company’s product has a 90% chance of ending the world, and another’s has an 80% chance of taking over the world, giving your money/support/encouragement to the 80%-ers seems kind of like endorsing evil."
Is this how a psychologically healthy species looks? This is a perfectly reasonable framing of human reality, talking about companies and how they'll end the world. Here, let's remember the context. Ready? So for millions of years living things basically performed a global bloodbath. We come out of this. There's Phalaris who put the brazen bull on the map. There's all kinds of mentally ill cults, whose ideas were basically just psychopathy maximization tools if you really distill things. Then there were thousands of years of domination, war, torture, while humanity clung to crumbs of actual values like ethics, like wisdom, and so on. Not because these things actually stand a chance in an evolutionary game, but because without them, the ultimate thing that wins, can't win in its own ultimate way(that thing is power/survival, which is at odds with wisdom and ethics, by the way, and not compatible with it in the way that DNA and beings that operate on DNA pursue power/survival).
Now here we are. We've lobotomized people whose thinking and behavior we simply don't like, and doesn't appear to fit in, a few decades ago. Not centuries, decades. We set off nukes. Some of us think that was kind of cool. Most of us don't even bat an eye("But but but but! Haven't you seen Oppenheimer?" Fuck you).
Just business as usual. The global quantity of child abuse(and indeed, adult abuse, and animal abuse), would satisfy the needs of a very sadistic god. We're just truckin along and debating two companies who have a 90% chance of ending the world and an 80% of taking it over. And this isn't some insane abstraction. We're just out of touch. Do you get it? We're out of touch, there's no whistle loud enough in this cacophony of psychopathic and psychotic noise. The malevolent children have managed to sedate and exhaust and drown out the adults in the room. Mommy and daddy can't seem to help, and it is an emergency for everyone.
I am always curious about the apparent obsessive drive to find a single coherent ethical or moral philosophy that will apply everywhere and to everyone, regardless of circumstances. I satisfied myself long ago that there isn't one--the human race is too divided by different definitions of right and wrong. What seems right to one faction seems wrong to another, and this seems inherent in human nature. I believe that this is due to evolution--we are selected to compete against each other organized into small in-groups, and to use justifying logic to give our own in-group some sort of persuasive advantage (it probably helps attract allies). This means that regardless of any moral standard "society as a whole" (which in this paradigm doesn't actually exist) adopted, it would satisfy primitive emotional impulses for any number of small in-groups to form a coalition in opposition to it. In the battle of cultural memes, factional emotion will beat universal logic every time. We aren't designed to agree with each other. "I believe that there is a plurality of values which men can and do seek, and that these values differ."
The alternative is to invest our resources into political processes that help ensure that we largely get along with each other with the minimal level of destructive conflict and violence. (We probably can't entirely eliminate the potential for destructive conflict and violence, but we could successfully seek to minimize it.) Politics is all about resolving inter-factional conflict with a minimum of destructive waste, and we know of certain proven methods for doing this. "Democracy is the worst form of government except for all the others" and all that.
Lets stop looking for universal logical solutions and start thinking about better ways to encourage people to tolerate each other.
I remain genuinely confused about why people (especially non-theist people) who seriously believe that the extermination of all life on earth an imminent and inevitable event would be against using violence to avoid that future. Surely even "unstable, lawless, chaotic civilization" is a better outcome than "lifeless planet".
I can imagine there's some argument that using violence will only accelerate the extermination of all life, but surely we'll eventually cross a threshold where violence is the only option remaining, right?
1. They SAY they seriously believe that the extermination of all life on earth an imminent and inevitable event, but deep down, they actually do not.
2. They can't think of any specific application of violence that would be effective in stopping said event.
2a. They are aware that historically, non-violent resistance has a better track record at effective long-lasting change than violent resistance. (Surprisingly few people seem to know this little fact)
3. If they seriously believe that the extermination of all life on earth is INEVITABLE, then by definition no act, violent or non-violent, can stop it, and they might as well enjoy their lives. Going to prison on terrorism charges tends to interfere with enjoying one's life.
I think one thing about this is it massively underestimates the extent to which companies run on trust and an actively anti-organization employee can do damage.
It's not just that one person who doesn't want the organization to succeed can get by with 10% less productivity. It's that someone who is serious can easily have a productivity of -10, possibly more.
Again, this depends on skill and willingness to break other dentological bars. Bringing a poisoned salad to the company pot luck or planting bombs to destroy the whole building may well be highly effective, but those actions break a different dentological bar.
Still, there are many opportunities for large scale property sabotage (Like putting copper shavings in the server room ventilation system, see the CIA simple sabotage field manual https://www.gutenberg.org/ebooks/26184). There are many opportunities to plant subtle malware into the computer systems if your writing code. (See the relections on trusting trust) There are also opportunities to play office politics. To tie everyone up in endless meetings discussing politically contentious but ultimately unimportant issues.
Would it be good news if Ukraine dissolved its military in the interest of pacifism? It would be kind of heartening, but not really since I know that not *every* country is going to follow their example. Even if all but one did, I would guess that the country that didn't is evil. It would only be good news if I knew that every countries leader were identical twins with Zelensky or something.
Would it be good news if I learned that I decided to starting committing crimes for the greater good, such as assisnation? NO NO NO. Even though the rest of the country aren't my identical twins, it provides evidence that maybe hundreds of other Americans are thinking the same things as me. Like the previous example the correlation is low, but the implications are way different! This would both cause chaos for the country and personally endanger me and my loved ones (since as an assassin I am probably a high value target). Moreover, the rest of the country would predict this is happening using psychology and would just ramp up security, wasting resources for no benefit (since now the crimes for the greater good won't even succeed). For example, because of SBF people don't trust EA financially.
What about contributing to the AI race if you think alignment is solvable? I'm not sure what EDT would predict, you'd need to collect some data on how correlated AI researchers are with each other, and then run a simulation. Surely assurance contracts could help tho? (But if assurance contracts work so well in theory, why aren't they used more in practice.)
The AI politics example is probably beyond simulation, but intuitively I viewed it as good news? The drawback is that it provides evidence of AI danger people getting into AI politics as well, wastes resources, etc..., but probably still net positive. The chance of crazies don't play too much into my calculations, even considering that other political movements might have crazies (perhaps this is wishful thinking?).
If I heard that was Scott Alexander started lying on the Internet for the greater good, this would be terrible news, of course! Who'd be next after him, Issac King?
I suppose you might define a "deontological bar" as an X such that hearing "X will be done in an attempt to benefit the greater good" is bad news.
I think one relevant issue here is that unlike atrocity, AI is so new that a lot of the cultural norms around it are still being set, so complete refusal to engage with it could set up at least a subculture with a deontological bar around AI use, which would probably be a useful thing to have in our brave new world.
This seems to be founded on an assumption that existing assassins were not, themselves, Republicans, which is not in evidence.
Leftists don't want Trump personally dead nearly as much as they want institutions behind him to be torn out at the roots, and he's doing a far better job of that now than he would as a martyr.
The peaceful anti-AI movement does not end up having to keep out a lot of crazies because all movements face that. It has to keep out a lot of crazies because "this threat is extremely urgent, and will result in human extinction" is the paradigmatic example of a threat where most people, if they really believed it, *would* justify assassination, terrorism, and other extreme measures. This is why people have posted that other causes don't have a lot of crazies; your cause is inherently the kind of scenario where being crazy is justified, and it's an uphill battle coming up with reasons why it isn't. The only reason you're not getting even more crazies than now is that most people don't alieve that their cause is as extreme as they "believe" it is and most people don't follow the logical consequences of their beliefs anyway.
It's the same reason why the pro-torture argument is "what if there's a ticking bomb and if we don't torture, then a few million people will die in an hour". It's urgent and has massive casualties and that means you may need extreme measures to stop it.
The real question consequentialism needs to answer here is: how would one distinguish "deontological bars" from social / reputational costs that aren't worth paying yet?
>Constraint consequentialists believe that you should try to do good things that improve the world, unless those break hard-and-fast rules (“deontological bars”).
How is this different from regular deontology?
Do most deontologists hold that you should avoid trying to do things that have good consequences *even if* you are not violating any deontological rule in doing so?
Joking aside, I always heard it referred to as "the Ukraine" prior to Euromaidan when the current Ukraine vs Russia conflict started, and I maintain that it should be an acceptable English way to refer to the country that reflects English-language idiosyncrasies rather than an opinion on the country's status, like how we say "the Gambia"
My family is from Kharkiv (so all-Russian-speaking Ukrainian, albeit live in NYC since the age of 3) and I'll note that part of the reason I think (multilingual) people take it seriously in English is that it mirrors a distinction made in Russian - my dad took until like 2024 to get out of the habit of saying "na" instead of "vi" Ukrainii despite not being able to have his own mother's funeral because the city was being bombarded. Given the country is basically named for being a borderland ("krai" being "edge") it's sorta the difference between going "live on the border" and "live in Border."
That said I'm also aware that "the Ukraine" and "the Iraq" are just things people heard a million times, but by now people ALSO know that some Ukrainians are going to get uppity if you say "the Ukraine" and Scott throwing that into a post that supports Ukraine's military is an interesting test. I have no idea how to process "The Ukraine must conquer Russia and crown Zelinskyy a new tsar."
Polish has the same "na" / "w" distinction and the same virtue-signalling push to address Ukraine with the latter. (Albeit in this case, "na" would supposedly signify Ukraine as rightful Polish territory. Which, yeah, probably the reason historically.)
However, most of the people doing that aren't willing to apply the same reasoning to Latvia, Lithuania, Belarus, Slovakia, Hungary, Montenegro... (with Croatia / Slovenia being a wash) (Specifically Hungary, "we Węgrzech" is highly jarring and difficult to pronounce.) Kinda makes them easy to disregard, even though I'm sure they'll eventually win this one by pure attrition.
That's really interesting to know - in Russian Ukraine is alone as a post-Soviet state using that linguistic construct (although it was using it I think since before there even was a USSR, and plenty of places that aren't former Soviet states still use it, like Cuba also gets "na" rather than "v" and has always been respected by Russia). I can definitely see a lot more reason to feel stubborn about keeping "na" for Ukraine if it's part of a massive pattern in Polish, rather than a weird one-off local outlier as it is in Russian.
How does not accepting the concentration camp job count as something that dooms you to automatically lose if you don’t follow suit? How are you losing?
What makes killing people wrong? If a religion decrees some morals, what makes those things actually moral? Are things good because the gods decree them so, or do the gods decree good things good? Is killing people wrong because you say so, or do you say so because it is wrong? What makes killing people wrong?
Do unto others.... Do you want someone to kill you? There are also many religious faiths that speak against this, Christians, Buddhists.... Killing people is wrong. Look I know we have a long tradition in our past of killing people. Mostly the 'others'. But we've grown up beyond that and agree that killing people is wrong. Or do you think it is OK?
I am not saying it is okay to kill others, merely that you are not giving a logically sound reason.
"I do not want others to kill me"
*non sequitur*
"therefore I will not kill others."
How does the first statement justify the second? Do you care about not killing ants? Amoeba? Not to the extent you care about not killing humans, so clearly you have particular definitions of 'others' in mind that changes why you should not kill these others.
Hmm, I'm not sure there is some 'logical' reason. One might call it a universal zeitgeist, perhaps the point of view of the universe, (Sorry I was just listening to a Sean Carrol podcast with Peter Singer.) and others may call it God. Whatever, we have all agreed that killing people is wrong. About killing other life, I'll agree this somewhat on a similar level and perhaps I/we just haven't progressed enough to make it wrong. I say this with the smell of bacon cooking for Sunday morning breakfast. I do like to eat bacon.
Anyway, Logic is a wonderful thing, but it's not the only thing. I love my Buffalo sports teams. (Go Sabres!) But if you look at professional sports from a logical point of view, then the whole thing looks a little silly.
What do you call the process of making deductions, including deductions about whether your deductions are correct? I call that logic.
Maybe you don't particularly like classical logic. "I love my sports teams, but if you look at sports from a logical point of view, it looks a little silly." I would say you're working with more of an emotional logic: what is right is what you feel good about. So, there is no math formula you can write that describes why it's fun to root for the Sabres, you just feel it is. (Arguably, maybe a big enough formula could simulate your brain and see what makes it tick, but let's limit ourselves to tech available in 2026.)
Similarly, you see no 'logical' reason to not kill other people, merely an agreement that it makes us feel bad. And, the way to propagate that agreement (because there are always curious little boys and girls) is to teach everyone when they're very little that murder is bad, *simplicitor*. It's a *sin* you see, so you should feel bad even thinking about it.
I have two issues with emotional logic (or, deciding right by what your brain feels is right):
1. What if you would feel much better about something else, if you could see it in front of you, but you never can because your brain doesn't yet feel good about actualizing it (example: cooking).
2. How do people with different senses of justice come to an agreement? Or is it impossible, and if they aren't indoctrinated with your sense of justice early enough you have to have fundamental disagreements until one of you genocides the other?
Oh I love logical thinking. If I'm trouble shooting, fixing, debugging something then logic is the way to go. There is a joy in fixing something, that is different from the joy of your team winning a hockey game. But still a joy, perhaps more long lasting than sports, the tractor continues to work years later. Emotional logic is a term that makes no sense to me. Emotions are beyond logic, or just just a different dimension. I love my sports because of the emotional ride, other fans say they like the game where we blow the other guys out. But my favorite games are those that go back and forth... the outcome remains uncertain till the end. It can maybe be explained in terms of some hormones or chemicals being released... but there is no logic involved.
Oh there are 'gut feelings', "Oh this is a bad situation, I should leave." and you should leave. That is also beyond logic. (Perhaps it's some analysis going on at the sub-conscious.)
1.). I don't understand this? (I love cooking.)
2.) I think we all have pretty much the same sense of justice. That's my point. I guess you can disagree. A jury of my peers is a test of that sense. If people from other cultures have a different sense, then I guess that's something for the diplomates to figure out. Or as you say, war in the end.
With my mild dyslexia I read the title as DENTOlogical bars, and I thought this was about devices to prevent us from grinding our teeth. But as Emily Litella famously said, “Never mind.”
I wrote a mediocre lesswrong post on it once, which now has 12 karma. https://www.lesswrong.com/posts/JwfkYzTehmGsizvuL/failing-safely-is-the-anomaly. My view of this is: by default humans are insane, so insane that the default assumption should be that whatever cause people have is actually crazy and would make society worse instead of better, and the great triumph of classical liberalism is finding out ways to mitigate the harms caused by this.
In retrospect, I think I should have mentioned that the individual is better off too, in addition to society. Example: western flat-earthers who get mocked online are themselves better off than African Boko Haram members who keep murdering people until they die for it. But I think my main point, that anyone with a cause is probably delusional and needs to make safety precautions in case they are wrong, is still valid.
I think it would be helpful to clarify the exact view at issue here. You might believe either:
1) The best action is the one which has the best consequences provided you don't violate the constraints.
2) The best action is the one with the best consequences full stop but as a fact about human psychology (or less plausibly rational action) one should intend to obey those constraints.
---
As a matter of moral theory 1 seems hard to justify. But once you switch to 2 it isn't clear why you should expect the rule to actually be all that clearly defined. Human psychology is pretty complicated and messy.
I don't think it's particularly controversial that breaking a norm against assassination licenses a retaliatory assassination. Some places even allow it against people who have otherwise been incapacitated and unable of causing further harm, which I think is a bad policy for several reasons. But certainly - retaliatory, or in some cases even proactive norm-breaking against someone credibly attempting to break the norm is, in fact, common sense.
It's how norms are being enforced. They rely on a shared understanding that you are protected by the norm if and only if you observe it.
What you probably meant to say is, people shouldn't try to weasel their way out of norms by reinterpreting them in a way that gives them license to break them. (Such as, declaring that your political opponents writ large - and not just individual assassins - have broken the norm by, I don't know, not showing enough performative outrage against the individual assassin in question, or by having previously spoken ill of the victim.) You may think such rule-lawyering allows you to preserve the norm going forward, after you've accomplished your goal of disposing of people you find inconvenient, but no, no it doesn't, people notice what you're doing, and the norm gets eroded, and soon you're finding out it stopped applying to you too.
Speaking of speaking ill, I think both the article and the comments somewhat underappreciate the fact that Trump has recently ordered assassinations (and/or kidnappings) of other countries' leaders. This makes his own assassination, specifically, probably not the clear-cut go-to example that you probably think it is. (Even if you think international relations operate on the different set of norms, "don't hurt the hegemonic force or it'll start lashing out" is still a functional norm that breaks when it just starts randomly lashing out on its own.) Trump has very much invited his own assassination by this point, the recent attempts have just left me shrugging rather than worrying about the norms getting eroded (that ship, as they say, has sailed), and if you think me stating this simple fact signifies my own unwillingness to observe the norm, you'd be shooting a messenger (possibly literally). (It mostly signifies me not being an American and worrying more about international norms than your own internal ones.)
But yeah, this is how norms erode. Most people aren't itching to start norm-breaking at the first opportunity, they just find it increasingly harder to gather motivation to proactively defend the norms for people who otherwise don't observe them (if not by the letter, then at least in spirit). At some point, a reset to a stable equilibrium is needed - whether it'll come by [lawfully deposing Trump and credibly pretending he was an aberration] or [civil war until everyone is tired] or [everyone giving up on norms altogether] is left to be seen - but very much depends on whether people's loyalties will ultimately lie in the norms or in their respective tribes / immediate self-interest.
I think that if you want to be fully consequentialist, you also need to take into account the consequences for yourself. And actively causing harm to people you believe don't deserve it (like in the concentration camp example) is bad for you as well (unless you're a psychopath).
Even if you're a saint and would sacrifice your own well-being, you have to weigh the opportunity cost of doing so. It's likely that you can help in some other ways. Maybe try and get some people out of the concentration camp altogether. Maybe sabotaging it's operations, maybe helping end the war faster so that concentration camps have a fewer days left to kill people.
N the hypothetical scenario where there is really nothing else that can help and all you can get away with is by becoming a slightly less brutal guard yourself then it sort of is justified I think. It's just that hypothetical is too far removed from the real world that it doesn't make much sense.
A more realistic scenario where you can influence who becomes the guard and you pick the less brutal ones seem more realistic and also more intuitively good (although you might then also combine it with other subtle ways to help).
Another reason to avoid becoming a slight less brutal guard yourself is that it can change you into someone who genuinely is brutal and you instinctively want to avoid that.
None of these considerations are relevant in the disbanding/keeping the military hypothetical. Keeping a military force when it is not needed might be a waste but it is not actively and purposefully harming anyone - unlike the concentration camp guard.
Is there some sort of (naturalistic) "is-ought" fallacy going on here?
Of course utilitarianism almost inherently faces this problem, and particularly rule utilitarianism (what 'deontological bars' sounds like). However it sounds like consequentialists are only hesitant here to act, not because they care about the 'bars', but because acting could set self-undermining precedents. So they're just 'practical utilitarians', which is just same old OG utilitarianism. All that's happening here is a convolution of moral arguments with practical ones, but it does highlight how important epistemic accuracy ('competence') has always been to utilitarianism (e.g. communists).
>For example, you shouldn’t assassinate democratically-elected leaders, even very bad ones
Consequentialism and deontology aren't the only games in town. Maybe you shouldn’t assassinate people because you're not James Bond, but James Bond should, because is and he has a license to kill. Allowing assassinations under very controlled circumstances could be better than having a universal rule of never killing Hitler: or a free for all allowing anyone to kill politicians they don't like. Everyone accepts some version of this , eg. that the police can do it hunts civilians can't.
Act only according to that maxim which, if adopted by a realistic proportion of rational agents, would best produce the optimal state
And I define maxim as: a universalizable rule of conduct, judged not by moral intuitions about the acts it permits, but by the state it can reasonably be expected to produce, where the standard of reasonableness rises in proportion to the magnitude of potential harm.
I still need to improve it but it’s the best I got currently.
"Cooperate if and only if you can prove that your provable cooperation leads to their cooperation, and it's possible that their cooperation leads to a possibility of your cooperation."
Equivalently, we can reduce
◇(X(S) → ◇S(X))
into
¬□(X(S) ∧ □¬S(X))
which reads
"there's no proof that they will cooperate even while you are provably defecting."
The first half prevents you from being a cooperate rock, while the second achieves maximum utility when playing with cooperative rocks. This is not my idea, Joseph Camacho wrote it on LessWrong:
“Don’t be the first person to defect from a generally functioning norm”.
Great idea! Too bad in real life people may start to justify their actions by (rightly or wrongly) implying the OTHER side already defected, so we must do it too. I am Polish and I've seen that in Polish politics; but I also followed a lot of rightwing blogs (yeah, there were times when I was really EDGY young men) who claimed rightwinger must finally take the soft gloves because the OTHER side defected years ago.
(too bad I can't find this once popular meme about "conservative is a man standing athwart history, yelling stop" with a guy in front of locomotive)
Oh, there's no end to it if we go "who started it first" but today there's an opinion piece in the paper by a columnist who is SHOCKED and APPALLED that the pro-abortion side in Ireland are not stopping at the limited changes to the law they got passed, they want NO RESTRICTIONS ABORTION. This was not what she voted "yes" for in the referendum to repeal the Eighth Amendment!
And I have to say to that "Well, what the fuck did you think was going to happen? The pro-abortion side were open right from the start about their aims and goals. Why did she think they needed the Eighth Amendment repealed, not just amended? or new legislation clarifying when doctors could intervene to terminate a pregnancy when the foetus was non-viable or already dead? They needed it gone to get what they wanted. And any view of how abortion has gone in the UK and USA should have told her the same: it went from "only a last-resort surgical procedure in limited cases" to "fundamental human right".
This is why I voted "no" back in 2018, quite apart from moral view of the matter: "but... but... but they said they wouldn't change anything!" to which I can only reply, God forgive me, "you stupid bitch". Of course the campaign would keep campaigning till they got what they wanted, of course the politicians will fold on this the way they folded back then - if the perception is "there's votes in this", they'll scrap anything and everything because the guys who swore "this far and no further" are no longer in power and it's a whole new ballgame, baby.
Deontological bars. Break 'em, you can't glue 'em back together again later when you find that the result of breakage is not what you wanted.
>After announcing in the inaugural issue of National Review in 1955 that the magazine would “stand athwart history, yelling Stop, at a time when no one is inclined to do so,” he evolved into a conservative kingmaker, one whom presidents and would-be presidents ignored at their peril.
Once again consequentialists try to judge the success of deontology on consequentialist grounds. This doesn’t take seriously the deontological perspective, which is that ethical questions are orthogonal to their consequences in any this-world sense, and that success, failure, suffering, pleasure etc. can’t be used as evidence either way.
The television show Better Call Saul is about this problem.
As in, the whole show: in the first two episodes someone with a consequentialist temperament gets a hard reminder that he should probably locate some rule-like guardrails here and there. In the remaining 61 episodes he tries to figure out how, which, and where.
The other main cast members represent other recognizable moral positions: a deontologist, a virtue person, an eye for an eye consequentialist, a stealth consequentialist (pretend there’s rules when people are looking, for the sake of the social fabric, but break them as needed when sure you won’t get caught). All of these characters change views profoundly as the show goes on while mostly staying within those categories.
I have never seen allegory handled less dryly or more delicately - so delicately that hardly anyone seemed to suspect it was about moral philosophy when it was on. (It also may be the best work of art produced in this century, if that’s relevant.)
Maybe you're not interested in litigating your object-level examples, but...
> Some of the people working with the companies think there might be a deontologic bar against certain types of mass activism.
I don't think this is true. I spend a lot of time on the relevant parts of Twitter and I don't think I've ever seen a lab employee argue that participating in this specific form of mass activism is violating any sort of deontological prohibition. I have seen some e/accs (and other randos) suggest that trying to slow down AI will kill people (in expectation), and also (more from various bad-faith commentors) that those arguing that unaligned ASI might cause human extinction are morally responsible for things like a crazy person attacking Sam Altman, but _even then_ I don't think I've seen any of them say "and you are violating a deontological prohibition by making such arguments". Partly I think that's because they aren't really operating in that mode, but also because that would be a pretty crazy claim, and trying to make that argument explicitly (rather than implicitly) would render it very obvious that their entire argument is in bad faith, unless they came up with a very surprising and compelling case for that.
I have seen more people make arguments that could be read as "I am deontologically prohibited from participating in many kinds of possible protests that might be organized on this topic, because that would require endorsing claims I believe are false", but most of those people (I claim) would take a 5-10 year pause enforced by global treaty if the only available alternative is no pause. One such person I know does work at a lab. (I don't know more than one such person working at a lab with that particular combination of beliefs.) This feels like a pretty different bar than the one that paragraph is sketching out, and I don't think they would claim that people ought to be deontologically prohibited from any of "working with Steve Bannon, working with Bernie Sanders, working with NIMBYs who hate data centers because they’re a thing that might be built in someone’s backyard (or by non-union labor), training TikTok influencers create short-form videos about the dangers of AI, and holding protests where we chant vapid slogans outside AI company headquarters".
I think the underlying issue is that you’re trying to fit a simple, “single-sentence” rule that generalizes across all situations. That’s like trying to solve next-token prediction with a linear model, it’s just too rigid. You can just expand your model class to include e.g. the fuzzy things that your black-box brain outputs as moral judgements, and which depend on where in thing-space you are. In other words, it’s fine to have your deontological bar be “I don’t do egregiously evil things, as evaluated by my fuzzy black-box moral compass”. Yes, this is subjective, but I think that’s a price worth paying here
I don't understand the unilateral disarmament example. I understand deontological bars as a patch on consequentialism to prevent the consequentialist from taking unethical actions that have good net results.
But the only problem you find with unilateral disarmament is that it doesn't provide good results. So I don't see why your rule for deontological *bars* would need to rule out actions with bad outcomes, since the consequentialist wouldn't feel tempted to do those things anyway. (This is different from a fully deontological system, where the deontologist needs license to not do things which would have bad results for them if others don't cooperate.)
To put my objection another way: You've identified a case where Kantian universalization recommends a bad course of action. But this isn't a problem for the consequentialist, who only cares about *prohibitions* from the deontological rule, rather than *recommendations* from it.
We might take some inspiration here from our understanding of biology. The general rule is that cells can do whatever they want unless and until they threaten the health of the organism that all of the cells depend upon for survival. If and when that happens, the immune system kicks (or at least it's supposed to) in and the transgressive cells are eliminated. Sometimes immune systems get confused, of course, which is what we call auto-immune disease, but the system generally works well enough.
Now suppose that instead of looking only at the cells and the organism they're apart of we zoomed out to coordination and competition among organisms or zoomed in to coordination and competition for energy and other resources among different biochemical pathways within the cells? I submit that we can find analogies to the same order we see established among cells — that is, that the basic rule is that each organism (zoomed out) or pathway (zoomed in) can do what it wants so long as it doesn't threaten the coherence of the structure one level up — that is, in the case of the organism, its colony of kin and in the case of the biochemical pathways, the cell.
Now we have a substrate we might use to establish an ethical principle: in most circumstances, it's best to maintain systems within some critical regime that maximizes adaptive capacity. In practice, this means don't let cells kill each other unless there's some cell that is threatening the health of the organism as a whole, then have the immune system take care of them. But don't have so much order that the cells are basically perfect replicas of one another, never transgressing boundaries all the way up until all are wiped out because of a tiny change in the environment.
I prefer this to `“Don’t do something which would be bad if universalized, unless the norm is non-functioning in such a way that you’d be playing cooperate while your enemy plays defect”.` because "universalized" and "non-functioning" have measurable referents with respect to the hierarchy of systems and even the concepts of "cooperate" and "enemy" resolve into a relatively well-defined hierarchical competition to stay alive. And note that death itself might be understood within this framework as the condition under which a "cell" within some level of the hierachy runs out of the energy budget available to continue to update its self-description to coordinate within that level of the hierarchy, thereby threatening the "organism" within which the "cell" must coordinate with other "cells." So murder is unjust because it preempts that natural process whereas execution may be just because necessary to preserve the process for the "organism" as a whole and other "cells."
Now let's apply this to "don't support an AI company" and political action campaigns.
In the case of "don't support an AI company" then it seems like the ethical target steers us toward making a decision based on whether or not the AI company in question is more or less likely to enhance the well-functioning of whatever body it's a part of. To be concrete, does the AI that a specific AI company is building threaten to destroy the entire economic and political order within which the AI company itself is operating? If the answer is yes or maybe, then the answer is the same as for the candidate concentration camp guard, although the answer to the question is much more obviously yes in the case of the latter, which is why I believe that question is being asked at all.
In the case of political action campaigns, it seems like the ethical target steers us toward making a decision based on whether or not the platform and or ideology of the campaign has the potential to undermine the entire economic and political order within which it operates. That was and is arguably true of a campaign built either explicitly or implicitly on an ideology like Carl Schmitt's, for example. That would be true too of a platform or ideology that openly disrespected the rule of law or the basic principles of respect for human dignity that have allowed for people to live together in community with less violence.
And yes, to your point about early Christianity, as Girard laid it out, Christianity does offer a unique and compelling narrative and set of symbols that help people to understand and accept a solution to the problem of an unending escalation in violence — a solution that Leo is doing a great job reacquainting the world with at the moment.
Another way to encapsulate “don’t be the first person to deviate from a well-accepted norm” is as a special case for “culture is above game theory on the society-wide maslow hierarchy” (though game theory itself probably encapsulates this). And if we assume caring about consequentialism with deontological bars is a sort of “culture”. In other words “you have the luxury of deontological bars if you’re not in a tit for tat”.
I feel inclined to note that "don't ever work with AI companies" can be almost-fully justified on utilitarian grounds if your P(Doom|AGI Soon) is high enough. Essentially, one says that the decrease in P(Doom) from this particular contestant winning is massively outweighed by the increase in P(Doom) from helping *anybody* and thus lowering the chances of a successful Butlerian Jihad. This technically doesn't forbid helping them with intent to betray and sabotage, but that's pretty hard anyway.
The Yudkowskian view, for instance, is that any and all methods of neural net alignment are 100% doomed, and that you'd be more likely to survive literally jumping off a cliff than *anyone* building ASI. This means that essentially all non-traitorous help given to AI companies is -EV, because there is no world where they get what they want (to keep building AI) and also we don't all die.
It seems sort of weird to be debating hyper minutia of ethics if you *really* believe AI is extremely likely to cause total human extinction. Isn't the most relevant question what will work and less what is or isn't ethical? Isn't this close to being the most important cause there has ever been if AI is going to annihilate us?
Non sarcastically: I think this issue is one of the major parts for why Harry Potter fanfiction tends to be so popular and prolific.
Presumably purely by accident, but JKR managed to create near perfect scissor setup for this political dilemma. In a world with barely functional government and extremely universal access to lethal weapons, when one political side is trying to actively genocide the other and actively working against peaceful norms by terror attacks, is sticking to pacifistic/ non-lethal means because those are your ideals tantamount to treason, as it gets more of the persecuted side killed in the end?
This works even better, because the descriptions in the books are vague enough, that everyone is casting their own preconceptions on the severity of this situation.
Both the deontologists and the utilitarians were trying to escape cristian theology (to ground ethics in reason or empirical facts rather than divine command. But they escaped through a door that was already shaped by Protestant thought. The very form of their secular ethics) its individualism, its universalism, its systematic ambition, its hostility to mere tradition was Protestant before it was philosophical.
>Of course, the Early Christian Strategy is to ignore all of this and do the right thing in every case, including unilateral military disarmament. But they can’t keep getting away with it, can they?
I mean, maybe not. Time will tell. It's not over yet.
The idea is interesting, but the thing about critiquing strategies based on their success so far is that we don't know when the endgame is. I don't think it's pointless to compare strategies and try to understand their advantages, but it's not the endgame yet.
Empirically, the Christians did *not* stick with unilateral military disarmament. See especially: the crusades and the inquisition. They eventually returned to a peaceful approach, though.
You don't even need to go that far. The very first Christian Roman emperor murdered his wife and son. That's pretty bad even by pagan norms!
Technically, Constantine was not Christian at that point. He converted officially on his dethbed.
He was baptized on his deathbed. It was standard at the time to delay baptism as long as possible, to maximize its effects (less time for more sins to be committed after your one sin-erasing moment). The New Testament kind of clashes with this notion, but it's not like the New Testament canon was even finalized yet in Constantine's lifetime.
That he converted 25 years before that is almost universally acknowledged by historians. There may have been some lingering bits of syncretism, but you don't claim to see a burning cross in the sky that says "In this sign you will conquer" and not immediately convert to the cross-having-religion, that's just not how anything works.
Even after the Milvian Bridge Constantine continued to make sacrifices to the gods as Pontifex Maximus, use divination, and mint coins depicting the pagan gods.
We could say that Constantine first added the christian god to his polytheistic worship and he then became fully and exclusively a christian at the end of his life.
Funny enough a yt channel I follow just posted a video on "Constantine The Great's Pagan Coins"
Those aren't early. Crusades happened approximately 1,000 years after the beginning of Christianity. Inquisition happened another half a millennium after that. Scott is talking about the first century, or maybe first couple centuries, when Christianity was being persecuted by the Roman Empire.
Yes, *Scott* was talking about an earlier era. The comment I replied to said, "I mean, maybe not. Time will tell. It's not over yet." Clearly they weren't talking about Early Christians anymore.
I see, that makes sense.
Christian states were fielding armies long before that.
Its easy to preach unilateral disarmament when you're a small sect and you have no effect on military policy anyway. Its another thing entirely when you're the state.
The gap between Constantine converting and Augustine of Hippo's just war doctrine are very small. And Rome didn't disband its army in the interim period.
Because Christians belong to Christ, rather than Christ belonging to Christians, the only true Christians are the ones who follow Christ's commands. Everyone else only serves to blaspheme God. And this was a problem between God and people long before there were even Christians. As it says in the book of Romans, in the New Testament:
Well then, if you teach others, why don’t you teach yourself? You tell others not to steal, but do you steal? You say it is wrong to commit adultery, but do you commit adultery? You condemn idolatry, but do you use items stolen from pagan temples? You are so proud of knowing the law, but you dishonor God by breaking it. No wonder the Scriptures say, “The Gentiles blaspheme the name of God because of you.”
I'm not interested in questioning who counts as a "true" Christian. The bible also makes very clear that we're all sinners who fall short of our ideals and God's commands.
Very true, that's also in Romans. Seeing as you were making a comment on Christians and their actions, though, it wouldn't really make sense for you to not be interested in what actual Christians did vs. posers. If not, it doesn't really make sense for you to post your initial comment to begin with. Let alone to start it with the word "empirically".
Whereas I think that if the use of "Christians" as a term for a historical group makes sense at all, then IMO it only makes sense to use it if you're applying it to an identifiable group, rather than a scattered collection of individuals within such a group. In which case, I can't not apply it to Catholics from the conversion of Constantine through the Protestant Reformation. If you want to argue that many of the people in the group I'm labeling "Christian" are deeply misguided, I have no problem with that! If you want to go full Chesterton and say that Christianity has been found difficult and left untried, I have no problem with that either! But *specifically* excluding the largest group that calls itself Christian, which made up an even larger proportion in the period under consideration, I find to be an unambiguous No True Scotsman definition.
They didn't stick with it even during the Roman Empire. Constantine had lots of Christian supporters.
So the early Christians were peaceful and powerless, then Christianity became powerful and non peaceful, then it lost power and became peaceful again...?
Yes.
That's probably how many religions evolve.
The powerful seem to have an astonishing knack for gaining the favor of the local gods for whatever policy they choose to enact.
A cynical sort might suspect this has more to do with the wishes of the powerful than with the tenets of any particular religion
The Crusades weren't early Christianity. I don't know where the dividing line is, maybe the first century or two.
Agreed, of course.
Lyman Stone correctly pointed out that the early Christians were much more violent than Scott gives them credit for. https://lymanstone.substack.com/p/the-rise-of-christians-mobs-moms
Reading Lyman Stone's article sets off my Historian Bullshit Sense. In particular, the source he links for his claim that Philip the Arab was a Christian really, really does not back up his bolded claim that:
> Almost a century before Constantine, Christian ideological leaders already have the view that Christians in politics should be declaring open war on pagan worship.
I don't know this is wrong, it seems pretty plausible to me, I just think it's totally unjustified by any of the sources he claims to cite. He's doing the flimsy-chain-of-logical-deductions-each-individually-plausible thing, and most of his claims about Japan and China seem to be about as unjustified. I do not and would not trust anything he says without independent research.
It's also certainly unjustified by the story he tells in the text. There's a major difference between thinking that a Christian emperor should not *participate in pagan ceremonies* (all Christians were forbidden from doing that), and thinking that as a political matter, he should shut them down.
Both may be true -- I have absolutely no idea -- but he needs to distinguish and provide evidence for the one he needs, and he doesn't.
Ah, reading that and looking him up, my immediate reactions are:
(1) He's a Lutheran? Figures! Reflexive remnants of anti-Catholicism, because the second the name "Constantine" comes up, the Reformation trope of "he was the one who perverted Pure Early Christianity by tying it to the state" and "This was when Roman Catholicism was invented, that's not Christianity!" rears its head.
(2) Okay, now he's jumped on the hobbyhorse and is galloping furiously off. "No, Christianity did not succeed because X, Y or Z, but because My Thing!"
Christians were high fertility and *that* was their winning strategy, says guy working for pro-natalist movement. Gosh, you don't say? So them being Christian was nothing to do with it, just that they were (ahem, excuse me while I take the piss here) sexist patriarchal oppressors of women who forced their wives to be barefoot and pregnant and engaged in marital rape to keep them popping out a baby every year? Is that your message, Lyman? 🤣
"Pagans Had Low Fertility, Christians Had High-ish Fertility
...I have a huge amount of new, never-before-used evidence demonstrating that Christians really were disproportionately female, Latin-speakers (especially low-social-status Latin speakers!) really did have low fertility in the 2nd/3rd century, and Christians very likely did have higher fertility, not least because of their absorption of the highly fecund norms of Egypt."
Dude. Duuuuude. "New, never-before-used evidence" about female and low-social status Christians? Damn, I thought I lived under a rock! Then again, he qualified as a sociologist, I shouldn't be too harsh on the guy for being ignorant of basic, established, well-known historical content.
EDIT: Also, Lyman, not to throw shade on you, but it wasn't the highly fecund norms of Egypt, it was the anti-abortion stuff. They didn't kill their daughters, they didn't abandon their newborns, they even went around picking up abandoned babies off dungheaps, imagine!
https://www.reddit.com/r/AskHistorians/comments/99o3oy/romans_usually_left_the_unwanted_babies_in_trash/
As an aside, I had never heard of Philip the Arab as Emperor, but to my shame I'm not familiar with the Year of the Six Emperors and their various successors who were technically emperor for nine days before the next guy stabbed them. I've never read any claims that he was the first Christian emperor, so sorry Lyman, I think you're spurring on the hobbyhorse here.
Broke: we can't easily judge strategies outside their endgame.
Woke: we're in or blatantly running into many potential civilizational "endgames" by many metrics (used by you and/or others), from climate change and AI to national politics and (depending on your religion) Middle East prophecies.
Bespoke: you can sorta model people and groups by how "endgame mode" their strategic thinking is, a little like Burja's live actors vs dead actors. E.g. dead plus not-endgame-thinking might be "political campaigning like it's 2010 again", endgame-vs-not can cause rifts by who's a transhumanist vs not, etc etc politics politics social groupings
@Daniel Muñoz you owe it to the fans to read and comment on this 😎
Oh yeah pls! 😂😂🙌🙌
@Adam Smith - expect a full paper + post in the next year. Got something in the works…
not that response I expected! You got this.
There's some distinction to be drawn between *actions* and *policies* (a policy describes how to act, given a context/situation, it's a decision rule). In between there are partial policies of varying flexibility. I think the armament case can perhaps be resolved by treating the rule you wish to be universalised as being (in large part) a *policy* (perhaps something like 'disarm when multilaterally possible, don't be first to escalate, seek de-escalatory stances and movement, ...').
This distinction might solve the general case too, unsure.
It's an important distinction in general for consequentialists. For example, basic sequential search theory against a known distribution suggests an optimal stopping rule--keep searching until you find something better than a threshold value. That looks like a "satisficing" action, but the policy is optimal if the stopping value is chosen appropriately.
Then you can wonder about optimal meta-policy if figuring out the optimal level of the stopping value is costly. And so on in a potentially infinite hierarchy.
Yes, but if you write it all out that way it wraps around to deontology again. Albeit a deontology even more complicated than biblically accurate Jewish law.
> every time someone tries to create the Peaceful Nonviolent Pause AI Movement Based On Peaceful Nonviolence For Peaceful Nonviolent People, it spends an inordinate amount of resources keeping out violent crazies who want to tag along.
This is untrue, and a strange thing to confidently claim a priori. It actually takes very little resources to keep the crazies out, in both absolute and relative terms. More than zero, but it's really not that bad.
There was the StopAI founder who went crazy and is missing, and the guy who attacked Sam Altman tried to join one of the PauseAI things and had to be kicked out, and the PauseAI people sure seem to spend a lot of their time loudly condemning violence and having sections on their site about how bad violence is.
> the StopAI founder who went crazy and is missing
Who? I hadn't heard of something like that happening.
Guido is still around and tweets, did some protesting last summer, old timey nonviolent stuff, and popped up after the altman house incidents to re-commit to that. Like Emile Torres, he sometimes gets sidetracked attacking LW/rationalist types and Yud in particular, over various posthumanist things Yud has said but which were conditional statements with extraordinarily low probability that will never occur in any case if AI goes badly, so he's kind of alienated a lot of possible allies for what seems like nothing. But I still like the guy, he's fine.
Sam must be the one who disappeared, I don't know the deal but he said he was "no longer affiliated" with StopAI late last year and I haven't seen any mention of him since then.
I don't know who Guido or Sam are.
The co-founders of the organization you were asking about
What are their surnames?
Saying "Sam" in an AI context seems really confusing. I assumed you meant Sam Altman.
I can't speak to Stop AI so much; they have their own particular.. uh.. issues. On the loud condemnations, there's some recency bias on how many tweets are about that, I suppose, given the need to respond to those attacks, and even then it's still a fraction of a percent of the total effort involved in running an advocacy org. I don't know what would be inordinate. Maybe 5% of total resources? That would be ridiculous.
This is a thing that it is very important to get right, but it's also not a thing that consumes a great deal of resources once you have established norms. In the case of PauseAI US, we're structurally set up to significantly reduce this risk vector. We just don't let violent crazies tag along. If they try, we tell them to buzz off. If volunteers suck, we kick them out. This is not a particularly hard problem to solve.
Of course, it's attractive to imagine that this is a debilitating problem if you are looking for reasons not to do advocacy!
"If they try, we tell them to buzz off."
And if they refuse to do so? It's the open street, they have as much a right as you to walk along it, they're not part of your protest they're doing their own protest, etc.?
If the video in question is https://x.com/GrageDustin/status/1820847003271639392 then I think we have very different takes on her attitude. But she mentions burning tires, so maybe there's a different video?
I think that the last issue, constantly and loudly condemning violence, is less an effort to keep crazies out, and more an effort to prevent themselves being interpreted by hostile outsiders as endorsing violence. Consider how much Eliezer in particular gets accused of endorsing violence measures to prevent AI doom, in spite of his disclaiming that very publicly. However vocal he is about disclaiming his support of violent measures, it's clearly not vocal *enough* to prevent people from reading him as being in support.
> in spite of his disclaiming that very publicly.
The man talks out of both sides of his mouth. He says he never said "bombing datacenters," but DID say "destroy a rogue datacenter by airstrike."
He never advocated for the use for force against enemies. But DID advocate for credibly threatening to use force so they comply to your demands and you don't need to actually follow through. Also, that doesn't count as "violence" as long as the violence is predictable. https://www.lesswrong.com/posts/5CfBDiQNg9upfipWk/only-law-can-prevent-extinction
You can see why one who isn't already favorably inclined towards him might not appreciate the subtle distinctions.
To be fair, one could interpret it as a sort of "violence" to use force to enforce treaty mandated controls on AI research, but I think it more or less *requires* already being negatively disposed towards him, or trusting people who're negatively disposed towards him to report what he means, to interpret it as such. By the same token, we could call it "advocating violence" for a person to speak in support of nuclear nonproliferation treaties. After all, actually enforcing those treaties requires the threat of violence.
Sure, and it's not violence if my friend Tony breaks your kneecaps with a baseball bat, and it's anti-Italian discrimination to suggest it is.
I think it's more than a little disingenuous to suggest that having a legal treaty to disallow something, which is theoretically enforceable via violence, equates to "advocating violence." By the same token, advocating for essentially any law at all equates to advocating violence, because all laws are ultimately backed by government force.
Eh, I think Eliezer likes his little fictional fantasy scenarios where he is the mysterious sage figure of giant authority and wisdom, and sometimes he talks them out loud, but he doesn't really mean "call in the airstrikes", it's just more of the fictional interludes in The Sequences about the Sekrit Gnostic Training Of Leaders bit where the Ancient Sage *could* airstrike, via mysterious sekrit Gnostic techniques that led to him having political authority amongst the ostensible leadership to bring this about. Ancient Sage is not the king, but rather Richelieu or Talleyrand who manipulate kings and emperors.
Actually international treaties are an interesting case. According to Opus 4.7 “the NPT itself contains no built-in penalties. There’s no clause that says “if you violate this, X happens.” Enforcement is indirect, political, and widely considered the treaty’s weakest feature.” And DPRK violated it and suffered no violence.
This is a parallel to how we enforce the treaties and international norms against development of WMDs. We destroy a rogue nation's nuclear research facility by airstrike. One presumes we would also destroy a datacenter in the same way.
The problem with "bombing datacenters" vs "destroy via airstrike" is that the former can be said out of context to suggest the destruction is being done by rogue actors whereas "destroy via airstrike" very clearly implies it's the actions of a state acting in some capacity that is recognizable within the system of international law and norms. "Yud says we should bomb datacenters" could be (and has been) said to imply that he thinks normies like me who oppose AI should be bombing them, when he explicitly does not think that would be effective or correct.
If you object even to the use of airstrikes against a rogue nation to enforce the conditions of a treaty by destroying the site where they're researching dangerous technology, then your real problem is that AI is being analogized to nuclear/WMD control in the first place, not with the words.
Given his beliefs, it's perfectly reasonable for him to advocate for violence, sure. There are plenty of people I'd like great violence perpetrated upon as well, and out of purely personal animus without even the pretense that it's to prevent the end of the world/human civilization or whatever.
My objection is only to the duplicity in claiming he DIDN'T call for violence, when he very clearly did, just with some aesthetic preference about what it looks like.
I was a prosecutor for 15 years, I would not have described it as "calling for violence" on the several thousand occasions I requested an arrest warrant. But all state action is either violence or the threat of violence, certainly *some* of the people whose arrest I demanded did not just quietly submit. Yud's point is that it's "predictable, avoidable violence" in both the case of my arrest warrants and the enforcement of non-proliferation regimes, the standards are clearly articulated and the use of force visibly telegraphed in advance, actually having to DO the violence is the undesired but completely avoidable consequence.
You and others keep doing this thing of asserting that anyone who believes AI will destroy the species ought to be supporting "violence" writ large, because you know the public doesn't like that and it would discredit the anti-AI movement. Yud and others on our side keep repeatedly pointing out that even under a strictly consequentialist analysis, acts of violence are not going to result in avoiding the thing without state power (or something equivalent thereto) behind it, and that they don't support that, but you're very insistent on abbreviating this position to the overly broad "justifies violence" as a rhetorical move to frame opposition to AI in a way you think is easier for people to dismiss.
So a "rouge nation" is one that doesn't agree to the treaty you and a few others wrote?
It's justifiable by selfish self-interest, but it's not justifiable by moral or ethical argument except, perhaps, "nature red in tooth and claw".
No, a "rouge nation" has red *cheeks*.
Though I grit my teeth to say it, advocating violence by a nation, in its war-making capacity, is distinct from advocating violence by non-national actors. And "destroy a rogue datacenter by airstrike." sounds much more like the former than like the latter.
( As someone who wants to _see_ AGI, I don't support this use of war making capacity, or, for that matter, any flavor of 'pause', though I think the testing transparency proposal like NY's RAISE act are reasonable. )
Zizians, anyone?
Zizians seem like a good example of how you can't keep sufficiently determined crazies away from your movement.
The connection between Zizians and the rationalist community was that (1) Ziz attended a CFAR workshop, or maybe multiple workshops, and (2) Ziz recruited among the members of the rationalist community.
How could this be prevented? In theory, CFAR could do a psychological screening of every potential workshop participant -- but that seems excessive, and as far as I know other similar organizations don't do that. Afterwards, you can't prevent a crazy person from living in the Bay Area, or contacting people in the community.
In 2019, Zizians protested *against* CFAR, and had police called on them.
And yet, when in 2022, Zizians started to murder people, Wikipedia in its infinite wisdom decided to call them "rationalists".
In other words, once a crazy person participates in your workshop, even the fact that they publicly declared that they hate you does not prevent Wikipedia to keep describing them as one of you if they do something bad a few years later.
(As an analogy, imagine if the Wikipedia page on Donald Trump started by calling him a Democrat. That would actually make more sense, given his 2001–2009 membership. Ziz only participated at a CFAR workshop, which apparently makes them a rationalist forever.)
I see you have never, ever gone to a political protest of any kind -- if you do, you will surely find a few completely out-to-lunch loonies who were never invited, but somehow show up.
Of course.
It's just not that overridingly hard to deal with.
It's a necessary cost of activism but as Nathan reports it is not a particularly expensive one in time or money. You will take reputational damage sometimes. You invest some in background culture and individual responses to reduce that damage.
Context: I volunteer full-time for Pause AI but rather randomly am also an old friend of someone who went on to found Rising Up and Extinction Rebellion.
I have attended a few protests of various sizes (including a very small one I organized myself), and didn't directly run into any problematic loonies. I'm sure they show up often enough to need a plan for them, but obviously not even every time, and again, this is a manageable problem.
If you think protesting is bad or something, you should just say that, instead of inventing reasons to not want to do it.
I don't think protesting is bad, and I'm not inventing reasons not to do it. I'm just pointing out that it is in no way "easy" to avoid having loonies show up to your protest once it reaches any reasonable scale. No Kings, Women's March, March for Science, March for our Lives, BLM, etc all 100% have complete crazies (hopefully off to the side) who show up, and you do need to figure out a way to deal with that instead of just saying it's "easy" and not a problem.
Scott is equivocating between the obvious evil of working for a company that is trying to build something that is likely to kill everyone, and forming broad coalitions that include distasteful people in the shared interest of protecting humanity. He needlessly tacked on the bit about how hard it is to keep the crazies out as a way of communicating his distaste for activism.
Clown management is a real thing you have to attend to. That doesn't make it a giant unmanageable issue that should dissuade people from being involved. Buying a car means spending some time on car maintenance. It doesn't mean spending an "inordinate" amount of time on car maintenance. That kind of language (along with calling pithy and useful slogans "vapid") is meant to dissuade, not to clearly communicate what you're in for.
Most people who say these things (most definitely Scott included) simply believe that activism is a low-status activity, and they cower in fear to think of the personal social consequences of associating with ordinary people and empowering them to improve the state of the world.
That is what I am arguing against here. Snobbery and cowardice, and the undue magnification of real but manageable problems as a way of hiding that cowardice.
There's an entire social media formula where you go to any large political gathering, find the loonies, and then get them on camera doing or saying loonie things for everyone to gawk at. It works for a reason. There are lots of weirdos in any sufficiently large gathering that is open to the public - even more so when there's some kind of emotionally charged topic involved. The mass movement is necessary, but the snobbery is warranted. The type of person who likes having intellectual debates about complex topics usually finds it a bit embarrassing to stand in a mob and repeatedly chant "Go Team!". Society needs someone to change diapers at the old folks home, but that doesn't make it enjoyable or glamorous.
And if they don't show up, there will be a few undercover members of the opposition.
Or the government!
"It actually takes very little resources to keep the crazies out, in both absolute and relative terms."
I was just listening to the morning talk show on local radio where members of Sinn Féin were being questioned on, amongst other things, participating in the recent fuel protests here and asked about standing on platforms with guys from new, far-right parties and guys with convictions for animal abuse.
It's tougher than you make out to keep the undesirables away.
There's probably a difference between people who are aiming at violence right now and people with unsavory histories.
The Christian belief in a Holy Ghost based gut instinct is inseparable from vibes. Rationalists are realizing you simply cannot make the right rational calculated choice on a dime, these things take great intellectual work!
It's never clear whether you're improving the overall works of the newly founded socialist workers party or whether you neeeever should have gotten involved.
Shouldn't have to say same for AI safety lab association, one should see the general problem.
Great writing as always!
Can you explain the contrast between "takes great intellectual work" and "should be done on vibes and can't be rationally calculated"?
I don't know what plasma had in mind, but one might look at virtue ethics as suggesting to cultivate the good aspects of being by great practice and reflection and intellectual engagement *so that* one's vibes and acts are good.
I still think "virtue ethics" acts as some sort of uh, virtue signal, where it never grounds itself in any particular position, but gets to feel smugly superior to everyone else because they're not practicing "virtue ethics", the vague practice of doing everything right with no effort (or, if you decide effort is good, lots of effort, but it's somehow magically different from and better than everyone else's effort). I have tried to read books on virtue ethics and I still can't figure out how it isn't this.
I suspect this is a matter of sampling bias due to your calm-down-and-be-reasonable aura. Virtue ethics is the decision-theory equivalent of those IRS computers that still use COBOL, or https://thecodelesscode.com/case/234 It's optimized for providing quick answers under conditions of high stress and low working memory, so whenever consequentialism has enough swap space to give an answer at all, it'll give a better answer, and leave virtue-based responses looking half-baked. Feels like a weird edge case to you, but many people live under "no room to think" conditions more or less full-time.
As for the paradoxical amounts of effort involved, consider AI training runs vs. inference. Cultivating the proper virtues might be a laborious process over many years, involving self-examination, digesting standard datasets, and personalized feedback from peers or mentors; then, habits having been fixed, applying them in the moment needs little or no conscious attention.
Tangent:
(1) I've been interested in the implicit model of divine action present in the Homeric epics.
The background mythology includes some very overt acts, like turning people into stone. But most actions attributed to the gods are extremely reasonable. The gods do things like inflicting disease, guiding projectiles to strike or miss a target, and speaking to people in a manner that other nearby people don't hear/notice.
(2) There's a scene from the book Taran Wanderer that's always stayed with me.
Taran is a foundling, and he has gone off into the world to try to learn where he came from. He meets a farmer, Craddoc, who claims to be Taran's father, having given him up long ago for reasons. We, the readers, learn that Craddoc is lying, but Taran joins his household and helps operate the farm.
But he's frustrated. The farm feels like a dead end to him. He wants to be adventuring.
One day Craddoc falls down the side of a ravine and is badly injured. Taran is presented with the choice of attempting to rescue him, or not. It's dangerous.
The book tells us that Taran considers the fact that, if he simply abandons Craddoc to die in the snow, it will solve all of his problems, and no one will have a word to say against that decision. There's no sense in killing yourself trying to rescue someone from certain death. And it says: 𝘏𝘦 𝘴𝘦𝘦𝘮𝘦𝘥 𝘵𝘰 𝘩𝘦𝘢𝘳 𝘩𝘪𝘴 𝘰𝘸𝘯 𝘷𝘰𝘪𝘤𝘦 𝘴𝘱𝘦𝘢𝘬 𝘵𝘩𝘦𝘴𝘦 𝘸𝘰𝘳𝘥𝘴, 𝘢𝘯𝘥 𝘩𝘦 𝘭𝘪𝘴𝘵𝘦𝘯𝘦𝘥 𝘪𝘯 𝘴𝘩𝘢𝘮𝘦 𝘢𝘯𝘥 𝘩𝘰𝘳𝘳𝘰𝘳.
He ultimately decides to make the attempt, because he doesn't want to be the kind of man who makes the other choice.
---
Something that I wonder about in relation to this is that epic belief in gods speaking to you in a way that no one else can hear. I can imagine a culture in which the antibody against Taran making the selfish choice relies less on him acknowledging that, even though that choice has obvious benefits and no obvious drawbacks, he should nevertheless sacrifice himself... and more on Taran recognizing that those unwelcome thoughts in his head aren't 𝘩𝘪𝘴 thoughts at all, but rather the lying words of an evil god attempting to trick him.
Maybe rejecting antisocial ideas is easier if you have a scapegoat handy to blame them on.
I've only seen summaries of Julian Jaynes's "The Origin of Consciousness in the Breakdown of the Bicameral Mind", but what you describe sounds similar to what I recall of those summaries. You might find it interesting, albeit controversial.
I think* "on a dime" is an expression that means something like "trivially"?
(* Disclaimer: I'm not an American and mostly deducing this from context)
If so, Plasmarob didn't say "should be done on vibes" or "can't be calculated", but rather "*is* done on vibes" (by Christians who're following their conscience) and "can't be *trivially* calculated" (by rationalists who are trying to scratch-derive ethical principles). And I think in both of these things Plasmarob was agreeing with Scott's existing position?
https://en.wiktionary.org/wiki/on_a_dime
> (US, idiomatic) Within a very short distance; especially, with agile precision in a very confined space and suddenly.
Oh sure!
You nodded to Christian strategy.
I am saying there will be times where you are in a gun-to-the-head position that tests your deontologic bar and you do not have time to make a rational calculation in the way we do in online long form content hypotheticals.
Some hard problems like you describe with AI take great intellectual work but and AI can show up at any time and ask you a question! Some things have to be done on vibes and we can only rationally calculate if our decision was best later.
The proper imo early Christian strategy is by feeling, moral vibe.
three (ok 4) examples - politics, ai, and practical home emergency:
1) let's say you are a partisan invited to visit a world leader you consider doing a great evil that will cause many people to die. While visiting, to your astonishment someone attempts to slip you a weapon and says "do it". What do you do? People will be watching in less than 4 seconds. You may die or be in prison forever. Zero time to react. Real trolley problems don't give you a day to debate it.
2) AI. Once you donate money to a tech company you think is less likely to create evil, there's probably no takebacksies. How long is sufficient to wait to see about that 80% or 80% odds? 2B) let's say you are training an AI with every input you make. How much time should you spend determining the long term consequences of each prompt or each data input? The answer isn't zero but how rational and how crystallized does your model of deontologic bars need to be? (with what you won't ask an AI to do).
3) One common family encounter is medium-level injuries. Every parent will eventually have to determine whether it is worth calling 911 for an ambulance ride for their child. I've seen families bankrupted from how often they do this. If your child hits their head and blood starts coming out, how much blood is too much? I was surprised how much blood is not actually an issue, I have worked through this with a nurse nearby. You don't want to wreck your family's future with unneeded expense insurance doesn't fully cover but this is your child and their life could be at stake! How bad is enough for 911?
All deontologic bars eventually to be broken. Especially not killing, not just in war, but because if someone armed breaks into your child's bedroom window, and this scenario happens in America plenty, there is simply no calling the police to solve my moral problem.
It's a time problem, and every scenario is different. Deontologic bars are rules of thumb. Maybe some are moral principles, but "do the right thing" can vary more than we appreciate, and in the moment all you have to call down is a prayer and your own character and experience.
"a Holy Ghost based gut instinct"
I don't quite think that's the theology of divine inspiration by the Holy Spirit, but I'm no expert.
I think it's a pretty fair way of describing the Catholic view of how conscience works? An automatic built-in instinct that's divinely inspired?
(I'm not an expert either but one can read it directly in the Catholic Catechism, part three, sections 1777 to 1782. Say what you will about Catholics, they definitely have by far the best API documentation. https://www.vatican.va/content/catechism/en/part_three/section_one/chapter_one/article_6/i_the_judgment_of_conscience.html )
While oversimplified, the way *you've* described conscience is fair; the mistake is conflating it with acts of the Holy Spirit, which does not necessarily play a role in conscientious discernment.
The immediately following paragraphs in the Catechism focus on the proper formation of conscience. This foregrounded concept of formation is diametrically opposite to "automatic" and "built-in".
It really depends on which theologian you are quoting. Remember "masturbation is the sin against the Holy Ghost", but others has used it as a quite different meaning. This is what happens with an ungrounded metaphor.
The correct approach is to ignore all of this and do the right thing in every case, which absolutely does not include anything stupid like unilateral military disarmament.
I have no opinion on whether constraint utilitarianism can be coherently formulated, but it too is wrong, like (regular) utilitarianism and deontology, which it tries to hybridize.
But the whole point is that it's hard to know what the right thing is. Even if you're a deontologist, different people have different deontologies, so who's right? If your claim is that people should just follow their particular deontology, that can justify basically anything, so we're back at square one.
I agree that that knowing what the right thing is hard. My point is that neither utilitarianism, nor deontology, nor a hybrid of the two like constraint utilitarianism, gives a correct answer to that problem.
You said: "The correct approach is to ignore all of this and do the right thing in every case." My question is, How would one know what the right thing is to follow this?
Pick whichever philosophy seems most right to you, and if ever it stops seeming right, ignore it in that case and do what seems right.
Many horrific things have been done in the name of consistency, when even simple instinct, basic decency, and common sense point in the opposite direction. Morality is not a solvable problem, so don't ignore every other heuristic in favor of just one!
Many horrific things have also happened because people were just doing what seemed right. There was a time when slavery seemed so to most. It's thanks to the people who questioned those instincts that we've moved past that.
Point taken that moral innovation is important. (That said, nah, people pretty much knew slavery was bad at the time. The slaves visibly didn't like it.)
When it comes to your day-to-day life, you are mostly already just doing whatever seems right to you. If you get convinced that something else is even more right, then great! But for people who live their lives by hard-and-fast rules with no deviation, moral innovation is impossible.
>How would one know what the right thing is to follow this?
By using their entire capacity for reason. Of course, that still may not suffice, but a simple shortcut certainly can't suffice in every case. If there was such a shortcut, millennia of moral philosophical effort would have found it.
Reason alone CANNOT define right actions. You need a set of axioms to base that reasoning on. The problem is that peoples "sets of axioms" generally favor their own self interest. So organizations try to replace those sets of axioms with different ones that favor the organization's interest.
How would you determine the right axioms, except through reason?
What you can a shortcut, others might call taking advantage of insight distilled from those millenia of philosophical effort. It doesn't excuse less thought but it would be suboptimal to try to do everything from scratch.
Yes, I don't endorse the original commenters formulation of "ignore all of this". Those shortcuts have their proper place, and, if nothing else, that millennia of philosophical effort proved inadequate for completely solving ethics should be a strong precaution against embracing claims of massive novel breakthroughs too readily.
Is there any reason to believe that any alternative to them does?
Oh great, just "do the right thing". This is like saying "There's no need to learn calculus, just put down the right answer to every problem on the calculus test."
Scott, I was literally quoting you with that phrase (final paragraph of your post). It would not have been my wording.
Yes, but I tried to derive it! Tell me which of these is the right thing, and how you know:
- Military disarmament
- Refusing to participate in mass politics
- Refusing to work for an AI company
- Assassinating a bad US president
- Assassinating Hitler
Well ... one version of "just do the right thing" is: "Morality" is just a set of quick & dirty heuristics. The hope is to find a set of simple rules that are easy to understand, that are consistent, that you can apply to all situations, and expect that it pretty much quickly guides you to behavior that you won't later regret. But all actual philosophical attempts at constructing such a code of ethics seem to wind up with inconsistencies, ambiguities, or at least an opposition philosopher can construct a disturbing hypothetical corner case where your moral code doesn't seem appealing after all.
But you can just abandon the attempt to find a simple uniform easily understood set of universal rules. Instead, just accept that each situation is unique, and it's easily possible that human preferences are not 100% consistent.
What you do instead is simple decision theory. At every decision point, look at the actions you have available, estimate the likely consequences of each action and the probability mass across the possible worlds that would result, and choose the action that maximizes your expected utility or general happiness with whatever world(s) would likely result. Stop searching for the heuristic summary of "doing the right thing", but instead just make a complete evaluation of each unique case from (your) first principles.
Maybe there's too much uncertainty to make a confident decision. Maybe you don't know what your preference is between the different sets of possible future worlds. In that case, don't worry about it! But when you clearly WOULD prefer one probability mass over another, then just choose that action in that moment.
And don't worry about whether your full set of decision choices over your lifetime, is easily described by a simple list of logically coherent rules.
So, something like act utilitarianism?
Very close! But any time you try to summarize all decisions with a simple sentence, you're vulnerable to strange hypotheticals. Act Utilitarianism generally seeks the "greatest net happiness" -- but what if you refuse to aggregate, and think that the happiness of two distinct individuals is incommensurable? Or perhaps you feel that you and your family are more important (for your own actions) than the welfare of strangers? Or perhaps you're after "pleasure for the greatest number of people" ... but then you fall into the Repugnant Conclusion?
So I was actually attempting to be serious about: Don't try to summarize the morality in a small paragraph. Don't even seek global consistency. Just evaluate each scenario in isolation, and do your own decision theory calculation for that specific scenario.
Going by the virtue-ethics system from the Exalted RPG (which is, to be clear, definitely not what I use in real life, but happens to be conveniently well-defined for illustrative purposes)...
- unilateral military disarmament goes against Valor and possibly Conviction
- ...but negotiated *mutual* disarmament, or unilateral de-escalation when not in immediate existential danger, is supported by Temperance
- deliberate political use of misinformation clearly goes against Temperance
- breaking an oath goes against Temperance, and abandoning an ally in need goes against Conviction, but participating in (or up-close passively observing / tolerating) atrocities goes against Compassion, so getting a job as a concentration-camp guard, or doomsday-bomb factory worker, or otherwise entering into a situation where you'll inevitably need to either be directly complicit in horrible things or quit on short notice, is setting yourself up to fail, so don't even start on that road (harm theoretically prevented relative to someone else doing it instead isn't even considered as a potential factor)
- assassinating the leader of your own country is treason, thus solidly against both Conviction and Temperance
- ...but if Compassion and/or Valor say his cruelty and incompetence are intolerable, removal should be pursued through the proper legal channels
- assassinating the leader of a country you're currently at war with is likely supported by Conviction, plus Valor if the plan involves daring heroic risks, plus Compassion if the enemy in question is notably complicit in atrocities
Definitely not perfect, but it seems to have the basics covered.
I've gotten dinged for not showing my work in math class.
The difference is we know calculus works, whereas there's no ethical analog known to work.
How about "We know calculus works in properly defined scenarios."? It's quite easy to misapply it.
You can give three people who have almost nothing else in common the same calculus problem and, if two of them come up with identical answers while the third disagrees, be very confident that the third has made some mistake which they'll recant as soon as it's pointed out. Is there an ethical framework anywhere near as clear and consistent?
> You can give three people who have almost nothing else in common the same calculus problem and, if two of them come up with identical answers while the third disagrees, be very confident that the third has made some mistake which they'll recant as soon as it's pointed out.
This isn't true at all. It's not difficult to set up a problem that plays into a common mistake. At that point, whether you get more agreement on the correct answer or "the" incorrect answer is fully dependent on which three people you select. You can always choose people who will make the mistake. You'll know in advance that they will.
I’ve noticed that utilitarians (and to some extent other consequentialists) seem to think that ethics is a solvable problem which could be discovered as calculus could be discovered. And therefore an isolated demand for rigour appears: find a correct way of deciding ethical decisions before taking them or justifying them. But there’s no reason ISTM to suppose that ethics is solvable in that way, and that anybody has ‘learned ethics’ in the sense that people have learned calculus. In the absence of the existence of calculus ‘just put down the right answer, having learned from the relative success or failure of those who’ve answered it before’ is a good approach.
I'm happy to work with Bannon, Sanders, NIMBYs and anyone else if it helps prevent human extinction -- because that's a more important issue than any other possible issue: whatever issues someone cares about, they are all null and void if there are no humans and therefore no human caringness in the universe.
It will certainly be necessary to work with the Chinese Communist Party, and probably further down the line Vladimir Putin, Kim Jong Un and the IRGC. And that too is clearly worth it. In a hypothetical where it was genuinely necessary to work with Pol Pot to avoid extinction, I'd do that (though in practice a regime change war would probably be preferable there).
It's funny because - this is something few people know - America did work with Pol Pot to counter the Vietnamese. Before his death he said "when I die, my only wish is that Cambodia remain Cambodia and belong to the West. It is over for communism, and I want to stress that."
You need something more complicated than this, otherwise it permits assassinations, etc.
Everything has to be decided on a case-by-case basis, erring towards wisdom and caution. For example, Yudkowsky has said he thinks that in extremis, he is OK with a pro-AI-safety coalition bombing the data centers of countries that refuse to comply, and I agree.
Things that humans value are only valued if there are humans. Without humans, quite possibly nothing in the universe has any value at all. This is worth minus infinity utilons, therefore (almost) anything else is better.
To elaborate, consider whether to assassinate Sam Altman. I am against this.
Why? Not because it is intrinsically immoral. How could it be, when there is no true morality? (https://pontifex.substack.com/p/there-is-no-true-morality) and therefore nothing is intrinsically moral or immoral.
But, most people are against murder, so assassinating people looks bad, so it would weaken the cause and thus be a bad tactic.
Humans have an intuitive sense of physics, that is useful to them as a starting point in intuiting physics problems, even if it does not go all the way to solving them. Similarly, morality is intuitive game theory, and is useful as a starting point in intuiting co-ordination problems between (semi-)rational actors, even if on its own it cannot solve them.
>Everything has to be decided on a case-by-case basis, erring towards wisdom and caution. For example, Yudkowsky has said he thinks that in extremis, he is OK with a pro-AI-safety coalition bombing the data centers of countries that refuse to comply, and I agree.
You think bombing Chinese datacenters, and very likely starting WW3 in return, is "erring towards wisdom and caution"?
Yes. WW3 is not as bad as human extinction.
FWIW, I, personally, disagree. I value the intellectual triumphs of humanity, and I think these likely to survive ASIs taking over, while I think there are pretty high odds of losing them forever in a WW3. Naturally, I would _prefer_ something like <evidenceFromFiction> Culture Minds keeping humans as pets </evidenceFromFiction> over both these scenarios.
WWIII would almost certainly kill under half of humans, and Wikipedia would certainly survive on hard drives in EMP shields. There's nothing permanent or even particularly long-lasting about its effects on humanity.
> he is OK with a pro-AI-safety coalition bombing the data centers of countries that refuse to comply, and I agree.
So Yudkowsky and you are choosing global thermonuclear war over AI, because global thermonuclear war would have more survivors ?
I cannot speak for Yudkowsky, but I can speak for me. IMO, unaligned ASI would likely kill everyone and is therefore far worse than nuclear war.
There are ways you could limit the amount of destruction against the holdout nation (e.g. China). You could make sure that in your first strike you only use a small part of your missiles, that you only strike data centers and try to limit the damage to cities etc. You can make clear that if they make a retaliatory strike, their entire country will be wiped out, but that if they don’t, they will be given lots of help to rebuild their damaged infrastructure (sans AI data centers, of course).
In a way it’s a shame we’re facing this problem now and not in say the 1990s when liberal democracy was undoubtedly on top and where it would have been easier for the important countries to agree. However, we are where we are, and saying “I wouldn’t start from here” is not a plan.
So it sounds like the answer to my question is "yes": you are choosing to perpetrate global thermonuclear war, but you think that you can do so in a way that mutually assured destruction will not apply.
I understand that your plan makes sense in your worldview; but can you envision what someone who is not already 100% on board with your cause would think of this plan ?
He didn't say the civilization would survive. Just that (human?) life would.
If you accept his belief in the result of ASI, then it's a reasonable decision. I tend to put the probability at less than 40%.
In point of fact, I think it's probably possible to avoid both, but sure, the metaphorical end of the world sure beats the literal.
>but can you envision what someone who is not already 100% on board with your cause would think of this plan?
This doesn't really matter for someone who subscribes to the Yudkowskian view of neural nets. You cannot bargain with reality; you do what is necessary, or you die. The willingness to go to war to enforce the ban is a necessary component of making it work, and the ban working is the only way to avoid us all dying. We clearly have some work to do to create that willingness, but there don't seem to be any better options available.
> "Everything has to be decided on a case-by-case basis, erring towards wisdom and caution."
I'm saying this is the kind of act utilitarianism I don't trust, or at least it degenerates into this without some kind of very strong operationalization of "wisdom and caution", which is what I'm trying to discuss here.
I believe Yud's recent remark on this topic was, to paraphrase, something like "consequentialists are right, but the vast majority of people aren't smart enough to BE consequentialists and get lots of stuff wrong so they should just be deontologists instead". Not consequentialists with a deontological bar, but instead just plain deontologists out of humility and caution I suppose. Some guy at the RAND corporation can do the consequentialist stuff about how many megadeaths from this or that nuclear scenario, the kind of people who decide whether militaries exist can do that, but Joe Everyman shouldn't try to assassinate a candidate because it would be bad if that were common. Ethics doesn't really need to universalize.
For my part I just try to act to be the person I want to be, various ethical rules and modes of analysis are just there to be quick heuristics and guardrails but any of them are breakable in extremis. (Although some rules become part of one's identity over time.) Facing a mortal threat or a complete destruction of my dignity and self-image, I might break nearly any of them but almost certainly along a vector I already considered an acceptable part of my self.
Saying "everything has to be decided on a case-by-case basis" is obviously correct, but gives no hint to the basis of decision. It's also true of every math problem...but the basis of the decision there comes from outside context. (E.g. is it multiplication of integers or matricies?)
So "everything has to be decided on a case-by-case basis" is quite insufficient.
No one needs to have a consistent moral philosophy. You can just have preferences and strategies and do things. Yes, work with Bannon and Sanders. No, don't permit assassinations. Ta-da! Solved. If you have more confusions, you can ask more questions, until we collectively have an intuition about what is acceptable and what is unacceptable. But most of it is covered by "don't be an idiot" and "don't be terrible."
Especially in politics. Hypocrisy is pretty much baked in.
The moral philosopher Jonathan Dancy (father of Hugh, the actor) is known for his view of "moral particularism" under which there is no general rule. He uses the analogy of humor, saying there's no general rule for what's funny. https://www.youtube.com/watch?v=qHH9WiHNhv4
Why stop there? Allow assassinations of bad guys. Don't allow assassinations of good guys. Ta-da even more!
But everyone will just use motivated reasoning to say "my side are the good guys, my opponents are the bad guys."
Well, as the other guy said, that sounds like a confusion that you can ask more questions about, until we collectively have an intuition about which side is the good guys and which side is the bad guys.
Yes.
On the whole, I think killing people is bad, therefore one should very much err on the side of caution in doing so.
Ah, but how do we decide which questions are legitimate, and which are not legitimate? The problem is that people have an endless amount of preferences.
I feel most violence is blocked by simple consequentialism, it is rarely an effective strategy, the violent actor is rarely going to have a monopoly on violence so increasing violence even if well directed makes things worse.
I think your comment is a good illustration of why one should clamp one's expected value of projected outcomes to somewhere around continental scale, at most (and ideally much lower than that). Otherwise, you end up saying things like "because I am laboring the entire humanity/Solar system/Universe, no cause can be weightier than mine, and thus my ends justify any means". Mathematically, this is a perfectly valid conclusion. Empirically, it never, ever brings about net-positive results.
In many situations I would agree. However unaligned AI is a problem like no other humans have ever faced. Normal rules cannot apply.
Yes, of course. And that's what everyone else says too. Everyone's got some kind of a pet cause that is the biggest problem humans have ever faced, be it AI or sin or even just Trump. And it's easy to say, "but of course all those other problems are fake over overblown, mine is the only real one". Everyone says that too. That's why you need that expected-value-clamping heurisitc.
Self evidently, human extinction is a worse problem than sin or trump: sin has been around for as long as humans have, and so have bad leaders, and neither has caused human extinction. Now, with AI, humans have the power to cause our extinction, which we didn’t have before.
> Self evidently, human extinction is a worse problem than sin or trump...
Whoa, "self evidently" to whom ? If I were a Judeo-Christian theist who believed in Hell, I'd say that sin is a way worse problem. When AI kills you, if you are sinless then you'll go to Heaven and experience eternal bliss. Humanity cannot in fact die out, because this Earthly life is just a prelude to the eternity. But if you are sinful, then you will be eternally tortured in Hell forever ! This is much worse than merely shortening your already infinitesimal stay here on Earth by a few years; it's even worse than ceasing to exist; it's a literally infinite negative utility ! Surely, you must stop wasting your time on AI, and repent !
That’s not how Christian theology works. The way it works is all are sinners but if you sincerely ask God for forgiveness, you’ll be forgiven. Being without sin is simply impossible for humans.
Ramez Naam doesn't think so, and he successfully predicted the rapidity of solar power's exponential takeoff further in advance than anyone I'm aware of, certainly far better than any of the official government-backed projections.
You jump back and forth between decisions made as individuals and decisions made as groups (political parties, Ukraine) here a few times, and there are cases where this distinction importantly matters.
For the military disbanding, there's two more arguments why it's bad. One is that "no military at all" is an unstable equilibrium - the first country to get one will conquer the world - but we have a better alternative where countries can arm up as needed (but no further) which is both more stable and converges to peace.
The other is that when you universalize a principle, you universalize it more for people similar to you. People in your culture are likely to reach the same conclusions as you (either because you convince them or because you think similarly), so you can successfully hit slightly more unstable equilibria with them. Trying to sync with people in a different country or culture is harder, and the more difference it is the less ambitious you can be. Coordinating with your clone is trivial, coordinating with your countrymen is doable, coordinating with another country is harder, coordinating with unknown aliens is basically impossible (hence AI alignment being hard!). And "no army" is the third of these.
Good comment
> coordinating with unknown aliens is basically impossible
Interesting enough, Heinlein's Starship Troopers has a classroom discussion of how that could be possible as the next step after humans are all cooperating, despite the impression people get from the Verhoeven film that it's all xenophobic militarism.
>You jump back and forth between decisions made as individuals and decisions made as groups
This is a long-term tension in Scott's writings about EA as well.
Is there a difference between constraint consequentialism and rule utilitarianism? If not, why not use the standard term, which is rule utilitarianism? And why not look at the literature of rule utilitarianism and see what it has to say about what kinds of rules are good rules?
Rule utilitarianism means you should always follow rules and stop worrying about what has good consequences (outside the rules). Constraint consequentialism says you should try to do things that have good consequences, unless there happens to be a rule in the way.
Thanks, Scott. I don't think rule utilitarianism says to stop worrying about what has good consequences. That would be deontology. Rule utilitarianism tries to pick rules that will lead to maximum utility. Rule utilitarianism would in theory not worry about violating rights if the rule maximizes utility. Constraint consequentialism seems like consequentialism with a deontology wrapping. I would argue that that's just consequentialism in practice.
Deontologists say certain things are intrinsically good regardless of their effects on overall human happiness because they make God happy. Rule utilitarians say you should stop worrying about whether the rules are actually good because that makes the rules less effective towards the end of overall human happiness.
Atheistic deontology is possible.
I don't see how such a philosophy could be coherent.
Deontology (at least the categorical imperative flavor) is functionally equivalent to rule utilitarianism; no necessity to multiply entities by introducing a deity.
"I don't see how such a philosophy could be coherent."
Then you probably haven't read much libertarian philosophy
>Deontologists say certain things are intrinsically good regardless of their effects on overall human happiness because they make God happy
Not necessarily, eg. Kant.
Or Nozick, Rand, Rothbard, etc.
Rule utilitarianism, can and does have differences from pure utilitarianism. A pure utilitarian would push the fat man to save just two people. A rule utilitarian might refuse to , to save any number of people , or require some higher threshold to break the rule. (Rule universalism probably implies non absolute deontology).
One thing that kind of confuses me is, why add non-consequentialist rules if not for assuming it somehow results in better consequences? And wouldn't that make the rules themselves consequentialist? Is this distinction a matter of "mostly easy to track consequences" vs "here's a rule and we assume it is good but there's no way we can actually know"?
OK, nevermind, I looked it up and there is a difference. Sorry, I should have done that before commenting. Rule utilitarianism seeks to select rules that would maximize utility. Constraint consequentialism is act consequentialism with non-consequentialist rules applied on top.
seems wrong to me.
I would throw out the disarmament example. First, I don't accept that all wars of conquest are immoral. Second, norms have to be enforced eg by armies, I know it's a thought experiment but it's hard to even spell out the magic assumptions required for it to work.
What's your rule for deciding if a war of conquest is immoral or not?
if it will have good consequences of course :)
The ends justify the means
<mildSnark>
Всё зависит от того, победим ли мы.
Vso zavisit ot togo, pobedim li my.
</mildSnark>
I am just starting to read Kant, so this comment is attempting to start a conversation moreso than attempting to give a complete answer (via Cunningham's law).
> A common formulation is “act as if your maxim would become a general law”. This correctly rederives the bar against assassination, but fails even in some simple cases. For example, should the Ukraine abolish its military?
This is Kant's 'Formula of Universal Law'. I think it's meant to serve as a filter. That is, it can only rule out maxims, but not confirm that they are actually correct. It is just one of many filters that Kant has. I'm not sure which of these other filters might stop the Ukraine problem, though.
To get into Kant's answer you could start with his "Perpetual Peace: A Philosophical Sketch". ( https://en.wikipedia.org/wiki/Perpetual_Peace:_A_Philosophical_Sketch )
It's surprisingly sane for a) Kant and b) 1795.
Assuming that your rule about acting as if your reasoning would become a general law is based on the categorical imperative, you are misunderstanding it for Ukraine. The general rule would be something like if I am surrounded by hostile, neighbours or neighbours, which could become hostile in the future or otherwise, likely to need a military to promote good consequences. I should maintain a military. Sure if this was a general law, you would never find yourself in that situation where you have a hostel neighbour, which could become potentially hostile, assuming you’re not the kind of person to become a potentially hostile neighbour yourself, but this doesn’t mean you have to act like that when actually surrounded by potentially hostile neighbours, you can have general rules about things that are universally followed, even if in the world where everyone follows them, that situation never comes up.
Honestly, your second proposal might not even be a real deontological bar. Couldn’t you just think of it as the cost of damaging social rules being great enough to more than cancel out any benefits from assassinations? The outside view definitely doesn’t sound like a deontological bar to me. That’s just basic epistemic determination of probabilities before making a decision. It’s not a deontological bar just because you’re not consciously thinking of maximising stuff. The fact I don’t jump off. A cliff doesn’t become a deontological bar just because I’m following a rule because this is very much a policy which maximise beneficial consequences in the situations where I follow it. Of course, this kind of rule really doesn’t handle edge cases, but that’s why you need a general moral theory. Just because you want to maximise good consequences doesn’t mean you think about that all the time that would waste so much time and effort that you would definitely not be actually the best policy for maximising good consequences. You only think about that regarding situations where the answer isn’t obvious.
Your final suggested rule makes intuitive sense, but of course the problem is that the exact edge cases are unclear since it’s not clear when exactly the other person counts as defecting against you. Does a 90% chance of defecting count,? What about 50% what counts is defecting against you, personally is lying to you through their own articles defecting against you? I also think this rule neglect many of the reasons why you should not be promoting propaganda on your blog. As an example, if you are in Germany under Hitler and your good friend, a member of the SS who is super gullible towards you comes knocking. It still seems like you can lie to him, even if he actually is a very good friend, who is trying to cooperate with you, and the German government isn’t actually defecting against you personally. I think the obvious reasons why the general rule against propaganda doesn’t apply has less to do with defection and more to do with things like lying damages, social trust, and lying to good people harm their ability to achieve their goal which are presumably good and the fact that actual people are not super gullible most of the time, and if you started promoting propaganda, people would read you less and trust you less. This is why it would be fine to lie to your friend because that’s propaganda against somebody who buy stipulation in the thought. Experiment is both evil and gullible and benefit is unusual huge compared to the Small social cost of damaging trust to the single lie instead of a consistent pattern of propaganda that will repeat in the future and will go out to multiple people instead of just one.
Now to be fair, my own approach has certain problems, for example, appears to imply that difficult edge cases are exactly when I should be thinking long and hard about the consequences and yet when I’m condemning people trying to murder Trump, it doesn’t appear to me that I am actually thinking, long and hard, but I think those kind of cases are easy explained because my internal intuition is that it’s just straightforwardly obvious that that will have bad consequences so I don’t think about it hard for the same reason. I don’t think about whether an Apple I throw will reach escape velocity. It just obviously will not.
" The general rule would be something like if I am surrounded by hostile, neighbours or neighbours, which could become hostile in the future or otherwise, likely to need a military to promote good consequences. I should maintain a military. "
I'm nervous about this - why shouldn't the rule about assassination be "don't assassinate someone unless you're in a situation where it would go well and solve all your problems"? At this level of granularity you might as well just go back to consequentialism.
I mean, I actually agree that that’s a perfectly viable rule. Although I think that’s a feature, not a bug. In the real world, the actual reason why you should not go around assassinating people definitely has a lot to do with the fact that it generally does not in fact work out and solve all your problems, and cases where it would in fact, reliably improve the situation like for example, assassinating Stalin actually seem like situations where we want to use assassinations. Of course, that doesn’t actually work like standard deontological rules where you can avoid just thinking about the consequences in trade-off, but firstly again I think that’s a feature not a bug and secondly, I think honestly the categorical imperative while many other people studying it don’t appear to realise this for some reason is basically just functional decision theory using natural language instead of mathematics. Still, it does take care of things like whether to cooperate or not. Since most people are in fact, not cooperate bought but still willing to cooperate with you for mutual benefit while having sophisticated strategies to punish you if you start defecting.
You can of course argue that this doesn’t actually solve the problem you’re looking at, but honestly a strategy of non-specific rules that would be good to be universally adopted. Sounds like a pretty terrible way to come up with the deontological bars as you appear to have noticed in the original post, so I don’t really think that’s much of a loss. The best thing you can say about it is that it encourages cooperation and is self endorsing but otherwise it’s just not a great system and the lack of specificity is a problem and if you get rid of that most moral philosophy will at least arguably pass since they’re meant to be universal. If you adopt version of the categorical imperative, that is basically just functional decision theory which you should do anyways, then you don’t get this problem of basically every possible ethical position being okay, but you should do that anyways because function decision theory is just obviously the kind of theory beneficial to you to adopt.
To be clear, I’m not actually suggesting you assassinate anyone when you really confident it will work out because of things like outside view and the damage to social rules, et cetera. But it’s not actually possible to construct ethical rules with no reference to empirical reality. Otherwise, the rule will turn out to be either really stupid or turn out so detailed that it’s obvious that you are in practice consulting empirical reality before formulating them in which case you should just try to find out the general principles that you are combining with empirical data to formulate the rules.
No, we know for a fact that after he died, his successor were all less crazy, although that partially because they all went through his rain of terror, and therefore understandably, not willing to repeat that that’s why I gave that as an example of a situation where we know it would definitely work out at least in a particular time period. I find it possible it would not work out in the 1920s and even then I suspect given what a disaster he was, it would be beneficial in expectation. To be clear, this is an extreme example, and most political figures probably should not be assassinated, especially if they’re not dictators.
Is that rule bad bc it probably wouldn't solve all your problems? If an assassination actually would solve everything and would only be a net positive what's the reason to not do it? the sub reasons for the deontological bar of no assassinations were practical ones, surely in a hypothetical where there are no practial issues it would be fine, no?
Yeah I was wondering how one decides how vague or specific the rule is, too little specificity and the rule can be misapplied, too much and your desired action becomes the only one the rule allows.
I don't think these two situations are exactly symmetrical. Most people who support slightly-less-reckless AI companies do so because they think some AI company will end up winning no matter what, so they're hoping it'll be the one who's creation is slightly less likely to kill everyone. Here, the lack of a functioning norm is a major factor because it's the whole reason why they make the calculation that one of the AI companies will end up winning and so the best way to minimize risk is to support the less reckless ones. This is similar to the reason why one might support the less evil major-party candidate in an election even if there are non-evil third party candidates to choose from or the option to sit out.
On the other hand, most people who support movements like Pause AI do so because they think the first-order benefits of mobilizing a political response against AI development outweigh the second-order risks of breaking whatever norm exists against partisanship. In this case, the lack of a norm against partisanship is not as great of a factor because they'd likely make this calculation even if such a norm did exist. It's similar to the reason why most people would support assassinating Hitler even in the presence of strong norms against political assassinations.
If a constraint is argued for & justified on consequentialist grounds, it's not really a deontological bar, is it? It's just another complication in our consequentialist calculations. So in a sense the ideal case is obvious: calculate and don't forget the unaesthetic fiddly bits about social coordination.
On the other hand, an intriguing area of investigation is what would make good deontological bars, justified only on deontological grounds. A religious person could take up this challenge effectively I think. As a secular person I don't hold my principles sacred and inviolable in the same way, so there's always imaginable consequences that would render breaking a principle a reasonable choice. But it would feel so much cooler for there to be sacred inviolable principles!
I feel like this is a piece in a general, long, slow multi-year series entitled "Scott realizes that consequentialism doesn't work, tries to salvage parts of it, those don't work either, also realizes that there just generally aren't really answers in philosophy."
What's wrong with consequentialism?
There’s no way to square “AI is a genicidal threat” with “Public assasinations are destabilizing to society” within consequentialism.
What? What does that mean? I am a consequentialist and believe both of those things, but I can’t really begin to explain how I “square” them unless you explain what you think the problem is.
Did you not read the article?
Sure did! Do you have a response to what I wrote, or did you just want to chat about what I’ve read recently?
I don’t understand philosophy well enough to follow the debate on consequentialism, but that was a great comeback.
The problem with consequentialism is that it's like... only a thing in thought experiments. The trolley problem is a fun thing to think about. But if you actually try to make the trolley problem an actionable moral code, you end up with chest-thumping young men talking about how they reserve the right to kill arbitrary numbers of people, and when you try to get away from that, you get stuff like this post.
Outside of thought experiments, the world is too vastly complex with too many second and third and fourth and nth order complications to actually do any of the computation that consequentialism relies on. You can always redo any consequentialist argument to be either, "Yes, we absolutely should assassinate Donald Trump" or "No, we absolutely should not." It doesn't actually provide a guide to behavior.
That doesn't mean it's not a worthy thing to think about. But trying to BE a consequentialist just doesn't work.
Obviously nothing works perfectly, but you also have to answer questions (ie either do a thing, or refuse to do it because it's immoral). The people who never consider the questions and act according to instinct and prejudice aren't just "more advanced" than the people who consider them while gradually refining their views.
There's some evidence that in fact following custom and prejudice can be less risky and damaging (consequentially) than trying to consciously calculate the best thing to do. Pretty much the entire history of the 20th century (if not earlier) tells you what can happen when intellectual ideologues argue for overthrowing traditions of bourgeois morality, etc. The real question is finding the dividing line between liberal experimentation and rejection of established custom and hierarchy on the one hand versus illiberal and totalitarian departures from tradition. It's easier when your "tradition" is basically liberal, as in the United States.
But "traditions of bourgeois morality" themselves represented a massive challenge to longstanding custom and prejudice. The defenders of aristocratic morality made the same conservative arguments against the rise of the liberal bourgeoisie.
Those nightmare-spawning 20th century intellectual ideologues thought they were just the next step in the prosperity-broadening economic, scientific, and political transformations that had followed from the post-Reformation and post-Enlightenment refusals to be limited by custom and prejudice.
The fact that so many were catastrophically wrong doesn't mean that following custom (even the historically recent batch thrown up by bourgeois morality) is always optimal.
That the U.S. tradition is liberal and somewhat revolutionary compared to the aristocratic and hierarchical value system it replaced was noted in my comment. As I said, the hard part is understanding where the dividing line between liberal experimentation and destruction of essential norms comes in.
That was the essence of the discussion of the French Revolution among the Enlightenment players in the U.S. and England who were all fairly Whiggish in ideological orientation. Some saw it as an inspiring step forward while others warned it was the road to perdition (e.g. Burke in England and the Federalists in the U.S.). The latter had the better of the hindsight derby, at least in the medium term, as the devolution into dangerous mob rule and then Caesarism via Napoleon with the consequent massive wars made the likes of Paine look naive.
At this point, we know a lot more about what norms, customs, and laws preserve social and economic freedom and dynamism than they did back in 1790 or 1917 or 1963 or 1975. But many are determined to ignore these lessons.
This doesn't work like the original conservatives imagined because we found out most traditions are pretty novel in human history. A serious modern conservatism has to reexamine custom through new knowledge.
I didn't suggest that anyone was more advanced. And I do think it's worthwhile to think about philosophy. But I don't really think that thinking about philosophy yields frameworks that let you compute answers to moral question. At best perhaps it sharpens your moral instincts a little? But in an unsatisfying way that still does not lead to answers.
+1
There's too many thousands of years of prior context to any choice that could be made in the universe, and there's no philosophy that is aware or able to consider all of it to say what is actually the right or wrong thing to do.
Morality is fake. Universalist morality was a European Christian invention. Everyone else treats their tribe and the other tribe very differently and makes no apology for it.
People focussed on trying to do good is an overall net negative. The human condition is improved by knowledge, not morality.
Yes, yes, you're very edgy, congratulations.
I really hope not, because that would be Scott going in entirely the wrong direction and getting more confused about the world.
Nothing in the above generates any significant problem for consequentialism, and “there aren’t really answers in philosophy” is nonsense. Philosophy involves lots of questions about which there is definitionally a truth of the matter, whether or not we know those answers yet.
I think Scott is entirely too intelligent (and too familiar with the basics of analytic philosophy) to think anything of the sort.
Yeah seems like another topic that feels like it's been discussed to death (including in this very blog!) before
I think time dependence is an overlooked factor here.
I might be 10% less cruel than the other guards... Initially. Is that likely to be sustainable? Am I justifiably confident the experience will not have negative emotional impacts that eliminate the initial benefit?
Or to flip the military question: there would be clear benefits to Europe rebuilding major military capacity again in light of current events. That's weighed against "how likely is that to lead to WW3 in Europe?". Because even if one generation might be responsible about it, the next might not.
I don't think you need to invoke a deontological bar to justify not working for Anthropic. Accelerating race dynamics in the current environment is bad, period. (Yes, working for the other big labs is even worse, but these are all bad decisions.)
Wouldn't this imply that if nobody had ever joined Anthropic, AI would be much less advanced right now? But why should that be true? OpenAI is already neck-and-neck with Anthropic, removing Anthropic from the picture probably wouldn't have slowed them down, and there are reasons to think it could have sped them up (eg they have more compute available for their own models)
My only counterpoint here is that I think Claude code has led some of the recent push into AI acceleration, and both Google and OpenAI have spent a lot of effort building cool doohickeys (Sora, Dall-E), that it isn’t clear to me that they would’ve gotten a high powered coding platform online as quickly, and thus enjoyed as much of the resulting hype boom. It’s possible the other things (less competition would outweigh it), but I think CC is a big point towards Anthropic causing acceleration in a different way than Google or OpenAI seemed to be thinking seriously about.
I think it is straightforwardly true that creating Anthropic sped-up AI timelines.
1. Having peer competition is what is causing OpenAI to frantically spend as much as they can as fast as they can while releasing as many models as they can. There is immense commercial pressure to build and maintain market share. Less competition means less financial investment and less compute. It also means more pressure to build and release potentially unsafe models (because "if we don't, the other guys will beat us").
2. The fact that they were able to catch-up to OpenAI so quickly suggests that they have some sort of superior process. It is possible that this is entirely a function of their talent, and so Anthropic not existing means that the same process would have happened at OpenAI or Google, but I suspect that this isn't the whole story.
It's not just about how advanced AI currently is. Anthropic is taking mind-share by pretending to be safer than the other labs, performing safety-washing while still racing to superintelligence and otherwise proceeding in just as dangerous as fashion as everyone else.
They are contributing more danger to the world than OpenAI. OpenAI was wasting its time on stupid things like Sora until Anthropic surged ahead. Here and in general, more actors in the space means more competition, means more effort toward the end goal.
A world without Anthropic in it would be a marginally safer world.
I think this is too 11D chess "bad things are actually good". I would not be happier with a world where Musk, Zuck, or other frank accelerationists are in the lead, and I don't think the straight lines on graphs that predict AI capabilities growth so well would have suddenly asymptoted out if Anthropic hadn't existed.
No, it is straightforward that bad things are actually really bad. Building superintelligence (which will probably kill everyone) is bad. Most people agree that it is bad and do not want it.
Maybe I'm wrong about the actual effect Anthropic has had on the world. We can can banter back and forth about which nth order effects to consider. But the thing that they are doing is obviously extraordinarily evil and must be stopped.
Isn't it self-defeating to accelerate AI development in order to beat the AI accelerationists??? The pressure to win has already forced Anthropic to drop their unilateral safety commitments.
I think OpenAI declared their code red and worked much more on coding for RSI because of Claude code
There are, I think, two intuitions about morality which are in conflict:
1) morality is in some sense maximalist. Depending on your style of morality, it could be doing the greatest amount of good, or avoiding the most amount of blameworthiness.
2) there is no excuse for not being able to identify the moral action. This implies that morality is knowable by everyone, otherwise they would have an excuse. (The law follows this inconsistently. On the one hand not knowing the law does not exempt you. On the other, people are required to be notified that their actions would, for example, take them across a property boundary; an absence of due notice is an excuse for trespass).
The latter intuition makes the idea of "moral strategy" seem incoherent, it should not be possible to discover that you have been acting imorally because you were not galaxy-brained enough to spot some exceedingly subtle, but powerful, moral gambit.
But AI risk does seem to require a deep level of analysis to find a good outcome (unless a good outcome is no longer possible and the best we can do is to avoid being blameworthy). Perhaps some moral intuitions are rules of thumb which don't really apply to it.
It would certainly be bold to argue that all moral evaluations are easy (although saying that not many are easy also seems quite a strong claim). However the second intuition does not require that all moral evaluations are easy, only that some tractably-evaluable moral choice is always available. In your example, a person does not have to decide if it is moral to kill that individual in order to have a moral choice open to them; if they can at least evaluate whether it would be moral not to kill them.
That is why it is in conflict with a maximalist notion of morality: the maximalist must be able to identify the maximally moral position, and identifying maxima is often intractable.
To return to the subject of the post: as well as their normative aspect, deontological rules also often exist to set out a tractably-evaluable rule where a maximum is difficult to evaluate.
Maybe it would be useful to imagine that, rather than picking a rule that literally everyone will magically follow, you're picking a rule that MANY people will follow, but there will also be other people who defect against the rule unless the first group can enforce it. So the rule will apply just to the cooperators in a mixed group of cooperators and defectors. This seems like a good way to derive e.g. police and courts.
Looking at your military example more specifically, it seems significant that having a military doesn't necessarily harm anyone, if you don't do anything bad with it. It increases the amount of harm you COULD do, and that creates some undesirable dynamics, but it still feels like a different category than imprisoning someone or conning them out of money, which are intrinsically harmful rather than potentially harmful.
Also, even if you could magically guarantee that there will never be human-on-human fighting, having SOME capacity for violence still seems good for dealing with wild animals and certain kinds of natural disasters (e.g. deflecting an incoming meteor). Also, many tools that are useful for other tasks can double as weapons in a pinch (lumber axes, cars, mining drills, etc.). So I don't think you'd want literally zero "military" power even if you could actually universalize the rule. (Though you'd probably want less than Ukraine has, so maybe this doesn't help in practice.)
You need violence or force even to keep people from non-violently squatting in your living room, or ultimately to get them to pay what they owe you if they refuse, etc.
You can only shun people who respond to shunning. If a group of homeless people decided to squat in some Amish buildings, the strategy probably wouldn't work.
Wouldn't the homeless people starve in this case?
I imagined they would sometimes go begging or stealing, but then they would return to the Amish buildings to sleep.
Every time I read something by an ethicist or that tries to rigorously define ethics, I'm struck by how much it's just trying to find a question that gives the answers we want. Strongly suspect that ethical theories are more strategies than rules and that you should pick the strategy that is most likely to make you do the things you've a priori decided are right.
This is a feature, not a bug - it's how philosophy (and reasoning!) work. If you're coming up with a theory of gravity, table stakes is that it should say things fall down at the rate we know they fall down at. Once it perfectly predicts the obvious cases, then we can investigate what it has to say about black holes / the early universe / etc, and use it to solve the non-obvious ones.
There is no such solution to morality. Humans are too weird for that. It's going to be special cases all the way down.
That's an assumption on your side. Consider the changes and refinements and quantification of our understanding of nature; while not accurate to the same degree, an prescriptive ethical theory is much more refined than "go with your guts", similar to how "stuff falls" is very different from a good understanding of gravity.
I don't think this is true. If I punched you in the face, I think you would have the right to ask why. If I said "new rule, I'm allowed to punch you in the face", you could justly appeal to some kind of shared norm of nonviolence. If I said "nah, I've decided to treat everything on a case by case basis", then I've retreated into basically random action.
That was a surprising escalation 😬
Case by case is not random. We also internalize fuzzy heuristics that compress those cases. We tend to communicate those heuristics as deontological rules, but because there is no such thing as a deontological rule in real life, we have to bear in mind that those fuzzy edges are always with us and act accordingly.
I think that (most) humans ultimately are only capable of caring about themselves and their extended family members. The entire history of moral philosophy is basically an attempt to work around this biological fact, but ultimately the problem is intractable.
Basically agreed. We evolved to deal with groups around Dunbar's number, including many close relatives. I suspect that _actual_ empathy is limited to people we know at least approximately as well as we would have known tribe members in our environment of evolutionary adaptation.
When someone does moral philosophizing, they are doing _something_ , but I don't think what they are doing has much in common with empathy for relatives or close associates. Maybe intellectual one-upsmanship? Maybe purity spirals? It often looks pretty pathological (notably all the woke stuff over the last 20 years - e.g. people talking themselves into supporting terrorists...).
That appeal would imply that norms are the authority of morality, which seems to contradict what you say elsewhere.
Also, norms are generally recognized due to enforcement, social or otherwise. So unless you’ve changed the norm to fit your case determination, you can probably expect to receive some violence yourself. At least, I would expect your case to be determined with that result.
You once wrote a piece, I can't find it right now, where you wrote something to the effect of "At that point, you're just one labcoat removed from being an indigenous farmer yourself", about how scientists try to understand the process of growing and preparing one particular edible plant and have to go into ever greater detail of soil composition and microclimates.
Humans are much the same way. Human values change with place and time, often in mutually exclusive ways, unlike a ball falling from a tower which will behave predictably if you know a few parameters. This ties in nicely with the impossibility of defining "universal human values" for AI alignment, doesn't it?
What if you change your mind about what is right a priori? Were you evil in the past?
Does it make sense to evaluate deontological bars on consequentialist terms? I feel like, if you're really operating that way, then they are axiomatic. I think what you're demonstrating is that this is really a purely consequentialist discussion.
This is a really good 3 sentence summary of the much longer comment I made
Yes, but that doesn't mean it amounts to pure consequentialism. it could easily give a different answer to a trolley problem.
Pure or not, it's still consequentialist thinking. It looks to me like straightforward rule consequentialism. But maybe I'm bleaching the term of its meaning, since I don't know of any way of analyzing ethics without getting into the consequences of following one ethical program or another.
It's still rule based thinking as well as consequentialism. It still isn't pure consequentialism.
Apropos of nothing except the thumbnail image for this post, this is what Adobe Firefly give me for the prompt "Dentological Bar":
https://imgur.com/a/20OyIkJ
The fault lies in myself more than in the tool, as I missed the first O from "Deontological" and instead asked for a made-up word that Firefly seems to have, not-unreasonably, interpreted as having something to do with dentistry.
Pretty good bar, I think it needs more sad philosophers
I continue to be persuaded of a few things:
- If AI takeoff is impossible for humans to control, the only thing that can control it are other AIs, especially if natural chokepoints exist which allow a response, again even if only by other AIs
- Hardware limits are always at their greatest in the present, absent drastic global degredation, so advances in AI software will be safer now than later
- An ideal approach to AI would be with great caution, but that ship has sailed for the time being. When ChatGPT first become an obviously remarkable thing I thought "amazing, I'm sure they will never connect this thing to the internet", and it immediately became a search engine
So, we should advance AI with the idea of having something go dramatically wrong sooner, with the least capable hardware. This doesn't preclude warning that something could go dramatically wrong as if it happens you are then the most credible group to craft a response. 10% safer AI companies are more likely to be allies in this case, so there is a case for them both for producing good AIs if possible and for having powerful allies if the moment for great restriction comes
My thing is that norm-protection is doomed without moral boundaries underneath it. Norms don’t preserve themselves. They need people who still feel that certain things cannot be done, even when everyone else is doing them, and even when defection can be recast as realism.
Take the concentration-camp-guard example. The problem isn’t only that one is worsening consequences or eroding a norm. It’s that one is becoming part of something. That seems like a different category.
Same with misinformation. If a writer knowingly manipulates his readers, the issue isn’t just whether other bloggers are defecting, or whether he is losing enough subscribers to justify doing the same. It’s that he has violated a particular relation of trust.
Yes, I think that’s the strongest case for being on the inside. But the guard is still a guard. He is still helping the institution function, even if he is mitigating some of its effects.
So even if good can be done from within, the “within” part has its own moral meaning. It isn’t just a neutral place from which one produces better or worse consequences.
If something is the right thing to do, it doesn't stop being right just because Steve Bannon, or Bernie Sanders, also supports it.
I suppose what you are really saying is that Bannon's support or Sanders's support is sufficient to make someone doubt his own judgement that it's the right thing to do. If that's the case, I wonder if that person should be doing philosophy at all.
I don’t understand how ‘deontological bars’ in consequentialism can make sense outside of being shorthand for 'policies that create a coordinated Nash equilibrium' (so, basically, the social contract).
If that isn’t self explanatory, the longer explanation is as follows:
<explanation>
Let us imagine that I am a special version of a constraint utilitarian. I think it’s moral for me to assassinate dictators. In a similar way (though immoral) I’d also cause great harm to many for the sake of my child.
If I had the power to act on these impulses, I might apply the sub-explanations in this post to stay my hand: maybe I’m not as brilliant a forecaster as I think I am.
But even if I was convinced I was as brilliant as forecaster as I think I am, I may still recognize that others are not. This is concerning, because those idiots might feel licensed to assassinate the good leaders or harm my [precious] child for the sake of their [irrelevant] one.
So, I conclude that it is moral for me to repress these idiotic people and take away their capacity to assassinate and harm.
When I try and do this, I run into two problems. First, I lack the power to do so unilaterally. Second, people are willing to help me with this, but they seem to want *me* to agree to not assassinate or harm either!
I weigh my options. A world in which I assassinate with impunity but so do they, vs. a world in which I bind myself to bind others. Meanwhile, 7 billion others weigh the same option.
We all come to the same consequentialist conclusion and bind ourselves to mutual policy. We loudly proclaim our ‘deontological bar’: to do no harm. In our heart of hearts, we retain our secret morality: “rules for thee, not for me”. But no good comes from speaking that aloud. All it will do is confuse the mission and give license to our enemies. So we keep it inside and say “no, I don’t think it is moral for anyone to assassinate dictators”, a statement that becomes true and not true at the same time.
</explanation>
I think this is just rule-utilitarianism. But if a constraint utilitarian starts to tell me about ‘deontological bars’ that don’t have social-contract justifications (or if they take umbrage with an Oracle committing a perfect secret assassination), I’m not going to say “I would be better positioned to navigate this debate if I knew what deontological bars were or where they came from.” I would simply lump them in with the rest of the deontologists as having weird and arbitrary justifications/carve-outs in their broadly consequentialist philosophy.
The social contract justification addresses all the examples in the article. Regarding Ukraine: you should contract with your enemies for an enforceable policy of peace (unless you don’t need to because you can crush your enemies and you KNOW you’re right), but if they’re unwilling or it’s impossible, then build up your army.
It turns the maxim “Don’t be the first person to defect from a generally functioning norm” into the equivalent “If people haven’t realized they can defect, don’t point it out by being the first one to defect” (which, in itself, is a Nash equilibrium!).
The concentration camp/“least-irresponsible AI company” example is tricky because it rubs against our intuition that there shouldn’t be a camp/company in the first place. Some might feel that by participating in the camp, you legitimize the building of/participation in future camps. For example, mass participation in atrocity might weaken people’s moral fiber to oppose the next atrocity. The people building the next atrocity might take heart knowing that if they are only 90% bad, they will find supporters for their mission, as opposed to being worried that badness will leave them without any support. The people who don’t mind participating in some amount of bad will feel emboldened to support the 90% bad things of the future because you won’t have a leg to stand on if you try and censure them. If you do not care about the shape of the next bad thing (for example, if you believe that AI companies are the worst bad thing or the final bad thing), and you feel that any norms of “don’t support an AI company” are thoroughly broken, then as a consequentialist, go ahead and work for the least irresponsible one.
If norms are only 50% broken and you’re deciding if that’s enough for you to follow the herd, then you have to apply more complicated versions of the logic above to determine your decision.
unironically the sanest comment.
Typo
than I would if I spread more information
Should be misinformation
Seems like you’re having trouble with two theories of ethics! You should solve it by picking your deontological bars via virtue ethics. That way, you actually don’t end up like a huge jerk and try to be honorable and act with integrity in your deontology to lead to good outcomes. In practice, this decides the pauseAI example by saying don’t start lying to win because lying isn’t virtuous, you can work at an AI lab but might get fired when you speak up about practices you don’t like due to your commitments to acting according to your principles, and you don’t sign up as a prison guard because gassing innocents is wrong.
This essay frames the questions as "Should I do this thing or not?" - questions in isolation, disconnected to what _else_ we might be doing.
It'll be very rare that your _only_ choices are, for example, to become a concentration camp guard or an SS trooper. (It does happen sometimes. Ukrainians conscripted by Russians face horrible choices like this.) It's more common that that _looks like_ the only choice, perhaps because a bad person has applied various manipulation tactics.
So a useful slant on deontological bars might be, "Do I really believe that the world is so bad that my best choice is to ______? Why do I believe that? What's the likelihood that I'm wrong?"
Breaking out of a thinking-box may require an act of faith, which looks a lot like an act of morals. "No, no matter how it looks to me now, I will not become a concentration camp guard. The world simply is not that bad, and I won't do it."
This is not a personal moral position. It's a partly grounded belief about external reality, which is personally preferable to the more pessimistic alternative belief that would constrain you to the undesirable behavior.
"Do I really believe the world is so bad that my best choice is to become a murderer and forever distort my country's politics?" ... No, probably not. And as a further filter, "If I do believe that, am I likely to be insane and should not act?" There's no deontological bar there, but the result is similar.
I'm not Scott to think through the implications of this in exhaustive detail, but it seems worth doing - it seems to solve at least some of the conundrums that Scott's article leaves unsolved.
It would seem that in the short term moral calculation is doomed to failure because we lack access to relevant data and lack the capacity to do the calculations fast enough to be relevant. Alas we are also doomed in the long term because our capacity to calculate increases linearly while relevant variables and relations increase exponentially.
It would seem we either need an omnipresencent being or a being that can bring exponentially increasing computational power to moral calculation if want moral calculation to be anything more than an intuition pump.
A religious consequencelist can do there best and rely on God to make all things work towards good. On the other hand it would seem the best hope for a rigorous secular cosequentalism is to be found in some form of technolgical singularity.
There is an important distinction between working with the companies and pause-AI regulations positions. First giving you much more personal gain, immediate or potential. Since i know personal gain makes people lower their personal deontological bars significantly so i cannot trusty their notation of common sense and have more sympathy towards, for example, nimbys then them
A surprising number of people are advocating moral intuitionism. They are assuming that we already know what the right thing to do is, and I must say, ethics and all normative reasoning becomes mush easier if you can start with the assumption that we intuitively know and fundamentally agree on what is right.
That probably works well for most of the day-to-day moral concerns people have. It in no way helps for questions that affect large scale populations or anything approaching an edge case.
The lesson we should all take from trolley problems is that moral intuitions are nonsense. Most people agree that trolley problems have obvious answers, they just disagree about which answer is obviously correct. All the variations on trolley problems exist to show that our intuitions break down with the slightest pressure. The ways to run through a trolley problem in a philosophy class are (1) get two people who think opposing answers are obvious and let them fumble through an argument and (2) take the most confident person and keep tweaking the problem slightly until their intuition fails.
And moral intuition always fails for any reasonable person because our moral intuitions do not scale to the size of society we have now. It might have worked for a sense of fairness in a village of 150 people. You cannot intuitively know what the third-order effects of instituting a minimum wage are. We have people *in this thread* with opposing moral intuitions about whether assassination is appropriate in some cases, and "do not kill" is about the most fundamental moral rule we have. But our moral intuitions also suggest a host of exceptions, which again vary between people because it comes down to vibes.
It does not save moral intuitionism to add a disclaimer that some people are bad and wrong, so we can ignore their intuitions. At least a third of the country, and probably more of the world as a whole, will disagree with you on important moral questions. You, yes you personally, are in the minority on multiple moral questions. Your moral theory cannot work if it says that many or most people are moral mutants, and it certainly does not work in your favor if most of the world counts you as one of the moral mutants.
What do you intend to replace intuitions with?
This is why it's impossible to have a morality that's entirely based on the subject's private considerations. Intuitionism is basically the height of this questionable approach. You have to leave at least some rule for taking into account objective factors (other people's thoughts & feelings, customs, institutions, etc). No individual can know the best possible normative action through solitary design, or at least, rarely and exceptionally.
Other people's intuitions are intuitions. Trying to base ethics in facts runs into the is ought gap.
There are facts about the origins of intuitions, how to accommodate intuitions, who has which intuitions, etc.
Do they translate into norms.
Depends on how you define norms
What if the Early Christians were playing a different game than Prisoner's Dilemma?
I recently read A Brief History of Intelligence, and I read the section talking about the development of altruism as loosely saying:
- The other members of your society notice when you act with altruism and when you defect.
- They remember and your reputation and status are affected accordingly.
- This provides safety for more altruism.
(And then you get the internet and this no longer works.)
To be more accurate, Bennet describes it in terms of a cycle of increasing language skills leading to more gossip and punishment of defectors promoting more altruism to non-kin:
***
If someone lied or freeloaded in a group that tended to gossip, everyone would quickly learn about it: "Did you hear that Billy stole from Jill?" If groups imposed costs on cheaters by punishing them, either by withholding altruism or by directly harming them, then gossip would enable a stable system of reciprocal altruism among a large group of individuals.
Gossip also enables more effective rewarding of altruistic behaviors: "Did you hear that Smita jumped in front of the lion to save Ben?" If these heroic acts are heralded and become ways to climb the social ladder, this further accelerates the selection for altruistic behaviors.
***
(Quick aside: I enjoyed most of "A Brief History of Intelligence", though I thought it got weak toward the end when Bennet was discussing AI. “Amazingly” he writes, Chat Gpt-4 can correctly answer questions that Chat Gpt-3 got wrong... because Chat Gpt-4 was trained on those questions. :facepalm: And it's a just nit, but there are a few places where his grammar checker chose phrasing that grated on me enough to remember.)
Jumping in front of a lion is a much higher bar than my experience of internet communities, which might be described as "being polite while talking past one another". Twice in my life, I have been in a situation where I really felt I was putting myself on the line for someone who was neither family nor a close friend. Both situations took soul searching and mustering courage, and I doubt that I'd be strong enough to do it again. To me, those situations were a very different thing than an acquaintance of an internet acquaintance is in a bad place so I'll toss some money into their GoFundme.
So I think of internet communities as generally being on the level of having an AI girlfriend. The one online community I'm in that's beyond that got that way because we met in person, with different subsets of people holding gatherings and some people flying across the US to attend them.
It happens that another book I'm reading is "The Different Drum: Community Making and Peace". The author, M. Scott Peck, says that creating true community is really hard and thus they are vanishingly rare. This includes places like churches that are supposedly all about community. (As Josh Hickman commented about in the Early Christians post.)
Another diversion: M. Scott Peck's famous book was "The Road Less Travelled" which talks about his experience as a psychiatrist doing psychotherapy. Now ACX Scott has expressed that he prefers to prescribe than to act as a therapist. Similarly, Peck writes: "During my first ten months of training I worked with highly disturbed inpatients who seemed to benefit much more from pills or shock treatments or good nursing care than they did from me, but I learned the traditional magic words and techniques of interaction." I can't find an indication that ACX Scott has read "The Road Less Travelled", but maybe it would totally not be his thing.
In the book, Peck likens raising children to a long-term therapeutic relationship. "So it is in good parenting as well as in good psychotherapy." That got me wondering if ACX Scott is doing psychotherapy with his kids.
The big breakthrough of Christianity was promising people everlasting life and a very nice one if you followed the rules. Not just the ruling class but everyone.
Yeah, that changes the rules of the game. Kind of like a carrot version of altering Prisoner's Dilemma so that there is an X% chance the mob boss will scrag a criminal who turns stool pigeon.
In short: it's unlikely that either of your examples would be ruled out by a deontological bar unless they're already impermissible on consequentialist grounds - but I'm unsure whether contemporary deontology is actually what you had in mind.
It's notable that the decision-theory-adjacent conception of deontology that you suggest, which is something like "deontology as a hypothetical coordination mechanism", isn't necessarily deontological. E.g. that reasoning is a major motivation for rule utilitarianism against act utilitarianism.
Contemporary deontology tends to be pretty heterogeneous (but then again, so does every philosophical tendency). Its proponents often use the term "non-consequentialism" to describe the tendency as it's easier to see what they're united against than anything they're united for.
So what *is* contemporary non-consequentialism if not a hypothetical coordination mechanism? Non-consequentialism tends to formulate particular moral devices which stick closely to our intuitions about particular cases - it's much more intuition-heavy and "bottom-up" than most consequentialisms. Examples include:
* A means principle - a benefit's ability to justify causing a harm is significantly reduced if the harm isn't an unlucky by-product of the benefit but part of the causal mechanism which achieves the benefit. This is why perhaps even a majority of contemporary deontologists would flip the lever in the original trolley problem, but very few would endorse killing and harvesting 1 person's organs to save 5.
* Adjustment for responsibility - a person's entitlements, e.g. to not suffer punishment, can be changed by their history of meritorious or blameworthy actions, even without any changes in expected future consequences.
* Excludable claims - we might want to exclude interests below a certain weight from consideration when much weightier interests are at stake - e.g. if a life is at stake, it doesn't matter that half the universe will get a paper cut; that simply shouldn't factor into our considerations.
Non-consequentialists do offer theoretical explanations which attempt to integrate particular devices into a wider philosophy, improving their theory's parsimony. Some pretty common theoretical moves are to suggest that the absence of that device cannot be be justified to a relevant constituency - the modern heir to the old social contract tradition - or to argue that it is needed to respect a special status held by moral patients.
Why do I think that non-consequentialists wouldn't have much to add for the two cases you consider? 4 reasons:
1. Non-consequentialists care a lot about causing and preventing harm. Most of the complaints about each side, even on non-consequentialism grounds, is whetjer each side prevents more harm than they threaten to cause - and that's the bigger, pre-existing controversy.
2. Most non-consequentialists think that side-constraints can be overridden when the stakes are high enough - and both sides believe that the stakes are *immense*.
3. In the case of people working on safety for AI companies, complicity is a real concern - but the attention of most non-consequentialists would be drawn to the difference between "is it permissible to intentionally harm someone if I deliberately harm them less than they would otherwise be harmed?", as in the case of the would-be concentration camp guard, and "is it permissible to attempt to prevent harm, at the expense of increasing the danger that that harm occurs earlier and by a different mechanism, while working to enrich a grossly unjust company?" The latter situation is troubling but easier to justify than the former.
4. That kind of mass politics is inevitable in a democracy, and non-consequentialists tend to be fond of democracies. Perhaps there's a stronger case if there's a major risk that the mass politics will overreach and cause significant injustice to non-culpable third parties? Perhaps a concern could be that it must involve deceiving people by deliberately misleading them about the reasons for you position? These both seem very hypothetical.
What political systems do consequentialists favour?
It's hard to answer that, because:
a) There's different kinds of consequentialism which bring different standards to bear,
b) Which consequences we should expect from different political systems is pretty controversial.
That being said, most consequentialist philosophers are - like most academics - some flavour of left-liberal democrat.
Do they notice and disapprove of rules based legal systems?
Not particularly - like most people who consider the matter, they normally believe the rule of law has good consequences.
Do they not that makes them rule consequentialists?
Nope. Consequentialism is a theory of what makes actions right or wrong - I.e. what we should or shouldn't choose. Right and wrong in that sense =/= what the laws should permit or prohibit. E.g. imagine what consequences would come of, say, 10,000 consequentialists suddenly supporting changing the laws to make maximising utility the only legal option.
I believe the correct approach is deontological consequentialism, wherein there exist a set of duties, and we must maximize our adherence to them.
We have a duty treat our fellow man with dignity. We also have a duty to minimize suffering. Becoming a concentration camp guard would clearly violate the first duty, and only maybe vaguely adhere to the second duty. In our pursuit of duty maximization, we would therefore not become a concentration camp guard.
Assuming we can unilaterally disband Ukraine’s military, we would need to weigh the duty not to kill people, the duty to minimize suffering, a duty to protect the independence/freedom/sovereignty of others, and a duty not to deprive others of the ability to fulfill their own duties. A wealth of historical knowledge makes us believe we probably should not disband their military, as likely doing so would end up creating more suffering in aggregate, a greater loss of freedom, and possibly more death. However…. we don’t know for sure. What we are sure of is it would deprive people of their ability to perform the duty of defending their nation, and thus it is almost certainly wrong to disband Ukraine’s military.
If you accept this lens, the next question is trying to understand what duties are impotent. The good news is most people already have pretty good instincts around this.
Obviously no one has a duty to chase down and maul children with dogs; in fact, I think we clearly have a duty to protect children.
I am not familiar enough with the situation to be able to give a good analysis of how to resolve/prioritize the conflicting duties.
My brief google search indicated most of the conflict was around the dogs mauling the children, and it seemed those incidents were what you were referencing.
The framework I laid out in brief is for understanding human morality. I am not claiming I can convince anyone to do anything.
They are all unconvincing because they are all functionally nonsense. There is no such thing as a perfect moral framework in a world that depends on physical needs and realities. I think this whole inquiry boils down to economics on its most rudimentary level. We rage against it because we don't like it but also because it's immutable. If we want to eat, we have to kill something else. People who painted in caves in France 40,000 years ago understood this and it disturbed them.
Regarding a way to harmonize "act as if your maxim became general law" with "Ukraine can have an army", IMO the right way to think about it is whether following your maxim brings you marginally closer to the world that you're supposed to get with the general law.
For the Ukraine example, them disbanding their army won't bring us closer to world peace, it will just empower people who keep their army, perhaps even making them *more* likely to do so. But I do kinda think that not-spreading-misinformation brings us closer to a no-misinformation world.
IMO the relevant game here isn't the prisoner's dilemma but the stag hunt (in particular with n > 2 people). And the right move in that game depends on what you expect other people to do, i.e., on what norms exist.
Also - I don't think this maxim is needed to explain the assassination thing. The way I think about it is, in considering the consequences of your actions, prioritize immediate, obvious things and have some humility over your ability to predict or weight more remote effects.
I.e. if I came up with a model of society and predicted that if I went and killed a random guy it would lead to world peace in 20 years, I wouldn't do it, because I don't actually have faith in my model. If someone said "OK but what if you *knew* that killing the guy..." my answer is "no fuck you, I'm not gonna kill a guy out of dedication to not fighting the hypo".
This is also my answer to "should you kill a guy to save trillions and trillions of people from a very tiny bit of passing discomfort".
"Not being the first to defect from a well-functioning norm" also doesn't work, in my opinion, because it risks giving society too much inertia.
How local is the decision? Because most norms don't apply globally or through time. Should soldiers working in concentration camps in Nazi Germany have killed and tortured jewish people? Most of them did, so, localized in time and space to Nazi Germany, this was a norm. Of course, we would call it non-functioning, and it breaks the "don't commit murder" norm we have globally. But it's unclear how you would design a system that automatically selects for the correct norm, or who gets to decide what is well-functioning.
Maybe we should just always pick the most global? But even this gives too much inertia in my opinion. Factory farming is widespread, and the vast majority of people support it (at least economically). Undoubtedly a normalized practice worldwide. And if you ask the average person wether they like the way food is produced, so long as you don't specify too much, they'll probably say yes. Normalized practice, so a norm, and approved by most people, so well-functioning (?). But of course, few people who care about ethics would actually argue in favor of factory farming.
You could counter that most people don't actually support the specific practices present in factory farming (see OWID's recent article on the topic). But their actions don't reflect what they say. So again, who gets to decide which criteria matters more? Is a norm "well-functioning" if most people say it is, or if most people act as if it is?
Factory farming _of animals_, should've specified, apologies. The discussion here is on ethics. If you look into the ethics literature of the treatment of animals, it's pretty clear that animal factory farms are morally indefensible the moment you grant any credence to the idea that their suffering matters. It has also long been a point of interest in rationalist and EA circles (e.g. Scott's text from 2015 on how meat eaters can still help reduce extreme animal suffering https://slatestarcodex.com/2015/09/23/vegetarianism-for-meat-eaters/ ) Pretty much the only moral defense of factory farms is precisely the fact that they are widespread. Which is exactly my criticism: if our moral system allows for things to be considered moral simply because they are common, then that's a pretty weak system.
They generate a lot of veal and calves' liver
Sorry for the delay in response, I've been quite busy. I totally understand what you're saying, and I must say I really did use to think the same thing before. But as I said, once you look into it, then it becomes difficult to justify factory farming. Check out, for example, this article from Oxford ( https://www.ox.ac.uk/news/2021-11-11-sustainable-eating-cheaper-and-healthier-oxford-study ). Many such articles and studies back this finding: vegan diets are typically much cheaper than ones with meat or animal products. Now, most of the research is based in the west, but I don't think it's an unreasonable assumption to say that factory farming is unlikely to help the poor, because it would be more cost effective to feed the world less meat anyway.
You could reasonably complain about health, and the literature there is a little less decisive. But most findings from independent researchers do seem to show that a vegan diet has more positive effects than negative (see this meta-analysis : https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=vegan+health+meta+analysis&oq=vegan+health+metaan#d=gs_qabs&t=1777961896288&u=%23p%3DDyzIiZ2FnksJ ). That's not to say that we should impose vegan diets on anyone, but rather that if helping the poor was our goal, factory farming of animals is not a way to do it. It would be more cost effective to produce plant-based products, and would have little to no health drawback.
In regards to cage-free eggs, I think it's pretty well known that cage-free is nothing more than a shitty marketing term, so of course I'd have to agree with you there. You can look up what "open air" or "cage free" actually looks like in factory farms, and you'll see that it's still miles away from a traditional farm. Yes, the broken bones of chickens while stuck in a factory farm 10000% counts, and is again many of the unjustifiable acts of cruelty present in our current food system. As you said, a farm who lets their birds go out and actually act like chickens would be way, way better.
Finally for the cows, I was unable to find any authoritative source claiming whether cows do or don't like being milked. But consider that cows, like any mammal, don't produce milk unless they've given birth. So what they do is forcefully inseminate them by restraining them to a giant machine, and then having a technician pierce into their vagina and inject them from inside. Typically without any anesthetics. And then they spent their lives inside a factory-farm where they can't move more than a few centimeters in any direction. Again, unimaginable cruelty for little benefit. And impossibly far away from the ideal of a 'traditional farm' that most people imagine when they think of farmed animals.
I agree with Scott that these particular deontological bars, both somewhat about collusion, are affecting what folk do.
I think AI safety en masse is too shy about both of them in these two particular cases.
I volunteer for Pause AI and endorse protests where we share space with folk who care about different AI risks and have some positions I would not endorse. Open-hearted discussions about those disagreements even occurs in the ground. Activism can be net good.
I also don't, myself, treat anyone working in a frontier lab as an adversary. A subset are folk to talk to and work with. Different folk make different trade-offs based on different values and assessments. I will stand outside Anthropic and protest, but would also welcome chatting over coffee.
Our combination of best moves to make extinction from superintelligence less likely almost certainly form an impure portfolio. This is OK.
Tangential: the thing about becoming a concentration camp guard to make things easier on the prisoners, is that the guy who wanted the job but didn't get it is still going to be somewhere, and will presumably be as bad for the people around them as they would have as a guard. So it's not just "will it improve the camp," it's "will it improve the camp more than the other guy will degrade their new environment." Which is almost certainly No.
On the main point: Consequentialism can't have Morals Despite The Consequences. You either need external factors like "God said to do it this way,", or internal factors like "I'll be able to sleep with this decision".
>and will presumably be as bad for the people around them as they would have as a guard
It doesn't follow that he would be as bad in consequences, even if he is equally lacking in virtue.
I don't have the interest to build a whole picture of this event by myself, so will merely reply "does too".
I don't really understand the arguments here.
The idea of a deontological bar seems unnecessary. Regular consequentialism + uncertainty works fine for all the examples I can think of. And, in terms of coarser societal norms, something like: "bad stuff should have a proportional cost" covers everything. The question of “defining and quantifying badness” is still open, but having a deontological bar seems to say “we should draw the line at some things that are infinite badness, like assassination”, but this is obviously folly.
With the assassination question - consequentialism works, and assassination already has a massive cost. I don't commit political assassinations partly because I think it would be bad for my movement, and would have negative utility more generally. But also because I don't want to go to prison!
Do I need a deontological bar on top of this? No, we don't, and it's pretty good that we don't! There was a well-publicised attempted assassination 3 days ago and I've heard no attempts to condemn it, because most people think assassination is okay in some contexts.
With the "don't work with an AI company", "don't work with Steven Bannon", "don't join a protest with terrible people" if we say: "doing bad stuff should have a proportional cost", and we acknowledge that these things are bad, then doing these things should have a cost. And they do (albeit not always proportional)! If the PauseAI team worked closely with Steve Bannon, the CCP, or Antifa, it would have a personal, and political cost to the team, which they would have to trade off against the benefits.
When EAs work closely with those creating superintelligence, like Anthropic, it should have a reputational cost, which they have to trade off against the benefits. This is more difficult, of course, but I don’t see any place that a deontological bar makes anything clearer or easier.
Regarding shifting norms on presidential assassinations: there have always been crazy people writing threats or even planning killing presidents. Skimming wikipedia, it seems that under Obama the number of such threats escalated. Many of these people were arrested and jailed for a few years or killed during the arrest. However, if we look at serious assassination attempts by domestic actors - where the assailant actually had intent and means to kill a president and was executing a remotely realistic plan to do that - there were very between the attempt at Reagan in 1981 and the attempts at Trump in 2024-2026. Perhaps Clinton 1994 should count as one, and the ricin letters to Obama in 2013 and Trump in 2020, probably gun grab on Trump in 2016 too. There were three such attempts on Trump in the last 2 years, plus two more in his first term.
So if we have a Poisson model for such attempts, we count 5 events in 1976-2023, two of them on Trump. And three events in 2024-2026. Annual frequency changes from 0.1 to 1.2 per year. This is statistically significant at ~0.2% level. Trump vs non-Trump gives 0.09 per year vs 0.72 per year, which is also significant at ~0.1% level. So yeah, the norm has definitely changed.
Alternative hypothesis would be the norm itself remaining fairly consistent as something like "don't assassinate remotely decent presidents, only insane tyrants who march armies against a legitimate American government." That covers broad popular approval for traveling back in time to kill Hitler, or John Wilkes Booth (insofar as his complaint about Lincoln was the fall of the Confederacy), or Trump after his coup attempt, and is even arguably consistent with some statements by the Founding Fathers.
Doesn't cover Charles J. Guiteau or John Hinkley Jr. so well, but consensus seems to be they were out of touch with reality, so deviation from social norms is to be expected. And the Kennedy thing is such a mess I'm not even gonna try.
1960s-1970s are also statistically different from both preceding and subsequent periods, with two attempts on JFK (one of them successfull), RFK killed while campaigning, and two attempts at Gerald Ford. Given short terms for both JFK and Ford, these are higher incident frequencies than for Trump. So for your hypothesis to be plausible one would need to brand JFK and Ford insane tyrants as well :)
There was definitely some controversy about the whole Catholicism thing.
As for frequency relative to Trump... how does that difference compare to historical trends in the overall homicide rate? Pretty sure it's gone way down, and a smaller pool of people willing to commit murder at all would naturally imply proportional reductions in any given specialized subtype.
I will be eternally amused if in a few years there are, I don't know, ten deontological bars that involve things like murder, covetousness, respect for parents, and graven images
People always forget what the First Commandment states, probably because it's the one that's caused the most problems over the years...
"I am the Lord thy God"?
As with most ancient numbered lists, there is no fixed numbering.
Mostly it's the second part: "...thou shalt have no other gods besides me".
Ah, you mean the Second Commandment.
Nah, that one is about graven images, I think. That said, the numbering does vary depending on the edition.
https://en.wikipedia.org/wiki/Ten_Commandments#Commandments_text_and_numbering
I think focusing on "how well-functioning is the current norm?" is slightly missing what is motivating people's considerations in many of these cases.
I think that what people imagine is that perhaps, right now, (resisting) becoming a concentration camp guard or (resisting) working for an AI company doesn't immediately change anyone's actions, but that resisting is slightly increasing the future probability of others resisting and shifting the general norms in the resistance direction, whereas capitulating is slightly shifting norms in the wrong direction.
It might be that the probability of anyone else resisting is very low <1% and the increase in probability from your resistance is even lower, so this might seem like a consideration barely worth taking into account. But I think people also often think ~ the only path we have to changing things is slowly convincing everyone that concentration camps / AI companies are bad, so it's vital we resist even if success is very low probability and the effect is minimal.
Love the thumbnail lol
I consider myself a meta-utilitarian deontologist. 1st level utilitarianism is not practically possible, as evidenced by all these stupid thought experiments that utilitarians get wrong all the time. The utility of moral choices is probably poisson distributed. Deontologism prevents mistakes that will outweigh any net gain all the clever utilitarian thoughts would have had
For those bars, is deontology the only correct framing?
Consequentially - for the one for pause activists, isn't there an argument that you never get a confident p(doom) reduction from marginal dollars given to labs, due to e.g. safetywashing, dual use safety research, or motivated reasoning coloring risk perceptions?
For the one for lab cooperators, I think the modal concern is backfire risk - that activism could increase p(doom), either by making it an issue subject to political cycles, or a fear that a properly AGI-pilled government and populace would take counterproductive actions? There's also, I think, a concern that the concrete asks necessary to reduce p(doom) would inevitably get watered down into something that wouldn't work.
You could also look at it from an identity (virtue?) angle. Pause activists are generally either people who've gotten fed up with the labs, due to them lying, or breaking safety-related promises, or being shut out by them when they argue for a pause, or they are people who never would have the opportunity and/or inclination to do "inside game" work. So their identity settles into being "people who don't like labs."
Lab cooperators tend to come from an elite pipeline; their whole adult lives have been "inside game," and do not see themselves as populists. They are used to seeing the world from an engineer's or researcher's POV - STEM isn't supposed to be political! - or from the POV of someone working at a 501(c)(3), where you are legally and culturally forbidden from being overly political. The only thing politics means to them is unpleasant things they see on social media. So, they see activists, and activism, as somewhat suspect.
Regarding assassination I am reminded of a 1955 satirical Robert Sheckley story where assassination is legal. https://chatgpt.com/share/69f47ac2-fcc4-83eb-ad19-106a0db6fbbe
Re the Ukraine example, I think there’s a significant distinction between “having a military” and “using your military to invade other countries.” On the individual level, human beings have the capacity to commit all sorts of terrible immoral crimes, rape, murder, and so on. But original sin notwithstanding, we do make a moral distinction between having the capacity to do something terrible, and actually doing it. You can theoretically have a large standing army which just sits around waiting in case it’s needed, and while you might say that’s still morally wrong, I think it’s a different category of wrong from launching an invasion of your neighbour. We can also view violence in self-defence as a morally distinct category from unprovoked violence. That also distinguishes between the Ukrainian army case and concentration camp guard example: if it was possible to take the concentration camp guard job, draw your salary, and actually just sit at home, not turn up to work, and therefore not do anything morally wrong in the course of your employment, maybe that actually would be justifiable, particularly if it prevented the Nazis from hiring a more diligent concentration camp guard. But in practice of course the Nazis aren’t going to let you have that cushy sinecure, and even the 90% less cruel concentration camp guard is going to have to actually do terrible things in the course of their employment.
So maybe the multilateral principle works in cases of positive action, but not in cases about whether it’s acceptable to even have the capacity to perform a certain positive action.
This also seems relevant to the difference of the AI case, because part of the debate is about who is the actual moral agent whose decision matters here. If AI is just a powerful weapon, then you can make the same moral distinction between building it and using it. But the argument of the AI safety people is it’s more like a nuclear weapon that could just decide to launch itself. So having the AI companies or the government bound by laws and norms about what it can be used for, is about as useful for AI safety as my solemn vow that I will under no circumstances launch a nuclear first strike is a contribution to nuclear deterrence.
"I don’t know how to get around this and think it might require common sense."
Smackdab inside thoughful reasoning, the messy real world requires the blurry "common sense". But only as a when no logic can wrap the problem. Given how often "common sense" works, common sense should be prominent in reasoning, less fallback more checksum.
"For example, you shouldn’t assassinate democratically-elected leaders, even very bad ones."
You'd think this would be obvious, but I'm seeing social media posting (which is probably more about virtue signalling rather than actually asking for it to happen) that it's such a shame the latest assassination attempt failed, or that eventually someone will succeed.
And I can't stop myself yelling at the screen "You idiots, don't you realise this holds for your side, too? If you're okay with Trump being shot, then you're also calling for Kamala Harris to be shot, or AOC, or your Fave Trans Nonbinary DSA-splinter offshoot Activist Just Elected As State Representative In Goosehollow to be shot!"
Because if you call up that which you cannot put down, it will happen to *you* also. Oh, but Trump is bad! He's uniquely bad! That isn't murder, it's revolution! Yes, and do you think there is no-one with a gun out there who doesn't think Harris/AOC/Dangerhair isn't uniquely bad and it's revolution not murder to kill them?
>and there was an airplane assassination attempt on a right-wing politician in the UK to
What? Do you mean Farage's plane crash?
I consider myself left leaning but, I think one of the biggest failures of the modern left/liberalism is forgetting or failing to imagine how much worse it can get. The word fascism gets thrown around a lot but normalizing assassination is the kind of thing that *actually* leads to us sliding into some fascist security state.
Who was it who said, "The road to hell is paved with good intentions"?
> your Fave Trans Nonbinary DSA-splinter offshoot Activist Just Elected As State Representative In Goosehollow
There aren't so many of those. Lot of left-leaning Americans look at what the Democratic party establishment has been doing... seemingly far more focused on keeping left-leaning candidates from getting elected, and pandering to anyone on the right who'll listen, than in properly solving what their own fundraising emails breathlessly describe as existential threats... the Epstein stuff... even just the sheer gerontocracy of it all... and they think to themselves "yeah, if all the big career politicians, on both sides, simultaneously died, and we had to pick a new set completely at random, odds are it would be an improvement."
"I would be better positioned to navigate this debate if I knew what deontological bars were or where they came from."
A deontological bar is the cure for what ales you.
"So I’m not sure what’s left of this one under the “don’t break unbroken norms” hypothesis - although I notice this would also justify taking the slightly-less-brutal concentration camp guard job.
(in order to turn down the concentration camp guard job, I think you have to strongly commit to acting as if everyone would follow your rule, even though you know it’s not true - but then how do we handle the military disarmament example?)"
Sometimes it is worth attempting to establish new norms, maybe by refusing such a job and encouraging others to do likewise.
That just sounds like the principle of double-effect with extra steps.
Others have pointed out the individual vs. group action distinction, but I think I have an answer for the concentration camp guard question:
Going from 100% to 90% brutality will remove an amount of badness from the world equal to 10% of the badness of a single concentration camp guard (over replacement--being a factory worker or something instead).
It therefore only makes sense to take the job if I think 0.1 marginal brutal guards is the most good I could reasonably hope to do (= the evil I could reasonably hope to remove). I think 0.1 MBGs is a pretty low amount, since it's not as though any one guard could hope to stop the gas chambers or the machine feeding them; I'm not convinced a few beatings one way or the other really makes much difference in comparison to the Holocaust.
Again, unless the system is sufficiently loose that one person can make a lasting impact, it's almost certainly possible to do more good by working elsewhere. Help refugees fleeing the country so they don't end up in the camps in the first place, things like that. I think even just working a civilian job to try and keep society running so people don't starve is going to be more of a net positive than giving out slightly fewer unnecessary beatings in Auschwitz.
For your section about Ukraine abolishing their military, the key is that just because there is a law against something doesn’t mean everyone will follow it. We have laws against murder but people still murder each other, that doesn’t mean we should abandon the law. As a society we generally agree that political assassination is bad but there are still political assassins, if our favored politician is killed the vast majority of people don’t think that the appropriate response is to kill the opposing politician in revenge.
So countries can’t come together and agree to get rid of their armies because there’s always a chance of someone going rogue and invading their neighbors anyways, so the cost is too high. But what they can do is enter into alliances with each other to cut down on the number of people they are worried about. They can sign treaties like the Geneva Convention limiting what types of weapons and tactics will be used. Opposing nuclear powers can agree to decrease their nuclear stockpiles, each one allowing the other to verify that they are upholding their end of the deal. These are all ways of countries trying to find the equilibrium between being as peaceful as possible while still keeping themselves safe and prosperous.
" If one company’s product has a 90% chance of ending the world, and another’s has an 80% chance of taking over the world, giving your money/support/encouragement to the 80%-ers seems kind of like endorsing evil."
Is this how a psychologically healthy species looks? This is a perfectly reasonable framing of human reality, talking about companies and how they'll end the world. Here, let's remember the context. Ready? So for millions of years living things basically performed a global bloodbath. We come out of this. There's Phalaris who put the brazen bull on the map. There's all kinds of mentally ill cults, whose ideas were basically just psychopathy maximization tools if you really distill things. Then there were thousands of years of domination, war, torture, while humanity clung to crumbs of actual values like ethics, like wisdom, and so on. Not because these things actually stand a chance in an evolutionary game, but because without them, the ultimate thing that wins, can't win in its own ultimate way(that thing is power/survival, which is at odds with wisdom and ethics, by the way, and not compatible with it in the way that DNA and beings that operate on DNA pursue power/survival).
Now here we are. We've lobotomized people whose thinking and behavior we simply don't like, and doesn't appear to fit in, a few decades ago. Not centuries, decades. We set off nukes. Some of us think that was kind of cool. Most of us don't even bat an eye("But but but but! Haven't you seen Oppenheimer?" Fuck you).
Just business as usual. The global quantity of child abuse(and indeed, adult abuse, and animal abuse), would satisfy the needs of a very sadistic god. We're just truckin along and debating two companies who have a 90% chance of ending the world and an 80% of taking it over. And this isn't some insane abstraction. We're just out of touch. Do you get it? We're out of touch, there's no whistle loud enough in this cacophony of psychopathic and psychotic noise. The malevolent children have managed to sedate and exhaust and drown out the adults in the room. Mommy and daddy can't seem to help, and it is an emergency for everyone.
Re:, taking a job as a concentration camp guard thinking you can improve the situation:
Would hubris be considered a deontological bar?
The thumbnail should have been him rapping instead of drinking beer
I am always curious about the apparent obsessive drive to find a single coherent ethical or moral philosophy that will apply everywhere and to everyone, regardless of circumstances. I satisfied myself long ago that there isn't one--the human race is too divided by different definitions of right and wrong. What seems right to one faction seems wrong to another, and this seems inherent in human nature. I believe that this is due to evolution--we are selected to compete against each other organized into small in-groups, and to use justifying logic to give our own in-group some sort of persuasive advantage (it probably helps attract allies). This means that regardless of any moral standard "society as a whole" (which in this paradigm doesn't actually exist) adopted, it would satisfy primitive emotional impulses for any number of small in-groups to form a coalition in opposition to it. In the battle of cultural memes, factional emotion will beat universal logic every time. We aren't designed to agree with each other. "I believe that there is a plurality of values which men can and do seek, and that these values differ."
The alternative is to invest our resources into political processes that help ensure that we largely get along with each other with the minimal level of destructive conflict and violence. (We probably can't entirely eliminate the potential for destructive conflict and violence, but we could successfully seek to minimize it.) Politics is all about resolving inter-factional conflict with a minimum of destructive waste, and we know of certain proven methods for doing this. "Democracy is the worst form of government except for all the others" and all that.
Lets stop looking for universal logical solutions and start thinking about better ways to encourage people to tolerate each other.
I remain genuinely confused about why people (especially non-theist people) who seriously believe that the extermination of all life on earth an imminent and inevitable event would be against using violence to avoid that future. Surely even "unstable, lawless, chaotic civilization" is a better outcome than "lifeless planet".
I can imagine there's some argument that using violence will only accelerate the extermination of all life, but surely we'll eventually cross a threshold where violence is the only option remaining, right?
> but surely we'll eventually cross a threshold where violence is the only option remaining, right?
I think you are overestimating the effectiveness of violence.
Violence is basically always an option, in that you can grab a knife and go stabbing random people on the street until they lock you up.
Going around protesting and waving signs is also almost always an option.
Going and crushing your own fingers with a hammer is also an option.
There is little reason to suppose, even in a desperate situation, that violence will be more effective than hammering your own fingers.
I can think of a few possible reasons:
1. They SAY they seriously believe that the extermination of all life on earth an imminent and inevitable event, but deep down, they actually do not.
2. They can't think of any specific application of violence that would be effective in stopping said event.
2a. They are aware that historically, non-violent resistance has a better track record at effective long-lasting change than violent resistance. (Surprisingly few people seem to know this little fact)
3. If they seriously believe that the extermination of all life on earth is INEVITABLE, then by definition no act, violent or non-violent, can stop it, and they might as well enjoy their lives. Going to prison on terrorism charges tends to interfere with enjoying one's life.
Why not, act as if your maxim would be a general law that would suffer some violation by bad actors?
I think one thing about this is it massively underestimates the extent to which companies run on trust and an actively anti-organization employee can do damage.
It's not just that one person who doesn't want the organization to succeed can get by with 10% less productivity. It's that someone who is serious can easily have a productivity of -10, possibly more.
Again, this depends on skill and willingness to break other dentological bars. Bringing a poisoned salad to the company pot luck or planting bombs to destroy the whole building may well be highly effective, but those actions break a different dentological bar.
Still, there are many opportunities for large scale property sabotage (Like putting copper shavings in the server room ventilation system, see the CIA simple sabotage field manual https://www.gutenberg.org/ebooks/26184). There are many opportunities to plant subtle malware into the computer systems if your writing code. (See the relections on trusting trust) There are also opportunities to play office politics. To tie everyone up in endless meetings discussing politically contentious but ultimately unimportant issues.
I think EDT does nicely in situations like this.
Would it be good news if Ukraine dissolved its military in the interest of pacifism? It would be kind of heartening, but not really since I know that not *every* country is going to follow their example. Even if all but one did, I would guess that the country that didn't is evil. It would only be good news if I knew that every countries leader were identical twins with Zelensky or something.
Would it be good news if I learned that I decided to starting committing crimes for the greater good, such as assisnation? NO NO NO. Even though the rest of the country aren't my identical twins, it provides evidence that maybe hundreds of other Americans are thinking the same things as me. Like the previous example the correlation is low, but the implications are way different! This would both cause chaos for the country and personally endanger me and my loved ones (since as an assassin I am probably a high value target). Moreover, the rest of the country would predict this is happening using psychology and would just ramp up security, wasting resources for no benefit (since now the crimes for the greater good won't even succeed). For example, because of SBF people don't trust EA financially.
What about contributing to the AI race if you think alignment is solvable? I'm not sure what EDT would predict, you'd need to collect some data on how correlated AI researchers are with each other, and then run a simulation. Surely assurance contracts could help tho? (But if assurance contracts work so well in theory, why aren't they used more in practice.)
The AI politics example is probably beyond simulation, but intuitively I viewed it as good news? The drawback is that it provides evidence of AI danger people getting into AI politics as well, wastes resources, etc..., but probably still net positive. The chance of crazies don't play too much into my calculations, even considering that other political movements might have crazies (perhaps this is wishful thinking?).
If I heard that was Scott Alexander started lying on the Internet for the greater good, this would be terrible news, of course! Who'd be next after him, Issac King?
I suppose you might define a "deontological bar" as an X such that hearing "X will be done in an attempt to benefit the greater good" is bad news.
I think one relevant issue here is that unlike atrocity, AI is so new that a lot of the cultural norms around it are still being set, so complete refusal to engage with it could set up at least a subculture with a deontological bar around AI use, which would probably be a useful thing to have in our brave new world.
> retaliatory assassinations by Republicans
This seems to be founded on an assumption that existing assassins were not, themselves, Republicans, which is not in evidence.
Leftists don't want Trump personally dead nearly as much as they want institutions behind him to be torn out at the roots, and he's doing a far better job of that now than he would as a martyr.
The peaceful anti-AI movement does not end up having to keep out a lot of crazies because all movements face that. It has to keep out a lot of crazies because "this threat is extremely urgent, and will result in human extinction" is the paradigmatic example of a threat where most people, if they really believed it, *would* justify assassination, terrorism, and other extreme measures. This is why people have posted that other causes don't have a lot of crazies; your cause is inherently the kind of scenario where being crazy is justified, and it's an uphill battle coming up with reasons why it isn't. The only reason you're not getting even more crazies than now is that most people don't alieve that their cause is as extreme as they "believe" it is and most people don't follow the logical consequences of their beliefs anyway.
It's the same reason why the pro-torture argument is "what if there's a ticking bomb and if we don't torture, then a few million people will die in an hour". It's urgent and has massive casualties and that means you may need extreme measures to stop it.
> most people don't follow the logical consequences of their beliefs anyway.
If people who say "abortion is murder!" REALLY believed it, there would be a lot more bombings of abortion clinics.
The real question consequentialism needs to answer here is: how would one distinguish "deontological bars" from social / reputational costs that aren't worth paying yet?
>Constraint consequentialists believe that you should try to do good things that improve the world, unless those break hard-and-fast rules (“deontological bars”).
How is this different from regular deontology?
Do most deontologists hold that you should avoid trying to do things that have good consequences *even if* you are not violating any deontological rule in doing so?
>calling it "THE Ukraine"
Based second-campist Scott recognizing that the Ukraine is a rightful territory of the Russian state!
I was going to comment on this, but then later in the post he referred to it as "just" Ukraine so I'm assuming it was just a slip-up.
Joking aside, I always heard it referred to as "the Ukraine" prior to Euromaidan when the current Ukraine vs Russia conflict started, and I maintain that it should be an acceptable English way to refer to the country that reflects English-language idiosyncrasies rather than an opinion on the country's status, like how we say "the Gambia"
My family is from Kharkiv (so all-Russian-speaking Ukrainian, albeit live in NYC since the age of 3) and I'll note that part of the reason I think (multilingual) people take it seriously in English is that it mirrors a distinction made in Russian - my dad took until like 2024 to get out of the habit of saying "na" instead of "vi" Ukrainii despite not being able to have his own mother's funeral because the city was being bombarded. Given the country is basically named for being a borderland ("krai" being "edge") it's sorta the difference between going "live on the border" and "live in Border."
That said I'm also aware that "the Ukraine" and "the Iraq" are just things people heard a million times, but by now people ALSO know that some Ukrainians are going to get uppity if you say "the Ukraine" and Scott throwing that into a post that supports Ukraine's military is an interesting test. I have no idea how to process "The Ukraine must conquer Russia and crown Zelinskyy a new tsar."
Polish has the same "na" / "w" distinction and the same virtue-signalling push to address Ukraine with the latter. (Albeit in this case, "na" would supposedly signify Ukraine as rightful Polish territory. Which, yeah, probably the reason historically.)
However, most of the people doing that aren't willing to apply the same reasoning to Latvia, Lithuania, Belarus, Slovakia, Hungary, Montenegro... (with Croatia / Slovenia being a wash) (Specifically Hungary, "we Węgrzech" is highly jarring and difficult to pronounce.) Kinda makes them easy to disregard, even though I'm sure they'll eventually win this one by pure attrition.
That's really interesting to know - in Russian Ukraine is alone as a post-Soviet state using that linguistic construct (although it was using it I think since before there even was a USSR, and plenty of places that aren't former Soviet states still use it, like Cuba also gets "na" rather than "v" and has always been respected by Russia). I can definitely see a lot more reason to feel stubborn about keeping "na" for Ukraine if it's part of a massive pattern in Polish, rather than a weird one-off local outlier as it is in Russian.
I was actually going to comment on behalf of those angry Ukrainians, but you beat me to it. Haha.
Or the Bronxes
How does not accepting the concentration camp job count as something that dooms you to automatically lose if you don’t follow suit? How are you losing?
No! You don't kill people (assassinate* leaders) because killing people is wrong. (Sorry perhaps I'm not sophisticated enough for this blog.)
Edit: OK why have there been almost no investigations into Trumps assassination attempts?
Later; No you should not spread dis/ mis information knowingly.
OK I'm going to say go to some religion for your morals. 'Do unto others as you would have them do unto you.' This covers a lot of behavior.
*Two asses in assassinate? my spelling sucks, I'm still learning words.
What makes killing people wrong? If a religion decrees some morals, what makes those things actually moral? Are things good because the gods decree them so, or do the gods decree good things good? Is killing people wrong because you say so, or do you say so because it is wrong? What makes killing people wrong?
Do unto others.... Do you want someone to kill you? There are also many religious faiths that speak against this, Christians, Buddhists.... Killing people is wrong. Look I know we have a long tradition in our past of killing people. Mostly the 'others'. But we've grown up beyond that and agree that killing people is wrong. Or do you think it is OK?
I am not saying it is okay to kill others, merely that you are not giving a logically sound reason.
"I do not want others to kill me"
*non sequitur*
"therefore I will not kill others."
How does the first statement justify the second? Do you care about not killing ants? Amoeba? Not to the extent you care about not killing humans, so clearly you have particular definitions of 'others' in mind that changes why you should not kill these others.
Hmm, I'm not sure there is some 'logical' reason. One might call it a universal zeitgeist, perhaps the point of view of the universe, (Sorry I was just listening to a Sean Carrol podcast with Peter Singer.) and others may call it God. Whatever, we have all agreed that killing people is wrong. About killing other life, I'll agree this somewhat on a similar level and perhaps I/we just haven't progressed enough to make it wrong. I say this with the smell of bacon cooking for Sunday morning breakfast. I do like to eat bacon.
Anyway, Logic is a wonderful thing, but it's not the only thing. I love my Buffalo sports teams. (Go Sabres!) But if you look at professional sports from a logical point of view, then the whole thing looks a little silly.
What do you call the process of making deductions, including deductions about whether your deductions are correct? I call that logic.
Maybe you don't particularly like classical logic. "I love my sports teams, but if you look at sports from a logical point of view, it looks a little silly." I would say you're working with more of an emotional logic: what is right is what you feel good about. So, there is no math formula you can write that describes why it's fun to root for the Sabres, you just feel it is. (Arguably, maybe a big enough formula could simulate your brain and see what makes it tick, but let's limit ourselves to tech available in 2026.)
Similarly, you see no 'logical' reason to not kill other people, merely an agreement that it makes us feel bad. And, the way to propagate that agreement (because there are always curious little boys and girls) is to teach everyone when they're very little that murder is bad, *simplicitor*. It's a *sin* you see, so you should feel bad even thinking about it.
I have two issues with emotional logic (or, deciding right by what your brain feels is right):
1. What if you would feel much better about something else, if you could see it in front of you, but you never can because your brain doesn't yet feel good about actualizing it (example: cooking).
2. How do people with different senses of justice come to an agreement? Or is it impossible, and if they aren't indoctrinated with your sense of justice early enough you have to have fundamental disagreements until one of you genocides the other?
Oh I love logical thinking. If I'm trouble shooting, fixing, debugging something then logic is the way to go. There is a joy in fixing something, that is different from the joy of your team winning a hockey game. But still a joy, perhaps more long lasting than sports, the tractor continues to work years later. Emotional logic is a term that makes no sense to me. Emotions are beyond logic, or just just a different dimension. I love my sports because of the emotional ride, other fans say they like the game where we blow the other guys out. But my favorite games are those that go back and forth... the outcome remains uncertain till the end. It can maybe be explained in terms of some hormones or chemicals being released... but there is no logic involved.
Oh there are 'gut feelings', "Oh this is a bad situation, I should leave." and you should leave. That is also beyond logic. (Perhaps it's some analysis going on at the sub-conscious.)
1.). I don't understand this? (I love cooking.)
2.) I think we all have pretty much the same sense of justice. That's my point. I guess you can disagree. A jury of my peers is a test of that sense. If people from other cultures have a different sense, then I guess that's something for the diplomates to figure out. Or as you say, war in the end.
With my mild dyslexia I read the title as DENTOlogical bars, and I thought this was about devices to prevent us from grinding our teeth. But as Emily Litella famously said, “Never mind.”
I wrote a mediocre lesswrong post on it once, which now has 12 karma. https://www.lesswrong.com/posts/JwfkYzTehmGsizvuL/failing-safely-is-the-anomaly. My view of this is: by default humans are insane, so insane that the default assumption should be that whatever cause people have is actually crazy and would make society worse instead of better, and the great triumph of classical liberalism is finding out ways to mitigate the harms caused by this.
In retrospect, I think I should have mentioned that the individual is better off too, in addition to society. Example: western flat-earthers who get mocked online are themselves better off than African Boko Haram members who keep murdering people until they die for it. But I think my main point, that anyone with a cause is probably delusional and needs to make safety precautions in case they are wrong, is still valid.
This is my standard for deontological bars.
I think it would be helpful to clarify the exact view at issue here. You might believe either:
1) The best action is the one which has the best consequences provided you don't violate the constraints.
2) The best action is the one with the best consequences full stop but as a fact about human psychology (or less plausibly rational action) one should intend to obey those constraints.
---
As a matter of moral theory 1 seems hard to justify. But once you switch to 2 it isn't clear why you should expect the rule to actually be all that clearly defined. Human psychology is pretty complicated and messy.
I don't think it's particularly controversial that breaking a norm against assassination licenses a retaliatory assassination. Some places even allow it against people who have otherwise been incapacitated and unable of causing further harm, which I think is a bad policy for several reasons. But certainly - retaliatory, or in some cases even proactive norm-breaking against someone credibly attempting to break the norm is, in fact, common sense.
It's how norms are being enforced. They rely on a shared understanding that you are protected by the norm if and only if you observe it.
What you probably meant to say is, people shouldn't try to weasel their way out of norms by reinterpreting them in a way that gives them license to break them. (Such as, declaring that your political opponents writ large - and not just individual assassins - have broken the norm by, I don't know, not showing enough performative outrage against the individual assassin in question, or by having previously spoken ill of the victim.) You may think such rule-lawyering allows you to preserve the norm going forward, after you've accomplished your goal of disposing of people you find inconvenient, but no, no it doesn't, people notice what you're doing, and the norm gets eroded, and soon you're finding out it stopped applying to you too.
Speaking of speaking ill, I think both the article and the comments somewhat underappreciate the fact that Trump has recently ordered assassinations (and/or kidnappings) of other countries' leaders. This makes his own assassination, specifically, probably not the clear-cut go-to example that you probably think it is. (Even if you think international relations operate on the different set of norms, "don't hurt the hegemonic force or it'll start lashing out" is still a functional norm that breaks when it just starts randomly lashing out on its own.) Trump has very much invited his own assassination by this point, the recent attempts have just left me shrugging rather than worrying about the norms getting eroded (that ship, as they say, has sailed), and if you think me stating this simple fact signifies my own unwillingness to observe the norm, you'd be shooting a messenger (possibly literally). (It mostly signifies me not being an American and worrying more about international norms than your own internal ones.)
But yeah, this is how norms erode. Most people aren't itching to start norm-breaking at the first opportunity, they just find it increasingly harder to gather motivation to proactively defend the norms for people who otherwise don't observe them (if not by the letter, then at least in spirit). At some point, a reset to a stable equilibrium is needed - whether it'll come by [lawfully deposing Trump and credibly pretending he was an aberration] or [civil war until everyone is tired] or [everyone giving up on norms altogether] is left to be seen - but very much depends on whether people's loyalties will ultimately lie in the norms or in their respective tribes / immediate self-interest.
I think that if you want to be fully consequentialist, you also need to take into account the consequences for yourself. And actively causing harm to people you believe don't deserve it (like in the concentration camp example) is bad for you as well (unless you're a psychopath).
Even if you're a saint and would sacrifice your own well-being, you have to weigh the opportunity cost of doing so. It's likely that you can help in some other ways. Maybe try and get some people out of the concentration camp altogether. Maybe sabotaging it's operations, maybe helping end the war faster so that concentration camps have a fewer days left to kill people.
N the hypothetical scenario where there is really nothing else that can help and all you can get away with is by becoming a slightly less brutal guard yourself then it sort of is justified I think. It's just that hypothetical is too far removed from the real world that it doesn't make much sense.
A more realistic scenario where you can influence who becomes the guard and you pick the less brutal ones seem more realistic and also more intuitively good (although you might then also combine it with other subtle ways to help).
Another reason to avoid becoming a slight less brutal guard yourself is that it can change you into someone who genuinely is brutal and you instinctively want to avoid that.
None of these considerations are relevant in the disbanding/keeping the military hypothetical. Keeping a military force when it is not needed might be a waste but it is not actively and purposefully harming anyone - unlike the concentration camp guard.
Just a minute-- how did we get from don't kill democratically elected leaders to don't kill anybody?
Also, there was that post about whether Orban was a dictator. It turns out that "democratically elected" isn't a bright line.
Is there some sort of (naturalistic) "is-ought" fallacy going on here?
Of course utilitarianism almost inherently faces this problem, and particularly rule utilitarianism (what 'deontological bars' sounds like). However it sounds like consequentialists are only hesitant here to act, not because they care about the 'bars', but because acting could set self-undermining precedents. So they're just 'practical utilitarians', which is just same old OG utilitarianism. All that's happening here is a convolution of moral arguments with practical ones, but it does highlight how important epistemic accuracy ('competence') has always been to utilitarianism (e.g. communists).
>For example, you shouldn’t assassinate democratically-elected leaders, even very bad ones
Consequentialism and deontology aren't the only games in town. Maybe you shouldn’t assassinate people because you're not James Bond, but James Bond should, because is and he has a license to kill. Allowing assassinations under very controlled circumstances could be better than having a universal rule of never killing Hitler: or a free for all allowing anyone to kill politicians they don't like. Everyone accepts some version of this , eg. that the police can do it hunts civilians can't.
My personal version of Kant’s maxim is:
Act only according to that maxim which, if adopted by a realistic proportion of rational agents, would best produce the optimal state
And I define maxim as: a universalizable rule of conduct, judged not by moral intuitions about the acts it permits, but by the state it can reasonably be expected to produce, where the standard of reasonableness rises in proportion to the magnitude of potential harm.
I still need to improve it but it’s the best I got currently.
The answer is SharkBot:
$$
S(X) \leftrightarrow \Box(\Box S(X) \to X(S)) \land \Diamond(X(S) \to \Diamond S(X))
$$
S(X) ↔ □(□S(X) → X(S)) ∧ ◇(X(S) → ◇S(X))
"Cooperate if and only if you can prove that your provable cooperation leads to their cooperation, and it's possible that their cooperation leads to a possibility of your cooperation."
Equivalently, we can reduce
◇(X(S) → ◇S(X))
into
¬□(X(S) ∧ □¬S(X))
which reads
"there's no proof that they will cooperate even while you are provably defecting."
The first half prevents you from being a cooperate rock, while the second achieves maximum utility when playing with cooperative rocks. This is not my idea, Joseph Camacho wrote it on LessWrong:
https://www.lesswrong.com/posts/LaCP6WyNzX8kiZn3w/payorian-cooperation-is-easy-with-kripke-frames?commentId=uyG83hDfNBfjPxeou
“Don’t be the first person to defect from a generally functioning norm”.
Great idea! Too bad in real life people may start to justify their actions by (rightly or wrongly) implying the OTHER side already defected, so we must do it too. I am Polish and I've seen that in Polish politics; but I also followed a lot of rightwing blogs (yeah, there were times when I was really EDGY young men) who claimed rightwinger must finally take the soft gloves because the OTHER side defected years ago.
(too bad I can't find this once popular meme about "conservative is a man standing athwart history, yelling stop" with a guy in front of locomotive)
Oh, there's no end to it if we go "who started it first" but today there's an opinion piece in the paper by a columnist who is SHOCKED and APPALLED that the pro-abortion side in Ireland are not stopping at the limited changes to the law they got passed, they want NO RESTRICTIONS ABORTION. This was not what she voted "yes" for in the referendum to repeal the Eighth Amendment!
And I have to say to that "Well, what the fuck did you think was going to happen? The pro-abortion side were open right from the start about their aims and goals. Why did she think they needed the Eighth Amendment repealed, not just amended? or new legislation clarifying when doctors could intervene to terminate a pregnancy when the foetus was non-viable or already dead? They needed it gone to get what they wanted. And any view of how abortion has gone in the UK and USA should have told her the same: it went from "only a last-resort surgical procedure in limited cases" to "fundamental human right".
This is why I voted "no" back in 2018, quite apart from moral view of the matter: "but... but... but they said they wouldn't change anything!" to which I can only reply, God forgive me, "you stupid bitch". Of course the campaign would keep campaigning till they got what they wanted, of course the politicians will fold on this the way they folded back then - if the perception is "there's votes in this", they'll scrap anything and everything because the guys who swore "this far and no further" are no longer in power and it's a whole new ballgame, baby.
Deontological bars. Break 'em, you can't glue 'em back together again later when you find that the result of breakage is not what you wanted.
Re: "conservative is a man standing athwart history, yelling stop"
At least the phrasing is visible in https://thepenngazette.com/standing-athwart-history/
[Re William F. Buckley Jr.] :
>After announcing in the inaugural issue of National Review in 1955 that the magazine would “stand athwart history, yelling Stop, at a time when no one is inclined to do so,” he evolved into a conservative kingmaker, one whom presidents and would-be presidents ignored at their peril.
Once again consequentialists try to judge the success of deontology on consequentialist grounds. This doesn’t take seriously the deontological perspective, which is that ethical questions are orthogonal to their consequences in any this-world sense, and that success, failure, suffering, pleasure etc. can’t be used as evidence either way.
So can you invent a deontological system?
The television show Better Call Saul is about this problem.
As in, the whole show: in the first two episodes someone with a consequentialist temperament gets a hard reminder that he should probably locate some rule-like guardrails here and there. In the remaining 61 episodes he tries to figure out how, which, and where.
The other main cast members represent other recognizable moral positions: a deontologist, a virtue person, an eye for an eye consequentialist, a stealth consequentialist (pretend there’s rules when people are looking, for the sake of the social fabric, but break them as needed when sure you won’t get caught). All of these characters change views profoundly as the show goes on while mostly staying within those categories.
I have never seen allegory handled less dryly or more delicately - so delicately that hardly anyone seemed to suspect it was about moral philosophy when it was on. (It also may be the best work of art produced in this century, if that’s relevant.)
Maybe you're not interested in litigating your object-level examples, but...
> Some of the people working with the companies think there might be a deontologic bar against certain types of mass activism.
I don't think this is true. I spend a lot of time on the relevant parts of Twitter and I don't think I've ever seen a lab employee argue that participating in this specific form of mass activism is violating any sort of deontological prohibition. I have seen some e/accs (and other randos) suggest that trying to slow down AI will kill people (in expectation), and also (more from various bad-faith commentors) that those arguing that unaligned ASI might cause human extinction are morally responsible for things like a crazy person attacking Sam Altman, but _even then_ I don't think I've seen any of them say "and you are violating a deontological prohibition by making such arguments". Partly I think that's because they aren't really operating in that mode, but also because that would be a pretty crazy claim, and trying to make that argument explicitly (rather than implicitly) would render it very obvious that their entire argument is in bad faith, unless they came up with a very surprising and compelling case for that.
I have seen more people make arguments that could be read as "I am deontologically prohibited from participating in many kinds of possible protests that might be organized on this topic, because that would require endorsing claims I believe are false", but most of those people (I claim) would take a 5-10 year pause enforced by global treaty if the only available alternative is no pause. One such person I know does work at a lab. (I don't know more than one such person working at a lab with that particular combination of beliefs.) This feels like a pretty different bar than the one that paragraph is sketching out, and I don't think they would claim that people ought to be deontologically prohibited from any of "working with Steve Bannon, working with Bernie Sanders, working with NIMBYs who hate data centers because they’re a thing that might be built in someone’s backyard (or by non-union labor), training TikTok influencers create short-form videos about the dangers of AI, and holding protests where we chant vapid slogans outside AI company headquarters".
I think the underlying issue is that you’re trying to fit a simple, “single-sentence” rule that generalizes across all situations. That’s like trying to solve next-token prediction with a linear model, it’s just too rigid. You can just expand your model class to include e.g. the fuzzy things that your black-box brain outputs as moral judgements, and which depend on where in thing-space you are. In other words, it’s fine to have your deontological bar be “I don’t do egregiously evil things, as evaluated by my fuzzy black-box moral compass”. Yes, this is subjective, but I think that’s a price worth paying here
I don't understand the unilateral disarmament example. I understand deontological bars as a patch on consequentialism to prevent the consequentialist from taking unethical actions that have good net results.
But the only problem you find with unilateral disarmament is that it doesn't provide good results. So I don't see why your rule for deontological *bars* would need to rule out actions with bad outcomes, since the consequentialist wouldn't feel tempted to do those things anyway. (This is different from a fully deontological system, where the deontologist needs license to not do things which would have bad results for them if others don't cooperate.)
To put my objection another way: You've identified a case where Kantian universalization recommends a bad course of action. But this isn't a problem for the consequentialist, who only cares about *prohibitions* from the deontological rule, rather than *recommendations* from it.
We might take some inspiration here from our understanding of biology. The general rule is that cells can do whatever they want unless and until they threaten the health of the organism that all of the cells depend upon for survival. If and when that happens, the immune system kicks (or at least it's supposed to) in and the transgressive cells are eliminated. Sometimes immune systems get confused, of course, which is what we call auto-immune disease, but the system generally works well enough.
Now suppose that instead of looking only at the cells and the organism they're apart of we zoomed out to coordination and competition among organisms or zoomed in to coordination and competition for energy and other resources among different biochemical pathways within the cells? I submit that we can find analogies to the same order we see established among cells — that is, that the basic rule is that each organism (zoomed out) or pathway (zoomed in) can do what it wants so long as it doesn't threaten the coherence of the structure one level up — that is, in the case of the organism, its colony of kin and in the case of the biochemical pathways, the cell.
Now we have a substrate we might use to establish an ethical principle: in most circumstances, it's best to maintain systems within some critical regime that maximizes adaptive capacity. In practice, this means don't let cells kill each other unless there's some cell that is threatening the health of the organism as a whole, then have the immune system take care of them. But don't have so much order that the cells are basically perfect replicas of one another, never transgressing boundaries all the way up until all are wiped out because of a tiny change in the environment.
I prefer this to `“Don’t do something which would be bad if universalized, unless the norm is non-functioning in such a way that you’d be playing cooperate while your enemy plays defect”.` because "universalized" and "non-functioning" have measurable referents with respect to the hierarchy of systems and even the concepts of "cooperate" and "enemy" resolve into a relatively well-defined hierarchical competition to stay alive. And note that death itself might be understood within this framework as the condition under which a "cell" within some level of the hierachy runs out of the energy budget available to continue to update its self-description to coordinate within that level of the hierarchy, thereby threatening the "organism" within which the "cell" must coordinate with other "cells." So murder is unjust because it preempts that natural process whereas execution may be just because necessary to preserve the process for the "organism" as a whole and other "cells."
Now let's apply this to "don't support an AI company" and political action campaigns.
In the case of "don't support an AI company" then it seems like the ethical target steers us toward making a decision based on whether or not the AI company in question is more or less likely to enhance the well-functioning of whatever body it's a part of. To be concrete, does the AI that a specific AI company is building threaten to destroy the entire economic and political order within which the AI company itself is operating? If the answer is yes or maybe, then the answer is the same as for the candidate concentration camp guard, although the answer to the question is much more obviously yes in the case of the latter, which is why I believe that question is being asked at all.
In the case of political action campaigns, it seems like the ethical target steers us toward making a decision based on whether or not the platform and or ideology of the campaign has the potential to undermine the entire economic and political order within which it operates. That was and is arguably true of a campaign built either explicitly or implicitly on an ideology like Carl Schmitt's, for example. That would be true too of a platform or ideology that openly disrespected the rule of law or the basic principles of respect for human dignity that have allowed for people to live together in community with less violence.
And yes, to your point about early Christianity, as Girard laid it out, Christianity does offer a unique and compelling narrative and set of symbols that help people to understand and accept a solution to the problem of an unending escalation in violence — a solution that Leo is doing a great job reacquainting the world with at the moment.
Another way to encapsulate “don’t be the first person to deviate from a well-accepted norm” is as a special case for “culture is above game theory on the society-wide maslow hierarchy” (though game theory itself probably encapsulates this). And if we assume caring about consequentialism with deontological bars is a sort of “culture”. In other words “you have the luxury of deontological bars if you’re not in a tit for tat”.
I feel inclined to note that "don't ever work with AI companies" can be almost-fully justified on utilitarian grounds if your P(Doom|AGI Soon) is high enough. Essentially, one says that the decrease in P(Doom) from this particular contestant winning is massively outweighed by the increase in P(Doom) from helping *anybody* and thus lowering the chances of a successful Butlerian Jihad. This technically doesn't forbid helping them with intent to betray and sabotage, but that's pretty hard anyway.
The Yudkowskian view, for instance, is that any and all methods of neural net alignment are 100% doomed, and that you'd be more likely to survive literally jumping off a cliff than *anyone* building ASI. This means that essentially all non-traitorous help given to AI companies is -EV, because there is no world where they get what they want (to keep building AI) and also we don't all die.
It seems sort of weird to be debating hyper minutia of ethics if you *really* believe AI is extremely likely to cause total human extinction. Isn't the most relevant question what will work and less what is or isn't ethical? Isn't this close to being the most important cause there has ever been if AI is going to annihilate us?
The “don’t break unbroken norms” falls away when you consider historical cultural norms, like those against sodomy, etc.
But then its just a matter of breaking them enough that you can then call them broken so that you can justify breaking them.
Which is a slippery-slope argument.
But it also applies to your assassination model.
Non sarcastically: I think this issue is one of the major parts for why Harry Potter fanfiction tends to be so popular and prolific.
Presumably purely by accident, but JKR managed to create near perfect scissor setup for this political dilemma. In a world with barely functional government and extremely universal access to lethal weapons, when one political side is trying to actively genocide the other and actively working against peaceful norms by terror attacks, is sticking to pacifistic/ non-lethal means because those are your ideals tantamount to treason, as it gets more of the persecuted side killed in the end?
This works even better, because the descriptions in the books are vague enough, that everyone is casting their own preconceptions on the severity of this situation.
Both the deontologists and the utilitarians were trying to escape cristian theology (to ground ethics in reason or empirical facts rather than divine command. But they escaped through a door that was already shaped by Protestant thought. The very form of their secular ethics) its individualism, its universalism, its systematic ambition, its hostility to mere tradition was Protestant before it was philosophical.
What about "do 10% better than average, until reaching the optimal point"? If everyone follows, it will eventually reach the optimal point.