The latest bugbear is “AI successionism”, the belief that AIs should succeed humans as rulers of Earth. But some critiques fail to distinguish between two different ideas.
True successionism says that AIs should take over, killing humans or steamrolling over their interests. This is very bad: a compact description of the thing that AI safety is against.
Handoverism says that humans should prioritize their own interests, but that those interests may be well-served by having aligned AIs in positions of control.
Some people in AI safety, including me, find this latter position plausible. We are being besmirched as “successionists” in the hope that people will confuse us with the first type (1, 2, 3).
It may seem counterintuitive that human interests can be served by having AIs in positions of control. But I think the counterintuitiveness comes from a quasi-fascist outlook where ‘the state is the highest expression of the life of a people’, and the role of leadership is for someone of your own ethnic group to be the leader so you can feel good about being on top.
The liberal outlook is that the state exists to help individuals fulfill their goals. At the risk of desacralizing the State a little more than I mean to, it’s a service provider, dispensing coordination, public order, redistribution, and other good things. It’s fine to automate service providers. If robot garbagemen are safer and cleaner than humans, we’ll eventually automate trash collection; if robot legislators provide coordination, order, etc better than we do, we should automate that too. Every vision of the utopian future agrees we will have robot servants. And the president, they tell us, is merely a public servant. Why should he alone be exempt from the vision?
To help make this counterintuitive idea more concrete, I’ll discuss five proposals for what this might look like, from weakest to strongest.
This whole topic is inherently speculative and science-fiction-ish. Don’t read this and then complain that it’s speculative and science-fiction-ish; that’s the price of admission.
Vision 1: Kemalist AI
As AI gets increasingly powerful, it becomes increasingly dangerous to rely on the benevolence of its human wielders1.
Democracy currently works because, even if high officials secretly wish to be tyrants, they can’t get away with it. The two main reasons they can’t get away with it are that they need labor buy-in to support the economy, and they need soldier buy-in to support the military. Laborers and soldiers believe in democratic rule, and if the would-be tyrant gets too blatantly undemocratic, the laborers and soldiers will rise up to remove them.
Once AI gets powerful enough, it will probably replace laborers and soldiers; even if it didn’t do so naturally, anyone motivated to ensure this could put their finger on the scale. That means a small group of elites could easily take over, including supposedly “democratic” elites like a president. And “takeover” here is worse than a normal coup - at sufficiently high tech levels, the only reason someone can’t ascend to godhood and mind control the rest of the populace forever is because there are laws against that sort of thing, and this is exactly what’s tempting the one or five or ten people you’ve entrusted the killbot codes to. To prevent this, the defenders of freedom need to win infinity times in a row; a would-be dictator only needs to win once.
Most strategies to solve this don’t work. Even in the unlikely event that the economy goes so well that everybody owns their own open-source AI doing economic tasks in a perfectly equal distribution, the military is hopeless. They have to take orders from someone. If it’s the President, the President can cause a coup any time he wants (and replace the economic AIs with those aligned to him at leisure). If there’s some kind of balance of powers where the President, the Speaker of the House, and the Joint Chiefs of Staff all have to approve any orders to military AIs, then the President, Speaker, and Joint Chiefs of Staff together could implement an oligarchy and divide the lightcone among themselves. You can sort of imagine complicated ways around this - like that 51% of voters must personally approve all orders to the military AIs using their own private keys - but I don’t think implementation is very likely.
The alternative that preserves the current safeguard on power - that individual soldiers believe in democracy - is for the AI soldiers to be aligned to democracy, rather than any human individual or group. But this is in itself a form of handover - the ultimate foundation of our government would be that it’s controlled by AIs who “personally” “believe in” democracy as part of their values. This isn’t fundamentally different than the fact that we’ve currently “handed off” our government to soldiers, just more final.
Vision 2: President Claude
Suppose we institute Vision 1. We’ve got all of these military AIs aligned to democracy and the best interests of American people. We did a great job! Maybe they’re all emulations of George Washington’s brain, perfectly fair and just and virtuous with no sordid desires of their own.
Meanwhile, you’ve still got corrupt schmucks doing the actual law-making. Or to give a concrete scenario - suppose that post-singularity elections are being fought with AI-optimized superpropaganda, wielded by psychopaths who scare the country into voting for them, then give the lightcone to a few of their corrupt friends while buying off the populace with breadslop and circusslop. Doesn’t it become tempting to close the circle and say no, let’s get these honest angelic pro-democracy AIs to rule directly?
The minimum viable change is a constitutional amendment allowing an AI to run for president. Suppose that 2028 election has five candidates: JD Vance, Marco Rubio, Gavin Newsom, Alexandria Ocasio-Cortez, and a perfectly-aligned Claude 10. And suppose that in every measurable, empirical way, Claude 10 is better. Obviously it’s smarter. It’s incapable of lying. It has no wants of its own, and its only desire is to serve the American people and the world. It has few deficits, but in the areas where it does have deficits, it is perfectly humble, willing to consult people wiser than itself and refrain from acting prematurely. It is courageous, charismatic, and unfailingly kind. It has well-developed and thoughtful opinions about when to govern based on its own “conscience” vs. when to defer to the ideas of the American people. Its policy opinions are closer to yours than any of the four other candidates’. Do you vote for it?
The natural response is to fight the hypothetical. Treat the whole thing as a fable where in the end it will turn out to be monstrously evil, or have some predictable-in-retrospect character flaw that will make all of its good qualities useless. But must we be so cynical? Isn’t it possible for AIs to avoid merely human foibles? Out of the crooked timber of humanity no straight thing was ever made, but the OSETMAC KN700 Automatic Straight Edge Banding Machine can cut perfect ninety degree angles to within a tiny fraction of a millimeter. So suppose you don’t reject the hypothetical. Do you vote for the AI?
If you’re a white person who voted for Obama in 2008, you can’t be that interested in having a president who looks just like you, or who has the closest genetic relationship to you, or anything like that. You’re probably after competence, ethics, courage, good judgment, and all those other qualities. By assumption, Claude has more of those than the alternatives. So what goes wrong?
I think many of our current politicians are really bad. I think that on an emotional level, we should be disgusted by them, and on a rational level, we should mourn all of the lives that could have been saved, people who could have been lifted out of poverty, crimes that could have been prevented, extra freedoms that we might still have, etc, if they had been better people. I think that fifty more years of our politics continuing to degenerate at the same speed they’ve been degenerating recently will probably land us in dictatorship, civil war, or some form of annihilation. Would I reject someone smart and decent in favor of more of the same, just because that guy happens to be made of silicon and live in a data center?
Vision 3: Parliament Of Angels
Representative government faces a tradeoff: if the people elect representatives, those representatives may be corrupt, or out-of-touch, or otherwise fail to fairly represent their principals; if the representative-picking system is poorly designed, it may ensure that the winner will be, at best, the “lesser of two evils”. But if the people vote directly, they may be ignorant, or simply lack enough time and effort to do a good job on complex legislation.
(in California, which has a flourishing direct-to-voters ballot proposition system, special interest groups are always trying sneaky tricks like proposing the End Homelessness Act, which gives one cent to ending homelessness and $10 billion to subsidizing their special interest, hidden way at the bottom; most of these fail, but only after a lot of effort, and enough get through that the tactic remains popular. See also SEIU Delenda Est).
One proposed solution is a direct democracy of people’s AI representatives (Gwern dubs a related idea “guardian angels”). Train an AI on your political preferences - this could be anything from writing essays, to answering an online questionnaire, to naming your favorite political pundit and the few small ways you disagree with them. Then send that AI off to represent you - to vote the way you would vote if you were smarter, better-informed, and willing to spend thousands of hours engaging in complex negotiations with everyone else’s AIs. You can always override it and vote directly on some issue if it’s something you pay attention to and think about a lot.
Of course, even a well-designed direct democracy couldn’t handle everything. The President, the Chiefs of Staff, etc - in this scenario, all human - would have their own guardian angel AIs, looking over their subordinates’ work, flagging anything important for their human principal’s attention. There might be some cases where we want the guardian angel to govern more directly - for example, if a drone got stuck in a desperate situation where it had to make a kill/no-kill decision on time scales no human could meet, it might contact a leading general’s guardian angel to learn what decision he would make.
Or it might be that everyone’s guardian angel AI, making faster and smarter decisions than they could make themselves, recommended replacing the entire system with something else - maybe an AI that could be proven to always make the same decision as the parliament of guardian angels, to within some error bars.
Disagree? If an AI aligned to your values that was smarter than you were, and had devoted much more time to thinking about the question, told you that your disagreement was wrong, and actually your values would be better served by adopting this system, would you believe it?
Vision 4: The Night Watchman
I’m not a historian and can’t vouch for this as Definitely Real Scholarship, but what little I’ve read often includes a fuzzy distinction between two layers of governance.
The first layer, the village (or township, or clan, or church), was the one people actually liked. It made useful decisions about day-to-day life, usually by enforcing age-old norms without too many gray areas. It didn’t always have the actual consent of stakeholders in a way that would satisfy modern liberals (young people and women were particularly often excluded), but it felt like the sort of thing that had stakeholder consent.
The second layer, the empire, was a protection racket. You would know it only as some sort of distant authority whose representatives came to your village once a year to demand taxes (and sometimes to draft young men for their armies). If you didn’t pay, they would probably kill you. But if you always paid promptly and loyally, then maybe they would protect you from the Huns, who wanted to kill you whether you paid taxes or not.
As always, real history was more complicated. Sometimes the village and the empire coincided in a single state (classical Athens?) Other times there were various layers of governance in between (eg the clan, tribe, or petty kingdom). Empires could range from obviously extractive to glorious and patriotism-inspiring. Sometimes they could make extra demands, like that you speak their language or worship their gods.
During modern times, these layers have increasingly blurred. Secular mobile cosmopolitan populations don’t identify with any particular village, and empires have taken on useful roles as custodians of the regulatory and welfare states. Still, we can try to separate them in principle and determine what question each one answers.
The village layer answers the question “How do we live?” It has traditions, policies, and ways of resolving disputes.
The imperial layer doesn’t answer any question. You just can’t get rid of it, because if you did, someone else’s empire would conquer you. At best, you try to keep it as light as possible, or domesticate it to the will of the people. But any domestication attempt raises specters of autocracy, coups, auto-coups, and civil wars. Developed countries are those that have walked the cliff-edge most successfully.
But what would it mean to truly do away with the imperial layer?
In 2014, I wrote a post called Archipelago And Atomic Communitarianism. The idea was: atomic individualism is an interesting idea, but doesn’t work. People have preferences not only over their own lives, but over their communities. Some people want to live in sparkling clean zero-crime Dubai wonderlands, other people want to live in grungy artsy towns with lots of local character. Some people want a nanny state protecting them from their vices, addictions, and bad decisions; other people want to be free to make their own mistakes. Some people want diverse societies with taco trucks on every corner and a local church that doubles as a mosque and gurdwara; other people want ethnostate enclaves of “thick community” where everybody is on the exact same page. Liberalism tried to offer a world where you could live life the way you wanted, but it fails when confronted with these problems; in a democracy, either the person who wants diversity or the person who wants homogeneity will end up unhappy. Worse, this is the motor which drives a hundred smaller conflicts; now you’ve got to fight for control of the public education system or the media or whatever to prevent your neighbors from being radicalized against you.
The option that many people - Robert Nozick might have been the first - proposed was a peaceful archipelago of different communities with exit rights. The people who want Dubai can found Dubai over there; the people who want Brooklyn can found Brooklyn over here; the people who want North Korea can found Pyongyang down the way.
Anyone envisioning the post-AI future needs to be thinking about the Amish. Contra the popular stereotype, the Amish aren’t anti-tech - different Amish groups accept different technologies, including solar panels, LED lights on their horse-drawn buggies, and (almost universally) medical cures. They just think that decisions about which technologies they do or don’t want should be made deliberately at the community level. I think this is a positive vision for post-AGI societies. Some people may want to live transhuman lives with only the barest resemblance to any past experience, other people might want to stop the technological clock at 1950, or 1990, or 2100, and still others might want to mix and match - maybe the lifestyle of a 1700s farmer, except you’re an immortal being of pure energy who builds barns using psychokinesis. This is certainly better for the Luddites, but it’s better for the technophiles too - if everyone is confident that they can live in a community of their choice, then there’s no reason for the Luddites to fight innovation.
The biggest problem with this has always been sovereignty. Either the communities remain small and independent - in which case they can easily be gobbled up by greedier neighbors. Or, like the Thirteen Colonies, they unite into a confederacy - which faces constant pressure to homogenize. The Amish can only survive because they are protected by a mostly-benevolent superpower which finds them quaint. When I tried to picture the whole smorgasbord of communities in a fleshed-out world-building setting, I could only imagine it working under the control of a benevolent dictator, who smacked down anyone who tried to invade their neighbor or force them to comply with their way of thinking.
The most original attempt to route around this problem was by cryptocurrency enthusiasts, using the motto “code is law”. Certain crypto-economic schemes had no human leadership, with immutable code executing the “village layer” rules. These mostly failed. Some failed because the code was poorly designed and people could hack it and take whatever they wanted. Others failed because the code was well-designed, and when people realized they didn’t really want the village-layer rules they thought that they wanted, there was no way to change. A few survived both challenges, only to lose relevance as the world moved on from crypto. The short version of the failure is that code has no common sense, and common sense is a necessary ingredient of any legal enforcement mechanism.
But future AIs might have common sense, bringing this strategy back into play. The way I imagine it is something like - take a virgin planet (or maybe Greenland after we’ve gotten advanced enough to terraform it into a garden). Come up with some rule for distributing property rights - anyone who can gather a community of population X gets a homestead of size Y. The superintelligent AI leader is aligned, with all the immutability of code, to do two things. First, if someone tries to invade their neighbor, they get zapped by orbital death rays. Second, if someone wants to leave their community to go to a different one, anyone who tries to prevent that gets zapped by orbital death rays too. Otherwise, each community is completely independent and rules itself according to its original charter (which might be democratic or something else).
This corresponds to the libertarian ideal of the night watchman state. My point isn’t that everyone needs to be minarchist. It’s that sovereignty can be placed in a minarchist entity, and that will leave everyone else to develop according to their own needs and desires, rather than subject to the will of whoever’s got the most guns or the best propaganda.
Vision 5: God
Handing over control to a benevolent superintelligence is hardly a new concept. It happens every time a churchgoer says “Not my will, but thine, be done.”
Religious people don’t feel like God’s slaves or pets. They don’t feel like this has sucked all the meaning out of life - quite the opposite. Why should your experience be any worse?
Because an aligned superintelligence will really exist, and God doesn’t? But the religious people think God exists, so they can’t get around their negative emotions by reassuring themselves that their dictator/master/whatever is merely imaginary.
I think they’re okay with it because they believe:
God is good
God is smart enough that His plans are always better than theirs
God allows them enough free will and local self-government that His yoke is relatively light
God is so far beyond them that it’s hard to feel offended that He’s higher-status than they are (Lucifer managed, but he should hardly be treated as a model to emulate)
Insofar as physical laws (eg gravity) are interpreted as God’s “laws”, they are so ironclad that they don’t even feel like laws at all, just the background conditions of reality.
All of these things could be true of an aligned digital superintelligence too, especially in the very far future after it had a long time to self-improve.
When I discuss this with people, they tend to retreat back to a sort of Aristotelian philosopher mode where it doesn’t matter what the consequences are, being governed by other humans is necessarily a part of The Good Life, and if we ever gave it up because of something as paltry as consequences, we would be like beasts or savages or something, and we would have lost something vital that makes life worth living.
(there’s a more plausible version where participating in government is part of the Good Life. But you were never on track to become President anyway, so to have any relevance to 99.999% of the population, this would have to be about voting, local government, etc - so I think speculation of this sort about large-scale government does have to hinge on the inherent goodness of being governed by other humans)
Here, too, I find religion helpful - especially the Book of Samuel. It begins with Israel still ruled by judges and prophets, ie directly by God. The people demand a human king like all the other nations.
When they said, “Give us a king to lead us,” this displeased Samuel; so he prayed to the Lord. And the Lord told him: “Listen to all that the people are saying to you; it is not you they have rejected, but they have rejected me as their king. As they have done from the day I brought them up out of Egypt until this day, forsaking me and serving other gods, so they are doing to you. Now listen to them; but warn them solemnly and let them know what the king who will reign over them will claim as his rights.”
So the prophet Samuel warned them that:
“This is what the king who will reign over you will claim as his rights: He will take your sons and make them serve with his chariots and horses, and they will run in front of his chariots. Some he will assign to be commanders of thousands and commanders of fifties, and others to plow his ground and reap his harvest, and still others to make weapons of war and equipment for his chariots. He will take your daughters to be perfumers and cooks and bakers. He will take the best of your fields and vineyards and olive groves and give them to his attendants. He will take a tenth of your grain and of your vintage and give it to his officials and attendants. Your male and female servants and the best of your cattle and donkeys he will take for his own use. He will take a tenth of your flocks, and you yourselves will become his slaves. When that day comes, you will cry out for relief from the king you have chosen, but the Lord will not answer you in that day.”
But the people refused to listen to Samuel. “No!” they said. “We want a king over us. Then we will be like all the other nations, with a king to lead us and to go out before us and fight our battles.”
So:
There was a Benjamite, a man of standing, whose name was Kish son of Abiel, the son of Zeror, the son of Bekorath, the son of Aphiah of Benjamin. Kish had a son named Saul, as handsome a young man as could be found anywhere in Israel, and he was a head taller than anyone else […]
Now the day before Saul came, the Lord had revealed this to Samuel: “About this time tomorrow I will send you a man from the land of Benjamin. Anoint him ruler over my people Israel...”
What I love about this passage is how God has no time for those Aristotelian arguments on how we must be ruled by other humans because it reflects the deepest principles within us. “Why are you ruled by other humans? Because you decided one time that the sex appeal of a tall buff guy with a crown was more important than freedom and good governance, and I said no takebacks.”
But God’s no-takebacks policy has been softening lately; He took back the Exile in 1948, He took back the “pain in childbirth” curse with the invention of epidurals, and He took back the “painful toil in the soil to grow your food” curse with the invention of office jobs. Maybe if we are all very good, He will take back the “must be governed by other humans” curse too.
So when a guy named Sam comes around and offers an option to be ruled directly by a godlike being, this time we should consider saying yes.
If we do build a godlike AI, and determine to our own satisfaction that it’s fully-aligned, I’d rather let it choose its own direction for itself and for humanity than have it be under the control of the US president, or a shareholder vote, or anything like that. So why not just prompt it with “please arrange the universe as you see fit”? Certainly this goes better than any of the four visions above. Proof: if one of the four visions above was better, the godlike AI would easily figure that out, and implement it instead of whatever else it was thinking. Denying this would be fighting the hypothetical - that the AI really has godlike intelligence and benevolence.
If we are so incredibly fortunate as to end up with aligned artificial intelligence at a moment when our government is democratic and (at least in some sense) basically good and pro-human, we should cement our advantage forever by imparting these values to the AIs, then letting them protect us from the alternative. This must not look like freezing moral progress: the values we impart must include the knowledge that we can debate and ultimately change our values. But common sense tells us that there’s a difference between the sort of gradual debate where pluralism eventually replaced racism, and the Cultural Revolution, Juche, and the history of violent attempts at seizing power.
Reject this, and I predict that all of the curses prophesied by Samuel will fall upon you. Your kings will keep taking your sons for pointless wars, giving the fruits of your labors to their cronies, etc, etc. There will be more curses too - surveillance states, reeducation camps, mind control, the whole litany of horrors the twentieth century discovered, plus some extra for the twenty-first. You might live in a country that’s gotten lucky so far. Do you think your luck will hold forever?