Motivated reasoning. Sub-conscious. 10% or whatever chance of AI Doom is such a scary idea, more so even than nuclear war etc, that I need to counter that fearful emotion. Hence coffee and halibut.
You beat me to it. You see this all the time in my field (teaching). I say, “Kids these days are not alright,” and then someone pulls up a quote from Plato saying that the kids in his day weren’t alright either, as if the fact that someone in the past was concerned about young people proves that any problems today are illusory.
I completely agree with your point - but disagree with your example of it! The point of the Plato quote isn't to say "kids in classical Greece weren't okay" halibut-style; the point is that Plato's description of kids is *so similar* to present-day descriptions of kids that people literally can't tell whether the kids being described come from classical Greece or present-day Europe unless they're told the origin of the quote - and therefore there probably isn't a downward trend.
A more interesting argument that can be made from the same observation is that the reason why Plato's description is so uncannily similar to ours is that "how kids look to an elder" is a sort of constant, and kids will always look that way to elders no matter what the kids are actually like, therefore we need proper metrics rather than just elders' impressions.
(Not endorsing any of these arguments myself, by-the-by, just setting them out because I think you aren't doing them justice!)
Which one are you thinking of? Because I’ve seen people quoting Plato (framed as Socrates):
> The teacher in such case fears and fawns upon the pupils, and the pupils pay no heed to the teacher or to their overseers either. And in general the young ape their elders and vie with them in speech and action, while the old, accommodating themselves to the young, are full of pleasantry and graciousness, imitating the young for fear they may be thought disagreeable and authoritative.
I know the Plato quote is fake, but it's worth remembering that Plato lived at a time of great Athenian declinel. The kids of his day were the ones who would grow up to lead Athens into a bunch of badly-concieved wars that would end with Athens as a Macedonian vassal. The philosophers who came before him were having civilised dialogues, the philosophers twenty years younger than him were lying around in the street wearing barrels.
<i>A more interesting argument that can be made from the same observation is that the reason why Plato's description is so uncannily similar to ours is that "how kids look to an elder" is a sort of constant, and kids will always look that way to elders no matter what the kids are actually like, therefore we need proper metrics rather than just elders' impressions.</i>
(a) That quote is fake anyway.
(b) Athens suffered a decline in power and importance during the fourth century which it never recovered from, so "Fourth-century Athens was in a simular situation to us" isn't quite as reassuring as the quotemongers seem to think.
(c) Even if the quote were actually true, you'd still need way more datapoints to show that ""how kids look to an elder" is a sort of constant", and the mere fact that everybody always brings up this one specific quote should make you suspicious that these other datapoints don't actually exist.
Agree with (a) and (b), but for (c) I think we actually do have some datapoints for the phenomenon: our experience of how we were regarded by elders when we were kids, and our experience of how we find ourselves regarding kids now that we're elders ourselves.
I do think there are also other works of literature that reference this phenomenon - Wodehouse and Jerome K. Jerome spring to mind, but I'm sure there are others besides - and suspect the "Plato" quote is favoured simply because it appears to show the phenomenon being present over the longest timescale (and possibly because Plato has considerably more cultural clout than P.G. Wodehouse..)
That being said: I'm personally not convinced the phenomenon actually exists - in other words I'm not convinced that "how kids look to an elder" actually is a constant - I'm just suggesting that, should one wish to argue that it does, one has more ammunition than simply that "Plato" quote.
The other literature you mention is all from the post-industrial West, which is historically unusual in many respects. To show that the phenomenon is actually universal, we'd have to look at date from a variety of different societies and time periods.
"I would there were no age between ten and three-and-twenty, or that youth would sleep out the rest, for there is nothing in the between but getting wenches with child, wronging the ancientry, stealing, fighting — Hark you now. Would any but these boiled brains of nineteen and two-and-twenty hunt this weather? They have scared away two of my best sheep, which I fear the wolf will sooner find than the master."
I read a book where a character was complaining about youth. "At age 10 they should be shut up in a barrel and fed through the bung hole. At 18 you decide whether to let them out or drive in the bung."
Heinlein. I think it's one of the Lazarus Long quotes (but I'm willing to be corrected). It's been misattributed to Mark Twain. And the lower and upper bounds of the age range change as it's been misquoted.
The real issue is culture drift, as Hanson has been exploring lately. The kids have, in fact, been getting worse continuously for several millennia, by all the standards both originally and incrementally. If you think they're getting better (or at least were until you were full grown) you can make an argument for that.
>The kids have, in fact, been getting worse continuously for several millennia, by all the standards both originally and incrementally.
Kids of the last couple of generations are stronger, smarter, faster, read more and write more than almost any generation before them. to the people of ancient Athens on average the kids of today would be unusually tall, buff, intellectually capable and their skin unblemished by pox-marks.
most of that is due to them being one of the few generations in history to get such good nutrition ,such broad access to information and lower disease burden but still.
I started out being mildly disappointed by your comment, then visually skipped down to where you were posting xkcd and smbc links and thought "oh yeah, ok".
No, friend, you skipped the actual point, which is where the standards are coming from. Yes, by your standards today being taller and handsomer and smarter is worth being utterly corrupt morally and worthless in terms of piety or physical courage.
I'm sorry that you hallucinated that I said "worse by all standards" but that's on you. Maybe reading comprehension hasn't improved by as much as people who make comics for a living think.
They blurred the line between morality and physical beauty so much that a sexy person stripping was a plausible defence because beauty was seen as a sign of having the favour of the gods.
And of course they'd view us as awful in some ways, we don't worship their gods, we don't follow their social customs. Awful!
The point is that social customs are always changing, generation after generation.
Some people interpret change as social decay, they see the youth are moving away from the almost-random set of morals they themselves embraced on the day when society finally mostly figured out morality just as they hit young-adulthood and decide that means society is going down the toilet.
Though that would of course also apply if time were reversed. Someone brought up in the 18th century catholic church faced with a future that included the Sacred Band of Thebes would see that as a sign of awful moral decay.
Because socially popular morality is an almost random walk over time. As flighty as fashion. depart in either direction through time and pretty soon society starts to look terribly immoral.
Now it sounds like you're partly agreeing with me, so I'll be charitable. Yes, all cultures drift over time (Robin Hanson has been writing some great stuff on this very recently). But specifically by the standards of BC 3000 the ancient Athenians were already hopelessly degenerated in ways at least analogous to how we are hopelessly degenerated by Athenian standards. If you just average the values of the human population of, say, the steppes on westward every 500 years or so, you'd observe a slow decline calibrated by the original standards of, say, the Yamnaya or by increments (each 500 year block looking at the more recent).
But no, I'm not talking about surface-level ideology (like which gods or what customs). Your bit about interpreting change as decay is the bog-standard explanation and requires no elaboration. I'm making a different point, which is that actual substantive change has been happening over time, and that this change would be seen as bad by people from the past. We think of ourselves as better than the people in the past - you wouldn't lynch a black man for being seen with a white woman, or (further back) capture an escaped slave at gunpoint to return him to the authorities. But -could- you, if you wanted to? Probably not. You definitely couldn't row a longship up an estuary to kill armed men, or live out of a wagon while conquering your way across Eurasia on horseback. We'd like to paper over this, but the fact is that our own ancestors would think we suck - unblooded boys who in their manners and customs are more like girls even than boys. And that's fine. That's what becoming civilized looks like. But let's not kid ourselves about "lol the ancients were just boomers who though the kids were no good".
Sure, it’s no rebuttal to a general argument that Kids Aren’t Alright. But I think he’s fair game when people are arguing about a specific failure mode. Specifically that the kids are a certain kind of wrong.
Good job patting yourself on the back for having explained away the outgroup's behaviour as a consequence of them having characteristic personality flaws?
The intro to Scott's post only works due to the arbitrary restriction to technologies in
> Maybe the argument is a failed attempt to gesture at a principle of “most technologies don’t go wrong”?
Drop the qualifier that the thing to go wrong must be technological, and you are one step away from a good candidate for a heuristic that people in the real world actually use - "most threats drummed up by people who stand to gain from you believing them don't play out". Surely this heuristic has its utility, especially if the Cassandra is sufficiently socially removed from the target that she could shrug off being revealed as a fraud later. The prototypical catechism on applying it takes the form of "sketchy outsider tries to scare you and then make you buy his product" - think Melville's "Lightning-rod Man" (which I actually had as school literature, for English class in Germany). I'd reckon this is why this anti-AI-risk argument is also often accompanied by hinting that AI safetyists are icky.
It's also unsurprising that people would feel betrayed if you pulled a reverse card on the heuristic - these sorts of simple rules are often perceived by the holder as hard-earned pieces of life wisdom that comprise their advantage over the younger and more naive, and to insinuate that they actually don't give an unambiguous answer is to devalue them and by extension the advantage.
The classic modern example of the character you're describing is the con man "Professor" Harold Hill, from the musical "The Music Man", who drums up a moral panic over any arbitrary recent change in order to sell them a bogus solution:
Not necessarily sub-conscious. Putting on your Talmudic thinking cap, first you decide what the right answer should be based on your experience, or what you think the world should be like, and then choose your evidence.
This is definitely correct, but I would say all reasoning is motivated, so you still have to look a little deeper to figure out what makes the motivation or reasoning Bad. Comparing the Coffeepocalypse arguments to Scott's counterarguments, obviously we both think Scott's reasoning is better and more convincing, but why? Well, he's goes through the steps of considering counterarguments and examples and then dismissing them for coherent reasons...but what makes those reasons/reasoning coherent? Well, I think one of Scott's main motivations is to have a worldview where his beliefs about how the World works actually align with how the world really works. In other words, his motivation is primarily to "Truth."
Of course, we also know Scott also has a higher prior than many that any sufficiently advanced AI will destroy or enfeeble Humanity, and he is also motivated by that. I think the Coffeepocalypse arguments are intended to lower that prior; its not convincing because the reasoning is Bad and someone of Scott's caliber will see right through that. But most people are not up to Scott's caliber of reasoning and therefore may lower their prior for AGI bringing Ruin, and thus will be less motivated to resist it. Again, I don't think their arguments are intended to be consistent, they're just trying to move priors one way or another. They don't work on me, but mostly because I share the prior that AGI is potentially dangerous, and one of my main motivations is having a worldview that would be accepted and explicable to Scott Alexander and his readers.
I think the entire argument is against expertise, especially regarding "new technologies". It's "Experts tell you something, but look, here experts were wrong about something similar, so just distrust experts in general".
And people more vividly remember "experts were humiliatingly wrong" stories than "experts warned us of something and managed to prevent it" stories. Also more than "experts didn't warn us and some disaster happened", those stories are least memorable cause there's no heroic protagonist - there's a lack of one.
I feel like if this was the argument, they would have made more of an effort to show that it was "experts" who were against coffee. Instead they blame kings, queens, and alcohol vendors.
At least to me these register as critiques not so much against experts, but towards predictions in general. ”People are bad at predicting things. Nothing ever happens, so you shouldn’t worry.”
I'm thinking of the person I know who always make a similar argument, and I think she aims it at the conjunction of experts + long term predictions, especially catastrophic ones.
Scott wrote:
"And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS)."
The person I'thinking about would say that none of those were predicted enough in advance to qualify as long term predictions.
There's the notorious documentary, filmed in 2019 and released on Netflix in January of 2020, that went into detail about how the Wuhan wet market was a very good candidate for the origin of the world's next big pandemic...
It’s worse than that. People are bad at predicting things, what they predict usually doesn’t happen, and when big things happen they usually weren’t predicted.
So, by this logic predictions of AI doom should make me update that it’s *less* likely to happen!
In the coffee example, you're right. But the usual argument is usually framed around experts.
Also, experts might be the wrong term here. Maybe it's closer to "elites", which Kings and Queens definitely fall under. I could imagine someone saying "President Clinton claimed X but actually Y happened", about, say, the economy. And this would count as proof that elites get things wrong, even if Clinton isn't actually an "economic expert".
There are much more concrete examples re. the economy. President Obama famously claimed that, without his proposed stimulus, unemployment would reach 8%, but the stimulus would prevent this result. We got his stimulus, and unemployment went over 10%, and stayed above 8% for two years. Now, I don't know whether any (let alone most) economists would have supported his claim in 2009, so this doesn't exactly reflect on economists' capabilities. But it sure does reflect on politicians' and other "thought leaders'" use of expertise claims to advance their agenda.
To be fair, presidents have a huge incentive to express overconfidence in their policies, since it directly relates to their electability. Unfortunately, we don't live in a society epistemically humble enough that people are more likely to vote for a candidate who expresses "The best evidence I have suggests this will probably help."
To an extent, what presidents try to do probably reflects what they think are good ideas. But what they say very likely isn't an accurate reflection of what they think.
Sure - politicians do politics. Maybe, sometimes, they really think their policies will benefit their constituents in significant ways (as opposed to getting more votes to get power, or to stay in power).
But my point was that people who claim the authority of experts to advance their agendas deserve fully as much skepticism as people who advance their agendas based on no support from anyone.
I know it's tangential, but creating a forecast that underestimated the unemployment rate by 2% doesn't strike me as an especially damning indictment of economists. I'm happy they got it within 5%.
The claim you attribute to Obama may be famous, but I haven’t seen any evidence that he actually made that claim.
If I may offer a guess as to what you are referring to: On January 9, 2009, Christina Romer and Jared Bernstein were not working in the White House but they had a semi-official status because they had been chosen for jobs in the White House starting on January 20, when Obama would be inaugurated. They wrote a paper projecting the effects of a stimulus package, which included a graph of projecting unemployment with and without a stimulus. It showed unemployment topping out at 8% with a stimulus and 9% without.
Their baseline projection turned out to be wrong: in February 2009, when the actual stimulus package passed, unemployment was already at 8.3%.
I almost take it as the opposite: moral panics by people with little-to-no understanding of the issue, but high status, are common. As an example of where this argument somewhat works, I'd provide most "normal" concerns about AI (e.g. taking jobs or misinformation).
Where it falls flat is in explaining Geoffrey Hinton, Stuart Russell, Yoshua Bengio, etc.
If you tried to make the analogy closer to Hinton, Russell, and Bengio you'd get the "moral panic" raised by Manhattan Project scientists relating to the nuclear bomb potentially resulting in human extinction.
Re: "the experts" - did you take counterfactuals into consideration, as in how planet Earth could have been had we not placed so much faith in experts, and thus perhaps had taken a less foolish path(s)?
This counterfactual is pretty easy, without trusting experts we'd still be living in caves. We especially trust experts every day of our lives. I flew on a plane just the other day!
>This counterfactual is pretty easy, without trusting experts we'd still be living in caves.
Well said, and quite literally true! Every time I step into a building, I'm relying on the structural engineers and the people who designed and ran all the processing steps for all of the materials in the building. _I_ certainly don't have the relevant tensile and compressive strengths and safety margins at hand.
More explicitly than just “experts,” it’s distrust of Silicon Valley tech bros who want to feel important and be at the center of the narrative. Crypto hype is fresh on everyone’s minds - the blockchain bros cried wolf, and then when everyone came to look and the wolf wasn’t there they said, “no wait you just can’t see it, only we can see it because we’re better/smarter/more technical than you!”
Most of the media is buying the AI-sooner narrative because it gets clicks, but people are also understandably exasperated at media and negativity bias and doomerism in general. There are a LOT of reasons to be skeptical of the incentives of everyone hyping this narrative, because they’re people/industries who desperately want/need to be the center of attention.
Bitcoin is still bullshit (its primary use case is facilitating illegal transactions, aka literally committing crimes), but it continues to remain expensive bullshit.
Even worse, people remember "experts warned us of something and managed to prevent it" stories AS "experts were humiliatingly wrong". Climate change deniers will cite ozone layer depletion and Y2K as examples of the experts warning about things that didn't happen.
As the Y2K deadline approached, not many people, including elites or experts, were saying that the problem was largely fixed, or that there wouldn’t be serious if not catastrophic problems because the appropriate people had taken appropriate action. In retrospect, the lesson seems to be that the expenditure of money, time, and effort to avoid catastrophe was sufficient to avoid most serious problems. (Of course, we can’t run the counterfactuals of spending 10% or 20% or 50% less time, money, and effort.) In any case, back in late 1999, there were still plenty of Y2K doomers who had more-or-less sophisticated (but, in retrospect, incorrect) arguments about why the Y2K problems had not been – and even “could not be” - adequately addressed. Still - it’s hard to know how to apply the lessons of Y2K to AI.
Yup. At the time, I thought that there was a lower bound on the amount of damage that Y2K would do, from the roughly 25% of the economy (roughly half of small businesses) that did nothing to avoid the problem. I was wrong - and I'm still not sure why the final impact, even on that sector, was so small. I've never been so happy to _be_ wrong.
That's interesting - is it possible that small businesses mainly used off the shelf software that had been fixed by the vendor, so even if the business didn't care about Y2K, if they happened to update their software before the millennium, it got fixed anyway? But then it's hard to imagine businesses that didn't care about Y2K would be diligent about software updates, especially because updating would be more manual and complicated back then.
Or maybe organisations that were small enough not to have done anything were also not that reliant on computers anyway, so when they ran into problems they could just do things manually?
>But then it's hard to imagine businesses that didn't care about Y2K would be diligent about software updates, especially because updating would be more manual and complicated back then.
Agreed!
>Or maybe organisations that were small enough not to have done anything were also not that reliant on computers anyway, so when they ran into problems they could just do things manually?
Could be, though even tiny organizations generally need to send out invoices and keep books.
> "And there are people who can argue circles around me. Maybe not on every topic, but on topics where they are experts and have spent their whole lives honing their arguments. When I was young I used to read pseudohistory books; Immanuel Velikovsky’s Ages in Chaos is a good example of the best this genre has to offer. I read it and it seemed so obviously correct, so perfect, that I could barely bring myself to bother to search out rebuttals."
> "And then I read the rebuttals, and they were so obviously correct, so devastating, that I couldn’t believe I had ever been so dumb as to believe Velikovsky."
> "And then I read the rebuttals to the rebuttals, and they were so obviously correct that I felt silly for ever doubting."
Presented with a flood of logical steps by someone capable of arguing circles around me, adding up to a conclusion that is deeply counterintuitive; and presented perhaps also with a similarly daunting flood of logical steps by someone else also capable of arguing circles around me, adding up just as convincingly to the opposite conclusion... what is one to do?
One can sway back and forth, like a reed in the wind, according to which super convincing argument one last read.
Or one can throw one's hands up in the air and say: "There are hundreds of incredibly specific assumptions combined with many logical steps here. They all look pretty convincing to me; and actually, so do the other guy's. So clearly someone made a mistake somewhere, possibly more than one person and more than one mistake; certainly that's more likely than both the mutually contradictory yet convincing arguments being right simultaneously; but I just can't see it. What now? What other information do I have? Well, approximately ten gazillion people have predicted world-ending events in the past, and yet here we all are, existing. So I conclude it's much more likely that the weirder conclusion is the one that's wrong."
Condense that to a sentence or two, couple with an average rather than expert ability to articulate, and you arrive at coffeepocalypse.
From the above essay:
> "Even the smartest people I know have a commendable tendency not to take certain ideas seriously. Bostrom’s simulation argument, the anthropic doomsday argument, Pascal’s Mugging – I’ve never heard anyone give a coherent argument against any of these, but I’ve also never met anyone who fully accepts them and lives life according to their implications."
AIpocalypse is just another idea to add to that list.
People who believe in AI doom, even a moderate amount (more than 1% chance) seem to me to be invoking a Pascal's Mugging, but vehemently disagree with my conclusion that they're similar.
In both cases I agree with you that nobody seems to have a good counterargument against AI doom/Pascal but unless they already agree with the conclusions are not at all *convinced* by that argument.
Telling me there's a non-zero chance of AI doom doesn't tell me how I should or shouldn't lead my life. There's a non-zero chance that an asteroid will destroy the planet this year as well, but almost nobody treats the possibility of that as a legitimate reason to live in fear or build underground bunkers.
Because total destruction of everything we value is part of the equation, all answers to the question of "how seriously should we take this" comes out to infinity for any doomsday prediction. We obviously cannot react to every such infinite concern with proportionate resources (we can't go spending all of earth's resources every day), and for every doomsday except one that we ourselves agree with, we think the actual appropriate level of resources to spend is in fact close to zero. The problem is multiplying by infinity, and the solution is to take infinity out of the equation. People who agree with the infiniteness of the problem don't do that, come up with infinite in the equation, and wonder why everyone else doesn't get the same answer.
a >1% chance of something happening isn't a Pascal's Mugging, it's a "it will happen at least one time out of 100". A Planet-killer asteroid hitting earth has an incredibly lower chance of happening anytime soon, and yet people do seriously worry about that (see eg https://en.wikipedia.org/wiki/Asteroid_impact_prediction).
We actually have *NO* idea how likely AI doom is. It could in fact be far far less likely than an asteroid. Also, with asteroids we can try to game out the possibility of one hitting us and wiping our humanity, but even knowing it's very unlikely doesn't tell us it's impossible even on short timeframes.
That's why I refer to people who believe AI doom may happen (even if they also think it pretty unlikely) compared to people who don't. Pascal's wager was originally about believing in God, and how it makes sense to believe in God even if the probability of God existing was low. As someone who already believes in God this argument makes perfect sense to me! But, I totally understand why this is not at all convincing to someone who doesn't already believe in God.
Then you are just arguing probability numbers. This has nothing to do with multiplying by infinity, people who argue coherently for AI risk do so because they assign a high probability of AI doom comiing to pass - not primarily because the outcome is infinitely negative.
People who are worried about infinite negative utility are clearly making a Pascal-class mistake, and it is not a correct argument against AI.
We have evidence for large asteroids hitting Earth in the past, so there is a non-zero probability of one hitting Earth in the future. You can argue about how likely and how soon, but you cannot argue with non-zero.
We have *no* prior evidence for AI, or AI doom. Saying AI is like humans is just wrong; we're pretty sure that we're not replicating human thinking in there.
Do we have evidence of AI being smarter than humans? Say in Chess or Go? Or maybe image recognition? Do we have evidence in our historical record of one intelligence wiping out another intelligence?
I'm saying that their belief that AI doom is likely is unjustified. They cannot provide evidence that the chances are actually high, because it's an unprecedented question that depends on future technological advancements that we cannot know are even possible. Recursive self improvement is an idea that may not may not be true, not a fact that we can plan around.
I believe that the counter-argument to the fact that AI doom is an unjustified belief is to talk about how serious it would be for us, if true. That's Pascal.
The issue I've come up against is that the people who argue for catastrophic AI risk tend to disregard other catastrophic risks that strong AI would hedge against. AI strong enough to pose an existential risk could also help fight climate change, find alternative energy sources, reduce car deaths, find cures for cancer, eliminate the need for human drudgery, and do polygenic screening which, when combined with embryonic selection, could reduce illness and death in general across a population. There is currently a 100% chance you will die at some point. What weight should you assign to addressing that issue? The accelerationist argument is almost always framed as 'accelerationists hate human beings' which is a severe weakmanning of the accelerationist arguments.
I do think we should be arguing relative probability numbers as well as we're able to. But in practice, infinity or no, people strongly tend to choose one goal or catastrophe and let it fill their entire mental field of vision. Most people just REALLY don't like to have to juggle 7 mental balls or consider that multiple inputs may contribute to a single result.
> There is currently a 100% chance you will die at some point.
Well, the heat death of the universe is probably going to be a problem, but the chances of a well-preserved cryonics patient being restored to health in some form or other is greater than zero.
Dunno about cancer and polygenic screening, but I think we've got adequate non-AI-based options lined up for energy, climate change, drudgery, homelessness, and car deaths, it's mostly just a matter of legislation and budgeting at this point.
First, broad regulatory exemptions for solar PV (and, optionally, wind) construction. Anybody objecting on environmental-impact grounds needs to show that a given project would somehow be worse than an equivalent coal plant, at least until there are literally no more grid-scale coal-burning generators operating anywhere in the world.
Second, Georgist tax on land value, replacing most other taxes.
Third, GPS tracker (and, optionally, ignition-lockout breathalyzer) as mandatory automotive safety equipment. All roads become hassle-free toll roads, no more DUI hit-and-runs, no more selective enforcement of traffic laws based on individual biases. Urban planners could experiment with complex schemes of road-use fees varying by time of day - or even outright banning cars from particular areas - at the push of a button, confident that an updated map would be on everyone's dash faster than you can say "in fifty yards, turn left."
Revenue would increase while enforcement costs plummet. Spend the initial wave of money (and surplus solar and wind power) on accelerated weathering: grind up a mountain's worth of olivine and dump the resulting powder in the ocean, thereby absorbing excess CO2.
Once that situation is stabilized, celebrate by implementing universal basic income. Anyone who doesn't like drudgery will spend some of their share on avoiding it, thus bidding up the price of that dwindling supply of drudgery-tolerant labor, until most dirty jobs cross the threshold of "more cost-effective to automate away," while remaining automation-resistant ones pay so well as to be prestigious by default.
Of course, people won't just sit around idle. They'll use that security and abundance to start systematically solving all the problems they were previously obligated to ignore, due to being too busy flattering some middle-manager's ego in exchange for subsistence, or too broke to do anything but get in arguments on the internet.
I like your last argument. Very clearly no one actually believes AI will cause the death of all humans because their behavior would be radically changed.
>because their behavior would be radically changed
It depends on the time scale and the probability and the person's time horizon.
I'm 65, so I don't lose sleep over much that happens more than 20 years from now. My default assumption is AGI succeeding and probably (with _HUGE_ uncertainties) eventually replacing humans (but possibly preserving large chunks of our culture).
I took a two thirds paycut and moved from a place I liked more to a place that I liked less in order to do my part about preventing the AI apocalypse; I think that counts as strong evidence that I believe in it.
[Why didn't I do <other thing you think AI risk implies>? Well, probably because I didn't think AI risk implied that thing, or it doesn't match up with my personal competencies, or whatever.]
Increasingly many people in the EA AI safety community have taken fairly drastic actions. But you're right that basically none of them think it's a 100%, or even >95%, certainty, and the possibility that they'll have to live with the consequences of their actions keeps them from doing anything even more drastic (e.g. spending all their money or getting themselves imprisoned or...)
every time I see someone make this argument, the actual changes they recommend are pretty nonsensical
the truth is, i have absolutely no idea what to do with this information. every "extreme action justified by extreme threat" i can think of has a negative expected value
i suspect those are the same extreme actions you are thinking of, but since you don't actually believe, you don't feel obligated to continue the chain of thought to its conclusion
Pascal's mugging is, and always has been, about prospect theory and human psychology. The solution is very simple: stop assigning non-negligible probabilities to extremely unlikely events. Pascal's mugging is about a problem with human psychology—it shows that drawing our attention to an extremely unlikely, but potentially highly-important event, causes us to hugely overestimate its probability.
I think the steelman for coffee or "failed predictions of disaster" arguments is based on Lindy Effect sort of things (well, humanity's been around a quarter million years, plagues, volcanoes, hurricanes, tidal waves, wars and genocides have all happened and we're still here, sooo....), and maybe for some few better read and better epistemic-ed arguers, pointing to nuclear-disaster style arguments about how hard it would REALLY be to actually kill every human.
I personally don't buy either of these arguments (for one thing, we've been through at least one and probably two significant "close to extinction" genetic bottlenecks during our time as humans https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2842629/), but they're a better class of argument than "we don't worry about halibuts" or whatever.
I don't think this study really relates to the question of human extinction - reading through it it seems to be more about the "genetic bottleneck" of a small group of humans moving either out of Africa or across the Bering Strait, not all of humanity being close to extinction ("we" haven't been through a bottleneck if "we" includes Africans).
You know, you're totally right, thank you. On further research, it looks like the bottleneck I was thinking of was around 900k years ago, before we were human, and the population went from ~100k individuals to ~1k, and stayed low for ~100k years. So still a "close to extinction" bottleneck, but for human ancestors vs anatomically modern humans.
I think the strongman of the argument they are making is, for almost every new technology there have been people making dire warnings and apocalyptic predictions about it (e.g. trains would kill everyone over a certain speed, video games corrupt our youth, nuclear power etc).
These arguments are easy to make because new technologies are scary to many people and it is very hard to definitively show something that is new and unproven is safe. Nearly all of these arguments have turned out to be mostly or completely wrong.
Therefore when someone says AI could kill everyone, your prior should be that it almost certainly won’t.
Whilst I can broadly accept that, it isn’t an argument to just ignore evidence about AI, just that before you examine the subject that should be your starting point.
You could also make the bolder claim that the burden of proof is on those claiming AI will cause calamity, given how often those sorts of arguments have been made and how beneficial new technologies have been on average. But I’m not sure I would go that far.
I think its even broader than this. There is a market for consuming doomsday prophesy. So, writers produce doomsday prophesy and are incentivized to make it as plausible as possible to gain market share. This has shifted over my lifetime from nuclear war, ozone, global warming, climate change, COVID, and AI. If you put my casual knowledge against any of the top writers and say, "respond to my specific arguments otherwise you are wrong", I loose every time.
In the paradigm, most doomsday authors will switch between the most salient doomsday narrative over time. And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI. I think their COVID record is good evidence that they are overreacting to AI. The historical overreaction to coffee shows that this is a normal thing to happen in societies.
> If you put my casual knowledge against any of the top writers and say, "respond to my specific arguments otherwise you are wrong", I loose every time.
+1. This is the core of it. I'm reminded of conspiracy theorists who insist that you're not entitled to disagree with them until you've watched [some random 4h long youtube video], or read Marx cover to cover.
Sneering about epistemic learned helplessness is dishonest, when it just isn't practical for me to spend months looking into everything in detail. I do take AI risk seriously, and I've engaged (admittedly somewhat superficially) with many of the arguments. What else do you want me to do?
As a conspiracy theorist, I have the same complaint about Rationalists, Normies, and The Experts.
> What else do you want me to do?
I would like you to wonder what is true, using an appropriate form of logic. Knowledge that there is more than one form is typically only cognitively available when discussing logic itself directly/abstractly, and this knowledge availability phenomenon applies to many things other than types of logic.
Yes, You get me to come back tomorrow, if you tell me how bad things will be. In my old age I find I've become less and less certain about anything. I don't know enough to have an opinion either way. And so for most things, "I don't know", is the right response. And arguing with other people who mostly know nothing is just a waste of my time. So why do it?
Because if all the wisest people take this option, all the actual actions will get taken by unwise people, and I for one don't want to live in that world.
(Cf that one line in HPMOR about how "good people" are often too busy wringing their hands over what is ethical to take actually helpful actions)
Do you have any examples of any of the people you are talking about (or anyone prominent, really) actually saying covid would end the world (a "doomsday prophecy")? It's been a few years, but my memory from early in the pandemic is a lot heavier on people who said things like "it will be over by May" and "I'd be surprised if there are even 10,000 deaths in the United States" than people who overestimated the actual number of deaths, which is in the millions in the U.S.
Scott advocates for: healthy quarantine, fomite prevention measures, home healthcare, praises Chinese lockdowns, advocates for contact tracing, wearing masks, wearing full respirators, and zinc supplements. He mostly ignores costs associated with these measures. Its an impressive list of predictions that would be adopted by global policy makers while also being ineffective and in some cases disasterous, especially to the global poor.
Meant "doomsday" as people predicting doom, not the literal end of the world. Maybe I am using the word incorrectly.
If everyone had followed those recommendations in March 2020 then we probably would have eradicated the virus entirely by July 2020 and got on with the rest of our lives, with far less total disruption, probably with fewer total days of lockdown.
No, they tried something that looked vaguely like it and failed. Locking down until eradication (and then keeping borders closed) was the best move at the time.
Locking down until almost-eradication is a huge fucking waste of time and money which just happens to look a lot like locking down until eradication.
>than people who overestimated the actual number of deaths
I was an (obscure!) overestimator.
In the early days, when, at one point, the death rate looked like 7% and the fraction of the population who needed to have gotten it and gotten immune to stop the pandemic needed to be around 50%, I expected the worldwide death toll to be on the order of 2.8x10^8. I was wrong. The final toll seems to be about 40X. smaller https://www.worldometers.info/coronavirus/ gives 7x10^6
But Scott was wearing a full respirator out in public in maybe Feb 2020. I didn't keep up with their output at the time, but my sense was that they were advocating for more caution and for lots of interventions social distancing, travel restrictions, healthy quarantine, vaccination, boosting, etc. But then corrected faster than other authors when these ended up not doing much.
> And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI. I think their COVID record is good evidence that they are overreacting to AI.
This seems like a pretty unkind thing to say, so it better be true. Do you have any actual examples of these people making "doomsday prophesies" about COVID?
The term "doomsday prophesy" is unfair, and unkind, and overly provacative. I like Scott and find him interesting and worth reading.
As for "overreacting", I think the history speaks for it's self. Here is Scott in march 2020 very intelligently advocating for all the ineffectual and counterproductive NPIs that became public policy.
I don't think that post matches the way you describe it here, let alone the sort of thing you claimed in your original comment. Most of it seems to hold up pretty well even in hindsight, let alone compared to the level of information that was available at the time. In general the course of the COVID pandemic seems like a big win for Scott and others in the community who were saying early on, "hey this could be a really big deal" over the people who were saying it was nothing to worry about.
Can you point to a specific claim in the post you linked that seems like a hysterical overreaction?
His advocacy for healthy quarantine, is a a hysterical overreaction. This was easily the worst public policy decision of the pandemic.
"The story I’m hearing from most smart people is that China has done amazing work and mostly halted their epidemic. This is very impressive...Nobody’s too optimistic that democratic countries can follow their lead, though"
There is also a section titled: "Self-quarantine starting now vs. later?"
China's early coronavirus response worked, as far as I can tell. Where they dropped the ball was that they didn't distribute the mRNA and other vaccines in favor of a Chinese one that was significantly less effective and continued to have a strict "zero COVID" lockdown policy long after vaccinations with Western vaccines would have made it no longer worthwhile.
"advocacy for" is a bit of a stretch relative to the actual content of the post IMO. But I will grant that that part was a bit overzealous.
Still, given the overall impact of COVID and our society's response to it (whether or not you agree with the latter), Scott comes out of it looking a lot better than the people who were saying it would be no big deal. I don't think it makes a very good argument that he's prone to making mountains out of molehills.
There also a difference between warnings and prophecies. Basically, warnings are conditional, prophecies aren't. If someone says X will happen ,and it doesn't, they were wrong. But if someone. Says "X will happen if we don't do Y", and we did Y, they weren't wrong.
> most doomsday authors will switch between the most salient doomsday narrative over time. And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI.
I don't see what happened as particularly fallacious. They've been concerned about AI, and there isn't a global pandemic, so they just talk about AI. Next, they're concerned about AI, and there's also a global pandemic, so they talk about both. Then the pandemic goes away, and they're back to AI.
These people aren't one-note pundits; they're talking about what's on their mind, and if the entire world is going nuts due to a pandemic, this is going to be reflected in their output, one way or another. And I don't recall any instance of them treating covid as being even a serious threat to human civilization, let alone human existence.
I'd also disagree with the characterization of them as "doomsday authors". Scott certainly isn't: he writes about a lot of stuff. Eliezer probably regrets not being a more effective doomsday prophet about AI, but he's not a general-purpose doomsday prophet: he saw the danger of AI early, and devoted a lot of his life to trying to stop it, and his advocacy is simply a part of that. It's the difference between a war journalist who travels the world from one war zone to another, and a war journalist who only happens to be covering a war because their country is currently being invaded.
> Eliezer probably regrets not being a more effective doomsday prophet
AI doomsayers need a mass market, inexpensive, public ritual, along with some really expensive remedies to really take off. The public ritual keeps the issue, "top of mind" for the public, provides controversies for the popular press to write about, and provides a mechanism for social enforcement. The expensive remedies create profitable businesses who can bribe politicians and fund doomsayers. For climate change these are recycling/green energy. COVID it was masks/medical care. Not sure what it could be for AI, but the first to figure it out makes billions.
So AI doesn't actually fit the pattern/ you are trying to make it fit?
Also masks were never a profitable business, so the model doesn't work there very well.
To be clear, I do think the point that there are lots of doomsayers provides a prior against these sorts of arguments in general, but I think it also clearly does not produce such a strong prior against them that we can safely assume they are all false.
It seems like a situation where skepticism but attention is warranted, and an awareness that there is something in a lot of our minds that really likes believing doomsday arguments, even when they aren't very good.
Having said that, I think the case for AI being dangerous enough that we should be very cautious in developing it is solid, and we probably would agree on a lot of specifics in terms of that if we pinned them down carefully enough.
> Also masks were never a profitable business, so the model doesn't work there very well
Masks were the public ritual and had nothing to do with money. The 50K insurance payments for treating patients w/ COVID, vaccines, Remdesivir, etc. were the expensive remedies.
> And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI.
If this is what you sincerely believe, then you fail at understanding quantitative arguments. The worry about AI is that it could kill literally everyone. (For "Scott, Yud, etc."; other people worry about other things, for example AI saying something politically incorrect, or failing to pay artists for using their art as inspiration.)
There was never a similar worry about COVID; instead it was that it could kill a lot of people... and it actually did, but much fewer than was expected. So the "doomsaying" at worst meant overestimating the number of victims by one or two orders of magnitude (and even this only assuming that all the vaccination etc. had zero effect).
There are people out there who worry about crime, car accidents, fires, floods, and various other things that sometimes kill people. Should we also include them to the list of doomsayers who merely follow the popularity incentives?
They overestimated the risks for COVID. So, I think its reasonable to assume they are are overestimating on AI.
I think they have brains that subconsciously fill in holes of uncertainty with doom. It's part of why they are popular, but also leads to overestimation of risk as a power law of uncertainty: 1-2 orders for COVID, ~7 for AI.
Why wouldn't you go that far? Why wouldn't the burden of proof be on the people making a claim, especially a claim that would necessitate significant changes in how every person on earth lives their lives?
Fair point I guess it depends on definitions, but in my head putting the burden of proof on the AI risk lot (I.e. showing decent chance AI likely to kill us all) would mean at a minimum making them show it to a UK civil trial standards (i.e. more likely than not). I wouldn’t be comfortable for life to go on as normal if there was a 49% chance they were right.
I would go further than that, proof that there's a 5% chance of doom seems like a decent place to put serious effort into preventing catastrophe. I've seen a LOT of conjecture about what the probability of doom is, and a whole lot of reasoning about how it could happen, but just about zero proof. If this was a court of law, the case would be thrown out as conjecture.
We can respond to conjecture, but such a situation doesn't require action, especially specific actions. I happen to think that being pragmatic about limiting AI is a good idea. For instance forbidding recursive training (AI learning from experience to train itself) is probably a good idea. But we should be clear that there's zero proof of AGI being possible, let alone any of the further concerns about how that could lead to doom (often also involving technology we don't know to even be possible, like nanites).
For the record, it wouldn't take sci-fi nanotechnology for an AI to be able to destroy human civilization using only an Internet connection. If enough novel killer viruses struck humanity at the same time, we'd go down like the American Indians did to European diseases. Creating a virus given a DNA sequence is well within the capability of current technology, and there are companies that you can pay to send you custom DNA sequences as long as they don't match a database of known dangerous viruses. If an AI *could* design killer viruses (admittedly a difficult task) and then bribe/threaten/trick people into assembling them (something much easier than designing one), it might not kill literally everyone, but it could certainly leave humanity in no position to oppose it.
do you not consider humans to be such proof? i certainly do. we seem to be fully generally intelligent, we certainly count as an existential threat to all beings less intelligent than we are, and we sprang out of incremental improvements to non-generally-intelligent designs
i have some trouble understanding people who are not immediately swayed by this argument, which seems utterly ironclad to me, but i guess if anyone has any novel arguments i'd like to hear them
Zero proof of any of the specific things (foom/takeoff, specifically) that would lead to AGI takeover. I didn't say, and wouldn't say, that AGI is impossible.
To put some of my cards on the table, I'm religious, so it's not at all weird to me to consider human intelligence potentially separate from machine intelligence.
Secondly, there is nothing at all weird with saying that we may be on the wrong track for AGI or that true AGI would be far more complex than anything we've even attempted at this point.
One reason: because winning arguments and optimal avoidance of risk may look identical, but they are not.
Another reason: you are considering "that would necessitate" to be necessarily factual - this may be fine if you're operating within a thought experiment, but we can't live within thought experiments. Or can we? 🤔
> Why wouldn't the burden of proof be on the people making a claim,
The burden of proof is indeed on those making a claim, which includes those who claim AI will NOT be an existential risk. Agnosticism on the outcome is the right logical position, but then you're still left in the position of evaluating how plausible a danger it is, and thus whether some investment in mitigations are warranted (like the Y2K bug).
> especially a claim that would necessitate significant changes in how every person on earth lives their lives?
AI doesn't exist yet. How could mitigations against something that doesn't yet exist necessitate significant changes in how every person lives their lives?
Well, consider the necessary mitigations. There are people around the world riding hell-bent for leather to produce human or superhuman level AI, and a regime that was capable of suppressing them would be…truly intrusive.
But research into AI safety has not borne much fruit yet — certainly no silver bullet. That very fact might, to some eyes, suggest that slowing down would be prudent, but there is no sign of that happening. So an oppressive regime might seem, to other eyes, the only choice.
There’s also the fact that AI research is pretty easy to hide, compared to, say, nuclear-weapons research. A global compact to restrict it would be so *very* easy to cheat on.
AI safety has also not seen much funding. What's the ratio of funding between "training/developing new AI" vs. "*ensuring* AI outcomes"? 100:1? 1000:1? possibly even worse.
My point is that a lot of people are working on AI so as to achieve specific outcomes. AI safety research generally aligns with the goal of *ensuring* specific outcomes, but it's not typically seen in that light, and most people have instead been content with "has a 2% chance of going wrong". There's value in closing that gap to zero, and I consider that within the realm of AI safety, because what is AI safety if not "does what it's supposed to and doesn't go wrong in unexpected ways"?
But like Making these giant neural nets takes serious compute. It's harder than making drugs, comparable enriching uranium. And the world is trying to suppress the first one, and mostly managing to suppress the second.
Russell may not have any reason to believe there is a teapot in orbit, but if someone walks up to him and says there definitely isn't a teapot there, he's absolutely going to ask for evidence.
Which either leaves us with nothing at all to say about anything we cannot directly observe, or working from other principles we can examine more directly.
The likelihood of a teapot being in orbit cannot be considered 50/50 just because someone claims it's there. Unless shown positive reason to expect the teapot to be there, we should start the conversation with something *significantly* less than 50%, likely some small fraction of one percent. This is necessary from logical and practical standpoints. Logical, given we have no mechanism for how a teapot would get there, and practical because we can't respond with action to every assertion made.
Assertions are practically free, while action is many orders of magnitude more expensive.
Given that starting point, we would be perfectly justified in saying he was wrong even if we can't prove the negative. Or, more likely, to ignore the hypothetical and spend no resources trying to prove or disprove the conjecture or remedy any issues the argument brings up. We should spend zero dollars trying to protect orbital satellites from this teapot, for instance.
I mostly agree, so now the question is whether Russell's teapot has the same probabilities as AI being an extinction level threat, so that you can clearly and obviously conclude which is more plausible and which requires the greater burden of evidence.
It seems clear that an intelligence greater than human intelligence is possible, as evolution would disfavour maximal intelligence for various reasons. Therefore greater intelligence is possible. We know of no reason why intelligence should be correlated with moral superiority, at least in a system that did not evolve as humans did, where cooperation was key to survival, therefore we should assume there is no intrinsic relationship between the two.
So it seems that an intelligence greater than human intelligence and that is not necessarily morally superior to humanity is eminently possible, and that we are plausibly on that path given the economic and strategic incentives there are to replacing and expensive fallible humans.
There are clear steps in this deduction that would permit empirical refutation, such as some intrinsic association between intelligence and morality, so it's falsifiable in the course of standard research on AI, particularly AI safety research.
Agreed that this is the strongman, further strengthened by the esotericism of the subject and how early we still are in the game. It's intuitively clear how nuclear weapons can destroy civilisation, and it's reasonably clear how a pandemic might, but not how and why the lovably-bumbling "glorified autocomplete" evolves into something that does, even by reaching for those other means. To persuade an ordinary layperson you have to keep hammering at the physical mechanism. Pascallian (or near-Pascallian) thought experiments are just not compelling, and stuff like the AI-box hypothesis is downright counter-productive.
The first instance of 'is' affirms my view of the correctness and truth of the claim in the comment I replied to. The two subsequent instances, coupled in contractions with their 'it's, assert common knowledge of how more stereotypical x-risks might play out. The final 'is' asserts my opinion of the negative persuasive power of the AI-box thing and by implication of other elegant but weird rhetorical techniques used by the doomers.
I'd do the 'are's, but, well, I can't quite be... bothered.
So, you are contemplating a model. Why not contemplate the thing itself? Isn't that the goal here, or is this more so a sort of social activity, that perhaps comes with unwritten rules?
Not exactly, the heuristic section talks about whether moral panics in general are correct (which is a fair point) but I think if you just focus on the technology worries and moral panics you can make a much stronger case that most of them have turned out to be nonsense.
In that section Scott is looking at this as evidence for the anti AI risk side (for which it is pretty weak), but if you look at it for what your priors should be before you jump into the evidence on AI it is much stronger.
One could have accurately claimed that cigarettes and automobiles would each have killed millions of people over the course of a century... Of course, millions of deaths _sounds_ apocalyptic but actually isn't...
True! Many Thanks! And the body count is smaller too, for automobiles (for tobacco in all forms, WHO gives around 7 million a year currently, so over a century it _might_ exceed WWII's 70-85 million - not clear how much of this is cigarettes specifically).
Interestingly, with most past technologies, the people making doomer arguments about it were the people not involved with it, while the people involved with it were the ones who were dismissive. With AI we seem to have the reverse pattern, which doesn't directly tell us anything except that it's different.
I think the argument isn't against disaster specifically. It's against our ability to predict anything accurately. The argument isn't "The smart guys predicted a disaster but no disaster happened, so no disaster will happen this time". It's "The smart guys predicted something and were wrong, so smart guys aren't as good at prediction as they think".
Bringing up examples where no disaster was predicted but a disaster happened anyway doesn't refute them. It supports them. Bringing up examples of accurate predictions refutes them. However, in practice this is hard to refute because predictions are rarely *completely* accurate. Even getting things directionally right is very hard, let alone getting the timing, magnitude and exact character of a big future change correct. Also, an isolated correct prediction can be written off as luck, or the "stopped clock" effect. You need a long history of correct prediction to show that prediction works.
I think the easiest way to gauge what side of this you fall on is the extent to which you believe in the Efficient Markets hypothesis.
Arguably, the world was indeed destroyed at least a couple of times, for certain values of "world" and "destroyed". The extinction of the dinosaurs is one obvious global example; the Black Death and any of the various genocides in human history are local examples. None of those cases depended primarily on new technologies, however.
War, famine, and disease were the historic (local) x-risks. Notably, war often leads to famine and disease by disrupting supply chains and capital infrastructure.
In risk-analysis I believe they have a system for assessing the probability of risks that must always turn up nil for observer-selection-effect reasons: I gather they look at the near-misses, then apply a correction factor based on the average ratio of near-misses* to incidents within similar fields.
(* presumably near-misses for less-observer-selection-effect-susceptible risks..)
For example: for Titanic-level maritime disasters, even though they're hard to do statistics with about because the frequency is less than once per lifetime, the incident:near-miss ratio is about 1:100 (source: vague half-remembered lecture I once attended, possibly whilst not entirely sober..) so if you see 10 near-misses in a given time-frame you can suppose a roughly 10% chance of an incident within the same time-frame.
Can we do this with AI doom? Probably not, because there probably aren't any incidents similar enough to AI doom with a known incident:near-miss ratio. However, to address Peter's phrasing: we do have data on nuclear war near-misses - and there are certainly more than enough of those to make one unwilling to just assume new technology won't destroy large parts of the world that one cares about.
Having read through other comments, I'm convinced that while superficially appealing, this is actually the *wrong* reference class, and it may explain why some of non-doomer arguments seem like a combination of willful blindness and philosophically-baffling moonlogic to the AGI pessimists.
"New technology" as a reference class implicitly means "tools" -- things that expand the capacities of individual humans or groups of humans while fully subject for all intents and purposes to their human masters.
The (or at a least a primary) doomer argument is that the reference class for AGI is totally outside of this paradigm. Instead it's "inventing a cognitively-superior organism that, for any conceivable agentic goal, will necessarily compete with other previously-existing species for resources to accomplish it." This is a story that *already happened* with natural GI and the outcome was that every other sentient species lost, hard. Hell, humans broadly *like* charismatic megafauna and they've still been devastated in population numbers where they weren't simply subject to extinction because humans don't like them enough relative to other priorities and would prefer to put their habitats (if not just their bodies) to other uses. Whereas one expects an agentic AGI to be as indifferent to humans as we are to bacteria, and for humans to have roughly the same capacity to challenge AI dominance as, like, parrots do to humans.
Meanwhile, the AGI's resource needs and ambitions have no natural upper bounds, which as a corollary means competition with humans that it will always, always win.
The relevant reference class isn't "new tool makes bigger bang," (scary though that is on its own), it's "the impersonal force of evolution results in the most cognitively-capable species being best adapted to its environment and winning resource competition, mopping the floor with every less intelligent species, and controlling whatever resources it feels like, to the exclusion of potential competitors and the deliberate extermination of pest species."
Yeah, the argument that is most striking to me is to ask where Neanderthals and Homo erectus are now. They could tell you about the danger from the competition of a superior intellect.
They seem to be content in their lot and to be thriving compared to most other species… pretty much ever.
To steel man your argument, it is that similar species competing for a given niche leads to a winner and losers/extinction This is indeed well documented.
On the other hand, complementary species could support and thrive together. Interestingly this would imply that the solution isn’t via slightly smarter AI, which could be in competition, but rather in substantially and more powerful and smarter AI.
There are thousands of people on this planet with enough power to completely derail or even eliminate your life with the stroke of a pen, and face zero negative consequences for it. The only reason they don't is because they don't care enough about you to do that. Unless you have the potential to be a billionaire or a world leader, then all people like you and I can aspire to be is a comfortable, placid resource for people with actual power. They're the only ones on the planet who aren't pets(at best) to someone else. And if you say no, you have dignity in this arrangement, well, you potentially can have that with AI overlords too.
I agree that an entity whose capabilities have "no upper bounds" is likely to be quite bad news for us (arguably, the *needs* of humanity likewise have no upper bounds, but our capabilities are bounded). However, it's not enough to merely posit such an entity; humans have been inventing gods and demons throughout pretty much our entire history, and there's no reason to believe that any of them do or can exist. If you want to convince me to take specific steps and contribute specific resources toward stopping your uber-entities, you must first demonstrate that they are at the very least plausible, if not actually likely. Unfortunately, by positing unbounded entities (either truly unbounded, or "merely" effectively unbounded, as in e.g. "capable of doing anything they want to anything within the Solar System), you are kind of shooting yourself in the foot, since your burden of proof becomes likewise unbounded.
The statement about lacking upper bound was about needs rather than capabilities — the idea being that the AI need only be superior to humans to win in any competition for resources, and (barring implausible artificial restrictions like “make exactly ten paperclips and no more” rather than “survive and make paperclips,”), there’s no upper limit to AI demand for resources and attendant competition with humans for same.
The specific statement was “Meanwhile, the AGI's resource needs and ambitions have no natural upper bounds, which as a corollary means competition with humans that it will always, always win.”
I am thus making a much less strong claim about the necessary capabilities of the AI than you are imputing to me — it needs to only be better than humans, not arbitrarily better than humans.
(I should note that I actually do think arbitrarily high capabilities in a practical sense are likely under a recursive self-improvement scenario, which seems ex ante extremely plausible, but I don’t think it’s a necessary element of the claim.)
Understood, and I withdraw my objection -- but I'd still argue that the needs of humanity are ultimately unbounded, seeing as we are driven to reproduce without limit other than resource constraints. In fact, given a zero-sum competition between two societies of humans, one with greater needs than the other, the outcome is uncertain unless you also factor in their *capabilities*. Or to put it another way, an unemployed bum has a greater *need* for a sandwich than a billionaire, but the billionaire is still much more likely to obtain the sandwich.
I would agree that needs of humanity are unbounded except that it seems like material success is correlated with sub-replacement fertility empirically speaking. But I think the issue is moot in any event: assuming infinite needs by both humans and an AI (the AI having unbounded needs basically because that's just the way that utility-function-maximization works for any plausible goal beyond "power down and stay powered down," and for the same reason being a lot more single-minded about working to meet its goals than humans empirically are about reproducing), if the AI is more capable you have both conflict over resources and AI victory.
If you propose someone or something that is reasonably smart, or strong, or fast, then the burden of proof is still on you, but it is quite modest. We can observe many such beings all around us already, after all.
If you propose someone or something that is the smartest, strongest, or fastest person in the world, then the burden of proof is much higher, but perhaps still doable. It is not clear whether intelligence, strength, or speed can even be sorted like that (e.g. bulldogs have extremely strong jaws but cannot lift heavy objects like humans can), but perhaps you could still make a compelling case, if you bring up a mountain of evidence.
But if you are proposing an entity that is not merely the smartest/strongest/fastest person in the world, but so superior to everyone else as to be literally unimaginable -- then I am not sure how you could justify it. What kind of evidence could you bring up for an entity that is definitionally incomprehensible ? You could propose it in an abstract, but you have just defined yourself out of the ability to comprehend it, which means that you could not possibly have sufficient evidence, since looking at evidence is *how* we tend to comprehend things.
> If you propose someone or something that is the smartest, strongest, or fastest person in the world, then the burden of proof is much higher, but perhaps still doable.
The Astronauts returning from the moon were the fastest humans ever. So are you saying that the Apollo missions had some exceptionally high burden of proof?
I mean sure, no one else is going that fast. But no one else is getting on a giant rocket.
Before the first atomic weapons were made, the scientists were talking about explosions enormously bigger than anything that had come before. The evidence they had was some fairly modest particle physics experiments. Yet they correctly deduced such weapons were possible. And if physics had been slightly different, and nuclear explosions were a trillion times more powerful than chemical instead of a million, the evidence required to show this would not have been greater.
> What kind of evidence could you bring up for an entity that is definitionally incomprehensible ?
Superintelligent AI isn't definitionally incomprehensible. I don't know what limits your imagination is under. But I can at least try to understand some aspects of it.
Deep blue is better at chess than me. So I can't predict it's exact moves (If I could, I could play as good chess). But I can understand what it's code is doing, what a minmax search tree with heuristics is. I can understand why such a thing would be good at chess. And I can predict it will win. That is quite a lot of understanding.
Reasoning about minds smarter than you is hard. Sometimes I use analogies involving things I understand but that dumber minds (human or animal) don't. Sometimes I use abstract maths.
There is a lot I don't know, but some things I can make a guess at.
> but you have just defined yourself out of the ability to comprehend it, which means that you could not possibly have sufficient evidence
If the AI produces a valid proof of the Rienman hypothesis and designs a fusion reactor, I don't need to know how it works internally. I don't need that much understanding. The concept of "very intelligent" predicted that the AI could do those things, and most other concepts didn't.
If you saw the fusion reactor in front of you, clearly designed by an AI and clearly working, would you say it was dumb?
> So are you saying that the Apollo missions had some exceptionally high burden of proof?
Yes, and it was met with sufficient evidence. The same goes for atomic science.
> And if physics had been slightly different, and nuclear explosions were a trillion times more powerful than chemical instead of a million...
Yes, if the laws of physics were completely different, lots of things would become possible (or impossible). I don't think you and I, not being physicists, can speculate on it in any kind of an informed fashion (well, at least I know I'm not a physicist).
> Deep blue is better at chess than me. So I can't predict it's exact moves (If I could, I could play as good chess). But I can understand what it's code is doing...
Are you saying that Deep Blue is "superintelligent" ? I don't think so:
> If you saw the fusion reactor in front of you, clearly designed by an AI and clearly working, would you say it was dumb?
It depends. I think that Deep Blue is, in a way, really dumb. It's a program designed to play chess, and that's the only thing it can do. I may not fully understand how it works (other than knowing how alpha-beta pruning works in general), but I am fairly confident that it's a relatively simple tool. A machine-learning system designed to build fusion reactors would be a much more complex tool, but it would still be exactly as dumb as Deep Blue.
But you are not proposing such a system; rather, you are proposing an independent autonomous agent that is not merely good at designing reactors or playing chess, but is good at doing literally everything. And not merely good, and not merely better at it than the best humans, but literally beyound (perhaps far beyound) the upper limit of human capabilities. This is why AI-doomers are afraid of AI: not because it could build a better reactor or play a better game of chess, but because it would be able to easily outwit literally everyone at everything. I think this does add up to being "incomprehensible", in addition to being completely unprecedented (outside of the realm of fantasy), so yes, I'm not sure what kind of evidence you could muster towards such a grant claim.
I acknowledge that an AGI (assuming such a thing is possible at all) doesn't necessarily have to be so overwhelmingly smart. Technically, it could be something like a top-notch physicist who is better at particle physics than any living human physicist. This would make the AGI technically "superhuman", but not much of a threat, because there's really not much you can do *just* by being able to think really hard about particle physics.
We have evidence in favor of superhuman capabilities for arithmetic, chess, go, poker¹, next-token prediction², maybe Dota2³, Atari games⁴, and protein folding⁵. We also know that evolution can design very efficient and even self-repairing objects, which humans can't. Existing computers are far below physical limits to computation⁶.
Humans like von Neumann and Ramanujan demonstrate the existence of general and specialized cognitive capabilities far above the mean human level, respectively.
A relatively small advantage in technology and tactics has been sufficient for crushing victory of a small group over a large group of humans⁷.
Yes, of course; but I could make an even longer list if I included the capability to multiply large numbers (and large sets of such numbers), route network traffic, manage fuel injection schedules in real time, and so on. I could in fact expand this list to the ability to crush hard objects or fly across continents or even extract energy from sunlight. I don't see how my list would be any more (or less) valid than yours. And yet you are not overly concerned about the dangers of hydraulic presses; so why are you overly concerned about the dangers of e.g. go-playing systems ?
Humans like von Neumann and Ramanujan do indeed demonstrate "superhuman" performance (I mean, they're technically human but I understand your meaning) on a variety of tasks, yet both of them were highly specialized in their areas. From what I can tell, even Von Neumann was fantastic at mathematics and physics (and computer science, arguably a branch of mathematics); pretty good at organizing highly specialized committees; eidetic memory (possibly); and pretty close to human-average on the vast majority of other tasks.
The only non-trivial point on your list that I can agree with is #7:
> A relatively small advantage in technology and tactics has been sufficient for crushing victory of a small group over a large group of humans
But this sweeps a massive amount of complexity under the rug. Terms like "technology" and "tactics" imply a degree of self-guided autonomy and adaptivity that no present-day machine learning system could remotely approach. It's like saying that swords pose an unprecedented level of existential risk because in a war the army with better swords will probably in (all other things being equal). This is technically true, but not because of some kind of an unprecedented level of danger posed by swords. It's the humans wielding them who are the danger (and sadly a well-precedented one).
I considered non-cognitive abilities, but those didn't seem super relevant. Yes, humans have created devices that are much faster/powerful than many animals, but speed/power is not the thing that gives humans advantage over other animals.
I put multiplying numbers in the arithmetic bucket.
The list was a collection of systems that have superhuman cognitive abilities in narrow ranges, which (according to me) are evidence that superintelligence is not impossible.
I think that von Neumann was actually much better than most other people in a wide variety of tasks, e.g. he was a consultant for a while, getting paid large sums by his clients, and learned six languages before the age of eight¹. I think that e.g. Ramanujan was probably a better mathematician, but much much more cognitively specialised.
> Prisoner’s dilemma says that he and his collaborators “pursued patents less aggressively than the could have”. Edward Teller commented, “probably the IBM company owes half its money to John Von Neumann.” (pg. 76)
> But this sweeps a massive amount of complexity under the rug. Terms like "technology" and "tactics" imply a degree of self-guided autonomy and adaptivity that no present-day machine learning system could remotely approach.
I don't necessarily disagree with that :-)
I was merely arguing that the gap between agentic dangerous AI systems and us would not need to be unboundedly large. It is an entirely different line of argument to show that we're anywhere near that.
> but speed/power is not the thing that gives humans advantage over other animals.
Maybe I'm being pedantic, but this is not the case at all ! Humans are endurance hunters, which gave us an early edge, due to being able to run down prey animals. We also have better physical equipment than many animals, while being deficient in some domains. For example, dogs have better smell but we have better vision. But yes, our cognition is also superior to that of any animal (that we know of anyway).
> The list was a collection of systems that have superhuman cognitive abilities in narrow ranges, which (according to me) are evidence that superintelligence is not impossible.
Right, but my point is that superhuman performance on an extremely narrow task does not translate into "intelligence" (insofar as the term has a meaningful definition). Calculators can multiply numbers very quickly, but are not "intelligent" in any sense of the word; and this would not change even if you bought yourself a huge crate of them.
Perhaps more importantly, "intelligence" alone doesn't buy you much. As I'd said before, a superintelligent box that sits under the desk and thinks very hard without interacting with anyone or anything might as well be equivalent to a dumb old brick. What AI-doomers are worried about is not "superintelligence" but rather "super-capabilities", and these are subject to much more stringent physical constraints.
> and learned six languages before the age of eight¹. ¹: I have not heard
Sorry, I'm having trouble parsing this -- not heard what ?
> I was merely arguing that the gap between agentic dangerous AI systems and us would not need to be unboundedly large.
That is true, but the AI-doom argument relies on this gap being unboundedly large. The AI is not merely smart, it is "super"-intelligent; it cannot merely position troops better than human generals, it can win every battle despite being completely outnumbered and outgunned, etc. The AI who is a slightly better general than its human counterpart would be a problem, but not an unprecedented problem, as the stories of humans like Caesar and Napoleon illustrate.
Since you put "no natural upper bounds", here's a pedantic list of upper bounds:
Holographic principle information processing density
energy available to cool everything on earth's surface
Speed of light limits on communication and cognition
Halting problem
Note that these upper bounds are """fairly far away""", but you know, failing to mention them probably also means that 10-20% improvement over human capabilities in all fields is also impossible. This is an unfortunate fact of how logic works, and scientists are racing to find new forms of proof that disallow this, because they have motivated reasoning, and I have every man reasoning.
Both you and Bugmaster appear to be misreading the actual context of that claim, which was regarding resource demands rather than capabilities having no natural upper bound. Capabilites need merely exceed those of humans, and then basic agentic goals like “survive and reproduce” or “make paperclips” have no natural upper limit as to the resource demands that they create so as to create conflict (won by the AI, because it’s smarter than humans) between it and other species.
I agree I misread, but by complete stupid coincidence those are also indirect constraints on the ability to gain resources, (I.e. AI will be bounded by the light cone, cooling will limit the rate and indirectly the total amount of resources acquired due to 2nd law of thermodynamics). Of course if that was the argument I wanted to make, I should have made it, instead of an unrelated one.
Oh also I was being sarcastic, and so think I am your mind clone when it comes to object level points. Let's echo chamber, I mean, combine our proofs.
>failing to mention them probably also means that 10-20% improvement over human capabilities in all fields is also impossible.
Specifically for the speed of light limit, specifically for communications, barring a surprising discovery in physics, we are already at the physical limit for communications speed, since we routinely use e.g. radio to communicate, so we can't get even a 1% improvement in the communications delay to e.g. our satellites and space probes.
I'm unsure whether I'm agreeing with you or disagreeing.
To be clear, I was being sarcastic, because I think naming physical bounds that are orders of magnitudes away as constraints on near term improvements is a dumb argument
Secondly, I said "communication and cognition". Yes the *latency* may be at or close to a physical minimum, but the *bandwidth* may not be, considering how little information English contains, and how slowly it gets produced relative to good thoughts (tm). Also yeah if hypothetically if we get a giant honkadonka badonker's brain (A REAL TERM USED BY A REAL X RISKER), we're talking more about "limited by bus speed" (aka Einstein traveling at light speed thought experiment train fast) rather than limited by extremely slow human reflexes (regular train fast).
I think we're just talking and neither actively agreeing or disagreeing.
I think part of the problem is exactly defining what risks AI pose. There is a difference between a hand-wavy nanobot apocalypse and something like “It could disrupt multiple industries completely and immediately, which will lead to instability in the economy”.
But big picture, absolutely none of this matters, because we can clutch our pearls here in the United States, or even pass regulation, trying to slow down technical progress (good luck), but the reality is progress will just march forward in Russia and China and India.
Seems to me the only safety we can get is to try to make the US be the leader of this technology, instead of de facto ceding leadership to China
Yes, this is a vital point that just gets whistled past. The closest I've ever seen a doomer get to addressing it is "the Chinese government is notoriously pragmatic and careful and would never take such a risk" which is so disconnected from reality it leaves one staring in awe.
As someone who spent most of their adult life in China, I am unsure how you think this is disconnected from reality, especially to that extent? China has been regulating AI more strictly than anyone, the Party prefers AGI not to exist since it endangers their rule, and as a country they are much more steerable than democracies are. Russia and India are so far behind in the game, and AI is so hardware dependent that I do not see how you could think that progress would move faster there. (answering to both carateca and Victor). The "But what about China" arguments ignore the fact that 99% of AI progress China saw is due to US open-source models, and on their own they make next to nothing. Read about their Chip manufacturing industry to see how corruption eats everything from the inside and how difficult advanced technology is to make.
I am not saying they are not a geopolitical player and that they need to be involved - but I also don't know serious AI Safety Policy people who ignore China? Still, I think the right proportion is 80% work on Washington, 10% EU, 9% China, 1% everyone else combined; based on impact it'll have.
Also we literally just lived through a global pandemic which was very likely caused by China's failure to be pragmatic and careful in regulating scientific research.
Note that even if they didn't actually create COVID, they were still engaged in the kind of high-risk research that *could* have caused something like COVID, which is sufficient to disprove the claim.
But basically the entire arc of post-Cultural-Revolution development in China, from Deng on (well, before Deng was formally in power, too) has precisely been reaction to Cultural Revolution and an attempt to ensure that China does not fall into similar chaos again.
I wasn’t specifically referring to government sourced AI, I was referring to the fact that if the US government tries to limit googles efforts into AI, they are just going to move their efforts overseas.
If increased regulation becomes reality, any small country in the world just has to have cheap power, and they can build themselves as the go to location for model training. Just like Ireland is a corporate tax haven, and Switzerland used to be a bank haven.
Data is easy to move.
In terms of the national security (including economic) risks of AI, my point is that they will happen regardless. The only question is what country profits from them both economically and geolopolitically.
And if it wasn’t clear, I don’t think acknowledging those risks or even fear mongering about nanobots is a good path forward for the US to be the primary beneficiary of this technology.
Data is easy to move, compute is not - it can be regulated.
Also, if the US says "only models which can legally be sold/used in the USA are those created in these countries" then ~no one will want to make models elsewhere, since US is a huge market and the main market to be on. This is not such an impossible task, there are very few places which can do it, very few companies which can do it, and if the US decided to do it it could sanction countries which tried to skirt this. US alone essentially has the power to control compute if it wished, it just does not right now.
Smart Researchers are not easy to move. If you tell some AI expert earning 6 figures that the company is moving to outer Mongolia, expect most of them to quit and get a regular programming job instead.
> The closest I've ever seen a doomer get to addressing it is "the Chinese government is notoriously pragmatic and careful and would never take such a risk" which is so disconnected from reality it leaves one staring in awe.
Actually, China is considerably *less* likely to create a doom AI. They are so obsessed with strictly controlling culture and access to information, that they will make *sure* their AI is aligned with communist party values, because if it's not, heads will roll. The places with free experimentation are the main sources of danger.
I didn't say it wouldn't be dangerous, I said a CCP-aligned AI would be less likely to be a doom AI that exterminates humanity. The "danger" I'm referring to is typically this sort of AI doom. Any AI likely has more mundane dangers, like misinfo bots from GPT.
Most Western attempts to align an AI will focus on getting it not to kill anyone, killing is bad, blah blah blah.
But a CCP-aligned bot is implicitly one that already thinks it's okay to kill millions of people. It may find it easier to generalise to killing everyone.
> Most Western attempts to align an AI will focus on getting it not to kill anyone
Or maybe a Western AI will be obsessed with eliminating racism, and so kill all white people. Gemini already erased the white founding fathers. Hard to say how culture will evolve these days.
> But a CCP-aligned bot is implicitly one that already thinks it's okay to kill millions of people. It may find it easier to generalise to killing everyone.
That doesn't sound like alignment to a goal of spreading communism to me.
I'm sure we can imagine all sorts of possible worlds featuring silly mistakes, I'm just saying that I can see fewer plausible worlds where a CCP-AI exterminates all of humanity.
This is just one of the many versions of the "it's inevitable so there's no point in taking any action" argument that I find deeply unconvincing. I find that a person only makes this argument if they don't actually think there's any existential risk. If your house was on fire and the doors and windows were boarded shut, most sane people would still be trying to break down a wall with their shoulders even if they understood that it was probably impossible.
China is also interested in controlling AI. International cooperation sometimes works.
Well china is mostly cracking down on any AI that ever mentions "internet maintenance day"
And Russia won't be making much AI because it's loaded with sanctions on electronics. Most of the smart people are trying to get somewhere, anywhere else before they are drafted as cannon fodder.
I'm puzzled by your puzzlement. The (indeed silly) "coffeepocalypse" argument just highlights how new things instinctively and generically cause panic, even if unjustified—and a fortiori when there's actually some inherent danger to them. It says: you should significantly lower your prior regarding the risk associated with new developments.
This might be persuasive if one could point to instances of AI-risk people being excessively cautious about technology in other domains, which would be a demonstration that their prior regarding the risks of new developments was set too high. But in fact you find that they're gung-ho about most non-AI technological development.
This presumes that there is a specific class of people who make bad panic-predictions, rather than bad panic-predictions being a characteristic of people in general.
Why do you require it be in a different domain? The duplicitously-named OpenAI's stated reason for not releasing GPT-2 (from 2019) was that it was too dangerous. Now, there are far more capable models freely available, I'd say they were "excessively cautious" if I didn't think they were lying through their teeth about "safety and security concerns" as Bootleggers in Baptist vestments.
Even if this argument about priors is accepted, don't you also have to look at the specific evidence for the specific risk? I don't think a typical AI-risk person would assign a high probability to a randomly-selected new thing causing disaster, so they aren't necessarily disagreeing with you about priors.
I think it would be best rephrased as "There are people who have a reflexive mindset against new ideas. It's a good idea to get rid of such a mindset". Global cooling and overpopulation may not be technologies but they are ideas, just as AI doom, AI paradise and pretty much every AI-related prediction and concept is.
One problem here is that it's probably a good idea that we do have at least *some* people who are true curmudgeons, willing to challenge everything and anything, simply because it's good for everything to go through a challenge. It would be great in general to acknowledge that the society benefits from a vast number of different mindsets and ways of reacting to things.
I appreciate the steelmanning, but isn't this really an argument aimed at fellow anti-AI-riskers to help them feel superior? I just can't take this charitably at all.
Yeah, the argument isn't the point, it's the comparison of AI doomers to people who were afraid of nothing and forgotten by history.
I don't find the linked Rutherford bit very convincing, but knowing nothing about the guy, I would assume he was prominent in the field, admitted he was wrong shortly afterward, and remained prominent in the field. In which case it's a much more charitable example, made stronger by the guy having enough expertise for people to think he knew what he was talking about.
While this is true, your decision to play or not to play depends on the size of the revolver's drum. If the drum contains one trillion chambers, and you know that only one is loaded, then you might as well play at least a few times (assuming there's a payoff involved).
For any reasonable payoff, even $1 a pull, it would make sense to pull that trigger enough times to make yourself filthy rich. 1/1,000,000,000,000 is really really low odds. You would win the big jackpot lotteries *multiple times* (there's about 4,000 times better chance of winning a multi-hundred-million lottery) long before finding the one chamber that's loaded.
Hmm, interesting hypothetical, pitting two common sources of irrational thinking against each other: very low odds and marginal utility. For example, is it worth that final pull of the trigger to get that last $1 to make yourself filthy rich when you could safely stop at almost filthy rich? After a few tens of thousands of pulls, it's difficult to weigh the tiny benefit of another buck against the tiny risk of kicking the bucket.
After a million pulls or so, (perhaps 2 weeks of semi-continuously pulling the trigger), the dominant tradeoff will be repetitive strain injury rather than the 10^-6 mortal risk. Come to think of it, a 100 year lifespan is about a third of a nanomort per second - even if one could pull the trigger three times a second, the hazard of just being alive outweighs the 10^-12 risk by about 100X.
Yes, I considered this angle, but within the hypothetical it makes more sense to talk about how many times we would take the risk, rather than thinking about repetitive strains.
Most people wouldn't want to be a billionaire from this plan, as the risk starts to be real, even if still incredibly small. Pulling $10 million would be such a low chance of finding the bullet that individual triggers between 9.9 million and 10.1 is nothing. Pick a number you would be happy retiring with and pull away! Anything less than about $100 million is nothing compared to a trillion.
I would like to have a choice about the game of chicken we seem to be playing with the climate and a variety of other issues on this planet and in my country.
Oh right, democracy will solve it, we can just vote our way to Utopia.
AI is also the only way we could ever hope to advance medical technology enough to end (or at least delay by a large amount) death in our lifetimes. (Not saying it is a guarantee, but without it we certainly won't get there)
So it's more like we've all been poisoned by a very slow-acting poison, and the russian roulette gun has bullets in some chambers and antidotes in others. You might get killed but you have nothing to lose.
Do you think there are other advances that could realistically get there in the next 50 years? I have no doubt things like cancer treatments will continue to improve, but even over the past 10-20 years the average life expectancy in the developed world has plateaued.
Ok, so you don't know what advances have not been discovered yet or where to even go looking -- but you are sure that they do exist ? What makes you so sure ?
I am not the one who claims to know what the future will hold. I am saying that it is impossible to know the future, unless you are Nostradamus or something.
>Do you think there are other advances that could realistically get there in the next 50 years?
Over that time frame, given sufficient investment, I think Drexler/Merkle nanotechnology aka atomically precise manufacturing would probably suffice to fix the vast majority of illnesses and consequences of aging. "Replace with known-good parts" goes a long way, even where we _don't_ know in detail how the parts function. In practice, the investment is not happening, while the investment in AI _is_ happening.
I doubt we'll be ending death anytime soon, AI or no AI. On the other hand, our lifespans had already increased significantly relatively recently; we might be hitting diminishing returns, but still, there's some cause for cautious optimism. AI is a very powerful statistical analysis tool, but it's no silver bullet; still, with more powerful tools we might get some of those returns, despite their diminishing.
On the flip side, if AI isn't powerful enough to end death, then I'm certainly not worried about it taking over the world or turning the universe into paper clips.
(By end death, I at least mean natural death - like if we had powerful AI it seems like it would be trivial to ask it to find the causes of aging in the human body and tell us how to prevent them)
Indeed, I agree that if we had an AI that was effectively omniscient, then we could ask it to "find the causes of aging in the human body and tell us how to prevent them". In fact we could ask it for pretty much anything (logically consistent) and get the correct answer. The same is true not only of an AI, but also of a fully-functional crystal ball, extradimensional magic-using diviners, clairvoyant psychics, and any other fictional omniscient entity that you could devise.
Analyzing the workings of a very complex physical system doesn't seem like something you need an omniscient being for, just a very powerful computer/intelligence with biological knowledge.
But I'm confused why you're concerned about AI risk, if you think AI is incapable of something like that. If it can't even solve aging then surely it can't take over the world.
Oh, I'm not concerned with AI risk -- that is, not in the AI-doom sense. I don't think that AI will either end death or kill us all; it's a powerful tool, not a pocket deity. However, I *am* concerned about what thoughtless or malicious people could do -- and are already doing -- with powerful tools. I think we should be focusing on *that*, not on some hypothetical superintelligent future AIs that are likely never going to happen.
> Here’s an example of a time someone was worried about something, but it didn’t happen. Therefore, AI, which you are worried about, also won’t happen.
Right, that's an obvious strawman. The real argument goes more like this:
"Here's an example of a time someone was worried about something due to employing a particular pattern of reasoning. The something didn't happen, and the manner in which it didn't happen helped to expose the flaws in the pattern of reasoning. You are worried about AI due to employing the same pattern of reasoning, riddled with the same flaws. Therefore, you are likely to be just as wrong as that other guy."
So what are these flawed patterns of reasoning? You imply that there is some kind of argument to be made there, but as far as anyone else can tell the same argument applies to "nothing ever happens".
The pattern that I often witness is people clinging too much to a good narrative without experimental data to back it up. Good narratives are incredibly useful for generating hypothesis that you can then experimentally test (including back-testing), but a good narrative alone is not sufficient to put forth as evidence for some future event.
The AI Doom people have a good narrative. It is a plausible story that one can easily imagine to be true, and in it might actually be true! You cannot debunk it and show it is definitely not true, which allows it to live a long life as a good narrative. It is totally fine to build this narrative and share it. The problem only arises when you want to act on the narrative without first doing the work of showing that the narrative is correct.
Lots of people in the past have come up with good narratives for how the future will unfold. Video games back in the 80s/90s were incredibly convincing. If you allow children to act out violence in video games, it will desensitize them to violence! We later realized that children are actually quite good at differentiating between reality and a video game, and we really should have tested the hypothesis that they can't before doing something like regulating video games.
With AI Doom the situation is significantly worse because the suggested solutions have a history (that you can back test against) of negative outcomes. Stripping people of civil liberties, going to war to stop scientific advancement, totalitarianism, etc. all have a track record of being bad for human flourishing, and we shouldn't budge on those because someone has a good narrative.
> The problem only arises when you want to act on the narrative without first doing the work of showing that the narrative is correct.
Showing the narrative is correct either is itself a source of the danger (develop AI and see how capable it becomes), or it involves funding AI safety research to prove that AI safety either is possible or impossible. I think plenty of AI safety people are just saying to fund that research, which doesn't seem unreasonable.
Certainly some people take it much further, like EY, but I'm not sure he's a representative sample.
Consider that instead of debating AI-doom, we were debating demon-doom. I claim that demons from Hell are about to invade our plane of reality through portals on Phobos, and this is why you must a). start stockpiling holy shotgun ammo, and b). stop using electricity, because electricity attracts demons. Let's imagine that I managed to spin that summary into a coherent narrative.
Showing that my narrative is correct is either a source of danger (open demon-portals and see what happens), or it involves funding demon safety research to prove that demon safety is either possible or impossible. So, how much are you willing to contribute to demon-safety research (and/or holy shotgun stockpiles) ?
Straw man. Demons don't exist, AIs do exist *right now*. AI danger is already *here*, even if it's not superintelligent AI.
The existence of superintelligent AI also follows from straightforward scientific arguments. Show me a scientific argument that demons exist, and then we'll talk.
"AIs" in the sense that you mean them (i.e. AGI) do not exist; ChatGPT is arguably the best we've got, and it's only slightly better than Markov chains. It's just a statistical tool that happens to work quite well at a narrow set of tasks. And the reason I compared "superintelligent" AIs to demons is because "intelligence" is poorly defined but also not terribly important; what you really mean when you say "superintelligent AIs" is "super-*capable* AIs that are functionally omniscient and omnipotent". There are good reasons to believe that such entities are either extremely unlikely to exist, or outright prohibited by the laws of physics (as we understand them today).
They don't. ChatGPT and its ilk aren't AIs in the sense of "AGI", and it's doubtful that the LLM approach is even capable of reaching anything resembling general intelligence.
> AI danger is already *here*
It's not, at least not in the sense that AI doomerists understand the term "AI danger".
I'm not opposed to people asking for funding for AI safety research. I probably won't provide such funding as I think there are better uses of resources, but if this was all that was being asked I would not likely care much at all.
However, there is a *very large* segment of the AI Doom population who think we should, as a society, make sacrifices to do things like slow down AI progress (Scott has an article where he indicates support for a slow-down). These sacrifices range from simple things like banning the authoring of certain types of software (a constitutional violation in the US), to ceding control over all future AI development and usage to large companies and/or governments (historically this hasn't ended well), to global totalitarian state (EY).
Where did you get the sense of the size of that segment, as opposed to how vocal and provocative those people are? By which I mean, are you sure it isn't a selection bias effect where the provocative positions and people just get more publicity for clicks and so that gives a skewed impression that they are widespread?
I would definitely call Scott vocal, but I wouldn't call him provocative. In fact, he often seems to go out of his way to avoid being provocative. I also don't believe that he says what he says just for clicks.
I have also spoken with other people at medium length who are active within the rationalist community who feel similarly. It is certainly possible that there is selection bias, but my selection mechanism isn't just randos on twitter or comment threads. It is people who spend a lot of timing thinking about these things and some of which have a background in AI/machine learning.
But the general flawed pattern of reasoning is the tendency to extrapolate all kinds of specifics from very little data. If you have two points (on a 2D plane), you can connect them with a line, a parabola, an exponential curve, or any other kind of shape you want, and draw whatever conclusion strikes your fancy. It also helps if you outright ignore several other points, because they don't fit on your preferred curve.
I'm having trouble seeing a difference between "you are following a general pattern of flawed reasoning in which you extrapolate too much from too little data" and simply "I think your evidence is weak."
Deciding whether someone is extrapolating "too much" or merely the correct amount requires actually weighing the evidence, right? It's not a fallacy like "denying the antecedent" or "ad hominem" where the argument *fundamentally* doesn't work and you can see that merely by examining the structure, without even considering the details.
The difference between "a flawed pattern of reasoning" and "your evidence is weak" is that the second case actually presents a *stronger* argument (relatively speaking). Saying something like "I drove my car down a steep hill with a tailwind, and based on this experience I believe it would get 1,000 mpg in daily use" is a mistake; but it's a different kind of mistake from saying "given that cars are getting more efficient and given that this process will continue indefinitely, and also given present-day mpg ratings, I estimate that the car I buy in 2030 would get 1M mpg". The second argument does consider evidence, but only peripherally; the first argument, meanwhile, is entirely based on evidence but falters in its interpretation.
I still do not see the distinction you are trying to draw, unless that distinction is "one of these examples predicts the performance of existing technology, while the other predicts the performance of future technology."
Things like "present-day MPG ratings" and "cars are getting more efficient" are both forms of evidence, and it seems like one could make some sort of reasonable estimate of future performance based on those or things like those. Obviously "this technological trend will continue indefinitely" is a dubious premise, but it seems dubious in basically the same way that "my downhill speed will continue indefinitely" is dubious, to me.
Yes, the dubious premise is indeed the difference, in that it is explicit and taken as given without being subject to any challenge. You could tell that 1,000 mpg guy, "hey, actually you forgot to factor in terrain, re-run your test while driving on flat ground or uphill"; and once he does it, he will correct his estimate. But there's nothing you can say to the 1M mpg guy to disabuse him of his estimates, because they are not based on any evidence at all (and are arguably contradicted by evidence). The fact that his prediction model also incorporates present-day mpg ratings is not very relevant, since they don't substantially change the predicted outcome.
I would identify the pattern of reasoning that the argument says is flawed as being essentially "trusting others' opinions on new technology":
1) I'm curious as to how much I should worry about new thing X.
2) I observe that many important and knowledgeable people worry about X quite a lot.
3) They probably know important things about X that I don't.
4) Therefore, I should worry about X.
So the argument is, "this pattern of reasoning would have gotten us the wrong answer in the case where X=coffee, so we shouldn't treat it as valid; in particular, we shouldn't use it in the case where X=AI"
But this ("trusting others' opinions on new technology") is actually a useful pattern of reasoning in the sense that it gives results better than chance. The coffee argument against it doesn't work; we always knew, with such a large reference class, that there would be cases where this reasoning has led to the wrong answer, so identifying coffee as one such case doesn't give us any new information about how often it's wrong.
This itself is an example of bad reasoning as it considers the particulars of each problem to be equivalent, that the underlying truth is derived from our predictions about it.
I only wanted you to clarify your comment, because I'm not sure what you meant by it. That said, trivially speaking, if my argument is "All X are Y; Z is Y; therefore Z is X", then you could point out the logical flaw in it without knowing any of the particulars. You could say, "No, Bugmaster, you are not justified in believing that Z is X", without knowing what any of the variables stand for.
"You are worried about AI due to employing the same pattern of reasoning, riddled with the same flaws."
The argument only works if someone is making the same claims about AI as about the other problem. It would indeed be disproven if someone describes how their reasoning is different.
The explicit claim is that *only* flawed reasoning is the underlying cause:
> "Here's an example of a time someone was worried about something due to employing a particular pattern of reasoning. The something didn't happen, and the manner in which it didn't happen helped to expose the flaws in the pattern of reasoning. You are worried about AI ***due to employing the same pattern of reasoning, riddled with the same flaws**. Therefore, you are likely to be just as wrong as that other guy."
The risk due to AI is a function of the particular attributes of AI, flawed reasoning is only one part of the potential for error.
Another way of putting it: it is not difficult to encounter a Rationalist engaging in realtime cognition who forgot that they are only an *aspiring* rationalist. Decompose the problem into an explicit object model with attributes, behaviors, interconnections with other objects/phenomena in the world, etc, and Maya will/may dissolve in front of your eyes.
Or, we can stick with Bayes Theorem: heuristics + sophisticated post-hoc rationalization, fantastic food for Maya.
The counterargument, included at the beginning of the article, is that the AI risk people are not employing the same pattern of reasoning. The coffeepocalypse person would then need to show how the reasoning was similar.
It's still arguing from an N of 1. "Here's _an_ example of...." is never a reasonable basis for sweeping judgments such as "____ will/won't happen."
If you're saying that two Ns of 1 each, pointing in opposite directions about a particular type of risk, rationally leave us with no basis for being sure whether or not that type of risk in a new instance is real -- that, I agree with.
But that is not what the real-world dialogue seems to be regarding actually a lot of big-risk questions. My experience is the same as Scott's: of people extrapolating from "here is an example where the worry turned out to be wrong and/or the logic behind the worry was wrong", to "therefore we know there is no risk." Which is absurd.
Why would it be absurd if the logic behind the worry was wrong? The argument is that the logic behind worrying about AI is different than the logic behind worrying about other risks.
"Why would it be absurd if the logic behind the worry was wrong?" Seriously?
I suspect now that you are trolling me but okay I'll play along....let's stipulate that the logic behind a worry about _________ being unsafe is wrong. You are seriously arguing that this stipulation actually _contradicts_ the worry? Proves _________ to be without meaningful risk?
No, disputing the logic that leads to a conclusion does not disprove the conclusion. Even if you arrive at a conclusion in an illogical way, that conclusion could still be true. But you don't know that it's true.
In other words, it would not prove ______ to be without meaningful risk. However it also would not prove that _______ has meaningful risk. Maybe you advocate treating everything as a possible existential risk until proven otherwise, but most people default to not treating everything as an existential risk.
No, it merely exposes your worry as being unjustified. It is possible that you should still be worried, but before you do, you need to find some legitimate reasons for doing so. Otherwise, you would be compelled to worry about literally anything anyone could imagine, and there's an infinite number of such things, so you'll waste all your resources worrying for no good reason.
A maximally hostile and bad explanation of the prevalence of halibut method: for any question that seems hard, most people won’t even try to engage with it at an object level. What they do instead is look for any similar disagreements they remember and which side won them and try to pick the winning side to maintain social status. They get helpfully halibuted into the ‘right’ side of the argument by the nudge in the problem presentation and from then on they are invested into getting their side of the argument to win, spreading halibut arguments further. Note that at no point did they reflect on what the actual answer would be - that would be waste of resources.
This doesn't feel hostile to me. It is certainly how I approach a many important life questions, and I would be better off if I did much, much more of this.
Never mind "seem hard", just "seem sufficiently implausible" is already enough.
Remember high school maths: generation after generation are taught to calculate carefully, methodically, showing their working at each step; then when the final result comes along, they are told to check it - at the very least, plug it back in to the original question, and ask themselves: "does this actually look sane, or did I make a mistake?"
When a chain of otherwise convincing reasoning arrives at a sufficiently weird answer, and there is no immediate way to verify the answer against anything but that chain of reasoning, the instinct trained into most people since childhood isn't to update their priors; it's to look for the mistake. A person might not have the time or the relevant expertise to identify a mistake, but the assumption there is a mistake remains hard to dismiss.
This instinct is also what protects from, e.g., proofs of God's existence.
Proof of God's non-existence is much more fun, but too much fun for most people to bear. It is excellent for debugging how the conditioned mind works, I believe there to be no better topic (at least one that can be mentioned in public). Science would be #2, Democracy #3.
Even more fun: whether people who claim there is no evidence for God have a burden of proof. Do negative claims have a burden of proof, or do they not? If not, who made that decision, where is that decision documented, and do we know (JTB) it to be logically perfect?
Contra the coffepocalypse argument: the coffepocalypse is real and true. It's a drug that infected the entire world, and even people who don't drink coffee are mostly addicted through other sources of caffeine (tea, coke, energy drinks, etc). We can't fathom living without it, we got dispensers in every company, college, rest rooms. We feel like shit waking up until we get our first dose and getting a hit together is one of our main way of socializing. Our world runs on caffeine.
And the case against AI is the same: we're facing a potential future where the entire human civilization is dependent on it. Of course the same case can be made of cheap energy, of cars, or something else (insert that picture of a 26-lanes highway for impact). We're living on layers upon layers of technology & infrastructures we can't live without. Can the pile eventually crumble under it's own weight?
I admit, it's not exactly (at all?) the original coffepocalypse argument. But I stand by it.
That argument, as you say, applies to every single technology ever developed including agriculture. If you're insisting we should never have come down from the trees in the first place, okay, but that decision is done. We put all our chips on technological civilization and if we don't want to see eight billion people starve we have to play the hand out to the end.
"Contra the coffepocalypse argument: the coffepocalypse is real and true. It's a drug that infected the entire world, and even people who don't drink coffee are mostly addicted through other sources of caffeine (tea, coke, energy drinks, etc)."
I was tempted to post up something like this, haha. I avoid caffeine as much as possible - but e.g. right now I have a cola here because I had a particularly rough night, so I need the kick of some caffeine to get back to baseline. But I went weeks without. (Not judging the people who need their caffeine constantly, I just wasn't happy with how my body got used to it and couldn't function without it, erasing all of its actual benefits.)
To be clear, though, I don't think this is actually much of a problem. It just amused me that the original post the article is citing was about coffee.
For me the best version of coffeepocalypse is as a demonstration that no matter how obviously harmless a new technology is, people will be extremely worried about it and say it will destroy everything. This is such a consistent rule that it even applies to *coffee* (but NB also all other new things) so you shouldn't take any information from people being extremely worried about a particular new technology. I don't think this is actually true or valid, but it's the most optimistic interpretation I can think of.
It's a social argument! It's not meant to be deployed against "AI is dangerous", but against "Various people say AI is dangerous, therefore AI is dangerous". If you don't trust your ability to evaluate the technical arguments, all you have is what various people are saying, and this kind of thing becomes a critical consideration. The news says "Geoffrey Hinton, very respected AI expert, says AI is very dangerous, so maybe you should think so too". In that situation "A lot of authoritative people have thought things were dangerous in the past and been wrong" is a point very worth considering.
Even technically minded people can do this. If you've never thought deeply about the technical details of AI risk, but heard some bad version once and dismissed it from that point forward, then when you find someone suggesting AI is dangerous, you may assume they got there by adopting the conclusions of others, and try to convince them that this way of reasoning is error-prone.
Edit: If this is the case, the most productive response may be "I don't think AI is dangerous because others believe it, I think it's dangerous for the following technical reasons that have nothing to do with deferring to others' conclusions..."
Edit2: In the same way, I have often used the Rutherford and Szilard argument *as a response to* the then-common argument "experts predict AGI is a long way away, therefore it's a long way away", to which saying "sometimes experts predict things are a long way away when they aren't at all" is perfectly reasonable
"In that situation "A lot of authoritative people have thought things were dangerous in the past and been wrong" is a point very worth considering."
I think what Scott's saying is that there's a symmetrical:
"A lot of authoritative people have thought things were dangerous in the past and have been right"
So that's only true if "very worth considering" means at most it'll only shift your priors very slightly and likely not at all, like in the Russian tanks example.
So it comes down to how you feel about the person who is currently speaking to you. It boils down to "I'm the kind of person you should trust, therefore trust my argument." I think that this is what most casual conversations about technical issues come down to.
It may shift you a lot, back towards your priors. To start with, any given thing being very disastrous seems very unlikely. Then you learn that some authoritative people think it's very dangerous, and you assume this is a reliable indicator so you update a lot towards it being dangerous. Then you learn that authoritative people have thought all kinds of harmless things are very dangerous in the past, so you undo most of your earlier update and go back to thinking it's probably fine.
Further learning that authoritative people are also often right doesn't change this much, that's where you started from.
"If you don't trust your ability to evaluate the technical arguments, all you have is what various people are saying"
For a more real world example. Worked in R&D for a big company. Many, but not all the technical people (PhD's mostly) understood what tech would work and what wouldn't. The managers and maybe half of the R&D team had no ability to evaluate technical arguments, so did evaluation based on social prestige (trade publication cover stories / keynote speakers). So, managers always had approx half of their R&D team saying the same thing as the industry buzz and reliably choose to invest effort in those areas. The technical people would be frustrated by this, but always lost socially. Occasionally, someone would be good at tech and socially, quit and go start a really successful company.
It’s not hard, but if it means you don’t get a vote. At some point, someone calls a meeting and says, we have to decide X today. If most people who don’t know say so, then the meeting ends up being dominated by either (a) someone who really knows; or (b) some Dunning-Krueger narcissist. Which of these is more common, I wonder…?
If we are going to use the ostrich as a metaphor, I think it would be symmetrical to have a fable for the other side also. Perhaps Chicken Little saying the sky is falling? That actually turns out to be very old; one of the jatakas has Buddha, in one of his past lives, calming the fears raised by a rabbit after a similar minor incident . . .
I have to admit I started out puzzled over the whole AI Safety issue, for several reasons:
1. Nothing I've heard of in the realm of AI comes close to anything I'd regard as "Intelligence", and I haven't heard of any major steps in that direction.
2. Since nothing we have, or are contemplating, comes close to "Intelligence", we really can't plan an effective protection against the next-several-generations version that might conceivably be a threat.
3. Who would ever give any automated system enough power to control important things in real life anyway? It makes some interesting premises for movies (Like Colossus: The Forbin Project, or War Games, or 2001: A Space Odyssey), but really those are more ridiculous than interesting.
I mainly started to take the idea semi-seriously because Scott takes it very seriously, and Scott is not only very smart, but very focused on good reasoning and careful consideration of evidence in context. At least in some areas. ;-)
But I keep returning to something I saw in one of the Substacks, which looked at the evolution of LLMs and GPT in particular. If I remember right, it extrapolated versions of GPT and concluded that GPT4 used some enormous amount of training data and computing power more than GPT3, and GPT5 would need very much more, and GPT6 would need more computing power than the world could provide, and still would be unlikely to approach anything like independent intelligence. So I think the dangers from AI are still (almost) entirely theoretical, and far in the future, and so far away from the current world that preparations are (probably) useless. But I'm nevertheless glad that smart, conscientious people like Scott are devoting some thought and energy to the issue, just in case it does turn out to be important.
I was amused that Scott mentioned my favorite current issue: global warming. I'm not sure of his overall stance, and maybe he was only listing issues that others might find relevant, but "People didn’t worry enough about global warming" seems like the opposite of his argument. Judging by media coverage and politicians' rhetoric, people seem to be worrying a very great deal about global warming, even though there's very little evidence that it has caused significant damage (at least compared to other environmental issues of the past and present) or that it will in the future. When I make this claim, I don't consider media reports to be evidence, even the media reports that say "Scientists say". I trace to actual scientific claims in actual scientific publications, as compiled by the IPCC, and find some reasons for concern, but no current or projected problems that would be anywhere near the magnitude of the problems caused by serious efforts to rapidly decarbonize the world's economy, which seems to be the only solution on offer.
But maybe my reasoning is no better than the head-in-the-sand example. I often point out that the sky-is-falling reasoning common in public discussion is unsubstantiated, and the proposed solutions won't come close to solving the alleged problem, but even if that's true, it doesn't prove that there's no apocalypse coming. So maybe it's good that people are willing to spend a few hundred billion dollars on things that probably won't make much difference. Except that I still think we could spend those hundreds of billions on other things that would make a difference in other areas.
Ironic, because Musk was one of the earlier AI alarmists. I think he was - at best - epistemically lucky. Anyone that starts their own AI capability research is not taking the risk serious. The likely reason he ever signaled caring is because AI risk is the sort of "cool" "sci-fi" issue that was on brand for him to care about. But lately he has found the Culture War more interesting.
On the global warming aside I think a lot of the angst is a hangover from the whole RCP8.5 whoopsie where it was widely assumed we were on track for a +5C world in 2100 (the effects of which would’ve been quite bad for humans and other animals). It would be very helpful (for me) to read a ranking of the main risks for our current trajectory of +2 to 3C. Coral reefs are supposed to one of the first major Earth elements to go down which wouldn’t be great and a shutdown of the AMOC would be quite disruptive. Other than that I don’t have a good grasp on the major impacts (crop failures?).
With respect to the cost of decarbonization it’s important to include the benefits from cleaner air. Every year we are learning more about the deadly and disruptive effects of dirty air. This imo significantly shifts the cost-benefit for decarbonization and clean energy.
Granted that there may be major impacts from 2 to 3C of warming, do you think these impacts would be larger than the impacts of rapid and complete decarbonization? If so, why? Or do you think we should have some more gradual and modest decarbonization? If so, how much would that help? All the projected impacts exclude the effects of mitigation. Economic prosperity has allowed very significant mitigation of climate effects any place that has become rich. Caribbean islands were pestilential messes in the 19th century, to the point that British soldiers assigned there were assumed to be given a death sentence. Although most of these places are not among the richest in the world now, they're rich enough to have effective public health, potable water, and air conditioning that makes them pleasant places to live and visit.
It's interesting that you would say the IPCC findings are toned down to sound positive, and include only findings that are "100 percent ironclad evidence wise." I have no inside knowledge of the IPCC process, but the published reports include findings that are "very high", "high," "medium," or "low" confidence, and "high", "medium", or "low" evidence. It seems to me as an outside observer that the Working Groups try to be honest in their summary of scientific understanding, but the Summaries for Policy Makers are "jazzed up" to imply more dramatic results and higher confidence than the Working Groups report. Then, the IPCC press releases make claims that are not scientific ("we must limit warming to 1.5C") while claiming that "scientists have determined" these claims. Are you saying that all of these are "ironclad evidence wise?" What are the papers with the "merely good" evidence for tipping points?
On biodiversity, there certainly are challenges, as there have been for the last few thousand years. Most of these problems seem to come from disrupting natural habitats. No doubt climate change adds to these challenges. But consider an alternative scenario: warming is limited to 1.5C by getting to Net Zero emissions by 2050. How much agricultural mechanization, irrigation, transportation, cold storage, and fertilizer would be available in this scenario? What would it imply for agricultural productivity? If agricultural productivity is significantly reduced, how will people react? It seems to me they'd have to use much more land for agriculture to feed the population. Wouldn't this cause bigger problems for biodiversity?
I agree that we understand the economy much, much better than earth systems. It would be nice to eliminate our impact on natural systems. This isn't possible, so it's important to reduce those impacts where and as we can. There are certainly risks that we can't quantify, which means we can't really have high confidence that we can determine the appropriate tradeoffs. But that doesn't mean we must do anything and everything necessary to limit warming to some figure. And it doesn't mean that there are easy and practical paths to reach that figure.
I don't have specific sources at hand, but my impression is that the most significant dangers now are significantly increased food prices leading to more widespread starvation, widespread drought, large scale population migration, sea level rise, and damage from increasingly severe weather.
Whether or not any of that counts as "significant damage" is a matter of opinion, I suppose.
Yeah, these are some of the dangers currently being quoted in the popular media. The scientific basis for these is thin, if you look at things like the IPCC reports. By which I mean the IPCC Working Group reports, not the Summaries for Policymakers, and certainly not the press releases. And all of the predictions include the caveat "without mitigation" - which ignore the benefits from continuing to spread a modern economy to the poor world.
My main source is the IPCC. Not the IPCC as described in media reports. Not the IPCC press releases. not the IPCC Summaries for Policymakers. The actual IPCC full reports. Most interesting are those from Working Group II and Working Group III. They're very long reports (Working Group II is 3068 pages), so I don't read them all. But any time I see some claim of pending doom, I consult the source to see what it actually says. For instance, many people claim that climate change will cause massive food shortages. IPCC Working Group II report, page 796, says something much more modest. I spelled it out in more detail in my comment below from Apr 25. If you have trouble finding it, it's the one that begins "I have a little different perspective".
I share your concerns about potential impacts - they could be much worse than I expect. But there's no science I'm aware of to say they're likely. Likewise, the boosters casually claim that we're in an "energy transition" that will soon have us all driving clean electric cars powered by clean electricity from sun and wind, with geothermal heating replacing oil and gas. The don't recognize that this can only be done at enormous expense, that would necessarily take resources away from every other personal and social priority. And no government, or any company, is acting as if such a thing is possible, despite lots of happy talk. Probably because public polling consistently shows that voters put climate change near the bottom of priorities for government action, and most voters aren't willing to pay even $10 per month to reduce the impacts of climate change.
No offense, but that seems pretty bold. Are you claiming to know how to summarize the report better than the people who wrote it? Can you point to specific sections and/or page numbers in both documents that you think are contradictory?
I'm perusing the technical summary right now, and while they do not put specific numbers on projections (How many people will die? What % of crop yield will decline?) the picture they are painting is pretty dire.
I have a little different perspective on the climate apocalypse narrative. As far as I can tell, it started with James Hansen testifying to Congress in 1988, when he predicted something like 6-8 degrees of warming (it's hard to be certain, because measuring against a pre-industrial baseline hadn't been started yet). This became the "catastrophe" scenario which justified Kyoto. By 1995, the IPCC reports were predicting something like 2-3C warming by 2100, but the "catastrophe" narrative had become established. The RCP8.5 scenario was never intended as a prediction: it was based on maximally pessimistic assumptions, but it fit the "catastrophe" narrative, so it got lots of attention.
If you want an assessment of the risks, I recommend the IPCC Working Group II report (https://www.ipcc.ch/report/sixth-assessment-report-working-group-ii/). It addresses each area, but doesn't attempt to rank them. The full report is 3068 pages, so you probably won't want to read it all. But each area forecasts rather modest potential impacts, most of which can be mitigated. In the area of crop failures, for instance, they predict something like 10-20% more people (65-130 million) will experience food insecurity due to climate change, without the effects of mitigation (p. 796). But "mitigation" would mean better crops, better fertilizer, better irrigation, better tilling, more mechanization, better storage, and better transportation, all of which are eminently practical, and all of which have contributed to vastly improved agricultural production over the past 60 years.
And yet most media reports still say "Scientists say we must limit warming to 1.5C to avoid the worst impacts of climate change." In fairness to the media, IPCC press releases say this as well. But the actual IPCC reports don't say anything like it.
I admittedly haven't dug into the details of the working group report as in depth as you, but I'm surprised by your internal framing of the quote you provided. Why do you consider 20% more people experiencing food insecurity a modest impact?
I found it strange that they talked about "food insecurity", rather than straightforward agricultural production, but the working group could only summarize the work of many people in the field.
I consider it a modest impact because it affects a relatively small number of people (1-2% of the world's population), and can be mitigated or reversed by technological and infrastructure improvements. But these improvements would be endangered if we focus on decarbonization. Actually, decarbonization would necessarily mean much less mechanization, irrigation, cold storage, and transportation, which I assume would have a much bigger impact on food insecurity (or agricultural production) than whatever impacts come from climate change.
I don't consider myself an expert, and I haven't gone through all the literature, but I check this stuff to compare it to what gets attributed to unnamed "scientists" who are quoted as saying we "must" do something. The actual scientific findings are much less dramatic than what gets reported.
Very interesting, that's an interesting perspective. Have you read anyone who's attempted to lay out an estimate of comparisons of those two things that you could share? This would be very different than my own internal understanding of the impacts which is that we would not be able to, with current technologies, mitigate the climate impact on food security in areas close to the equator so I'd love to read more.
I'm not aware of anyone who's attempted a rigorous analysis comparing impacts of (a) anticipated climate change given current economic activity vs (b) economic changes driven by some decarbonization path. Even stating the problem means that you'd be piling speculation on top of assumptions before you started the analysis. Whatever results you got would certainly not convince anyone who already has a stake in one outcome or the other.
Bjorn Lomborg has long been making this argument - that we'll be in much better shape economically, ecologically, and in human flourishing, if we keep using fossil fuels (reducing where it's practical) than if we try for immediate and drastic decarbonization. For this, he's been labeled a "climate denier".
To address your last point - yes, cleaner air would be a benefit. It's worth pointing out:
1. Rich countries today have cleaner air than they've had in the last 150 years. Cleaner air is not incompatible with a modern energy system.
2. Poor countries mostly suffer from "indoor air pollution" that comes from burning wood or dung for heating or cooking. Getting access to electricity, regardless of the source of the electricity, is the best thing they could do to get cleaner air.
3. There are currently no plans I've heard that credibly promise a reliable energy system with dramatically lower carbon emissions. The areas most committed to green energy (Germany, Spain, California) seem to be bumping against the limits of renewable incorporation, with occasional negative prices for solar energy (that is, paying other areas to take surplus solar energy) without coming close to eliminating fossil fueled energy.
Well, yes. I'd say the problems with a nuclear future are (probably) more political than technical, but no one I'm aware of is even trying to build support for a lot more nuclear power, and I'm not aware of any country where the politics of nuclear are workable. Except China and North Korea, I guess.
>Who would ever give any automated system enough power to control important things in real life anyway?
You don't need to directly plug the AI into your nuclear defense systems or whatever. If people are making decisions based on the output of the AI, then the AI can choose it's output to manipulate the people into doing what it wants them to do. The answer to "who would give the AI that sort of power" is "everyone who might build the AI in the first place", since an AI you do nothing with is a waste of money.
And if the AI is connected to the Internet, it can do a lot of unintended actions more directly. Current "AIs" are on the web, how confident are you that we will stop doing that once we get close to true "Intelligence"?
I'd like to push back on a few things in your comment that seem mistaken to me:
> GPT6 would need more computing power than the world could provide
This might be true (although I think the cutoff would be higher than GPT-6) if we were stuck at current efficiency and current computing power. But the world is aiming to add much more computing power (qv Sam Altman seeking reportedly trillions to build chip fabs), and current AI technology is getting more efficient quickly so that it takes something like 8x less compute to reach any particular capability level this year than it did last year (more accurate numbers & citation on request).
> Who would ever give any automated system enough power to control important things in real life anyway?
I think we're just barely reaching the level of capabilities that begins to enable this, but I'd say we're already starting to see it. See for example Israel's 'Habsora' system for selecting kill targets in Gaza (https://bit.ly/3Qhj0dn); a human is in the loop in theory but not so much in practice. Ukraine has started to use autonomous drones to carry out lethal strikes without a human in the loop even in theory (https://bit.ly/3QjQojs).
> Nothing I've heard of in the realm of AI comes close to anything I'd regard as "Intelligence"
That seems surprising to me; I would claim that current state-of-the-art systems are already intelligent by many definitions of intelligent. That's especially true because I think of intelligence as (approximately) a scalar value; a cat has intelligence although not as much as humans, and certainly GPT-4 seems more intelligent than a cat in many ways.
I imagine it's a definitional issue? I'm curious about what you mean by intelligence such that current systems have none of it.
"Who would ever give any automated system enough power to control important things in real life anyway?"
Your enemy manufactures at scale a bunch of super cheap ($500 ish) AI powered cloud networked military bots. They are pretty stupid and easy to break (like star wars drones, but less humanoid), maybe some fly, some a quadrupeds, whatever. But there are just lots and lots of them, and they are good enough to take out trained soldiers at a 50:1 ratio and civilians at 5:1. You drop a million of them out of a airplane and they now control a city. So you decide to give control of your military to your own automated bots.
I feel this equivocates on "give power to control". If you drop *dumb* bombs on something, you're giving power to gravity, ballistics, and unpredictable bounces. These are all impersonal forces of nature that never have qualms about killing anyone.
Dropping a million bots on a city is similar to dropping a million bombs, except "we don't know exactly what square foot of enemy city the bomb lands on" is replaced by "we don't know exactly who the bot will kill". Perhaps you oppose war altogether, or at least bombs, but having a weapon that indiscriminately kills within a certain context is not a new thing and not something we don't know how to handle.
Sure - there are some things that approximate some level of intelligence. For example, some simple programs have been able to separate different varieties of flowers based on things like the sizes of different flower parts. Or facial recognition software is a sort of intelligence.
What I was referring to was something more like autonomous will - the ability to form an intent beyond its programming, to develop courses of action to pursue that intent, and to implement the courses of action.
I suppose GPT-4 might be more intelligent than a cat - it can recognize things as words, and can generally put the words together in a meaningful sequence. But I don't think it can recognize any concepts behind the words.
So, it seems that there *have* been steps made towards intelligence, contrary to what you said before? Because when you and I were young there were not computer programs that were as intelligent as cats in this respect.
> autonomous will - the ability to form an intent beyond its programming
I'm not, personally, entirely sure this cashes out into a coherent concept (unless you're a mind/brain dualist); it's like asking whether humans have the ability to think a thought beyond their brain. But in any case this is orthogonal to what I think of as 'intelligence', which may have been the source of my confusion.
> But I don't think it can recognize any concepts behind the words.
In my opinion this is an outdated (though commonly held) belief that traces back to the 'Stochastic Parrot' arguments from linguists; current language models are already doing concrete things (like math) that those authors believed they would literally *never* be able to do.
I think it's very much worth thinking about what, in your view, would falsify a claim that GPT-4 can't recognize concepts. The tests that I personally thought would distinguish (happy to talk about those if that's useful) are ones that LLMs have now passed, and I've updated accordingly.
Thanks for the reply. I should have made it clear from the outset that I know many people know a lot more on this subject than I do, and have thought in much greater depth than I have. You are obviously one. I'd appreciate your thoughts.
Is anything currently being done getting into the realm of AI Danger?
I've certainly thought about it a lot, & am currently transitioning into doing AI safety research full-time, but that doesn't mean I'm not wrong! Smart, thoughtful people disagree wildly about this topic, and I don't mean to make any sort of argument from authority.
In particular, I've been thinking a lot about the Stochastic Parrot view and why people hold or don't hold it, and what falsifiable claims it implies, so I'm very interested in hearing from people who do.
> Is anything currently being done getting into the realm of AI Danger?
That's a big question. We're certainly at or near the point of starting to see potential dangers from misuse, but I assume that's not what you're thinking of.
My main near-term worry is about bad effects from scaffolded LLMs. LLMs can mostly be thought of as not having goals, but it's not that hard to wrap them into systems that are goal-ish, and that can make persistent efforts to achieve those goals (see https://github.com/yoheinakajima/babyagi for a simple example). Successfully working toward a long-term goal requires a lot of reliability, and current SOTA mostly isn't that reliable. But I expect that sort of agentic system to work better within a year or two, especially since at least some of the big scaling labs are actively working on them.
Mid- to long-term, I worry more about systems that are more intrinsically goal-ish, which I suspect can be achieved pretty well by reinforcement learning on top of an LLM base (we're already doing RL on LLMs, namely RLHF, but that's quite a thin overlay). We know that LLMs are already pretty decent at deception and persuasion (refs on request), and once we have a more intrinsically goal-oriented system it can be *strategically* deceptive and manipulative, as well as having the usual set of instrumental goals (resource-seeking, self-preserving) that come with any terminal goal. I mostly don't expect terminal goals to emerge spontaneously, but I expect some people to give these systems goals that can cause real problems, deliberately or thoughtlessly.
That's a rough, off-the-top-of-my-head sketch of the medium-term (the next five years or so). There are other plausible paths for problems to emerge, but I think that's already plenty of reason to worry, especially as these systems rapidly continue to get more intelligent (for most ways of measuring intelligence).
I do think we got lucky that the approach that's working best so far is one that's very non-goal-ish. If it was RL that had gotten this effective and was advancing at the current rate, we'd have substantially more cause for medium-term worry in my view.
Again, lots of smart, thoughtful people have substantially different expectations; this is just my own current thinking.
Thanks for your thoughtful response. I now know a little more. As I noted below in my response to Donald, I still (mostly) choose rational ignorance - I recognize that I'm not likely to know enough to have definitive views, but I'm glad other people are worrying about it.
I recognize that LLMs are pretty decent at deception - I assume the recent dust-up over Gemini is an example. I assume something very similar is behind the auto-censoring that Facebook, Twitter/X and Google use. I don't like them, but it sounds like your view of people who give the systems goals, which sounds more like a problem with the people running the systems than anything else. Don't we always have such problems? In what way does AI magnify them?
I can imagine someone using AI to try to access automated systems to alter them - to steal from banks, to alter election results, to change real estate registries (I don't think these are on line yet, so good there), etc. Isn't there already such risk without AI? With or without AI, we need reliable online security, which is a never-ending escalation between the crooks and the guards. In what way does AI change the situation?
I think the AGI argument is dependent on the concept that AI will accelerate AI advancement, leading to AI that can independently accelerate its advancement
I find this plausible and scarier than coffee. We assume AI will not be able to independently defeat humanity but a more likely strategy is to enlist people's incentives such that AI plus 30% of humanity work against our true general interest. Cortez had a lot of native allies
An interesting conclusion of the proposal that the AI threat is completely impossible for humans to confront is that it would appear our best option is to accelerate AI while hardware limitations are at their greatest and seek to solve AI alignment with AI. Chat GPT3 was used to comprehend GPT2's underlying logic. Even if we don't succeed in creating a team of superhero shield AIs we could at least experience a hopefully very partial AI risk scenario and thereby recognize that serious steps actually should be taken
> it extrapolated versions of GPT and concluded that GPT4 used some enormous amount of training data and computing power more than GPT3, and GPT5 would need very much more, and GPT6 would need more computing power than the world could provide, and still would be unlikely to approach anything like independent intelligence
So basically, you're saying that people are compiling huge data sets for these next gen GPTs, and that we're possibly only one generalization trick away from a post-GPT AI that can distill better knowledge from less information, but you're somehow reassured by this state of affairs?
I won't pretend to understand how you define intelligence, but on only a strictly "capabilities" basis, AI is already dangerous to societies from an information manipulation angle, and the military is already commissioning AI fighter jets and weapon systems. We do know that AI that can generalize better from less information is coming, we simply don't know the magnitude of that jump.
So maybe we need a better definition of AI. The US military started work on a system called SADARM (Sense And Destroy ARMor) in the 1960s and deployed a weapon in 1995. It was an artillery-fired shell that included submunitions with sensors to identify armored vehicles and explosive-formed penetrators to kill the identified targets. Would this count as AI?
I assume that current technology can do more things more independently, but is this a difference in kind?
I'd say what falls under AI should be general purpose than special purpose devices like the one you're describing (like LLMs for language, computer vision for sight, etc). I suppose that depends on whether the sensors are special purpose ones designed specifically to detect armored vehicles, like a thermal sensor is specifically designed to measure temperature, or just straight up camera system and the detection is done via some kind of computer vision model. That could possibly be considered rudimentary AI.
Great comment, Brian. One minor push back… how are you defining “intelligence”?
I see AI as very intelligent in at least some narrow ways and getting both more intelligent and broader every day. I also see it as amplifying human intelligence. But I define intelligence as “the ability to process information so as to efficiently solve problems and exploit opportunities.”
I'm far from an expert on the topic - I don't have a well thought out definition. My main idea in this area was "smart enough to develop a malevolent intent, and an agenda to implement it."
I recognize that AI can do some things well that have always been considered areas for human expertise. But I don't see those applications as a danger, unless the AI can develop a malevolent intent. Of course, humans could take over the AI systems (or their reporting systems, or inputs) and do malevolent things, but I don't think of these as "AI Safety" issues. Should I?
> 1. Nothing I've heard of in the realm of AI comes close to anything I'd regard as "Intelligence", and I haven't heard of any major steps in that direction.
Computers went from literally 0, to chatGPT. Now the scale of intelligence that a lot of people seem to think on goes from dumb human to smart human. And so sure, we don't have anything that seems smart to us (ie is smarter than the average person) yet.
2. Since nothing we have, or are contemplating, comes close to "Intelligence", we really can't plan an effective protection against the next-several-generations version that might conceivably be a threat.
That means it's somewhat harder to plan and protect. Not impossible. Some people are doing things that look like they might help.
3. Who would ever give any automated system enough power to control important things in real life anyway? It makes some interesting premises for movies (Like Colossus: The Forbin Project, or War Games, or 2001: A Space Odyssey), but really those are more ridiculous than interesting.
There are 2 threat models. The model where you connect the AI directly to something important. And the model where you try to seal the AI in the finest Airgapped faraday cage, and it hacks it's way out.
In reality, people are often hooking these things up to the internet. And the internet is connected to a lot of important things.
Humans aren't in charge because the other animals decided to hand us something important.
Is our security good enough to keep the AI from touching anything important?
> and GPT5 would need very much more, and GPT6 would need more computing power than the world could provide, and still would be unlikely to approach anything like independent intelligence. So I think the dangers from AI are still (almost) entirely theoretical, and far in the future, and so far away from the current world that preparations are (probably) useless.
It's hard to work out how smart GPT5 might be, not having seen it yet. But sure. I will grant you that just scaling GPT level tech quite likely doesn't get to dangerous levels of intelligence.
GPT tech is something that was only starting to appear 4 years ago.
There is lots of research happening on various improvements. If there was some new more efficient algorithm was invented next week, we maybe get a couple of years of impressive headlines before a singularity in 2027. New algorithms can be and are invented reasonably frequently. We have no strong evidence against a near singularity.
Don't assume that progress will be just scaling up, not algorithms.
Thanks for the notes. I was writing my initial thoughts, and why I was initially puzzled by the whole concept of AI Safety. I recognize that I must have been missing something, or smart, level-headed people wouldn't worry about it as much as they do.
I opt for rational ignorance on the topic - I recognize I don't know nearly as much as people who take AI and AI Safety seriously. I don't think I could learn enough to know whether any particular view is convincing. I'll let various better-informed people continue to act and argue among themselves. I won't try to figure out how real the thread is. Unless someone starts using it as a basis for fundamentally restructuring the world's political and economic systems.
This argument feels weirdly outdated. It's hard to imagine claiming GPT-4 today doesn't "come close to anything [you'd] regard as intelligence". This is basically the argument I made pre-GPT4. At the time I also thought "I'd never give an AI any real unchecked power" only to discover when GPT-4 _actually_ came along just how valuable it is to say "Write a script to go through this repo, `dvc pull` and parse all the onnx files, and list the sorted, dedupped names of all inputs you find alphabetically to stdout" and then run the code it produces without even a preliminary glance at it. Why waste time checking it's work when I could just run it and see if it works?
It's simply too valuable to have a tool which does the coding work for you and that value is lost if you constantly have to check its work
Fair enough. My terminology was overly broad - I was focusing on AI Safety, not AI in general. What I should have said would have been more along the lines of "Nothing I've heard of in the realm of AI comes close enough to independent action coupled with malevolent intent to indicate a problem fundamentally different from buggy software."
Also, I was describing my previous attitude, and why I didn't understand the issues of AI Safety. I did mention that I take the topic more seriously now, and I'm glad people are paying attention to it.
On your software use: What's your distinction between "run it and see if it works" and "you constantly have to check its work?" Do you run it once to make sure it doesn't crash? Do you assume it's doing the work you want, and not something else? Do you check different conditions to make sure it can handle them all? I accept it's a useful tool, but some users have definitely trusted it too much. For example: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/
> a problem fundamentally different from buggy software
I don't think AI doom _is_ fundamentally different from buggy software. Imagine you have some flight control software that has a model of what angle the plane should fly at and then makes fine adjustments to keep the plane at that angle at all times. There's a pilot to take over if something goes wrong in the flight control software. Some bug causes the flight control software to decide the plane should be pointed straight down at the ground. All the other pieces of the stack work _correctly_ in that they cooperate to keep the plane pointed in the direction the model points it, counteracting whatever the pilot does to try to correct it.
This is not an AGI, it's just buggy software, but it got everyone on the plane killed because of a _high-level_ bug. Contrast with a low-level bug where it misinterprets one of its sensors or uses the wrong multiplier for one of the jet speeds or something like that.
Misalignment seems to me like a similarly _high-level_ bug in AI software. The parts of the stack that make it able to reason intelligently and path-find and whatever else work _correctly_, but the part that decides _what to do_ with those capabilities goes wrong somehow. I don't think it's necessary to find "examples" of malevolence. Unless you reject the assumption our AI software will have bugs, the result will be "incidental" malevolence like in the plane example. There isn't much in the way of "independent action" right now, but that's likely to change in the coming months.
Check out Devin https://www.cognition-labs.com/introducing-devin to see what people are already doing. When you think bugs in AI, don't think bugs in ChatGPT. Imagine ChatGPT works correctly, but _Devin_ has a bug. How is that realized?
In my software use-case, "see if it works" just means "see that it produced a non-empty list of tensor names and that it didn't raise an exception". If it did that, I generally trust it didn't miss any. I'm imagining a one-time use-case; not a general tool, so what circumstances it doesn't work in doesn't really matter so long as it works this time.
In particular, if the code _also_ goes into my `.ssh` folder, grabs my not-password-protected private key, and uploads it to a server so that some other piece of software can start amending early commits to my open-source projects on GitHub in order to inject security vulnerabilities, I probably wouldn't notice.
I just played with GPT4 again, asking it some questions about molybdenum chemistry. https://chat.openai.com/c/c279d911-e0ac-419a-b4a3-d472386d7c48 It said some reasonably sensible things, but it _also_ said that molybdenum in a MoCl4- complex was in a +4 oxidation state, when it should have said +3. It also needed prodding to consider chloride complexes at all - and this is in 6N HCl !
The coffee argument attempts to bolster the idea that risks that seem ridiculous are ridiculous and that things suggested to pose theoretical risks via novel predicted pathways are usually wrong. It differs from plague and war in that plague and war are bad in very predictable ways and that coffee and AI were suggested to be bad in heretofore unforeseen ways that sound stupid.
"Computers will make computers so smart they could outsmart everyone at everything, at which point they'll kill us all" is in fact a novel and stupid sounding argument. I just happen to think it's probably true.
Yup I think this is totally on the money. I think it's a combination of this and a bit of the scale component. It tries to argue that both "risk from AI" and "risk from coffee" are silly by implying that just because coffee was so ubiquitous (like AI will be) that it must be dangerous is a bad argument.
My take is related to ""Twitter derangement syndrome"". I often see smart people on twitter transition from reasonable (but ofc with biases etc) tweets, who avoid the dumb on the other side of any argument and focus on the interesting ideas of the other side of an argument. They transition to meme broadcast. Musk for instance.
I think this happens because (a) takes on issues aren't independent correlations, (b) the algo pushes you to the pattern of reading the smart people on your correlation side, and dumb people on the other correlation side.
So you get ego-massage ("I also think what this very smart person thinks - I could have said that if I had more time ...") and outrage ("I can't believe whats in the heads of the people who think the opposite to me.")
At this point you may start to become tribal, *you* change. Whats the point of reason if the other side don't accept it? Just firehose them with vibes.
So my take on the coffeepocalypse argument is they are no longer trying to persuade you. They are trying to influence the 99% of people that are tribal that they see also make dumb arguments.
Could it be that the more smart people act tribally, the more oneself does. TDS is infectious.
Perhaps the 'grey' tribe subconsciously operates with the world view analogous to trickle-down economics: The tribe will adopt what their own thought leaders think. And those thought-leaders are susceptible to argument as the redistribution of the wealth of the rich-in-reasoning. But social media isn't well-designed for that to work well?
This is all black and white. I don't mean it so - we all have these elements running through us at different times of day and different issues. TLDR, it's not you the person is trying to convince. That's not where the engagement reward comes from anymore.
So you're saying that because others listen to them, some of which may form policy, that it is important to refute the arguments? It seems to me, rather, that those that form policy who are swayed by such arguments are the main problem here, and ought to be removed.
But arguing against absurd arguments accomplishes aught.
"[...]because others listen to them, some of which may form policy, that it is important to refute the arguments?"
I think yes? At least to make sure to avoid forming a consensus-by-default? Example: "all these people are saying the Earth is flat, and so far I don't see anybody contradicting them, therefore they may be right". To follow on this example, I don't think one should argue with flat-earthers per se, it is impossible, but at least to state unequivocally that "the Earth is round" is sometimes useful. Not to them, but to others.
It probably doesn't help that despite spending years patiently and logically explaining why the threat of an AI apocalpse is nebulous but real, the ACX comments are regularly filled to the brim with people who apparently wilfully refuse to consider the issue properly. They're obviously not just 'intellectual weaklings' across the board or they wouldn't be reading ACX. Refusing to take weird/new/scary/theoretical/left-field arguments seriously just seems to be such a baked in glitch in human reasoning that even smart people fall into it a lot if they're not making a conscious effort to be open minded. I find this puzzling, infuriating, and terrifying, and if I thought I could reason my way into believing it wasn't that simple, I'd leap at the chance.
You're probably right that comments offer an impression of readership that skews toward the contratrian, hostile, and hare-brained. It's certainly true on YT, and hopefully it's true here as well.
Oh god I thought I was going insane, because I regularly see people say "I found lots of convincing arguments against AI risk" and when pressed, they link to something as bad as coffeepocalpyse, and then 5 other AI risk skeptics come in and say "this proves the gigantic blind spots the doomers have, this comments section is too pro risk".
I think you are right about the comment skew. However, I think arguments are also strong that:
1) humans are on a trajectory to catastrophe without AI, and higher intelligence can both accentuate and mitigate this (IOW we need to consider the catastrophe reduction potential of superior intelligence and coordination)
2) there is an inevitable arms race toward AI, so it will not be stopped. At best it can be partially managed for the when (not the if).
3) regulatory bodies are as likely to make things worse as better based upon their track records.
Agreed on 2. Idk about 1 and 3, but even assuming they're correct, encouraging people (especially the people who actually work on AI) to take this issue seriously seems like it will very likely have some kind of positive impact on expected outcome. It might not move the needle as much as we want, but it's still something, which with these kinds of stakes is very worthwhile.
It's always good to try and understand why people who make arguments you disagree with are wrong because what if they're right and you should change your mind.
"someone who demands that you be open-minded about homeopathy would not be your friend. They would probably be a shill for homeopathy and best ignored."
Is it possible someone might have new information about homeopathy you haven't considered that would prove it is actually effective? I have to say, it is possible, but so unlikely it doesn't merit my consideration (since I have previously considered homeopathy, and new evidence must overcome my well-earned priors). One only has so much time to devote to research, and far too many hypotheses are proposed to research all of them, especially personally.
The assumption here is "once people were worried about coffee, but now we know coffee is safe. Therefore AI will also be safe." This is so obviously flawed a chain of logic it doesn't merit consideration. It is, perhaps, an interesting question as to how people could use such "logic" to form opinions, but it is in the same vein as superstitions, so a more general class of thought processes.
People worried coffee would be dangerous, and were wrong - correctly leads to the conclusion that *its possible to worry about new things being dangerous, and be wrong* - but it absolutely *does not* lead to the conclusion that *its possible to worry about new things being dangerous, and be wrong, therefore any time anyone worries about new things being dangerous, they are wrong*.
The Halibut example is a little confusing because its reverse the negative (not) to are. Suppose it was this, leaving the NOT in place:
A guy thought Halibut was a type of cow. He was wrong. Its not a type of cow. Therefore AI is not a type of cow.
Now, its almost certainly true that AI is not a type of cow (unless the Mootrix hypothesis is actually true), but the fact that AI is almost certainly not a type of cow does not logically flow from the fact that Halibut is not a type of cow. The two statements are unrelated.
So too does the fact that coffee turned out not to be dangerous (except in edge cases where people drank over the LD50 which is about 10grams or 100 cups of coffee and died of a heart attack). The fact that coffee is mostly not dangerous is completely independent of the as yet unknown safety or danger of as yet uninvented AI.
The better argument for me is to think about the Omohundro AI drives argument, with relevance to human intelligence. Omohundro argued that without specific reasons not to, any AI with motivational drives would behave like a human psychopath in its pursuit of resources. AI at that level doesn't exist yet, but there are plenty of examples of the Unfriendly Human Problem, aka human psychopaths, or just plain mean people. The fact that Unfriendly Humans *can exist* strikes me as far more relevant to the hypothesis that Unfriendly AIs *may be able to exist*.
By analogy from humans with specific types of brain injury (my field), most present AIs are analogous to being disabled in many ways. ChatGPT is blind for example, you can't show it anything, it can only respond to text. Its also pseudo in its theory of mind capabilities - you could cry while typing a message to it and it wouldn't know, although it might respond sympathetically if you typed about how sad you were. By extension, it can't really empathise, no matter what it "types" in response. It is psychopathic by default because no one has solved the AI empathy problem yet. If we solved the AI problem for general intelligence but didn't solve the empathy problem, we might be in trouble. The fact that high IQ psychopaths do exist means that psychopathic AI *might* be able to exist, but is not a proof or certainty.
A guy thought Halibut was a type of cow. He was wrong, because his conclusion that it's a type of cow came from a bias in favor of thinking things to be cows. So any claims that AI is a cow should be treated skeptically and treated as possibly biased.
If there actually was a nutcase running around calling things cows, that probably would be how you'd react.
Likely this was said above and I missed it, but the Russell argument from nuclear fission is (arguably) from a much smaller reference class: the first time we invented a plausibly catastrophic technology, some experts confidently predicted it was impossible. Thus, Russell’s should be more convincing than the coffee argument, which is presumably sampled/found from a much larger class of worried.
For me, it's about remembering past moral panics. I'm not saying that AI couldn't be really bad. I'm just saying that lots of ostensibly smart people believing it's going to be bad is not, in itself, a strong argument. This is especially relevant considering the information cascades that do happen in our community. "Eliezer believes in AI doom and he's really smart in [some other domain], therefore I'll update towards his position on AI doom".
I'm pretty much an AI bear. While I agree that AGI is not impossible, I don't see strong evidence that it's around the corner. In fact, I have the same frustration as Scott in reverse. Lots of people gesturing wildly at stuff that is clearly not AGI, then expecting me to freak out, without addressing the core of my skepticism.
P.S.: My favorite moral panic I remember from childhood is the idea that we were going to bury ourselves in mountains of trash. I looked up some old news clips from the late 80's in the archive of my local tv channel and it was quite funny to watch.
From the inside, freaking about GPTs isn't about thinking that GPTs were bad, but that a fairly core part of my "oh yeah AI is just far away" model around 6 years back was: "yeah, all the people who confidently said that Kurzweil was dumb by extrapolating Moore's law, they seem to be right. We don't have any kind of scalable algorithm, adding more hardware only leads to incremental advances (like 10% less loss, or sentences that are slightly more coherent in a sea of nonsense) for much hardware cost."
Maybe I am overreacting, but it seems concerning that my and many other people's intuitions were too pessimistic, and if I start examining other AI constraint intuitions, and start comparing them vs actual AI progress, it seems the entire constellation of intuitions were all too pessimistic. Since the bias was pervasive in the world model, I don't think it's surprising that I would change my mind about the total. (Compare to estimating the cost of building a skyscraper, and then finding out that you were pricing everything's cost at 10x their value and effort. That bridge price is going to change by more than 10x, because the crews working on it will also be spending less time)
>P.S.: My favorite moral panic I remember from childhood is the idea that we were going to bury ourselves in mountains of trash. I looked up some old news clips from the late 80's in the archive of my local tv channel and it was quite funny to watch.
"I would like to understand the mindset of people who make arguments like this"
I think a lot of people who do this aren't making logical arguments. They're signalling some combination of
1. Relaxed and confident leader: "don't be anxious about this stuff, be chill. Look, I'm chill."
2. Be optimistic despite things in fact being bad: *artillery raining down on trenches "We've always gotten through wars in the past, this time we'll make it too."
3. Caution against the village weirdo, adversary, and/or "boy who cried wolf."
4. etc.
These things are social signals and/or "just-so" stories (the heuristic arguments you allude to). Yes, on further inspection they aren't logically correct, but they do have utility, which is why they have survived as standard ways of communicating. I'm not defending their behavior, just observing!
In terms of Zvi's simulacra levels, people are operating on different simulacra levels and jumping around the levels. When you engage on level 1, suddenly it's a snap to reality instead of social signals and other goals, so it's a sudden incongruous incompatibility with what they're saying. You're talking about different things. It's like the fearless leader getting annoyed "Hey what are you doing saying we might lose the war, I'm trying to calm everyone down so we can keep people showing up to work."
Then there's also stuff like "do I want to think of myself or be perceived as the sort of person who would lie or be incorrect about something" and/or true curiosity about the truth, which is why they don't have a response to you or are even surprised/annoyed that you've said what you've said.
In terms of behavioral biology, you're snapping on the executive function/logical reasoning parts of the brain and asking them to allocate more compute here when someone was going more off vibes, social calculation, feels. This can be uncomfortable or even perceived as socially adversarial.
Disclaimer: I made it all up and you probably know most of what I wrote. To be fair, I understand a large purpose of the post was to introspect about our own ways in which we make these same mistakes, rather than point out that people make a certain category of mistakes.
There's also the question of the degree to which humans have free will, in general (at their best), and during each individual instance of cognition.
ie: it seems plausible that humans do have some control over the degree to which they will engage System 1 for analyzing a problem, but if one is raised in a culture of intellectual laziness, they may tend to take the easy way out.
As they say: "Ain't nobody got time for that", "Time's a wasting", "Thought is time", "Everybody wants to save the world, but no one wants to help Mom with the dishes".
People making these kinds of “Thing A didn’t happen therefore Thing B also won’t happen” assume that (or act as if) your worry about Thing B is based solely on feeling the emotion of apocalyptic fear for the very first time, and submitting that alleged novelty of emotion as your only evidence that worrying about Thing B is rational. So, they believe that if they can attack the novelty of that feeling (“people have felt apocalyptic fear before!”) your argument has crumbled. They are not thinking of your apocalyptic fear about Thing B as a downstream consequence of other evidence at all, so they don’t think they have to address the upstream evidence.
Could it be that someone just wanted to write about coffee and knew that pretending his coffee-story was actually about AI would get more clicks for the coffee-story?
I think people like Eliezer and Conor Leahy predict doom so confidently and occupy so much attention space here that they cause these arguments to look good as existence proofs.
Even putting those cases aside, people typically ascribe high confidence to most public claims people make and this implicitly lowers the bar for what counts as acceptable criticism. Original claim-makers are trying to model the world; critics are just trying to raise considerations.
I think this is a typical failure of thinking probabilistically. A lot of people basically think all probabilities are zero or 1; they hear someone say the probability of AI killing us all is non-zero, and hear that it's 100%. The coffee argument is a perfectly fine argument that the probability of AI killing us all is not 100%; having demonstrated it's not 100%, they conclude it's zero.
I was surprised Scott didn't mention this, as I think it is one of the strongest arguments for his side of this debate; catastrophic risks are not unimportant just because they are tail risks. A 1% probability of (say, catastrophic global warming) is a very serious risk, even though it is likely to to not happen, and people will stupidly point fingers about how "wrong" the experts are.
But where does the low probability come from? I would be skeptical about somebody who said there was a 99% chance that AI would kill us — where do they get that number? I am similarly skeptical of someone who says there is only 1% chance.
I have noticed this, too. It comes up a lot in personal assessments of risks; lots of people will shrug off a seatbelt thinking "it won't be me in a crash!", others develop phobias of typically harmless things due to one particularly bad run-in ("i will never approach a dog again, they're all rabid little bite machines!") See also Scott's excellent post The Psychopolitics of Trauma.
I'm not registered with twitter (and don't want to be). All I see at the 'read the full thread here' link is the tweet with the coffeecup image. I might be missing the link on the page, but I have looked and can't see anything obvious. Is there a way for me to read it?
As a general tip, sometimes you can unroll Twitter threads with sites like https://threadreaderapp.com/, but it depends on whether the site has seen the full thread before.
First, most people haven't studied AI or any of the other potential global threats in detail, so they default to the easiest and most reassuring response. Anticipate this sort of reasoning and when you encounter it, engage them in a discussion that brings out the bad things that HAVE come true. Keep in mind that people don't want to seem ignorant, so this is also a way for them to avoid a discussion they know, they don't know too much about. But they actually know more than they think. You can tease out what they do know and provide them with facts they didn't know using all sorts of techniques such as storytelling etc. You can even discuss literature and movies which is an easy way to get people to engage. Talk about Isaac Asimov and the three laws. Or the movies Colossus and War Games.
Second, be sure to acknowledge their other, more legitimate concern, which is that it is difficult to know what to worry about or what the best solution for a problem is because science, by its nature, changes as more is learned. Vitamin C is good for you, (Linus Pauling is a genius), then it's not, (Linus Pauling is a crackpot), then it is good again. If you worried about every possible bad thing that might happen, you could drive yourself nuts. So just be optimistic and focus on every wrong prediction when some distressing topic comes up.
Third, you need to steer the discussion away from heuristics so it addresses the known and unknown facts. AI may pose a threat, but governments are trying to work together to head off the problem. Then discuss why it is important for the public to understand the threat so that they can encourage their lawmakers to enact legislation or enter into treaties to address it. The same can be said about climate change. Make it personal. For instance, I am old enough to remember when we had something of a winter in South Florida. Moderate temperatures for over a month instead of one day in January. We need to be better prepared for the next pandemic. My sister died because of Covid. We have to stop Putin--most of my family in Europe were killed by Hitler.
Don't be put off by wrong thinking. Just get people to think better.
Coffee is pretty clearly not a threat to anyone's rule. The threat seems to have come from coffee𝗵𝗼𝘂𝘀𝗲𝘀, not coffee, but this is also difficult to attribute.
Gathering was recognized as a real threat long before anyone had heard of coffee. That's why the Roman emperors banned firefighting clubs. The problem wasn't the firefighting - everyone knows that's good - it was the clubbing. Other types of clubs were also banned, but we have better documentation on the firefighting clubs specifically because someone thought they were such an obviously positive force that they might get an exception, which they didn't.
What I find weird about this is that public restaurants are far, far, far older than bans on social gathering. I see no real difference between bars and coffeehouses.
If you believe my analysis, I see basically two options:
(a) Coffeehouses were no more threatening than bars, the revolutions were sparked by other things, and if the coffeehouses hadn't been open, revolutionaries would have met in other locations.
(b) The set of existing social norms around bars prevented them from generating the same social bonds that are expected in clubs, and that actually formed in coffeehouses. In this case, the coffeehouses do have some real causative force, but the revolutionary effect would have been easily avoided by regulating them more harshly.
If we had psychedelics houses, or philosophy houses, or maybe a combination of the two, would 100% of the risk to existing power structures derive only from the houses?
> I see no real difference between bars and coffeehouses.
Of course. Do you form any conclusions as a consequence?
Well, opium dens were a real thing. Opium is not a psychedelic and my understanding is that the characteristic behavior of the clientele was to basically just lie there in a stupor. My first guess is that psychedelic houses would be similarly nonthreatening.
To the extent that coffeehouses were threatening, it seems to have been because they not-so-secretly functioned as philosophy houses where you could also have an exotic beverage. (Could you get food?) Philosophy houses would presumably be even worse.
But philosophy practiced on an individual basis is also fine. That's another thing that is very old and that has been handled well by various large civilizations.
There is a book I liked very much, Patricia Crone's Pre-Industrial Societies: Anatomy of the Pre-Modern World, which describes a dynamic that seems relevant here.
Crone characterizes societies as tolerating a certain level of wealth accumulation and a certain amount of thinking. ("Philosophy", in what I take to be the terminology of your comment.) The structures of any given society, according to her, can accommodate a certain amount of each, but tend to fall apart when those bounds are exceeded. At that point society is reorganized into something that can operate well at larger scales.
You could make the argument that the coffeehouse revolutions are an example of this, that the increasing wealth of Europe gave people more time to sit around thinking about politics and this eventually made the surprisingly small-scale social structure of medieval Europe untenable. But this would be a pretty clear example of option (a) that I gave above. The coffeehouses would be just as irrelevant as the coffee.
> If you were to use a methodology other than guessing, do you think you may get different results?
No. That's why I think guessing is adequate here.
> Also: did you notice you didn't actually answer the question? I did.
Well, you edited your comment after I'd responded to it, which isn't a good look if you're going to go around insinuating that people have no responses to your points.
Then again, you don't really appear to have points, either. What do you think I missed?
1. The threat was explicitly characterized as coming from the gathering and not the coffee.
2. This comported with more than a thousand years of recognition that gathering could foment revolution.
3. It also matches our retrospective analysis in the present day.
4. Making it easier to revolt doesn't automatically mean people will revolt.
5. Making it harder to revolt doesn't mean that people won't revolt.
6. This makes it difficult to attribute revolts. But in most cases, they are overdetermined; any single factor whose absence would have prevented the revolt will be glaringly obvious.
7. Psychedelics houses do not pose a plausible risk of societal destabilization.
8. Philosophy houses do.
9. To the extent that philosophy in the absence of social clubs (or, synonymously, factionalism) poses this risk, (a) coffeehouses posed the same risk independently of coffee, and (b) so did everything that wasn't coffeehouses; the destabilization was going to happen with or without the coffeehouses.
10. To the much greater extent that philosophy houses pose a risk because they are social clubs, coffeehouses posed the same risk in exactly the same way, and this obviously has nothing to do with coffee.
All of that is already contained in my two previous comments. You might note that in both cases, 100% of the risk specific to the establishment is attributable to the fact that it is an establishment.
Philosophy houses aren't going to overthrow the king because the promotion of philosophy leads to better, more effective philosophy. But they can do it by giving people a safe space in which to discover that they share certain unflattering views of the king. If the king is going to be overthrown by the raw power of philosophy in practice, the houses won't contribute to that.
Psychedelics houses aren't going to overthrow anything, even in extremely favorable circumstances, because the customers don't organize.
There absolutely is a backlash against AI - an online community I'm part of has outright banned any use of AI generated images and is vehemently against AI art. People also worry about generative AI being used for cheating on homework, taking peoples jobs and more.
And I think it's fair to say some of those fears are legitimate.
However, the twitter thread's argument is that technologies are sometimes feared for their potential to change the world, but then become accepted for their benefits - even if those changes do actually occur.
There /is/ a brief dismissive mention "doomer predictions of the end-times" towards the end, yes. However I think this is basically incidental, it's not really what the argument is intended to be about.
Yeah, so I think you've been mislead by Scott saying that it was an argument against AI killing us all. What I'm saying is it's not talking about that at all. I think we can assume he isn't concerned about that.
When he talks about "massive backlash" and "battles" over AI, he's instead talking about the actual mainstream concerns of AI taking peoples jobs etc. (At least, this is my interpretation.)
The thread with the coffee analogy is saying people fought about other disruptive technologies, but after they've been around for a bit and accepted, people largely forget about the controversy.
That's presumably why he was talking about coffee rather than a more recent disruptive development - say, the automated loom - more people still know about the disruption that caused.
>he's instead talking about the actual mainstream concerns of AI taking peoples jobs etc. (At least, this is my interpretation.)
Hmm... Well, labor-saving/livelihood-ending technologies have be incrementally added ever since, as you cited, the automated loom, and I don't think _that_ aspect of AI is going to be stopped, backlash or no. While I'm generally not fond of the phrase "the wrong side of history", it is a reasonable approximation to where opponents of labor-saving/livelihood-ending technologies have found themselves...
Rather surprised that not one has brought the caffeoylquinide story up yet. And it appears gone from the dark corner of the network where it was once found. So it shall be narrated from one's memory. Gather before the simulated firelight, if you wish, and don't mind the clowns on fire jumping out of their helicopters.
---
1) Most of the Old World calories at the time of the advent of coffee were from grain. Wheat, among others. Wheat contains gluten; gluten metabolizes into many things, including something called gluten exorphin b5. Exorphin almost sounds related to morphine perhaps, or ephinephrine. This is not an accident. B5 acts on opioid receptors, with all the requisite effects of other opioids.
2) Coffee famously contains the psychostimulant caffeine. But it is not just a biosynthesis machine for some perk-up stuff. Created by a process as mindless as it is merciless, it also contains hundreds of other substances, many of them not important enough to study, and a few that are. One of these is something called 4-Caffeoyl-1,5-quinide. Something that actively blocks opioid chemicals from working as usual -> https://pubmed.ncbi.nlm.nih.gov/15088081/
Taking the narrative here at face value, when the substance gets shipped to Europe, a population that is constantly low-key microdosed on an endogenic opiate receives a sharp kick of a potent psychostimulant mixed with an opiate inhibitor, all distributed to their intellectual elite in highly concentrated locations.
Suddenly it becomes painful to think of the injustices of the world. Suddenly there's a feeling of strength to do something about them. And just as suddenly, there's cohesion with others who think the same.
Negarestani, the fable concluded, wrote about the wrong black substance.
The important part is the similarity of characteristics.
For the nuclear chain reaction example, the point isn't just that 'someone was wrong'. It's that even the most informed experts, who had all the available evidence, and who were completely convinced, where embarrassingly wrong. It's also that nuclear fission was cutting edge science at the time. That papers like to publish strong and sensational claims. That the scientific community reacted the way it did. The governments reacted to it a specific way as well. It's all this squished into a specific example.
The idea is that you're trying to work out whether something like AI risk is overrated or underrated by comparing it to other claims that share characteristics.
Its like this: "People were wrong about coffee and this whole AI situation feels kinda similar to the coffee one. It's similar because the same kinds of people are saying the same kinds of things and reacting in the same kind of way. It similar because the reporting reads similarly and the arguments for it have similar characteristics and so on. It walks and quacks just like a duck, so it probably is." That's the steelman.
The AI situation does share a lot of characteristics with other apocalyptic predictions and they're characteristics that make it harder to believe. (Just like we find it difficult to believe claims that share characteristics with pseudoscience, cults, scams, phishing, etc).
To most people, ai risk sounds like this. "Right now, thousands of the world's brightest minds are reckless working towards creating software that will, most likely, wipe out humanity. We don't know exactly how, but we're very certain that it'll be a disaster. All these smart people are being foolish in thinking the software is safe and helpful. It's not bad how social media is bad (soecietal transformation), its bad as in meteor hitting the planet bad (civilization ending)."
That's a wacky sounding claim that sounds a lot like loads of other wacky sounding claims.
This is my view of the argument. The point is that the characteristics of the old argument presented as an example are also present in the modern argument being discredited, and the modern argument is stupid for the same reasons the old one is.
I think your basic summary of the argument is wrong: You rephrase it as "Here’s an example of a time someone was worried about something, but it didn’t happen. Therefore, AI, which you are worried about, also won’t happen."
But instead it is the somewhat similar argument "Here's an example of a time someone was worried about something, but it didn't happen. Therefore you should be less confident that, AI, which you are worried about, will happen"
In other words, it is an argument against "we should be certain AI is dangerous" and not an argument for "we should be certain AI is harmless"
Unsurprising that if taken to mean "we should be certain AI is harmless" it falls flat.
Isn't this only a semantic difference and Scott would agree with you that the argument was against "you should be certain AI is dangerous"? I think what Scott meant by "AI, which you are worried about, also won't happen" is referring to the world ending scenarios of AI, or AI doom and not arguing for AI to be harmless.
Hm. I don't quite follow. Maybe I'm just not reading you right. Let me try to clarify:
You read Scott as saying "... Therefore, AI, which you are worried about, also won’t happen" and meaning "Therefore, the AI apocalypse, which you are worried about, also won't happen". You also think that Scott and I basically agree, just different words or maybe minor differences. Is that accurate?
If so then I agree that Scott is saying that the "past panics have not borne out" argument is intended to be an argument against AI doom specifically.
But I still think that this constitutes a misunderstanding of the argument, or at least it differs from my reading of it.
For me "past panic wrong, therefore AI apocalypse wrong" is a statement about the world, asserting that AI apocalypse is objectively unlikely.
OTOH I read "Past panic wrong, therefore certainty about AI apocalypse is wrong" as a meta argument. It says "there is a recurring pattern in humans to be really confident that things are going to be terrible and in many instances they were confidently wrong. No matter how certain you feel on the inside, the outside view suggests that most people in your position were wrong so maybe you should be less confident that your judgement is correct".
And of course there can still be argument about whether the claim "most people in your position were wrong" is actually accurate but in any case this seems like a substantially different argument from "AI doom is objectively unlikely".
Yes, your summary of what I was trying to say is accurate! Thanks for that. And I think I see what you're saying now better than before. However, I am still struggling to see exactly the difference between the two. Isn't the meta argument just the supporting evidence for the argument around how objectively likely/unlikely AI apocalypse is?
That is, past panics have usually been wrong ----> certainty about AI apocalypse predictions is likely to be wrong now ---> AI apocalypse is not likely to be true.
I think we agree though the the main question is evaluating the claim "most people in your position were wrong" as regardless of whether the key question is the meta one or the one about objective probabilities, that is the evidence utilized to make the claim.
I feel "AI apocalypse is not likely to be true" isn't really the goal of the argument but the practical end result is often similar / the same?
past panics have usually been wrong ----> certainty about AI apocalypse predictions is likely to be wrong now ---> we have no idea how likely the AI apocalypse actually is
is what I'd say the argument is. And that usually comes paired with "I'll only care about / invest a lot into AI security if good reasons are provided" with "we have no idea how likely the AI apocalypse actually is" often not being good enough a reason.
And in that regard "most people in your position were wrong" is probably the key claim, I agree with that. If really most people in similar positions were wrong then certainty in AI doom is probably unwarranted. And if there's not some reasonable level of certainty then why care / spend a lot of money.
I'm surprised you don't get it. It's motte-and-bailey!
Bailey: Look at the people I liken to my opponents! Look how dumb they are, and by extension my opponents!
Motte: Here is a curious story that undeniably happened. I think it's similar to AI worries that we all already know are unfounded, and story similarity is subjective, so you can't prove me wrong.
> And as some people on Twitter point out, it’s wrong even in the case of coffee! The claimed danger of coffee was that “Kings and queens saw coffee houses as breeding grounds for revolution”. But this absolutely happened - coffeehouse organizing contributed to the Glorious Revolution and the French Revolution, among others. So not only is the argument “Fears about coffee were dumb, therefore fears about AI are dumb”, but the fears about coffee weren’t even dumb.
Fears about coffee weren't even dumb, point #2: just look at what it's done to our own society. We've used coffee to normalize the notion of drug addiction. "I can't function without a cup of coffee in the morning" has gradually gone from a punchline to a simple "yeah, I hear ya" statement, because it's not just that weirdo over there who's an addict; virtually everyone in our civilization today, even a surprisingly high percentage of young children who we would never think of giving alcohol or tobacco to, is addicted to caffeine.
Forget marijuana; the true "gateway drug" is coffee.
> it's not just that weirdo over there who's an addict; virtually everyone in our civilization today, even a surprisingly high percentage of young children who we would never think of giving alcohol or tobacco to, is addicted to caffeine.
You must know a radically different subset of "virtually everyone in society" than I do.
Really? How many people do you know who 1) don't drink coffee and 2) are not Mormon? I honestly can't think of very many at all, off the top of my head at least.
Almost everyone I know. The main problem is not an objection to caffeine, it's that coffee tastes and smells absolutely disgusting.
However, while I actively seek to avoid caffeine outside of exigent circumstances, and this puts my consumption well below most of the people I know, I would not expect any of them to display the signs of addiction such as a splitting withdrawal headache. Having a Barq's root beer instead of a Mug when that's the only root beer option in the soda fountain just isn't "addiction" levels of consumption.
Huh. Wish I knew more people like the people you know.
Maybe it's my line of work. As a software developer, the notion that "programmers are machines that convert coffee and feature requests into code" has attained a certain level of "ha ha but serious" status, often leaving me looking around in disbelief, going "our brains are our livelihood, our most important asset. What are you *doing,* tampering with them with addictive drugs like that?!?"
About half of the people I work with drink coffee in the morning. We work in an open space so it's obvious. I drink coffee maybe once a month if I get a craving.
As an object level point, it does seem like a lot of kids drink caffeinated soda regularly. I don't know if you looked up statistics it'd be concerning, or that they reach the level of addiction, but it seems like a not unreasonable off the cuff intuition.
(I have no other stance on the merits or demerits of caffeine, just seemed relevant)
"I think it has to carry an implicit assumption of “…and you’re pretty good at weighing how much evidence it would take to prove something, and everything else is pretty equal, so this is enough evidence to push you over the edge into believing my point.”"
Kind of tangential, but this is the same kind of naive Bayesian fallacy that I was getting at in the lab leak vs zoonosis post. "Weighting evidence" implies that you have a small set of hypotheses and are adding log odds to promote them relative to each other, but this has the problem that it's not taking dependencies into account.
What you really need to use the evidence for is something like building a high-fidelity graph of what factors are relevant. There's tons of ways that Russia has a bunch of legible military advantages, but ultimately these just cash out to the one fact that Russia is bigger than Ukraine (and maybe also historically invested more into its military). Once you take that into account, you already expect each piece of Russia's military to be bigger than Ukraine's, and so it's of ~no evidentiary value to realize this for the particulars.
Rather, the evidence would have to come from other independent sources of information, e.g. morale, allies, quality, etc. (which turned out to favor Ukraine). The key here is that they have to be independent of each other, yet correlated with the outcome of interest, which happens most easily if they are the largest-magnitude causes of the outcome of interest.
Not sure my points here have any straightforward connection to the coffee argument. I guess the coffee argument is basically just telling people not to Aumann-update on this because it postulates a general factor of technology opposition which makes the discourse worthless as a signal.
In a sense, you could say that the coffee argument is a proposal to #cancel people who worry about AI risk, by proposing that AI risk worries are an authoritarian power-grab.
They're... Probably gonna succeed? At least, here's my current model for how cancelling works (epistemic status: I've only recently started thinking in these lines so I'm low-confidence):
It used to be that everyone who worked against the local powers got cancelled. The exact pattern of cancelling was complex, but up to my knowledge was mostly determined by the local powers having misaligned interests (e.g. polluting too much, as expected from externalities theories) plus traditional allegiances (e.g. religion).
For various reasons, this was untenable, e.g. the local powers opposed environmentalism too harshly. This lead to a lot of smart people building a movement that could overrule the local powers, which was mostly formed around "trusting the science" because that's what was the most developed justification for environmentalism.
But 90% of everything is shit, and this includes science as well as the non-scientific parts of this movement that was built. The "trust the science" movement elevated a lot of this to power, leading to poorly-informed impactful decisions, including much of "cancel culture" and "wokeness" partly because of lots of poorly-done science in the grievance studies.
There's the discussion of whether wokeness is dying or not, with various takes. Recently I saw some graphs where the more aggressive/negative approaches (like accusations of racism/transphobia/etc.) were declining while the more agreeable/positive approaches (like inclusion) were constant. This seems logical both because the backlash is especially concentrated against the aggressive forms of wokeness, and maybe also because the "woke purge" is essentially complete in the sense that the largest corporations all agree that they have to handle climate etc., so insofar as climate was the original motivation for the aggressive forms, the motivation is complete.
AI risk worries are strategically similar to "wokeness" in that they involve meddling in other people's profit-driven business for moral reasons. But it doesn't get carried along by the new "trust the science" ideology because the empirical signal for serious risks is too weak to show up scientifically, and instead discourse about worries about AI are going to be focused on the dying racism/etc approaches that are too unimportant and too unpopular too win, and which are too different from AI xrisk to be useful even if it does win.
I don't want to defend the coffee argument on its merits, but it feels like a much less stupid claim than "medicine doesn't work" which you took much more seriously for some reason.
“once people were worried about coffee, but now we know coffee is safe. Therefore AI will also be safe.”
I think this is not the argument at all. In fact, what you identify as the conclusion is really a PREMISE of the argument. To me the argument reads as:
1. Worrying about a coffeepocalypse is absurd.
2. People expressed worry about a coffeepocalypse once.
3. The reason was that these people were not honest but pretended to worry about it for an ulterior motive, namely power.
4. This is evidence that people sometimes malign beneficial technologies for ulterior reasons even if this is absurd.
5. Worrying about AI is absurd.
6. So the best explanation is that AI-doomers are dishonest and really driven by ulterior motives, such as power.
I think this is still a bad argument, but this is what it actually says.
This also makes more sense if you go back to Loris's point above. Reading the thread, I tend to agree with that point.
The coffee comparison isn't arguing against AI apocalypse. It's arguing against "AI will take the jobs of writers, AI will be used to create fake news, AI will make fan artists go bankrupt," The idea that AI would destroy the world is barely even on their radar.
"Doomer" to a normie is a metaphor. It rarely means "literally destroy the world". (Notice that the coffeepocalypse doesn't involve destroying the world either.)
This is really interesting to me. The argument intuitively makes sense to me and your "got you" response would seem like arguing in bad faith if I encounted it in the wild. I am also very much not the target group for this website, this is my first time posting, and I'm not sure if I can explain it in a way that would make sense when the original wouldn't. I'll give it a try though!
I think you're fundamentally misunderstanding the argument. You're taking this way too literally. I think translated it means something more like,
"Most people's objections to new things and new technologies are not data based. People have an emotional reaction to new things, simply based on them being new, but this is bad practice, a bias if you will. Not all new things are bad: here is an example. You should judge AI safety on its own merits, not just because it's new."
I think many of the people would have the the implicit argument that they probably think that the other evidence for AI safety doesn't stack up, that there are no heavy arguments for it to use your terminology apart from fear of the unknown. But I don't think that's true for everybody saying this.
Most people's fears about AI are simply just because it's new. Yours are not, so this argument isn't aimed at you. They're basically saying, I think you're only worried about AI because it's new and people are scared of new things. A good faith response would be more like, "I have reasons X and Y to be worried about AI" rather then changing the discussion to be about whether people should be scared of new things or not.
What they’re really saying is pretty simple: “It’s perfectly obvious to me that <coffee,global cooling,…> wasn’t bad despite lots of very-smart-people thinking it was, and I’m saying the same thing about AI. Since my track record on these other matters is so good, you should believe me on AI too.”
An “argument from prior track record” is actually a solid argument – eg it’s the argument for trusting any scientific theory. Unfortunately in this case the track record is being arrived at after the fact and highly selectively.
“There are nearly always predictions of widespread doom as a result of new technologies, even when they’re obviously innocuous like coffee. Given how rarely new technology triggers an apocalypse, we should be skeptical of doomer arguments about AI.”
Obviously this leans on coffee doomsaying as emblematic of something that happens constantly, but it absolutely does, for every major new technology I can think of.
Stuart’s argument about nuclear skepticism can be construed the same way—there are pretty much always scientifically qualified skeptics of new technology, so that an authority figure thinks something won’t work shouldn’t provide a major update, unless you understand the mechanism by which it’s impossible (laws of physics for perpetual motion) yourself.
The claim that there were moral panics that turned out to be nothing is rightly dismissed as an argument against AI concerns.
The claim that we missed some consequences that we should have panicked about before is also true, but tells us nothing about the specifics of AI doom scenarios, which have to be argued separately.
I'll admit I only skimmed the original thread, but it doesn't seem to me like the coffee analogy is an *argument*. That is, he's not saying "AI will be fine *because* coffee was fine." He's just saying "AI will be fine, and after it's fine it will seem as normal as coffee does now."
Like most posts online, it's not meant to convince anyone. It's just an anecdote for the people who already agree to feel good about.
>That is, he's not saying "AI will be fine because coffee was fine." He's just saying "AI will be fine, and after it's fine it will seem as normal as coffee does now."
Yes, I agree with your reading of the thread.
Frankly, depending on what counts as "fine", I'm skeptical that
"became an unremarkable part of daily life" implies "fine"
is even a reasonable view. For decades cigarette smoking was an unremarkable part of daily life in American culture. And the routine estimates of how many Americans it killed per year were on the order of 300,000. Personally, that is outside of what I consider "fine".
I'm not expecting AI to have an analogous effect - there are a variety of possible hazards, but none in that specific direction. But the general idea that the presence of something in daily life is _sufficient_ to conclude that it is harmless is _wrong_. At best, one can conclude that it isn't immediately fatal or rapidly, undeniably and severely damaging.
edit: The
>It's already happening. My step mother was using it to help with her writing and to spit out ideas about ideas for a raffle. My friend's five year old daughter talks to the GPT voice interface in her native language and they create stories together.
looks like it is arguing that AI is becoming part of daily life.
> Frankly, depending on what counts as "fine", I'm skeptical that "became an unremarkable part of daily life" implies "fine" is even a reasonable view.
I think it's reasonable, at least on the level of a personal belief or heuristic. Sure, we can use present information to say that past crowds were wrong about smoking. But if you were making a personal decision about whether to take up smoking, you wouldn't look at the past crowd - you'd look at the present one. And because the present crowd has present information, smoking is much less common. Ultimately, while it's a little helpful to know the crowd *can* be wrong about something, it's just not actionable without some expertise to tell you they *are* wrong.
That's why I think the context of the original thread is important. As an *argument*, "normal = fine" crumples the instant it hits an expert opinion. But as a naive assumption made by some guy in a thread about something else entirely? Not a mortal sin, IMO.
>Ultimately, while it's a little helpful to know the crowd _can_ be wrong about something, it's just not actionable without some expertise to tell you they _are_ wrong.
That's fair. I think we are just emphasizing different parts of the set of outcomes. What you wrote isn't really inconsistent with my
>At best, one can conclude that it isn't immediately fatal or rapidly, undeniably and severely damaging.
How does this sound:
There are _many_ components to daily life. Depending on how one breaks down the choices in daily life, there could be thousands of optional actions. Most of these really are harmless, and have never been linked to adverse effects. At any given time, there is probably an unwarranted panic being promoted on a few of them. But, also at any given time, there are probably one or two reasonably pervasive components that _do_ have significantly adverse effects, as smoking did. E.g. whatever the underlying cause (whether it is sedentary lives, junk food, whatever), the increased incidence of obesity (writing from the usa) is likely to be detrimental to health and is likely to be caused by (possibly a set of) unremarkable part(s) of daily life.
It's because doomsaying is an especially powerful form of persuasion, as it can justify people or society taking actions that would otherwise be completely insane or taboo (in our case, violating individual liberties, massive economic extraction, etc - in the past/other places perhaps it justified eliminating all the idolators, or everybody leaving the island, or rebelling against Rome, etc).
Therefore, those with agendas that are unpopular can exploit doomsaying to advance their (perhaps unrelated) goals, and it makes sense to build a strong epistemic immunity to believing any random doomsayer who wanders along.
Climate change is vulnerable to this. Many sincere people believe that it is a major threat that requires society-level disruption to solve. Many others want a particular *type* of societal disruption, and argue that we must implement their program because it's the only way to prevent climate change. That, in turn, makes it much easier to dismiss the sincere & concerned.
It doesn't help that many doomsayers are straight-up cranks (Mayan calendar/2012?), which makes the immunity relatively easy to develop.
Exactly. In fact, I think it's not only a powerful tool of persuasion, it's among the only 2 remaining political tactics in use in western world since WWII: fear (not doing what I want will have catastrophic consequences for many people including you) and guilt (not doing what I want will keep this poor group suffering). Doomsaying is just a form of the common fear tactic (sometimes with a pinch of guilt - with stuff like the more vulnerable will suffer more. Not for AI (at least I did not see it yet) , but guilt is less powerful and often comes after, when fear did not work)
With the regards to nuclear chain reactions, I feel like its a more valid form of argument - the statement there is "x is impossible because expert said so" which is then disproven. This is basically just restating the appeal to authority fallacy with an example.
However, just because someone has used an appeal to authority fallacy doesnt change whether or not the underlying thing could be impossible. You can not use an appeal to authority to prove whether nuclear chain reactions are possible or whether ai is dangerous. By the same token, showing its an appeal to authority fallacy does not prove either of those statements.
The difference between the statements is:
NCR being impossible were an appeal to authority which was wrong, thus we should ignore all appeals to authority.
Coffee being bad was an appeal to authoirty which was wrong, therefore all appeals to authority are wrong, therefore ai is good because anti-ai arguments have used appeals to authority.
Surely the steelman version goes something like...
* Humans keep doing this thing where they see something new and get hysterical about how [new thing] is going to bring about doom. This even happens when [new thing] is 100% harmless and cannot possibly cause doom, like coffee.
* Therefore, humans clearly have a significant false positive bias when it comes to calculating whether new things will cause doom.
* Therefore, the smart thing to do is have a strong prior against any claims that some new tech will cause doom.
---
Like, more than anything else this just seems to be an argument about the false positive rate. If some new medical test comes back saying you have cancer, but you then find out that the cancer-ometer ALWAYS says 'cancer' no matter what, you're probably going to ignore the result. The coffee argument is the same.
Dr S: We've put 'AI' into the new doom-ometer to see if it's going to cause doom. The results are scary: it reports a very high possibility of doom!
Dr J: Remember when we put a cup of coffee into the doom-ometer, and it reported a very high possibility of doom? I'm starting to think that thing isn't very reliable.
Random hat into the ring: it's a proof-by-contradiction, because they (I) see the standard AI Doomer position as "AI *could* be bad, therefore it *will* be bad". Thus, pointing out all of the other times people freaked out about "could happen==will happen" and then it didn't happen, weakens that type of claim in general. It's very much a pop-debate "dunking on Twitter users" type of argument, but I think that's the core.
If this was a genuine attempt to understand this reasoning, I'm concerned.
Humans have pattern-seeking brains. They also have threat-detecting brains. Therefore new ideas and technology is often met with threat predictions based on hypothetical reasoning built absent concrete empirical evidence. Often this reasoning seems silly in retrospect. Therefore when we see this kind of argument we should be concerned we are falling into a known bias pattern.
This isn't the knockout argument proponents think it is but it also isn't an insane one. I'm not well-integrated into the rationalist community but I've spent a lot of time trying to understand this position of (relative) certainty on AI risk and still every AI risk argument sounds more or less like this to me: https://xkcd.com/748/
Just hypothetical built on hypothetical until the end-result is exciting and world-changing and disastrous. There are lots of reasons I feel this way but one is for sure that "here's a danger scenario I've come up with based on motivated reasoning, I now think it's more likely than it is" is a known failure state for the human mind.
Scott, this is embarassing - how are you missing the argument so badly? Your article is the worst kind of bias argument; you can’t even give a brief Twitter thread an honest reading:
1) coffee is technology
2) coffee was once controversial technology
3) governments in the past tried putting a stop to the technology by going after distributors or declaring the technology itself unsafe
4) businesses feeling threatened by the technology displacing them formed opposition groups
5) the true motivation of these governments and opposition groups was preservation of power.
6) 3-5 are a pattern that plays out repeatedly for any controversial technology, and so the fear/doom around AI is likewise just people trying to preserve power.
The “tl;dr was tl;dr” is: the doom is just a power struggle.
Note: the argument has *nothing* to do with the *actual* level of safety or unsafety of the specific technology. You could plug in “nuclear weapons” for 3-6 and it would still work.
This is why it is *imperative* that AI safety arguers realize the vast, vast, vast gulf of popular opinion to be crossed if you want to not be seen as the new Chicken Little / Boy Who Cried Wolf.
And honestly: “wondering how people could be this stupid…”? You can be better at argument than this. Calling a placeholder for “common layperson opinion” stupid is the worst kind of own-goal reasoning.
> This is why it is *imperative* that AI safety arguers realize the vast, vast, vast gulf of popular opinion to be crossed if you want to not be seen as the new Chicken Little / Boy Who Cried Wolf.
...I'm sure they do realize that, which is why they were so surprised when a lot of laymen actually turned out to be worried about AI risk. Not that it'll matter, of course. Those people aren't the ones calling the shots.
Honestly, why do all these AI risk people care so much about whether humanity kills itself off or not? Do they seriously think that they can save a species that they also think is stupid enough to cause its own extinction?
Whether we can is a much less important question than whether or not we should try. Some of us don't just give up at the first opportunity and quit because it seems too hard. And some of us value humanity much more than you do, if my bad memory of your comment history is anything to go by.
In their minds a wild, low-probability idea vs something that we can see concretely. Obviously the thing we can see concretely has far more weight. It's like how we argue from real life to justify something in a storybook or refute it as absurd.
My hypothesis is that the vast majority of people making these kinds of arguments aren't actually trying to be right--they're trying to be interesting. In fact, I'll go ahead and propose that most conversational assertions aren't primarily meant to convince anyone of anything; they're meant to garner attention to the person making the assertion. In other words, they aren't primarily informational statements at all, they are bids for social status. "Coffee was safe and therefore AI will be too" doesn't mean what it appears to mean, it actually means "I'm an interesting person who says provocative things, pay attention to me." This carries further implications regarding what "winning an argument" actually means to most people. The rise of social media can have only exacerbated this effect.
We're just lucky we live in a universe where being factually correct is sometimes interesting.
The counter to the Coffee Argument is the Tobacco Argument: "people argued that tobacco was bad for you, but society in general decided 'hah, those joyless fools, tobacco is awesome and indeed beneficial' but it turned out the warnings and doom-sayers were right".
So sometimes the worry is not warranted, and sometimes it is. The hard part is telling which time is which.
The link between smoking and cancer is real, sure, but that's not why there was a panic. There are loads of common consumer products with well-known negative health effects, and the usual reaction is a shrug and a "if you decide to use it, that's your problem". The reason people are terrified of tobacco in particular is because of a massive blood-libel campaign claiming that thousands of helpless victims were getting cancer and heart attacks from breathing in other people's smoke second-hand.
(In reality, the data on whether secondhand smoke poses any dangers at all are still inconclusive, and even those studies that have found significant effects find it only for those who are exposed regularly in confined spaces for many years. There's no good evidence supporting the hysterical "NO amount of smoke is safe! Even being around a smoker for a few minutes could KILL YOU!" claims that people heard in PSAs.)
Stochastic risk, like smoking/cancer is quite counterintuitive. The AI discussion is not about a stochastic risk, is it? I don't know shit. Where is N.Taleb when I need him?
Well said! In the USA, the usual estimate for the number of people dying from smoking was around 300,000 per year (at least when I was growing up - the fraction of the population smoking has dropped since then). Yup, something can become an unremarkable part of daily life and still be damned lethal.
I think that supports my argument, actually. It's quite possible for a small group of "influencers" to each argue opposite sides of a debate and both profit, by gaining attention and status with an audience. I've even suspected that many public controversies seem choreographed somehow because neither sides is arguing based on the merits, because the merits of the issue aren't the point.
'Course, that doesn't mean that there isn't a real issue behind it all, or that real world consequences aren't attached to what we do.
Very true. The only thing that mitigate this is to check back those declarations for truth, and stop paying attention (which means getting outed for power figures) when the track record is poor.
Checking past declaration should be easier now... But it's not what happen, because many learned the art of being interesting And vague, and media never check their own track record obviously. Hence COVID amnesia for example....
The coffeepocalypse argument is a cousin of a thought exercise cognitive therapists ask people to do when their conviction that something is going to happen is based mostly on a *feeling.* So for instance very depressed people often do not try going out for a walk or calling a friend because it feels so true that doing either not only will not make them feel better, it will make them feel worse. So you ask them if they can think of any times when something felt very plausible to them, but they later discovered they were wrong, and most people can think of one. The point of the exercise is to help the person move feeling out of the evidence category, where it doesn't belong.
So I think that is the way to AI Coffeepocalypse argument is supposed to work. It's only suitable for people who were genuinely worried about their coffee consumption because the idea was in the air that it was bad for you, and are genuinely worried about AI for the same reason -- the idea of AI doom is in the air. You might think that's a pretty small subset of people, but actually I think that for most people, including me, it is simply impossible to make a decent prediction about how the whole AI thing is going to turn out, and also very hard to just have a wait-and-see attitude, so we fall back on our natural fear of the unknown, plus the AI worry that's in the air, and treat them as evidence..
So while I agree that the coffee argument is not exactly intellectually rigorous, I think it has a point, & the point is to get people to remind themselves that their uneasy feeling about AI is not evidence. So for practical purposes, the argument should be useful to almost everyone who got worried about coffee when there was chatter about its dangers, and is currently worried about AI. I guess we should also exclude the tiny fraction of people who understand AI well enough to have grounds for thinking they have some ability to predict how things will work out. (I'm inclined to think they don't have much more ability to make accurate predictions that anyone else, but they think they do.)
I think part of what motivates people firmly in this camp, like Marc Andressen, is that it seems the current technology has moved in a somewhat different direction than Safety folks in general thought the technology would look like. Correct me if I’m wrong, but didn’t Elizier not see the whole “inscrutable matrices”* thing and especially LLM’s making it this far? So I think the thing folks in his camp aren’t outright saying is “these specific thought patterns don’t seem to be predictive.”
I really like reading Elizier because he lights up parts of my brain that don’t light up elsewhere, and he takes me in directions I wouldn’t go on my own. But I also don’t overall agree with him on a lot of things. A lot of that disagreement comes from experience building complicated stuff (I’m a Product Manager, I know, barf, and I readily cede probably almost everyone here is smarter than me on a lot of different dimensions) and having an experience of just how often reality pushes back at you and says “Oh wait, this is even harder than you originally thought, even after you took into account it was going to be harder than you originally thought.”
I’m sure people like Marc Andressen, who build companies, have these feelings on steroids. Whenever you try to make something do one specific thing,, it’s really hard to get it to settle into that state. And I know the safety argument is that you don’t need a specific state to kill everyone but I just don’t agree.
It’s so hard to get “things” to do “stuff” that I have a hard time accepting that a nascent superintelligence will be able do almost anything on purpose without a lot of effort. Especially when what seems like superintelligence today isn’t agentic,** but is taking the form of these sort of weird Thought Crystals that no one anticipated. I’ve read the Instrumental Convergence arguments, big chunks of the sequences, etc, and while I do think there are *major* dangers probably not too far away from where we are today, I don’t think the shape of them is what a lot of people think they are.
Anyhow, if I can’t steel-man a particular argument someone is advancing, i try to see if there’s a nearby argument I can steel-man and I think this is it.
My own system of building emotive intuition pumps (I spend a lot of time thinking about what it’s like to *be* a data packet in my day job, and I”ve gotten pretty good at doing that to catch weird timing errors, etc, other people don’t catch) tells me we’re probably going to get weird emotionless golem agents at first. They won’t precisely want anything and commands like “figure out what to do and then go and do it” will probably have very limited efficacy while at the same time asking specific questions or giving it requirements and having it fulfill them will get breathtakingly awesome. I think all intelligence looks like “inscrutable matrices” at least in some part, and the next step beyond the one we’re at right now will be building structures on top of those to shift the weights around to pre-set states. And then a bunch of breakthroughs and we’ll get real time updating and that one in particular alarms me but by that point I think our theories of mind are going to get generally better.
*Mechanistic interpretability seems to be making them somewhat more scrutable, so you could say this was another wrong prediction.
**I do think LLM’s have general intelligence, but are not agents in a meaningful sense
"Doing stuff is hard" is the strongest counterargument to the rationalist position on AI in my opinion. There's this bizarre assumption that at some point of effectiveness, it's possible AI will get monotonically easier to improve forever, either because it's so smart or because it gets so good at acquiring resources or whatever.
But nothing monotonically increases forever. Like, by definition. To even make that a possibility in my mind, I'd need strong strong evidence that it was possible. And none has been provided.
I think there are things that are pretty scary that are possible and that we definitely need to be safeguarded against them, but yeah. Doing specific stuff is hard. Like if I wanted to make a machine to kill every person on Earth, I bet that would be really difficult and I would give myself away by doing it. Intelligence gets you a lot. Gets you scary stuff. But I don’t know that it can get you *as* much as it might seem. You not only have to be intelligent, but motivated, etc. Then you have to have specific motivation. Then you have to be good at building stuff. Then you have to build something and no one can figure out or guess what you’re trying to do.
I have a dumb argument, I call the argument from bears.
Most bears will leave you alone if they see you in the wild because they don’t have a history of hunting you specifically. Almost all bears can kill you but most won’t.
Polar bears will kill you on sight because they’ve had to learn how to eat anything edible.
I think my priors are that it is much harder to specifically make a polar bear out of the space of possible bears, but it’s easily possible to make something with dangerous potential.
To a large extent "doing stuff is hard" is self referential. It was difficult to impossible for evolution to make a macroscopically big wheel, because in order to get a wheel you need to grow it, but the wheel itself needs to not be connected to the axel until humans made it easy by just putting a damn stone on another stone, it was difficult to impossible to talk about how to make communications over noisy channels work out until information theory was developed and you can come up with the idea of checksums, it is difficult to impossible to push a feature to a large production codebase, written by people with completely different incentives, who aren't in every single meeting and who aren't good at all the sub skills to shipping a product until.....
If you systemically only think of things that are barely doable on the margin for humans, you have to have a good proof for why it would also be inherently hard for things that are more capable than humans.
I think it's fine to say something like "oh yeah, AI probably won't that much closer to a Carnot Engine wrt to efficiency for combustion engines", but much less fine to say "and therefore, there are no better ways to move".
No one is saying forever. All we're saying is "far enough to pose a risk." Which is really not that far considering the fragility of world economics and geopolitics.
one of the top 50 most annoying things about "AI discourse" is that people love to "argue" by debating which metaphor applies
"AI is like coffee"
"nuh uh AI is like a malicious god"
"nuh uh AI is like cold fusion"
"nuh uh AI is like this one movie"
"nuh uh AI is like this entirely different movie"
i still hold out hope that somebody involved in this wretched pit of a "discourse" will actually work from the facts we already know (AI is algorithms running on computers, we know quite a lot about the capabilities and limitations and resource needs of algorithms running on computers, we know your brain is not an algorithm running on a computer...) but i'm not holding my breath
"But then how does any amount of evidence prove an argument?"
The only thing that *proves* an argument about some future event is actually observing that event. Russia can have a 3:1 manpower advantage, better generals, better tanks, and every expert in the world could predict that they'll win easily, but none of that is proof. The proof is in the pudding, as they say.
And I think that might be the root cause of your frustration about AI arguments. No one has made the super-intelligent AI pudding yet. I think that where people come down on the debate about AI risk largely reflects their own temperament and deeply ingrained biases, and both sides have people that are making bad arguments to confirm their biases.
My advice is stop trying to understand all these arguments as well thought-out and rational attempts to discover the Truth, and start trying to understand them as flawed attempts at persuasion by appealing to emotion and common biases, very much like political debates.
The average person doesn't know anything about AI and is not in a position to evaluate the data (I certainly am not), so they instead have to decide "do I trust Set of Experts A, who tell me 'AI is coming and it could be a threat', Set of Experts B, who tell me 'AI is coming and it will turn the earth into a paradise' or Set of Experts C, who tell me 'AI is impossible and will never happen so we won't be living in either the paradise or the hell future'?"
Hence why the duelling arguments over "predictions of doom in the past which didn't come true" and "predictions about this tech is impossible which were shown to be wrong", to try and convince Average Joe and Josephine.
I think you should go with Set of experts D, who say, "so-called AI is coming and it's going to make some jobs obsolete while creating some new ones, but overall things will keep going as they always were, which means that our lives will get a little more comfortable on average in the long term". Admittedly, that's a less sexy prediction than any of the others...
It seems like the basic problem here is trying to find a general heuristic for evaluating some class of arguments without digging in and thinking them through. That's a necessary part of life--there are absolutely people who will claim that *everything* is an existential threat to humanity, if they think it will win them their argument. But the best it can do is give you a very rough guide.
Over the years, I've probably heard a million different claims about things causing cancer--chocolate, or red meat, or saccharine, or Red Dye #2, or whatever. Most of these were probably nonsense, driven by p-value fishing and a media file-drawer effect. And yet, smoking and breathing in asbestos fibers and getting sunburned a lot all actually *do* cause cancer. You can and should dismiss most such claims with a heuristic (media sources always report on crap like this and it turns out to be one n=14 study in mice involving 1000x the dose of the carcinogen humans are ever exposed to), but you also need another heuristic to decide whether to take the claim seriously enough to dig in a bit.
Once you're to the point of writing a think piece on why your quick heuristic for dismissing a claim of this kind makes sense, it seems like you've kind-of gotten yourself wrapped around the axle. The heuristic is useful for keeping you from changing your diet or spending hours digging into evidence every time the local TV news reports on that n=14 study in mice, but it isn't useful for actually evaluating a claim that something causes cancer. That can only be done by digging in and thinking hard and evaluating evidence.
Another way of thinking about these heuristics is as a kind of base rate. Prominent claims that X is going to lead to the extinction of mankind don't have a great track record, so we should start out skeptical. But then you need to dig into the available evidence. (Also, there's an anthropic principle thing going on there that makes claims that something will wipe out mankind a little funny to evaluate--everyone who ever evaluates such a claim will find that it never happens, even in a universe where {hostile AI, an engineered plague, an asteroid impact, an ecological catastrophe} does eventually drive mankind into extinction.
Of course, the problem wrt AI risk is that the evidence is hard to understand and interpret. We are speculating about capabilities of computers that we don't know they will ever have, but that they might eventually have that would be really dangerous for humans.
There's a parallel argument about AI risk that says more-or-less "we thought AI would never do X, but now it's doing X better than humans, therefore people who argue that AI will never turn us all into paperclips are also wrong." As a heuristic for dismissing arguments, it's not so helpful here. But maybe it can inform our base rate or priors.
My sense, as someone outside the AI field, is that it's hard to make convincing arguments that AI will never be able to do X (for any X), but that doesn't mean that AI will eventually do every X we can think of.
The argument you agree with, but question if it may be fallacious:
"People were wrong when they said nuclear reactions were impossible. Therefore, they might also be wrong when they say superintelligent AI is possible."
*is* a form broadly misused/abused - for pseudoscience or outright supernatural, e.g. Electric Universe, Velikovsky, astrology, ancient aliens, God.
"Is this form of argument valid?" isn't the right question here, as it most argument forms can be used for the invalid. It boils down the independent question of whether AGI/ASI in particular should be considered in the same category as pseudoscience, and clearly here we'd all point to other evidence to show that it's different.
To me, it sounds like a informal "Bayesian" argument. The author of the tweet is presented with a conundrum of the form:
- There is a subset of clever people that, applying a stylized model, have come out with a potentially catastrophic outcome that would materialize in the middle or long-run.
Then, he wants to estimate how often a group of clever people that came out in the past with a catastrophic scenario have been correct. Of course, such base ratio is pretty difficult to compute unless you can survey all intelectual discussions in recorded history (without significant bias), and then determine what is a "catastrophic prediction" and which ones actually came true . Thus, he take a small, quasi random, sample of N=1 catastrophic predictions and since that one didn't came true, he concludes that "clever people predicting terrible things" are usually not right.
I think the minor point, that clever people using simplified models of the World are usually wrong in their long-term predictions, is broadly correct and I could back it up with a large N. Whether this says anything about AI risk, I'm not so sure. At last, whether AI risk is real or not isn't a random event. But here, many of you defend "non-frequentist" probabilities and use base-rates in your reasonings that are similar to what the author of the tweet is doing. As an example: suppose you have a neuroscience study. You can not properly evaluate it and as such you don't know whether the result would replicate. After a quick googling, you find a survey of N=300 papers in the area, not including the one you are interested in, that says that only 40% of them replicate. Would it be so incredibly wrong to estimate the probability of the study you're interested in replicating as 40%. What about N=299 papers? Lower the N enough and you get to the Dan Jeffries twit.
I understand and sympathize with your frustration, but this is just another example of a common conundrum many people here face. Once you have some basic understanding of epistemology, most people seem like drooling idiots.
So there's a lot of people outside the rationalist community who don't think "x-risk" when they think about AI, and if you mentioned it to them they'd probably say it was some kind of techbro delusion. Even so, they still hate current AI technology. I've seen people react with visceral disgust at anything that seems AI-generated or uses AI—they're not *worried* about it, they're *offended* by it.
The coffeepocalypse argument thread opens specifically with reference to *backlash*, suggesting it's not really addressing risk but talking more about those kinds of reactions. This is given explicitly a couple of times: "Both forces fought back against the society-shifting power of coffee just as we are seeing media companies battle AI companies today for the societal shifting power of AI." "This pattern plays out again and again and again in history. New technology. Resistance. Resistance overcome."
(Of course this doesn't invalidate your general argument about using isolated incidents as proof.)
The story got mixed up. It was Szilard conceiving of the idea of self-sustaining nuclear chain reactions the day he read of Rutherford mocking the idea.
I don't think we can use COVID-19 predictions as a useful metric to judge intellectuals because we still can't agree on what happened. Its not like there is universal agreement whether or not Ivermectin works, the vaccines were safe & effective, the lockdowns were warranted, or the virus leaked from a lab.
4. All combined NPIs (masking, social distancing, quarantine efforts) were not effective at getting R to < 1
5. Age was a very strong predictor of COVID morbidity and mortality
6. Age adjusted PFR, IFC, and CFR are now known.
Its pretty easy to judge authors on most of these by casual observation. If an author supported school closure they probably had wrong ideas about 3, 4, and 5. If they were sharing their door dash sanitation protocol they were likely wrong about 1 and 4. If they were saying that vaccine roll out would end the pandemic by mid 2021 (like I was), they were likely wrong about 2, etc. If they were saying, "COVID is a government conspiracy", or "everyone should panic non-stop for the next 2 years" they were likely were wrong about 6.
I’m inclined to agree that there was little or no expertise. it was seat of the pants stuff - the lockdowns were not something that the WHO recommended prior to the epidemic, in fact they warned against it. It’s been forgotten now but masks were actually derided as ineffective for a few months, during which time old websites recommending masks against viruses were deleted, then that changed and people who refused to wear surgical masks were the epitome of evil.
In some countries surgical masks were derided as useless, like in Germany, while they continue to be recommended in the U.K.
Antigen tests were declared useless until they were recommended.
PCRs were the gold standard until that changed and they were considered to amplify the virus and therefore were useless.
Washing hands, even wearing gloves, was recommended at the start but ignored at the end.
Two weeks to flatten the curve was clearly nonsense in retrospect, but also you have to ask why only epidemiologist would’ve assumed that it would have worked – surely the epidemic was going to spread again once we left lockdown.
New Zealand wasn’t considered bat shit crazy, but only mildly crazy for continuing to believe in zero Covid for more than a year, or two.
The vaccines were definitely going to give us herd immunity, until it was clear that wasn’t gonna work so people started to say that it was never even claimed to begin with.
Luckily Omicron comes along and the severity declined. And luckily no further more dangerous mutation has occurred. (Although I suppose we are all somewhat immune now anyway. )
I think something completely different is happening:
1. "I'm trying to predict something in advance."
2. "Here's an example of a failed/successful prediction.
3. Given we now retrospectively know the results of this other prediction, it should be possible to have confidence in this new prediction prospectively."
The logical error, which is extremely common, comes with the assumption that a reasonable person could have had the same prospective clarity as the retrospective observer. We should know these are qualitatively different states, but we mix them up all the time.
In the same way it was impossible for someone to have confidence in coffee, given the prospective uncertainty, it would have been impossible for someone to have confidence in nuclear chain reaction. Nobody KNEW... until it happened and suddenly everyone knew! It's difficult to wrap your brain around this concept without practice, because once you know a thing it is hard to unknow it to the point where you can understand the person who is still struggling under uncertainty. Very easy to spot in someone else.
If the target audience (or even the poster) doesn't have a grasp on the specifics of the AI, the actual argument for why or why-not AI-doom is simply "do I think these people is right?"
Many people, especially smart people, will see other smart people supporting AI-doom, but rather than engaging with the logic, they are trying to evaluate the people (trust-the-science types maybe).
If your evaluation of AI-doom is "are these people usually right?" then it's quite reassuring to see that smart people have been wrong about doomsday scenarios in the past. It doesn't have anything to do with the underlying logic of the doomsday scenario.
Perhaps taking them as “arguments” instead of rationalizations and virtue signaling is the problem? With a social expectation of “having an opinion” in our contemporary dystopia, most people (who in my observation rarely think deeply about anything) reach for familiar justifications that resonate with them. Perhaps it seems more erudite to them than simply saying “I’m with #team-utter-bullshit not #team-absolute-tripe”? 🤷♂️ At least in the USA, we’re accustomed to bipolar arguments defined by the extremes with demonization and dismissal of the opposite as the standard modus operandi.
Tweet #5s "What they were and are really arguing about is power." is an important and true point. Like Chapman belabors in Better Without AI, we were more "intelligent" than chimpanzees for a long time without being appreciably more powerful than them. Our power came from building cultural, colonial knowledge stores, which is often what we mean by "technology". (It's not just a matter of technology, but also maintaining the institutions that can reliably translate those ideas into changes in the physical world.)
In the case of coffee, this new change in the world ended up having a democratizing effect; more cognition available to the average Joe without specific dispensation from kings, politicians, and alcohol manufacturers. I think Prof. Russell is trying to take an accelerationist, technophilic approach that basically all new technology that serves as cognitive prosthesis is democratizing in this same way. I personally don't agree with that take, but I can see how this is an attempted pattern match on a deeper level than "here's a wrong prediction, so there will be others."
So I agree that we should look at power, not intelligence, because intelligence does not trivially map to power and the correlation is weaker at the high end, not stronger. I also think that coffee has obviously different affordances than AI and so their relationships to power are dramatically different. Coffee just serves to stimulate you out of the low end, giving you a bit of intelligence that gives you a bit of power on the low end where they still correlate. AI is a case that can pollute or take control of our collective knowledge stores, and is reaching much higher levels of intelligence, so the relationship to power is much more idiosyncratic and needs to be evaluated on it's own merit. But I do think it is coherently arguing "better thinking technologies uplift us all" (even if I disagree with that argument) and does not reduce to just "here's one failed prediction, so this other prediction must fail".
The coffee/AI thing sounds like an appeal to absurdity. AI danger sounds absurd to them, so they compare it to something they think you'll agree is absurd, but this doesn't work well because the underlying reasons are so different.
If I were generous, I think the argument intended is actually a clumsily communicated version of "the likelihood is low, based on the recurring trend of alarmism over new tech/stuff typically being overblown". Yes that is not an adequate means to evaluate risk which ought to be case-by-case, but that is all the layman has to rely on here (which is nearly all of us, because we aren't flies on the wall privy to all the right information). Call it agnostic and brushing off the question with an affirmation that "everything will be ok", as opposed to a serious evaluation. Even sifting through all the data we have, they're not going to make a confident assessment themselves. I know I can't.
When all you have is spurious speculation on incomplete information, the temptation is to bias to not panicking. These events aren't *exactly* completely out of the public's control qua pressure, but they might as well be if there is so little confidence about danger any which way.
> Whenever I wonder how anyone can be so stupid, I start by asking if I myself am exactly this stupid in some other situation.
Here is a candidate scenario: try writing a piece on the absolute risk of democracy *as it is* in the US, as compared to alternative forms of government, including the one we have except transparent and truthful (behaving as it is claimed to behave), as well as other forms of government that have not been tried.
Unfortunately, there is a "do you have the balls to write it" aspect to this challenge.
Nit-pick on one of your examples: it's actually not hard to counter a 3:1 troop advantage, as long as you're on the defense. (Well, it is hard, but only in the sense that everything about war is hard and awful.) Defending troops are expected to lose about 1/3 as many as attacking troops. Casualty ratios in modern war have much more to do with who is on the offensive than other things you'd think would matter more.
I believe it's a case of the boy-who-cried-wolf phenomenon, combined with a failure to recognize the significance of fat tails--a dangerous combination.
(1) I don't think people are anti-AI safety, they're anti-'the sky is falling, the sky is falling!' warnings about "if we don't solve AI safety, we're all going to be turned into paperclips"
(2) This is because it has not yet been established that the Paperclip Maximiser AI is possible. There are a *ton* of Underpants Gnomes arguments about "AI is coming and it'll get out and take over the world and that's why we have to be sure we align it to human values*!", at least to my admittedly jaundiced view.
(3) Hence why one side relies on "all these previous examples of dire warnings never came true, therefore I am not going to worry about AI Doom" and the other side relies on "all these dismissals of new technology were wrong, and thus the dismissal of agentic AI is wrong".
(4) As I see the current implementation of AI, which is "replace human customer service with chatbots", "use AI to create clickbait articles" and "sign up for our seminar on how to implement AI in your business to sell people even more crap", I find it harder and harder to reconcile this with the fears/hopes that "once we crack AI, it will recursively self-improve to be super-duper god tier intelligence level and take over the world since it will be so much smarter than us mere humans, it will solve all the problems we can't solve due to not understanding the deep laws of the universe and will turn the world into utopia/dystopia".
*These just ever so coincidentally turn out to be the exact same as "21st century liberal San Franciscan values"; nobody ever seems to think we need to align AI to spread the Divine Mercy Devotion, for example.
Yeah, in another comment I tried to explain that people are probably evaluating AI doomerism based on it's similarity to other forms of doomerism that have/haven't panned out in the past. Since everybody has some sort of heuristic for this, we should interpret the coffeepocalypse argument as saying "AI doomerism looks more like [goofy doomerism] than [plausible doomerism] ".
I feel like you've essentially elaborated a bit on what the internals of the doomerism differential heuristic might look like.
My guess about the psychology here is that people who make this sort of argument see the world as a place where things *just tend to work out*. The course of history follows a trend of general improvement; we have (self-evidently) not yet destroyed ourselves, despite people occasionally worrying about that; therefore they see concerns about the possible reversal of this trend - the possibility of a future being catastrophically bad - as disproved by the whole general history of civilization. The coffee thing is just a synecdoche for that whole history. And it's a good example for their purposes because it sounds particularly absurd (people thought *coffee* would lead to disaster???), and for them the whole idea of worrying about catastrophe is absurd.
From this perspective catastrophe is impossible. The psychological appeal of this attitude is pretty obvious, I think.
In my admittedly poor experience, people seldom debate; they just fight and play signalling games to make alliances and minimize the costs they must incur fighting. Therefore, blatant fallacies are a telltale that they’re communicating at the upper simulacrum levels (<https://www.lesswrong.com/tag/simulacrum-levels>). It can be translated as, “Just shut up and support my side, or else”. By trying to debate, you’re challenging them, and they respond accordingly.
Of course, given how much you’ve written about all sorts of rationality topics, and how much I’ve learned from you, I’m pretty sure you’re well aware of these facts, so there’s probably some signalling game going on here, too, that I fail to perceive as I clumsily reply to the object-level question.
I think even if you admit this is a signalling game, signalling games are still *based on something*. For example, the thread author isn't typing out a random string of characters in support of their argument, and presumptively is self deceived in some way which makes them think it was a valid argument (normal people by and large still think they are being sensical even when they made almost exclusively signalling). I think it's bad to replace "I don't know what they're doing" with the word "status" and not acknowledge that you haven't really explained a lot ( you have explained some!).
I think to steelman these arguments you need to interpret them as counterarguments to arguments from authority. "Yes, there are high status people who are worried about AI X-risk, but high status people are not always right, see coffee and nuclear fission."
Most AI X-risk arguments are not produced as arguments from authority and they offer pretty detailed substantive arguments, so you might question why anyone would make this counterargument.. The reason for that may be that even a very elaborate substantive argument is perceived as an argument from authority if one is too bored to read further than the title and author's credentials. So it is not inconceivable that the the vast majority of consumed AI X-risk arguments are consumed as arguments from authority.
Why does the coffee argument work and the halibut argument doesn't?
1. Most of us don't put halibut in our bodies on a regular basis.
2. Coffee obviously contains a drug that impacts both mood and metabolism. Where there are obvious impacts, there may be hidden impacts. To summarize, coffee is an active substance in a way that halibut is not.
So, coffee is an analog for a powerful and possibly covert substance, while halibut is an analog for an inert substance.
The only exception is when men get slapped with a fish on Monty Python.
"And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS)"
WWII is definitely fitting given the very explicit arguments that another war would emerge without the League of Nations or some other coordinated peace effort. I feel less comfortable with the other two as analogies but I don't know the full history leading up to them. I don't think it's possible to steelman the original argument especially well, but I think that you're missing a distinction between mass death events that are unsurprising causal agents which are unsurprising (e.g. disease, war) and things that are not (e.g. global cooling, AI).
It's basically a calibration/bias argument. The same argument as one we regularly deploy against claims about some new miracle cure or religious miracle -- we have good reason to believe there is a bias here and you should update moreon the possibility you are being affected by it.
Basically, the argument has the form:
1) We've seen that people seem to be particularly inclined to *think* that new technology poses a grave risk (or is all good but I'll just describe this side).
2) However, in fact people who make these arguments have repeatedly and regularly been wrong. Maybe not universally wrong but wrong far more than their assignments of probability should allow.
3) You are a person and we have every reason to believe all the considerations that caused those past judgements to be overconfident/poorly calibrated affect you so therefore you should do the same thing you would do if you found out that your bets on sports systematically assigned your favorite team an unduly high chance of victory and adjust downward.
--
And I think this argument has a great deal of force. I mean why is calibration from the past predictions you made different in kind from that made by other people.
Yes, maybe you are more different from those people than you are from your past self so you should be somewhat less confident the adjustment is beneficial but still somewhat (and unless you have specific reason to believe you are less likely to be affected lower confidence shouldn't affect the magnitude or direction of the best adjustment)z
Good reasoning. One question. You wrote: "People didn’t worry enough about global warming, OxyContin, al-Qaeda, growing international tension in the pre-WWI European system, etc, until after those things had already gotten out of control and hurt lots of people."
How has global warming hurt lots of people? I see how it could in the future. What is the evidence that it has already done so?
"Think AI Doomers are absurd? Well they have more of a chance of winning than you think, because of historically identifiable neurosis. So you should really care about defeating them."
As with most communication, it's necessary to give it the aesthetics of refutation.
> physicist Ernest Rutherford declared nuclear chain reactions impossible less than twenty-four hours before Fermi successfully achieved a nuclear chain reaction.
It's not a particularly useful evidence on its own, but it's an effective rebuttal for "Here is a domain expert who thinks that artificial super intelligence is nothing to worry about, therefore you all should chill".
So other comments are already saying this, but I might as well throw in my 2 cents corroborating it: this Twitter thread doesn't read to me as about AI safety in terms of x-risk at all.
The point in your footnote seems relevant to me. The argument, such as I understand it, isn't that the French monarchy were wrong to be worried about revolutionaries. It's that the French monarchs were terrible and the coffee-fueled revolutionaries were right to be revolutionaries. The monarchy = AI safety, stuffy walled-garden elites who just want to preserve their power; OpenAI (or whoever) = the coffee producers fueling the revolution that will make a better world for all heedless of whether or not it supports the dominant elite. In other words, it's e/acc.
This argument makes no sense if it's applied to the Eliezer Yudkowsky form of AI safety, because a proper agentic superintelligence isn't like a coffee-fuelled revolutionary at all. But it makes a lot of sense applied to the forms of AI safety I am more likely to see farther away from that, which is much less about extinction and much more about unemployment. In particular I'm thinking of visual artists who *HATE* AI art with a furious, burning passion. This is a group I interact with every day, and their AI safety opinions have nothing to do with LessWrong or the associated diaspora. They're not afraid of the AI itself, they're afraid of other people using the AI to take their jobs. The coffeehouse analogy seems appropriate here, even if you don't agree with the conclusion: control of visual arts right now really is a kind of walled garden, and AI tools really do offer outsiders a way to be productive outside that system of control.
From the thread: "After a technology triumphs, the people who come later don't even realize the battle happened. To them it's not even technology anymore. It's a part of their life, just another thing. They just use it." This is cold comfort if you visualize the post-singularity world as a mass of undifferentiated gray goo/cloud of sexless hydrogen. I want my species to stay alive and it doesn't help knowing that the gray goo considers its life "normal"! But Daniel Jeffries clearly isn't visualizing this, considering his post-singularity future has human kids in it at all. He's visualizing a world which is much like ours, only with a powerful new technology (safely under the control of humans) and a new historical footnote about the failed attempt to prevent its spread.
What your response to the thread really makes me think is that AI x-risk cannot get away with de-emphasizing the "existential" aspect of it. If I were responding to this Twitter thread, I'd be saying: no, dude, this form of coffee isn't being used by other humans with some different ideas about who should run our institutions, it's being used by an *unfathomable greater-than-human intelligence that nobody understands or knows how to influence*. It's not a concern about what our grandkids will think, but whether they will have a chance to ever think anything at all.
To put the point differently, suppose you, Scott Alexander, had predicted 50 times in the past that Bitcoin prices would plummet (somehow specified) in the next month with 90% confidence. In fact, in only 10% of those cases did you turn out to be correct.
Surely, the next time you make the same prediction it's reasonable for me respond: but every time you make this prediction it only turns out to be correct 10% of the time. Therefore you should adjust your confidence based on the observed data down to 10%. It wouldn't be very convincing for you to reply: how is the fact I made an incorrect estimate about these other things (what Bitcoin will do in 8/22, 9/22, ...) relevant to whether my estimate about this different thing (what Bitcoin will do in 5/24).
So how is this situation different than that example?
Well the predictions are about less similar things than in this example. Fine, but they are still all about new technology causing problems so there is a common description that one could reasonably appeal to (just have to think that these judgements are likely to be similarly affected ... reasonable prior)
You didn't personally make them, some other person did? Again, this seems only a matter of degree. You might be more different from those ppl than your past self but it doesn't destroy the argument only makes it slightly weaker.
(year 2500, humans are long extinct, the AI society is conflicted on a new potential risk) see, in the early 2000 there was a panic about AGI, but it turned out great!
Most people are using the totally wrong category of reference classes to think about this all together.
This article and the comments around it are sort of emblematic of this discussion being "when was the last time you made a screwdriver that ended the world? No? Okay stop worrying."
People broadly consider the A.I. to be an inanimate thing, a simple machine, thats just particularly clever or powerful, like a computer or atomic bomb.
The real intuition pumping question should be "Do smarter agents conquer dumber ones?". And here the reference class is chock full of affirmative examples. Homosapiens sapiens, smartest of the hominids, more or less extincted the neanderthals and other at the time extant relatives. Humans, smartest of the animals, have conquered the entire world and quite definitively control the lives of the chimpanzees orangutans and bonobos who differ in DNA by only a few percent. Advanced human cultures assimilate or eradicate less advanced ones. In a thought experiment, if we took the 10% most intelligent humans and gave them one half of the planet, and the 10% least intelligent and gave them the other half, and eradicated the middle 80% of humans, the smarter group would fully and definitively control the fate of the planet, the human race, and that of the dumber group. There's a popular shared intuition that if an alien race shows up and it's smarter than us, we should be scared (and we should be).
A.I. isn't a fancy screwdriver, it's an Alien on the horizon and we aren't sure if they are smarter than us or not. That drives the risk intuition.
The argument then has to become "is it even possible to construct an artificial agent that is smarter than humans". That may be debatable, but the heuristic not to worry about the coffee or the video games or the slate tablet doesn't apply.
We may feel like the most superior organism of the planet, but we are a *very* long way from a decisive victory. Could we wipe out all of the other life on the planet if we put our minds to it? Maybe. Could we do it without killing ourselves in the process? Probably not yet.
We don't have evidence of a vastly more intelligent agent completely wiping out all less intelligent agents. They may win a specific battle that is important, but historically they don't wipe out all competition.
I feel like this makes the opposite point. If we decide for some reason biomass means anything at all, the human ratio of "control of the planet" to "total biomass" makes the outsized affect of our intelligence even more poignant.
Please note that the question is not "totally eradicate" it's "conquer or control" . We haven't totally eradicated the bonobos, but the bonobos are not the ones in control of their lives and we would never let them out of their zoos if they'd do things we don't want.
We may not instantly be able to use our intelligence to wipe out all mosquitos say, but we are using our intelligence slowly and effectively to control them in a way they are totally incapable of responding to.
The historical record is full of the smarter agents coming out on top, shaping the world in their image, outcompeting and controlling their competitors.
As the agents get more and more capable, the stakes get higher and the space for coexistence smaller.
Even if the A.I. doesn't wipe us out, I don't want it to sequester humanity in a zoo or flatten our natural habitat to make room for factories, and these are absolutely things we have tons of precedent for smarter agents doing to dumber ones.
I think your view of our superior position (control of the planet) does not align with mine.
Every day I fight plants, animals, bacteria, viruses, bugs, etc. here in the tropics. I am vastly more intelligent and capable than them, and I have the whole of human knowledge at my disposal. Yet, I still have an ant problem, the jungle still keeps growing where I don't want it, I still suffer from bacterial/viral infections (and there is a high probability that I will one day die from one), and the local lizard population keeps pooping everywhere it wants. I utilize extreme tactics like the ant killer stuff that tricks the ants into taking it back to their hive as food and then it kills the whole colony, but this has just lead to a strain of ant dominating the local area that is immune to the lures of this kind of bait.
In any individual battle I win quite soundly. No ant has yet to defeat my thumb and even though a battle with a viral infection may take me out for weeks I kill a billion of them and survive to tell the tale. You could make the argument that I'm on top here, but I think there is a reasonable argument to be made that I'm on top only if you narrowly define the terms of "top".
Humans certainly *have* wiped out entire species before, but there are plenty of species that we have tried and failed to eradicate and there are billions of species that we don't even bother trying to eradicate because we know it is going to be hard.
In another thread somewhere in these comments I think I brought up the concept that humans have dominated their evolutionary niche, which is on the 1 meter scale, and I think you and I agree on this. For other things on the 1m scale (e.g., Bonobos) we are pretty much in control if we want to be. However, there is a whole lot of stuff in the world/universe that doesn't exist within our evolutionary niche like ants, bacteria, plants, etc. and I don't think that we dominate/control those in a meaningful way.
It is entirely possible that AI will not share a "niche" with us. Maybe they dominate the digital space and "allow us" to sometimes inhabit it (like a zoo). I'm not convinced that that we will necessarily share a meaningful space with future AI such that there is significant resource competition between us.
I really appreciate this comment. The whole AI is to humanity as Homo Sapiens was to prehistoric humans analogy is flawed. Modern and prehistoric humans were relatively very similar and occupied the same ecological niche, competing for the same limited pool of resources. AI minds operate nothing like a human brain, so far as current AIs can even be said to possess a mind. The idea that they will eliminate humanity because they want... our food? Mates? Territory? It's silly. You can make better analogies than this.
They don't want anything of ours, but they like doing computations to accomplish whatever they want. It is considered ancient Chinese wisdom that if there are 10 suns' worth of heat aimed at you, you are in for a bad time. Please listen to this ancient and esoteric piece of knowledge.
If humans do decide to do something about that before it happens, any AI capable of maintaining long term persistent effects would be able to backchain from that fact and decide to do something at a more opportune time, in the same way that Chess Grandmasters can see your attempts to checkmate them and just not do it(tm).
I think the intuition that is not shared is that actions taken that do not explicitly say "hurt humans", can still end up hurting humans, just like how actions that do not say "make rainforest animals extinct" can still result in subsistence slash and burn farming destroying habitats.
If that's what people mean they should say so. Multiple commenters on this very article used the Neanderthal comparison, which I don't think can be interpreted in a useful way. If humanity is supposed to be the metaphorical wildlife choking on a plastic bottle ring, use that comparison. It conveys the risk of an alien intelligence that doesn't care about our wellbeing much more effectively.
Plenty of smarter animals coexist with stupider animals. It only matters if they are competing for an evolutionary niche. Which was true of Homo sapiens sapiens and other humans.
> it's an Alien on the horizon and we aren't sure if they are smarter than us or not.
That remains unproven, LLMs are already more knowledgeable than all humans, more articulate than most, smarter than nearly all - and are in no sense conscious.
We have every reason to expect we would be competing for the same resources as an agentic A.I.
LLMs are not agentic so I agree they aren't dangerous in this way and not really what were talking about as risky.
It's an agentic A.I. that you have to worry about (hence "on the horizon")
If you want to argue agentic A.I.s smarter than humans aren't possible, fine, but show your work because it would sure be weird if by chance evolution just happe ed to make humans as smart as was physically possible
> We have every reason to expect we would be competing for the same resources as an agentic A.I.
What resources are you imagining competition for here? Are we also assuming that a super-intelligent AI is unable to reach space to access resources there (which are many orders of magnitude greater than the resources available on earth)? Perhaps the theory is that the AI will become a K2 civilization where planets don't matter but the AI consumes all our sun of so we are left on a cold rock in space?
I think the specifics matter a lot here, because it changes the counter-argument significantly. For some examples:
* If the AI is just a bit smarter than us, but still stuck on this rock then it means it isn't really that "super" intelligent. It is just perhaps smart, and that is not necessarily enough to win.
* If it is smart enough to get off the planet, then we aren't competing for resources meaningfully until it reaches K2 civilization. Then we are competing with it for energy from the sun, but this assumes that it sticks around long enough to fabricate a dyson sphere rather than going out into the broader galaxy to explore more interesting things. Also, the timeframe required to build a dyson sphere even with nano-bots is still pretty long, so there isn't a whole lot of immediacy to that risk.
* If it becomes a K3 civilization, it probably doesn't care about our tiny little star and rock. They are insignificant compared to the energy of the galactic core and we can (by then, millions/billions of years in the future) perhaps just fly our little rock away and it won't bother us.
* If it is a paperclip maximizer, there is a lot of low hanging fruit (trillions of years worth) before it makes sense to deal with the rock that resists. Better off not wasting resources on war unnecessarily when you can just go farm the entire universe of lifeless rocks first. Deal with sentience last.
The above are just some simplistic counterexamples meant to illustrate that the specifics of resource contention matters a lot.
> If you want to argue agentic A.I.s smarter than humans aren't possible, fine, but show your work because it would sure be weird if by chance evolution just happe ed to make humans as smart as was physically possible
I did say they were already smarter in some ways. Micah answered the question about resource competition better than I could.
Scott, have you tried literally asking some of the coffeepocalypse people about this? That is, reaching out, pointing out why the argument seems extremely weak to you (and me), and asking why they find it compelling?
No thoughts on the coffee analogy, but I do have a minor point to make about the bad things that did come true - both the forecasted and the unforecasted:
> And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS).
> People didn’t worry enough about tobacco, and then it killed lots of people. People didn’t worry enough about lead in gasoline, and then it poisoned lots of children. People didn’t worry enough about global warming, OxyContin, al-Qaeda, growing international tension in the pre-WWI European system, etc, until after those things had already gotten out of control and hurt lots of people.
The Black Plague and AIDS are like past plagues. WWII is like past wars. Tobacco and lead are like past unintentional poisons. OxyContin abuse is like past opiate problems. al-Qaeda is like past terrorists. Pre-WWI tensions are like past pre-war tensions.
The only exceptional example is (human-influenced) global warming. Even there, only the cause is exceptional; a changing climate is a challenge humans have faced repeatedly.
I think this is part of why it's hard to shift low priors on existential risk from AI.
The important feature of the coffeepocalypse discussion is not the argument. There doesn't need to be an argument (in the philosophical sense) for it to do its work. The argument is orthogonal to the work.
(TLDR: Scott is working at level 1; coffeepocalypse is at level 4.)
I'm about to say a bunch of unprovable stuff. To know this stuff, you have to have done some emotional work to discover it for yourself. It may seem wrong to many people. That's fine. Maybe it'll be a guidepost to somebody, of the kind that I needed once up on a time, but didn't find.
Much of politics operates via taboo. Arguments reproduce mimetically, during which political actors attempt to fasten various taboos to them. Who is allowed to make such-and-such argument? What kind of person is allowed to draw which conclusions? What evidence is it permissible to consider in connection to some particular issue? These are questions that, philosophically speaking, are meaningless failure modes called ad hominem. In politics, however, the involvement of taboo with an argument is a powerful force.
I'll give one example, from far afield. It's not a great example, because it's a weak taboo -- but I can't give a better one. (It's taboo to call a strong taboo a taboo -- strong taboos are "just being a decent person" -- so I would be violating taboo by mentioning the strongest, most important ones here.) Here's the example: it's often asserted that men shouldn't opine on the issue of abortion. Philosophically, this is an ad hominem that any freshman could identify. (The gender of the person making an argument doesn't make the argument better or worse.) Politically, however, it's very relevant. Whether, where and how much this taboo is considered to be in effect is a political issue that people fight over.
If you want to see taboo in action in politics, look at tweets starting with the word "Imagine". These are almost all accusations of taboo-breaking. A tweet like that has a bunch of functions: it informs the reader of the existence of that taboo; it shames the target; it informs bystanders that they, too, can shame their neighbors for breaking the taboo. If the tweet falls flat (say, if the person tweeting gets ridiculed by lots and lots of people), it tells bystanders that this taboo is no longer in operation. Most of the time when taboos are implemented, it's much, much, much more subtle than these "Imagine" tweets. It happens in omissions, minor rudenesses, who gets left out of an event, in the structure of jokes, etc.
(Many taboos are good! I am not against taboos. We need them. Also, we should understand how they work.)
One of the most common (and subtle) ways taboo is used politically is to limit which ideas can be brought to bear on an issue. Because it takes work to search around and find all the relevant concepts involved in considering an issue, it's easy to taboo people into not doing that work.
Similarly, one of the features of moral panics is that the people promoting the panic use their social power to enforce a taboo against referring to the idea of moral panics in connection with their issue. When there's a real moral panic going on, you cannot call it a moral panic without consequences. If you're American, you can definitely think back and find examples of this. I'm not willing to break taboo publicly, though, so that's work you'll have to do yourself.
So, the coffeepocalypse discussion serves a bunch of purposes, just by publicly connecting the idea of moral panic to AI risk. We can gauge the strength of the moral panic (if one exists) by the response to the coffeepocalypse discussion. People who have been in a bubble where nobody ever connects the idea of moral panic to AI risk (if such bubbles exist) might realize that it's a relevant idea, and now they have a new tool to use in their thinking, even if they end up deciding that AI risk isn't a moral panic.
Also, now that the idea of moral panic has been spread around in connection with AI risk, it will be that much harder for a group that wants to start a moral panic about AI risk to enforce the standard taboo against thinking of the issue as a moral panic -- because lots of people have already had the taboo thought. It's like an inoculation.
In conclusion: just reminding people that moral panics exist is productive, because there's a whole avenue of political action in which powerful people use taboo to make people forget, ignore or downplay concepts that aren't favorable to them. If you're somebody who thinks AI risk is very real, even you don't want a moral panic about it, since they're so frequently counterproductive. Just keep on presenting your evidence that the danger is real, and try not to do stuff that makes you look like you're in a moral panic.
I think a possible point of the coffee argument is something like, “The arguments you’re making that AI poses an existential threat sound as plausible to me as the arguments that coffee posed a serious threat to society back then or that global cooling or overpopulation were existential risks in their time. That is, most of these arguments depend on constructing a long chain of reasoning, a bold just-so story, of how something terrible will happen because of this newly discovered thing (or newly appreciated risk). We know very little about this new thing and have no precedent of it causing harm at all, yet you want us to cede control over it to the state or embrace Luddism to ameliorate the risk. If you tried hard enough, you could construct equally plausible (or implausible?) theories of how attempting to avoid the risk might itself result in catastrophic harm. Note that these supposed risks are vastly different from the risk of pandemics or wars causing harm. There is historical precedent for the destructiveness of those. Not so for AI, global cooling, or population growth.” Maybe that’s not what the coffee argument really means, but it is something I would like an answer to from proponents of AI risk.
This version of the argument seems especially lazy. The worry about coffee wasn't that it would kill everyone (which it didn't), but that coffeehouses would be a venue for independent thought and would undermine the monarchy. If you look around the world today, we *don't* have that many monarchies left. So maybe the people were *right* to be worried about coffee.
But I think what these arguments *want* to be are like the ones where they point out old opinions about newspapers causing problems that sound eerily similar to modern complaints about cell phones. Like if your argument says "A, B and C therefore we should be worried about AI", but they point out that 500 years ago people said "A, B and C therefore we should be worried about coffee" and those fears turned out to be unfounded, it suggests that your argument is not as sound as you think it is.
I guess this still puts it in the existence proof category, but instead of an example of an existence proof of "not all technologies kill everyone" (which is really weak since you don't think that all or most technologies kill everyone, just that this particular one might), but instead "not all technologies for which your arguments apply kill everyone" (which would be somewhat stronger since you presumably think that your arguments about AI ought to be enough to imply that it is at least moderately likely that AI will kill everyone, and therefore a counter-example to this should substantially lower your confidence).
"He pointed out that physicist Ernest Rutherford declared nuclear chain reactions impossible less than twenty-four hours before Fermi successfully achieved a nuclear chain reaction."
This is somewhat garbled.
1) Szilard theorized nuclear chain reaction shortly after reading Rutherford's "moonshine" declaration in the newspaper, but it took about 10 more years for Fermi to build Pile-1.
Also some mitigating (?) factors in favor of Rutherford:
2) The moonshine declaration was in the context of a recent experiment that had split the atom using protons. This is much harder than using neutrons, since they are repelled by the nucleus. A nice and somewhat thematic way to finish this point would be to add "And he didn't consider using neutrons because the neutron hadn't been discovered yet". But in fact Rutherford's colleague had been awarded the Nobel prize in 1932 for discovering the neutron, and Rutherford himself had theorized their existence beforehand; the source I see for the moonshine quote is 1933. I'm not sure why Rutherford didn't consider using neutrons.
3) Part of the reason he was proven wrong so quickly was that Szilard read what he said in the newspaper and decided to prove him wrong.
If one wishes to be charitable to Rutherford, in 1933, "chain reaction nuclear fission would require applied unobtainium" is still a reasonable statement. The key discoveries of the next few years would demonstrate that uranium (or at least one of isotopes) has the relevant properties to be that unobtainium.
The alternate universe in which fissile isotopes have negligible terrestrial abundance is not obviously different from the one we live in, in 1933.
It's true that the missing ingredient for chain reaction was "isotope that emits > 1 neutron on average when struck by a neutron". Given that there are 200ish relevant isotopes I think calling the idea of such an isotope "moonshine" would be a bit aggressive, even at the time; there's just a lot of known unknowns.
I think it's like when you try to convince someone with a saying, like "Better safe than sorry." Of course this means nothing - sometimes a risk is worth taking and sometimes it's not. But if you've already decided that the risk is not taking, then you use the saying to illustrate your point.
Regarding the footnote, one of my more minor unpopular opinions that I like to float from time to time is that widespread caffeine addiction is a very under-discussed problem that is probably actually causing a significant amount of harm to society.
>widespread caffeine addiction is a very under-discussed problem that is probably actually causing a significant amount of harm to society.
Any specific evidence? AFAIK, evidence of _medical_ harm is weak enough that the advice on whether coffee is good for you or bad for you flips back and forth every few years. Re addiction: My personal experience is that if I miss my morning coffee, if there is a craving for it, it is so weak as to be imperceptible. Yes, tolerance happens - but is it an important effect for most people?
The medical advice on coffee that read is that it’s good to great in moderation. I literally forced myself to start on it as an adult, as I come from a predominantly tea drinking society.
I think it's ultimately just a bad extrapolation of an argument that is, or at least can be, fairly compelling.
If a particular person, or group of people, "P," has repeatedly predicted that event X will happen by a specified date, only to have said date come and go without incident, it really does seem as if one can safely discount P's next prediction that X will happen. I mean, they've gotten it wrong a bunch of times before, why should anyone pay attention to them this time around? Stated abstractly, the principle is "We can safely disregard predictions by P about X."
To make the argument concrete, use "climate doomsayers" or "preachers predicting the date of the Rapture" as P with corresponding values of X.
The mistake is that instead of disregarding predictions by P about X, the coffeepocalypse seems to be saying "Because we can safely disregard predictions by P about X, we can also safely disregard predictions by anyone about Y."
The reason it can seem superficially plausible is that it's a misapplication of a useful principle. I think you skipped over that, hence your confusion.
The coffeepocalypse argument is still bogus, but I think I understand how/why the mistake happens.
Dunno, I'm not so worried about autonomous AIs destroying humans, but rather some humans figuring out how to use AI to take over the world. Something like the "western steppe herder" culture, which about 5,000 years expanded across Eurasia, taking 9/10 of the preexisting male population out of the gene pool.
My best guess is that the logic is not actually "people were worried and didn't need to be, therefore they don't need to be worried about this." I think it's more an affirmative belief in humanity's collective ability to react and adapt to problems, which generalizes more to future problems than just remembering moral panics.
I think the steelman is that these aren't formal arguments at all. In the best light, they are invitations to consider a situation through a lens you might not have yet--heuristic activation, as you say, the kind of reasoning you only really need to be exposed to once. Suggestions that the situations /could/ be analogous, not that they definitely are, which might cause someone to nudge their credence away from the zeitgeist if they have not yet considered the possibility and don't want to drill into the particulars at all.
That said, I've seen this argument made in other contexts (relentlessly, on the harms of smartphones/social media) by people who absolutely do want to drill into the particulars and convince you that they are correct and this is another 'moral panic' cycle. (Aside: don't they have to be grounded in fear of moral violation to be moral panics? Aren't they just...panics, otherwise?)
Perhaps they think that AI is obviously a good thing (you can easily think of its benefits), in the same way coffee is obviously a good thing (you can easily think of its benefits), and they are not concerned about something obviously good going bad. The same doesn't apply to AIDS, climate change, Al-Qaeda, etc., that most people would not have thought as obviously good things in the first place.
The argument is really "They said it before and it didn't happen", rather than, "It's been said before and it didn't happen." It's an argument against the argument from the authority of elite public opinion. For the common people, the argument for X from elite public opinion is pretty good. "They" say chemotherapy delays death from cancer. "They" say Man really landed on the Moon. But even for a low-IQ person, the counterargument, "But they've always been wrong when they're forecasting doom" is a good one. It can be elaborated as "They've always been wrong when they're talking about technological doom and they make money on the supposed solution."
I'm an example of this myself with AI, though in a modified way. I'm high-IQ. But I don't know much about AI, and even less about AI Doom. I won't take up that difficult topic just because I hear some smart people are worried. Smart people are too often foolish, e.g. Y2K destruction of the power plants unless we pay millions to programmers, global frying unless we pay millions to climate scientists.
Come to think of it, my comment above also replies to the idea that people are "irrational" in the Kahneman-Tversky sense, and ignore small probabilities. Suppose there is a 0.1% chance of AI Doom, and AI Doom is hard to understand. It would be irrational for me to invest in learning about AI Doom, since in addition my personal world influence is too small for me to be much help averting it. So I will stick with my prior and refuse to learn more. My "prior" is really 1/1000, and I will very slightly modify my actions in life, but I won't worry much.
Yesterday, I had a minor panic episode. During the episode, I remember thinking, "I feel so stressed out right now--that *must* mean something in my life is horribly wrong, or else why would I feel this way?" I was taking my belief itself as evidence for my belief.
I think we sometimes do this collectively too. We have a moral panic about something and we tacitly think, "Well, if it wasn't a big deal, we wouldn't be having a moral panic." We take the collective belief as its own evidence. The legitimate use case of a coffeepocalypse argument is to remind us that the prevalence of a belief by itself can't be all our evidence, because sometimes folks are just dead wrong.
But, or course, people use it much less carefully than this legitimate use case.
Here's a proposition for you to consider. Everybody making these arguments is actually failing to articulate their real position. What they actually believe goes something like:
"AI doomerism looks an awful lot like these other sorts of doomerism and, sure, I've heard some compelling stuff about how frightening AGI might be, but these arguments are mostly inscrutable in a way that's commensurate with my goofy doomerism heuristics. Therefore I'll be putting AI doomerism in that basket for now."
They're effectively trying to tell you that, from their perspective, AI doomerism looks more like [Y2K panic] than [Donald Trump getting elected panic]. Most people aren't able to articulate their broad heuristics with any degree of precision, so I doubt you'd get anything coherent out of pushing them on this.
I think the best analogy is with conspiracy theories. If someone comes up to me and says that china is secretly controlling the world with radio disturbances based out of Antarctica how much time should I spend digging into this claim versus dismissing it out of hand? The reference class that may be relevant is how often are “wacky things people have said online” such as coffee apocalypses worth investing time into seriously considering. Pointing out that people make a lot of weird sounding claims on the Internet that turn out to be wrong is relevant. This isn’t an argument that’s going to be convincing to someone who has looked into these arguments, but I think the steelman argument here is that the person is deciding whether you’re a crank they can ignore.
The irony about the coffeepocalypse thread - is that the fear mongers were right. Coffee houses did help brew revolution - revolutions which have since overthrown the political power of nearly every monarchy on the planet!
I think the coffee thing is a story telling example which is being used to imply a base rate that most predictions of doom are false. You got very close to that here 'to gesture at a principle of “most technologies don’t go wrong”?' and then got distracted by talking about how the argument has also been used for things which aren't technologies. This may be true, but: 1/ if the base rate is predictions of doom rather than technologies going wrong it doesn't matter. 2/ They are talking about technologies here so it doesn't matter where else it has been applied, just whether it holds here.
The coffee argument reminds me of why Twitters “for you” feed is so annoying to use: a large number of viral tweets are making a similar annoyingly false statement that falls apart if you think about it for a second. Some accounts are genuinely dumb, some (most?) are doing it on purpose to generate clicks.
The Notes feature kind of helps but sadly your account doesn’t suffer significant penalties for posting clickbait BS so people keep doing it.
Every time Scott links a Twitter post, I am reminded of how completely different people's timelines are. The most obvious difference being that I barely get any people with blue checkmarks on my timeline.
It's honestly funny how quickly the blue checkmark became a red flag after it became purchasable. There's basically only three reasons to actually pay for a subscription: 1. They're a bot, 2. They're an "influencer", or 3. They're a moron. Combine that with the fact that you're more likely to see bad posts by people with checkmarks because the algorithm artificially makes them more visible, and it becomes completely inevitable that people associate blue checkmarks with garbage.
I think the footnote ("coffee really was dangerous") makes clear that you've slightly misunderstood what that thread is going for. He's not arguing that sometimes things aren't dangerous! He's arguing that sometimes things are inevitable, and we struggle against them but they just kind of happen and eventually we accept the new world. He says this explicitly mid-thread.
This is still a coffeepocalypse argument, broadly speaking, it's just a different coffeepocalypse argument. It's a single example of people worrying about a new thing and then *getting over it* (as distinct from "admitting they were wrong").
In terms of Aaronson's "Five worlds of AI," the thread is grappling with the question of Futurama vs AI-Dystopia, while dismissing Paperclipalypse out of hand
>He's arguing that sometimes things are inevitable, and we struggle against them but they just kind of happen and eventually we accept the new world. He says this explicitly mid-thread.
I'm reminded of part of the final scenes from "Planet of the Apes"
Julius and Zira look at each other, befuddled. Once more their eyes
follow the retreating figures of Taylor and Nova. The beach separating
the water from the canyon becomes narrower as they move downstream.
I'm 63. The societal fear of something that will end the world has been pretty constant all my life. The expected cause keeps changing, but the fear/anticipation has been constant. Lately it's been global warming. Before that overpopulation, nuclear war, etc.
So I've grown to think that this is some kind of cognitive bias humanity as a collective has. AFAIK, all societies on average expect the world to end soon. Christians have believed it for 2000 years, and still keep the faith. Maybe it's some kind of projection of the fact of our individual mortality, IDK.
So now that the AI-pocalypse starts to fill this need among a section of the population, my first reaction is "oh, the end of the world cause is shifting again".
All that said, I of course agree that this time *could* be different and the broken clock might be right this time.
Yup, (65 yo). The list of things that I know about and can have a useful opinion on, is tiny when compared to all those things for which I know almost nothing about. I therefore reason that most people's opinion on these questions are worthless.
(I'm writing this before looking at any of the comments, because I want to get across what I think *I'm* doing with this sort of argument and not be primed or influenced by what other people think *they're* doing with same).
This is very interesting, because I make this kind of argument constantly, and I'm surprised you don't understand its purpose. People really do think in very different ways.
You seem to think the argument is "people were worried about x and x didn't happen, therefore nothing people are worried about ever happens". When it's actually "people had a certain kind of argument for worrying about x, x didn't happen, you're using the same kind of argument for worrying about AI, therefore your worries about AI can be ignored."
It's an argument that there must be a hidden flaw in this kind of argument (even if we can't see it) because it's failed before.
Analagy: I say my electoral model says Trump will win this year's election. You point out that my model also said he'd win in 2020. Should I paraphrase your reply, like this post, as "I can think of one time Trump didn't win, therefore Trump never wins"? No, I shouldn't. What you're saying is not "therefore Trump won't win", it's "therefore your model is demonstrably flawed and should be ignored".
If people were saying in the 1950s that nuclear weapons, if not banned, would destroy us (based on facts that follow from the demonstrable power of the weapons and the nature of technological development), and we weren't destroyed...then anyone saying now that AI, if not banned, will destroy us (based on facts that follow from the demonstrable power of AI and the nature of technological development) should be ignored. It's not enough to simply explain their reasons for thinking AI will destroy us, even if they look airtight (since the nuclear reasons may have looked airtight). It's not even enough to give reasons why AI is completely different from nuclear power, because people in the 50s could have given reasons why nuclear power was different from every other past technology. It's necessary to give reasons why the *fundamental structure of the argument* being made now is different from the one being made in the 50s. (For example: this time there's a mathematical a priori argument, not just an evidential one, or something similar). Replace the nuclear example with any other you like.
Now none of this applies if you're making a probabilistic argument like "60% of the times circumstances x have occured there has been a catastrophe, and circumstances x are now occuring". But like you said that's almost impossible to count up, and it's usually not the AI-risk argument. The argument is usually "for these plausible-sounding reasons the prima facie assumption should be that AI is a threat." And once it's pointed out that claimed threats with similar plausible-sounding reasons have not ended up being threats, and thus the reasons are not so plausible after all (even if we can't see why), the burden is on *you* to either explain how the structure of argument is fundamentally different, or to rework it as a probabilistic one. Until you do that, the prima facie assumption for worrying about AI stands refuted.
> why nuclear power was different from every other past technology. It's necessary to give reasons why the *fundamental structure of the argument*
You're sneaking in a hidden reason, which is that the reasoning says the fundamental structure is the same generalizes well. If it turns out, for example, your idea of fundamental structure similarities is "this idea has doom in it", then you have neatly made it impossible for any doom idea to change your mind, and further actual arguments don't work, because they would all "share the same structure". This is known as the sin of pride, and has made many people named Walter very upset.
I don’t think that worrying about nuclear bombs is a moral panic on the same level as the panic about AI. It’s pretty clear how nuclear bombs can destroy a good portion of the inhabited earth, but the AI doom scenarios are fairly fanciful. Often not fleshed out at all. AI will become sentient. Result: annihilation. Path to result: uncertain and unexplained.
And, not to depress the party, people worried about nuclear bombs or proliferation have only been proven wrong, so far. That threat ends when all or most bombs are decommissioned - which won’t happen.
I think it's just an attempt to signal leadership and competence. Some bad economic event happens, and there's one guy running around saying "this is going to destroy the company!" Another guy is like "nah man, something like this happened in '87, it didn't destroy us then, it won't now". That's not a good argument to make, the other thing happened 40 years ago, a lot has changed. But that second guy is implicitly saying "I know about things like this, keep cool, we'll get through it, and obviously somebody like me who has this attitude should be the leader of that response rather than the crazy panic guy over there."
Reacting dramatically to a possible danger makes you look childish, and dismissing concerns with an appeal to past experience makes you look like the experienced adult leader. I don't think it's a claim about real probabilities.
I think it's a side effect of stronger resistance to fear (and guilt) arguments which are the de-facto standard way most lobbying groups G (government or other) attempting bulk influence in western democracies since WWII.
The pattern is "you should do Y to avoid the catastrophic X outcome", or "not doing Y cause many to suffer X". In then end, doing Y does not seem to change X much, but profit G while making the life of those doing Y harder.
This pattern is so constant that public resistance to it increases, leading to more and more extreme versions of the pattern and more and more resistance. I see it like an immune system response to fear and guilt propaganda.
In the case of AI, maybe the immune response is unfortunate: the fact those who should do X is not the general public, but very profitable megacorps, makes it more likely to be a genuine risk rather that yet more propaganda...But on the other hand, the megacorps are quite immune to fear and guilt. So expect research to continue as usual anyway :-).
Okay here's my steelman of the argument. Suppose we're arguing about whether resources are going to run out. Someone points out "here are all these cases of people boldly predicting that resources would run out and them not doing it." That would seem a perfectly kosher argument.
Now, in the case of AI, the inductive trend is a bit different. They're arguing something like "it's easy to argue some future thing is dangerous and likely to majorly disrupt the world and to find smart people making long arguments for it, but it's usually not." Now, pointing to one case doesn't show that, but most ways people have predicted the future will be significantly worse have been incorrect.
I think you are missing the heuristics that people are using to make these arguments.
When they point to the coffeepocalypse types doomers being wrong a lot, they have two heuristics that makes AI risks also seem like something they don’t need to worry about:
1. Predictions are really really hard. If someone is worried about something far in the future they are almost always wrong. If you look at what people were worried about 100 years ago about today, 99.9% would be completely irrelevant. This doesn’t mean you can completely ignore them but you should downgrade your concerns.
2. It’s very hard to predict human innovations. If the prediction is actually right and creates a big problem, it leads to a flood of human capital trying to solve the problem. This has solved countless problems in the past and while it is impossible to know ahead of time what the solution will be, it likely someone will find a solution. This isn’t an argument to stop investing in AI but actually argues for the opposite. However, it’s a heuristic why the typical person isn’t worried.
3. Even if the prediction is right, also have timing risks to consider. Even if someone rightfully predicted that AI safety would be a big problem 50 years ago, there very little anyone could do to lower the risk at that point. There a good chance people are worrying about this too early which makes it even less likely that it’s something to worry about right now.
I think the coffee thread would be more fairly summarized as:
AI is like coffee and other past technologies because it is currently facing opposition from powerful institutions ("media") who feel threatened by its potential to disrupt existing power structures and industries. And since the controversy is "really about power," it will play out the same way coffee did.
Russell's example makes sense to me because it sounds like he's arguing for "Clarke's First Law" which is pretty well known
> When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
I'm still not sure what the coffeepocolypse version is trying to argue for though
I think global warming is the stong version of the coffeepocalypse
My understanding is that current costs of warming may be net positive. A Lancet study reported that cold stress deaths have reduced twice as fast as heat stress deaths have increased in the last decade or so
Not only may we be wrong that warming represents the general violation of our Edenic past with all weather getting worse all at once, but the assumption that this is so means we are utterly blind to the costs
It is very possible that corn ethanol subsidies have singlehandedly outweighed all warming costs through starvation in food stressed countries, yet there is so little interest in the possibility that when I attempted to estimate the potential number of starvations I found that there has been essentially nothing since about 2010 that I could find even considering it in the literature
AI may be dangerous, but the assumption that it is probably infinitely dangerous is inherently dangerous itself. It may be right but if we allow the concern to go unexamined then we will experience some kind of corn ethanol subsidy cost times ten without thinking carefully about it. I don't think this invalidates AI risk mitigation but it shows that we need to not only estimate the necessary steps to hedge against AI risk, but also the likelihood that they will have a net expected benefit
A military strike destroying the capability to produce AI enabling GPUs may be necessary to credibly frustrate dangerous AGI - but does it actually have a chance of success commensurate with the guaranteed cost? Should we spend infinite money to prevent 10% more warming because of tail risks or is that actually a poor choice at some margin because we may also be preventing the tail benefit of delaying the next ice age and we just aren't thinking about it?
AI, like fossil fuel energy, clearly has huge upside potential for humanity, possibly larger than any prior technology ever! Interfering with AI development has huge potential costs and mistakes and can actually be self destructive, not unlike the environmentalists’ interference with cleaner, safer nuclear energy.
I think what might be going on here is that reporters reported on a real thing and then industry people responded by doing something and then the apocalypse didn't happen and reporters never reported on that part. So people assume that the problem wasn't real, not that someone fixed it.
They aren't implicitly making an argument. They're implicitly sharing a personal narrative.
When I was a kid, elites told me that I should be really worried about overpopulation and role-playing games. And I trusted them, and worried about those things. Then I found out that neither of these things were problems at all. When I was older, elites told me I should be really worried about violent videogames and assault rifles. But it turns out lots of people play videogames and are just fine, and most crimes are committed with handguns, not assault rifles. Then, most recently, I heard this crazy idea from elites that I should be worried about coffee. But this time at least I knew better and ignored it. And sure enough coffee isn't a problem. So when you tell me that elites are worried about AI, I don't buy it. Elites are always saying we should be worried about something but I've learned to ignore them.
The critical thing is that the teller of the narrative is not really distinguishing between different camps of elites; but this is not something so easily done.
> I always give the obvious answer: “Okay, but there are other examples of times someone was worried about something, and it did happen, right?
How about limiting it to examples of successful predictions of catastrophe-inducing events, civilization-ending events, human extinction events, or eschatological events? The history of experts predicting these events has been spotty at best. ;-) In my lifetime. I've lived through at least half a dozen EOTWAWKI (End Of The World As We Know It) events that scientific* experts predicted. And I've lived through three of four religiously predicted eschatological events (that I know of)...
..."And I feel fine," to quote the popular song.
If you were to say that AI will have unintended consequences, some of them negative, I'd be more inclined to take you seriously. Heck, I can already see it's distorting our knowledge base with "hallucinated" data. But predicting that coffee, horseless carriages, flying machines, and Lindy Hopping will have negative social consequences is not on the same order as predicting an EOTWAWKI-level event.
*scientific in the rationalist sense of the word — as opposed to religiously motivated doomers.
I think Scott is seriously missing the point of that thread and then the straw-manned version can't be steel-manned again.
The actual content is not "once people were worried about coffee, but now we know coffee is safe. Therefore AI will also be safe." Rather it is something like "Once people tried to heavily regulate coffee under various pretexts (and there were some true believers in those pretexts). But really the powerful people who made that into a political question rather than some cult's weird food taboo wanted to regulate it because social change threatened their position. In the long run it didn't work and couldn't have but it hurt the places that tried it. Same with AI now".
The guy doesn't try to engage the true believers on either AI or coffee doomerism, because he takes neither group serious, but that doesn't round to any of Scott's version of badly engaging the true believers.
I use the "here's a specific example of something not happening" argument as an attempt to call to mind what I believe to be a general pattern. I do try to make it clear that I'm making the argument that there is in fact a pattern, e.g. "Remember how in the 50s, leading AI researchers thought that machine learning could be fully solved by a few people in a few months? And 50 years later, thousands of engineers working together finally made a modicum of progress? Yeah I think it's going to go like that. The rate of progress on brand new technologies is always wildly overestimated, often by factors or hundreds or thousands, and I don't see why this would be different."
The first half of that argument is the specific example, and the second half is the attempt to generalize. I also would not be convinced solely by e.g. providing specific counter examples such as the progress on nuclear chain reactions, but I would potentially be convinced by a specific counter example, plus a reason to believe that the provided counter example is more analogous to AI than the "machine learning will be done in a few months" example.
1. Most people can't evaluate the object level arguments for AGI risk. Even attempted brief explanations tend to assume a ton of background reading, and more general math than the general public actually knows.
2. Doomers usually address this by citing experts. Lots of top people in AI have expressed concern, and you don't need a lot of personal understanding to tell that the guys who literally wrote the most popular college textbook on AI know a lot about AI.
3. But such experts don't have a great track record. A real evaluation would be hard (but maybe ACX worthy) so here's some anecdotes.
4. With neither object level arguments we can understand nor experts we can trust, we're left with our priors: most things don't destroy the world.
No one will ever be able to prove the negative, that an AI doom scenario definitely will not happen. The onus remains on the AI doomers to demonstrate their scenarios represent material threats.
IMO, the important part of the linked thread is the tweets about *why* various groups opposed coffee houses, based on political incentives. (eg "Kings and Queens saw them as breeding grounds for revolution")
Many people use a sort of political reasoning about everything - and by that, I don't just mean about red tribe vs. blue tribe, I mean that instead of reasoning about evidence for or against a position, they think it's most important to think and reason about *who* is on each side of an issue, and what their underlying political, economic, or psychological motivations for that position might be. The implication is "here's *why* the tribe arguing AI risk to you isn't trustworthy, because their views of the issue are corrupted in the same way as the coffee skeptics"
This lens is even more obvious when people make the arguments that "AI doomers put intelligent on a pedestal because they are nerds, and fundamentally *want* intelligence to be so important", or "AI doomers are scaring people to hype up their technology, or trying to bring about regulatory capture". Some of these arguments make more sense than others, but they are all fundamentally arguing about who to *trust*, not arguing the merits of the issue itself.
This can be maddening if you want to debate the substance of the actual issue, but I would say that for lots of situations, it's a workable way of reasoning about the world. If you're deciding whether or not to take a new drug, or put your retirement savings in a new investment vehicle being pitched to you...some people with the right educational background and time on their hands can dig into the issue fully and come to a rational conclusion based on the facts, but most people should and do start with "who do I trust about this".
Scott, sometimes you engage in strawmanning on stilts, all the while purporting that you are trying to steelman. I don't think you do this deliberately. I think you may be sincerely trying to steelman. But boy are you really bad at it when you don't sympathize with the argument in question.
The meta-argument here is: Humans have a powerful tendency to catastrophize, not generically, but about observable material trends, among which technological change is a prominent but not exclusive example. This tendency shapes the interpretation of evidence and results in motivated reasoning, especially the motivated construction of rational-sounding models to support the story. Not every concern about observable conditions or trends fits this pattern, but some do. When one observes a intellectual movement that pattern-matches strongly onto this pattern, it is reasonable to take that matching into account in the critical evaluation of its arguments.
The object-level argument would then be: AI Safety pattern-matches this tendency in ways X, Y, and Z.
Whether or not such an object-level argument is valid would need to be worked out on its details. But the attempt to dismiss the consideration at the meta-level is not at all convincing.
If you want to understand the argument, you could do worse than start with the mission statement of the Pessimists Archive, which collects evidence of past moral panics: "Pessimists Archive is a project to jog our collective memories about the hysteria, technophobia and moral panic that often greets new technologies, ideas and trends." https://pessimistsarchive.org/
Coffeepocalypse is like stereotyping or triggers. If you're taught to fear black people, your gut impression of the odds a particular black stranger is hostile will overstate the evidence. Almost all humans instinctively fear apocalypse from transformative new things, so you have to discount their fear a certain amount to estimate the true weight of evidence they've seen that a transformative new thing will be apocalyptic. It's not that there's no evidence, but their emotion is a bad guide to the evidence they've seen.
What was early realization of "humans creating potentially catastrophic climate change" in the epistemic environment like? Does it feel similar to the slowly dawning AI X-risk? I suppose the major difference is most normies are inherently vaguely fearful of "AI" whereas they weren't of climate change.
I respectfully disagree with your steel man arguments. My steel man version of the coffee thread:
1) New technologies regularly face backlashes, attacks, and opposition by authority figures and competing interests
2) Improved intelligence is simply too important to be stopped, therefore it won’t be stopped.
I do agree that the author just assumes in the end that it will all be OK. This is a really bad assumption. It is possible that AI will be overwhelmingly positive, that it will be extremely influential but in both negative and positive ways (mixed), that it will not be influential at all, or that it will be catastrophic. That said, a lot of the backlash will be by partisans, competing interests and threatened authorities. And it is inevitable and it will not be stopped. It will at best be partially managed, with the potential that it is mismanaged just as likely as managed. A better example here would be nuclear energy rather than coffee.
I am yet again begging that someone who has more free time (or research assistants/interns) than me make a dataset of supposed tech panics, and include variables like "did the people panicking about X include people who were creating X?" and "were the claims in some way accurate (e.g., did coffeehouses threaten monarchies)?"
I think you wrote about this once, but some people are incentivized to get engagement on what they write, not make good arguments. Coffee is a common thing lots of people are familiar with, it seems extremely boring, so it invokes an easy emotional response to say "people once worried about coffee." So they don't even ask questions like the ones you do here--once they've found something that might make a good article/tweet, they just run with it, because it doesn't matter. Probably also combined with standard confirmation bias + signaling tribal allegiance/outgroup disagreement.
(I think it's also the kind of thing that appeals to certain people, specifically nerds, who are likely to be interested in the history of random things and also to have an opinion on AI risk... I wonder if this was the result of someone reading about the history of coffee and then loosely tying it to AI risk for engagement).
Going off at a tangent, I had a vague memory of something to do with coffee and the Church.
Yes, turns out there is an urban legend that coffee, when first introduced into Europe, was regarded as a heathen drink of the infidel and some in the Church wanted it banned, until a pope (allegedly Clement VIII) tried it and liked it:
"Coffee aficionados often claim that the spread of its popularity among Catholics is due to Pope Clement VIII's influence. According to the legend, there was opposition to coffee as "Satan's drink", leading to the pope's advisers asking him to denounce the beverage. However, upon tasting coffee, Pope Clement VIII declared: "Why, this Satan's drink is so delicious that it would be a pity to let the infidels have exclusive use of it. Clement allegedly blessed the bean because it appeared better for the people than alcoholic beverages. The year often cited is 1600. It is not clear whether this is a true story, but it may have been found amusing at the time."
First coffee, then cocaine? I think stimulant enjoyers have much to thank the Church for!
"Vin Mariani was a coca wine and patent medicine created in the 1860s by Angelo Mariani, a French chemist from the island of Corsica. ...Between 1863 and 1868 Mariani started marketing a coca wine called Vin Tonique Mariani (à la Coca du Pérou) which was made from Bordeaux wine and coca leaves.
The ethanol in the wine acted as a solvent and extracted the cocaine from the coca leaves. It originally contained 6 mg of cocaine per fluid ounce of wine (211.2 mg/L), but Vin Mariani that was to be exported contained 7.2 mg per ounce (253.4 mg/L), in order to compete with the higher cocaine content of similar drinks in the United States.
...Pope Leo XIII and later Pope Pius X were both Vin Mariani drinkers. Pope Leo appeared on a poster endorsing the wine and awarded a Vatican gold medal to Mariani for creating it."
Well-stated. There were "experts" who predicted that AIDS was being overhyped for political reasons and other "experts" who predicted that AIDS would depopulate the entire continent of Africa. AIDS was (pre-treatment) a serious disease that caused illness and death in many places, but Africa is not depopulated. Perhaps some of the difficulty with the AI-safety debate is that it is always framed as an existential threat to humanity. Even if different threats have to be analyzed on their own terms, it seems reasonable to set a high threshold of evidence and analysis for the most extreme claims. Other examples of failed predictions of catastrophe seem like they might offer some lessons, although I can't deny that logically each threat is different and you can't reach a conclusion based solely on an imperfect analogy.
Which examples or analogies seem compelling seem to depend largely on what reference class people (typically unconsciously, intuitively) place AI risk in.
While the disagreements resulting from this difference are particularly strong and noticeable in the case of AI risk (and for the record, as other commenters have suggested, to me the best reference class seems to be, roughly, "what happens to an ecosystem when a more intelligent species appears in it", which, plus a separate argument for why this is likely to happen within a timeframe we care about, is cause for concern), but I suspect that the problem of finding a good reference class may also be behind many other sources of confusion and/or disagreement.
Here I find myself unsure of how to check whether one's choice of reference class is good, or what sort of things to do to improve it. Presumably someone has written about it at some point, so if anyone has any relevant links I'd appreciate it.
To answer your question, the original use of "outside view" was as a specific counter to the planning fallacy, where people will say things like "I'll write my paper in 5 days or so, 95% worst case, 10 days" and then they take 11 days. However, if you asked them first "how long did it take last time" they'll say something like "oh yeah, 11-20 days" which results in much better predictions.
Now (and this represents a leap on my part) Philip Tetlock says that one skill that superforecasters have is that they have good reference classes to previous similar events, and they use it to inform their base rates before they start doing research on political forecasting.
That's where the facts as I know them end. Here's my opinion:
I don't believe reference classes have been proved to work on long timescales, with abstract connections between classes. It works for the planning fallacy when *you* are trying to predict what you would do in the future, I don't think it would generalize well if you tried to get a "reference class" of paper turn in times. For Tetlock, what's been extensively tested has been political event prediction in the scale of 1-2 years in the future, not arbitrary predictions.
I think most cases of using reference classes is about trying to associate high or low status to certain ideas, and used not at all similar to how reference classes improve predictions in either of the above two examples. Why do I think I know this?
1. People like to use abstract principles to decide reference classes when they do the above, instead of really simple rules like "is there something that is almost exactly the same as now".
2. People treat choice of reference class as the final, or biggest update you could make, and tend not to be open to further argument after reference classes are established. This is not how superforecasters treat them! Any predictive value you might have had from using a reference class is getting wiped out by not changing your mind!
3. Duh, the people trying to use reference classes like to pick things that are extreme on their status implications, and do not actually seem to be interested in constructing some notion of a similarity measure, or seeing how sensitive their choice of reference class is.
All this to say: reference class tennis is bad and I don't think it's a fruitful route of investigation.
I think the for coffee it is an appeal to scale / shock factor that is supposed to be convincing. Look how even something so huge as one of the most popular drinks in the world (coffee, drank by over a billion every day), was thought to be dangerous. It's trying to break down the idea that although the potential impact of AI is so massive, that we should be skeptical of it based on the history of coffee as a similarly large scale but pointless fear.
I don't agree with this logic for similar reasons as you've described but that's my take on what the thinking is of people who make it.
I think there’s general skepticism against certainty of either total doom or total utopia. There’s an association between pundits and general self-promotion with these sorts of predictions. “Market collapse”, “super volcanos”, etc. It seems to me most of these people just want something for themselves, like status, instead of actually trying to save humanity. I don’t trust them.
I believe in AI risk because I’m a student of the topic. But I’m generally skeptical of doom predictions. I can see others doing the same if they don’t know enough about AI.
== Overpopulation / global cooling as proxy for AI Safety: ==
The implication is: “these are obviously as relevantly similar as two things can be”. People making this claim experience a strong intuition that the implication is true, and are surprised that you don't. They're also dismayed that you ask them to go the considerable length of justifying, or even explicating this.
== Coffee: ==
Attention grab, nothing more; unlike the other more serious, intellectually honest (if incompetent) claims.
== Nuclear reactions: ==
This goes to a specific aspect that people might actually be ~100% sure about: a lower bound on how long before something big happens it can no longer seem impossible. For nuclear reactions, people might think a year, a month, a week. You say it's less than a day, you make them think maybe they should also update this lower bounds for AI Safety.
== Russian tanks: ==
At least in this case, there's also the “good optimizer” assumption: if Russia has the resources to make its tanks that much better, it would be stupid not to put some resources into being ok in most other 99 aspects. Doesn't work for AI Safety though.
I think the implicit argument of the coffeepocalypse thing is that people in general tend to overestimate the probability of disaster, and that therefore if people are predicting, say, a 10% possibility of disaster that should be taken as evidence that the actual likelihood is much lower than 10%, like 2 or 3.
In most of his writings that I've found, he focuses less on the claims of existential risk and more on the the smaller cultural battleground of AI harming education, mental health, sundering the social fabric, etc.
In a substack essay he wrote with a similar tone, he lists various technologies that "people thought were going to devastate society or cause mass destruction":
> Television was going to rot our brains. Bicycles would wreck local economies and destroy women's morals. Teddy bears were going to smash young girl's maternal instincts. During the insane Satanic Panic in the 1980s which blamed Dungeons and Dragons and other influences like Heavy Metal music for imaginary murderous Satanic cults rampaging across the USA, people actually went to jail for abuse of children that did not happen. (https://danieljeffries.substack.com/p/effectively-altruistic-techno-panic)
And then he throws in in the supposition that existential risks are a weird fantasy in the same vein.
I think the issues with his line of argument tend to boil down to false equivalency:
- Most of his examples don't have anything to do with existential risk, but he uses these examples to bolster an argument which ends up dismissing existential risk (e.g. I don't think anyone postulated that the invention of bicycles would cause "mass destruction" in the way we understand it, so it seems weird to lump it in the argument)
- It's only in the past century we have the technology to actually wipe out the world (Genghis Khan-tier military tactics are small-scale compared to atomic bombs) so older panics tend to have not had as much existential gravity
He also seems to reflexively dismiss caution toward hypothetical scenarios that we cannot currently fathom. That is, if they did happen, hypothetical problems could probably be fixed by hypothetical solutions, so we shouldn't be scared by the hypothetical problems.
You're misunderstanding the target of these arguments. They're not trying to discredit the AI risk position as a platonic rhetorical object -- they're trying to discredit *the people holding that position*. Essentially, their belief is that anti-AI is a position that has been reached not through rational evaluation of facts, but by fear of the unknown. The argument is then:
1) We now understand that there's no reason to fear coffee (or whatever).
2) However, some people at the time were afraid of coffee, and here are the arguments they used against it.
3) Since these arguments turned out to be laughably false, the people making them couldn't have been acting rationally; they must have been poisoned by fear or some other emotion that got in the way of clear thinking. Maybe they were even being purposely dishonest out of self-interest.
4) You should notice the structure -- not just the substance -- of these badly-motivated arguments so that you're better able to recognize them when you don't have the benefit of hindsight.
5) Anti-AI crusaders use the same argumentative structure as anti-coffee crusaders.
6) Therefore, anti-AI crusaders are probably motivated by fear of the unknown or material self-interest, just like the coffee people we laugh at today.
7) Therefore, we should treat them the same way we treat other ill-motivated people like partisans and cranks -- by not even giving their arguments the time of day.
You're susceptible to your own version of this for the same reason everyone is susceptible to ad hominem: you hate the target and want an easy way to mock and dismiss them without thinking too hard about what they're saying. Ad hominem will always be the most rhetorically effective fallacy.
Russell's argument works because it's implicitly much more specific—it's focusing on "this seems sci-fi"-incredulity. A stronger, more explicit form of it would be either the rock with "nothing interesting happens, ever", or Yudkowsky's examples of people calling the idea of powered flight ridiculous (because it seemed "sci-fi").
Every technology is science fiction until it's invented, at which point it becomes science fact. "That seems sci-fi" is a fully general counterargument against any kind of substantial technological progress, ever.
That they said this or that could not be done (or was science fiction) and yet it was done isn’t a great argument for people always being wrong in negative predictions - that’s a selection bias.
Remember that most things they said couldn’t be done, and that weren’t done, are forgotten.
This seems like a better example of the opposite: if someone, at some point, said "by year X we'll have Y", people will complain about it forever, even if it was an obviously bad idea from the start.
Examples include jetpacks (congrats on burning your legs off) and flying cars (they're called helicopters, they're way too fuel-hungry to make any sense, and dear God you do not want to give every person a helicopter license the way we do for cars).
Good point. I was being needlessly sarcastic. But flying cars are in the news. And the Chinese may be ahead of the US! — queue outrage and a Congressional inquiry. Just keep 'em out my house's airspace, please!
As I mentioned in the comments, I see the opposite much more often—people never, ever forget all the things people predicted would be invented that weren't (e.g. "where's my flying car?"), then use it to argue against any future technology that sounds weird (is found in sci-fi movies). This argument historically has an extremely poor track record—science fiction movies also predicted the internet, smartphones, maglev trains, and universal translation software.
The real lesson to take away is that if it can be done (according to the laws of physics; and a human-level computer is clearly possible, because humans are human-level computers) and doing it would be profitable, it will (eventually) be done.
Reminds me of a volcano story on wikipedia: "Captain Marina Leboffe's barque Orsolina left the harbor with only half of his cargo of sugar loaded, despite shippers' protests and under threat of arrest. Leboffe, a native Neapolitan, reportedly told the port authorities, "I know nothing about Mt. Pelée, but if Vesuvius were looking the way your volcano looks this morning, I'd get out of Naples!" (https://en.wikipedia.org/wiki/1902_eruption_of_Mount_Pel%C3%A9e) All people in town except for three died in the eruption.
I think what your opponents are trying to do is point out the similarities between the false apocalypse predictions and the AI apocalypse predictions. Your job is to point out the similarities between the true apocalypse predictions and the AI apocalypse predictions. Also, the black plague, in the grand scheme of things, was more of a necessary thing than a bad thing (with apologies to the perriwig maker).
AI Will be good for mankind but bad for psychiatrists. Sort of like how the black plague was good for the enlightenment but bad for the church. Your opposition to AI is a manifestation of your innate survival instincts. You are a much better writer than psychiatrist so you should just embrace the change. If you think about it, you can help far more people by your writing than you can with your psychiatry practice. Even if AI does kill off a lot of people, it will probably do so in a very humane way (like persuading them to embrace hedonism in lieu of reproduction, something many western governments already do).
I agree that the coffee/global cooling/whatever arguments aren't very good, but I also think you are sort of fundamentally misunderstanding the work they are doing. You are treating these as arguments by which a person reasoned their way to a conclusion, using illogic. But that's not what people are doing at all. Rather, they have already arrived at a conclusion and they are using analogies to undermine the credibility of opposing viewpoints.
In other words, you are treating the argument as this: "People once incorrectly thought that overpopulation is a problem. Therefore AI is safe."
What people are actually saying is this: "Concerns about the dangers of AI are overblown for a bunch of reasons that I am taking as given. People, including experts, have often in the past incorrectly freaked out about new technologies, and you should be wary of such claims."
I am not surprised that you find this form of argument frustrating, because it doesn't actually engage with the substance of AI safety concerns. But it explicitly isn't designed to. These are statements by people who otherwise aren't convinced about the danger of AI, warning other people not to be seduced by credible-sounding arguments made by people with highly esoteric knowledge.
As others have pointed out, it very much is a sort of anti-expertise argument, but not in the mold of "experts are an evil/degenerate cabal trying to fool you." Rather it's more, "being an expert doesn't automatically make you right, especially in emerging domains of knowledge, as these examples show." And this is actually true! I get why it's annoying to have this point constantly waved in your face by people who aren't actually saying where they disagree with your arguments, but that is what is going on.
Scott, sometimes the other side is just stupid and wrong, and you need to squish them instead of persuading them, because they have a vested financial interest in NOT being persuaded.
It would be nice if maybe YOU could update YOUR priors along the lines of "85% of the population mean well and can be persuaded by reason and logic, but 15% of the population is just evil, stupid, or insane."
You're right. You should perhaps be more frank in your view that the Coffeepocalypse argument is idiotic. And this comes from a guy very skeptical about the AI catastrophe.
Even if he is idiotic, we can still use his dumb corpse to improve our lives. No part of the idiot wasted!
In this case, Scott sees an obvious reasoning error out in nature. He sees that it's by a person who could potentially, but by the Grace of God, could have been him, and since the person has not noticed the obvious reasoning error, he wonders if there's a similar anti memetic mental block that he has internally, hence trying to test the justifications in the nuclear bomb case.
And to point out, a repeated claim made across forums and comments sections is that AI riskers "have blind spots", so Scott is likely on guard for sightings of these supposed blind spots. I don't personally think this blind spot narrative is made in good faith, but also I'm way less modest and less nice than Scott, and you can't get Scott thoughtfully trying to analyze other people's viewpoints unless "am I also making this mistake too" is a reflexive thought.
I don't think you're giving a complete summary of the argument in the linked thread. I would call it a form of bulverism, since they're describing possible motivations for bad faith AI risk arguments. Similarly to how the beer lobby spread unfounded concerns about coffee to protect its market share, groups today could be spreading unfounded concerns about AI.
I think that's true. There's a debate between Ben Shapiro and Tucker Carlson that touches on AI driven trucks. Tucker said that he'd prefer that the government make up some safety risks if the alternative was the sudden destruction of trucking industry employment. Even if AI had no risks at all, there would be incentives to regulate it citing false risks.
As the term "bulverism" describes, this is all removed from whether AI is dangerous. They're not arguing that AI is safe, they are describing some bad faith reasons people would say its dangerous.
This made me laugh because I use the exact same thought experiment. "Whenever I wonder how anyone can be so stupid, I start by asking if I myself am exactly this stupid in some other situation." It is so hard to be aware of one's own blind spots that any strategy that allows one to look in a convex mirror around the corner is valuable.
We are talking about prediction. One hopes to have confidence in the accuracy of prediction. Yet with multiple variables and incomplete knowledge of all variables and forces at play, outcomes are generally uncertain. I could be hit by a truck this afternoon or the killer bees could get me like they got D. O_ in our town.
People fear bad outcomes. They desire control over outcomes. They desire relief from the anxiety about bad outcomes. For many people, a comforting story is all it takes, perhaps merely as distraction.
We do our best, and try to monitor outcomes in progress so quick course corrections are possible.
As a curious atheist, I like the attitude of the Old Order Mennonites, who pray not for outcomes, but for the grace to accept God's inscrutable will.
> The people I’m arguing with always seem so surprised by this response, as if I’m committing some sort of betrayal by destroying their beautiful argument.
This is actually a pivotal issue in explaining AI safety. People don't like being utterly decimated by another person, especially not when talking about objective reality.
I had a theory. In ground truth, over about a year there are now about 1000 AI pause advocates. Meanwhile Nvidia sold out a stadium and the industry announced, in just the last few weeks, plans to dump 650 billion! more into AI over the next 2-3 years.
There are obviously millions of new AI fans added monthly. For right now, where errors from AI are mostly harmless while the benefits increase monthly, doomers lost. There's nothing to discuss, no reason to engage with them. Hence comparing their position to people fearing coffee.
It's a dismissal, in the same way nobody bothers anymore to convince Amish to use technology. There is nothing to discuss and the evidence speaks for itself which side is right.
You already got it, it is the heuristic thing. See also, the Ancient Greeks complained about youth culture, so all complaints about youth culture are spurious.
I am pretty sure you have written on this before, with, birds-of-a-feather, and opposites attract. People will uncritically use little aphorisms to support their points even while equally believable totally contradictory aphorisms exist.
I have not commented on a SSC post in about ten years, so forgive me if my language is antiquated, but I said my objection to this post on the 'scord and was told it was wise and I should post it here.
I think that, in general, the most parsimonious explanation for a disproof of a seemingly nonsensical claim is that somebody is actually making the claim. For some reason, you seem not to have not considered this, despite it being the most obvious and straightforward explanation.
For example: even limited to my own browsing history, if I type "guardian.com ai" or "futurism.com ai" into my address bar, I will be confronted with about ten separate articles that claim AI will "do harms", "perpetuate harms", "pose risks", et cetera -- and that AI is in some way associated with pedophiles, scammers, Nazis, bros, et cetera.
Most of the time, there is not a real coherent argument being advanced in these articles -- it's just a random incident of a dumb or malicious act being somehow connected to AI; sometimes they quote a Rent-an-Expert who talks vaguely about "real harms". The general idea, typically made by implicature but sometimes made explicitly, is that AI is a thing which exists, and it didn't exist before, and it will clearly change the world in some way, and there are some harms, so it is bad, and should be (considered shameful / heavily regulated / made illegal / etc). If you'd like, I can provide a couple dozen articles and tweets and blog posts that articulate this vague point.
I think this is a pretty dumb argument to make, but people really do make it, hundreds of times, every day, and I think the most obvious explanation for the coffee disproof is that it is a response to this.
My assumption about what's going on here is that someone found some interesting historical facts about the social reception of coffee, and found that a good way to make their interesting historical facts go viral is to tie it to something that people are already predisposed to argue about. The tie-in to AI apocalypse is weak, but primes it for virality, and then when you put actually fun facts on the inside (even if they're not well-connected), it succeeds in going viral.
Prediction - if you could somehow do the same thing with cute pictures of capybaras rather than actually fun facts about the history of coffee, it would go equally viral.
Alarmist arguments about human extinction or massive disaster have a long pedigree. For example, just before the LHC was turned on, doomers claimed that it'll destroy the world through micro black holes or whatever. Given that, all those predictions failed to come to pass means that the chance of some new technology will cause human extinction must be very small. I don't know why AI doomsters like you obtusely fail to see this simple point.
I'll give a somewhat parallel scenario from the history of science. One reason the first law of thermodynamics was accepted as true was the long history of failure of every perpetual motion machine. After this uninterrupted series of failures many science academies stopped considering any claim of perpetual motion. When physicists wondered if the absence of perpetual motion machines may be a natural law they birthed the science of thermodynamics.
To take another example, people tried for 2000 years to prove the parallel postulate of Euclid. All those attempts came to naught. Then finally in the 19th century, mathematicians accepted the futility of such attempts and developed non-euclidean geometry.
Similarly you should recognize the failed past predictions of doom and accept the fact that p(doom) must be very low.
> Similarly you should recognize the failed past predictions of doom and accept the fact that p(doom) must be very low.
This isn't true for nonstationary time series (i.e. if there's a reason why the probability of an event changes systematically over time). A good example would be anything affected by technological improvement. If I predicted a 99% chance of an extreme recession by noting that GDP has only been over $70,000 in one out of the past 100 years, this would clearly be silly. GDP is trending upwards over time and I can't assume past observations about the distribution of GDP say much of anything about GDP tomorrow.
The probability of human extinction has clearly been trending up over time as we've developed technologically. Prior to 1945, we didn't have any technology that could conceivably end the human race. Then we developed nuclear weapons. Over the next 30 years, we developed biological weapons that could also do the trick. We invented climate change by accident too. That doesn't look like it'll be an extinction-level threat, but mostly because we lucked out w.r.t. the amount of methane trapped in Arctic permafrost.
The more general case is the "People were wrong in the past, therefore you're wrong now" argument. It's probably about 40% of internet arguments.
It's actually a good argument though, in the sense that it really does change people's minds. (It's not a good argument in the sense that it's more likely to change their minds towards truth than towards falsehood; it's not an asymmetric weapon.)
It can be very effective against an irrational person suffering an irrational fear, which is why people reach for it by default. If your kid is worried about monsters coming in the night, you can point out all the previous times that they've been worried about the exact same thing and that it hasn't happened. Or an adult hypochondriac worried that their itchy forehead means they have cancer will inevitably be reminded of the hundreds of other tiny symptoms that they've previously worried were cancer which turned out not to be cancer. Heck, I've used this argument against myself when I've had irrational worries.
The kings who were scared of coffee were obviously correct. Coffee-houses were breeding-grounds for revolution, they became gathering places for radicals pretty much immediately. Caffeine helps people think and one of the things they thought about was "how to overthrow the king".
I think this guy is probably exaggerating the degree to which there was a moral panic about coffee but, also, people at the time were completely justified in worrying about the social implications of a powerful new stimulant that everyone was going to drink all the time.
What about tobacco? There was no moral panic about tobacco. Europeans seem to have accepted it pretty quickly as a cool new foreign habit. But, tobacco turns out to have secret qualities that poison and kill a huge percentage of the people who consume it. The introduction of tobacco was actually incredibly dangerous, and we would be better off if there had been a huge clampdown on tobacco the second it was introduced.
On the other other hand, there was an early panic about the potato. People suspected it was poisonous (which it low key kind of is under some circumstances) and spent a couple of centuries refusing to cultivate it on a large scale. But the potato is actually a fantastic calorie-dense crop and not dangerous at all, so probably the unwarranted fear of the potato was incredibly bad and resulted in mass peasant starvation.
Are there any early modern examples of moral panics getting it right? Like did they discover crystal meth in 1600 and say "yeah, we're definitely not using this"? I don't know.
The lesson seems to be that there's nothing to learn from any of these incidents about whether fear of new technologies is inherently good or inherently bad. You probably just have to look at each thing on a case-by-case basis.
In terms of your model of what this guy is saying, though - he's making an argument that doesn't stand up because it feels right to him and it justifies something he wants to believe. That's 99.9% of all human political activity. Personally I find it difficult to model 99.9% of all human political activity, so I guess I can sympathise, but I'm also not sure there's much point agonising over it.
It seems very unlikely that coffeehouses were causally critical for revolutions. I doubt anyone here wants to claim that Europe would still be under absolute monarchies today if not for caffeine.
Of course, it’s not clear that people fretting about coffee at the time really expected it to lead to revolution either. The questions of 1) exactly what concerns people had, and 2) whether they were justified, are under-evidenced both in the original post and these comments.
>The questions of 1) exactly what concerns people had, and 2) whether they were justified, are under-evidenced both in the original post and these comments.
Very true! I would prefer to see more information on exactly what the concerns were too. Amongst other considerations, I've seen comments which assume the concerns the original post was addressing were existential ones and other comments which assume the concerns are nearer-term issues such as job losses. I do _not_ know which the author of the original post actually intended to address.
>Are there any early modern examples of moral panics getting it right?
(hides) I asked ChatGPT. It gave examples (albeit not _early_ modern ones) of the bans on leaded gasoline and on chlorofluorocarbons.
But I _also_ asked it about the definition of a moral panic, and one of the criteria it cited is _disproportionate_ alarm. If that is taken as necessary, then any case of taking prudent action against a real hazard get excluded, probably including the two bans it cited... The definition would exclude "getting it right"...
Steelman: "the fact that there exist people who are very certain AI will kill us is at best weak evidence that AI is dangerous, because there have been people who think X is dangerous for every X in history, so I shouldn't update on the fact that you hold this opinion."
(this is not an argument against the *content* of any argument you make for AI being dangerous, but such arguments can be very hard to evaluate)
Well there's the obvious problem of arguing against multitudes. But trying to break it down:
1: You think that AI poses an existential risk but you are wrong and it doesn't. I know that the possibility that you might be wrong must seem very implausible to you, but it is actually very plausible. As proof of this please consider this example of people being wrong.
2: You are saying that AI poses an existential risk, but you are arguing using reason and rationality which never works. You should argue using analogies instead. And if you do use analogies, like the one comparing AI to aliens or something then you're using the wrong analogies. To be safe you should stick to analogies about coffee.
3: You are a prophet of doom. Prophets of doom have a very bad track record. People who predict doom are usually wrong, and when doom happens it is rarely predicted except in the vaguest of terms.
4: You are not my people and I need to signal to my tribe by dunking on you.
5: Roll to disbelieve. I know it may seem like I'm abdicating my duty to fairly interrogate your arguments so here are some examples where I would have been right to do so.
6: People like you are so stupid, just like these other people who were also wrong! I just can't get over how wrongity wrong WRONG you are!
7: I think your arguments are suggestive and it makes me uneasy but I don't feel like I have the capacity to properly evaluate the evidence or your arguments, and I don't think you do either. Until we have better evidence in hand it is important to avoid causing a panic. Here are some examples of people overreacting to things that turned out to be false or overblown.
I think this is just rhetoric and they're not being epistemically rigorous. The idea is this line of reasoning feels similar to this line of reasoning and was wrong so this line of reasoning is likely wrong. This is rhetorically effective but should probably be discarded, although I admit I have made these arguments in the past I think I was doing the equivalent of an emotional flourish to the underlying argument.
I think that the reason you can't read any effective logic into it is that there is no logic to it. It's better to understand the argument as an appeal to emotion via pattern-matching.
It's not meant for people who are trying to think through things. It's meant for people who are already convinced they've thought things through enough, and it's meant to evoke a sneer. The implication is that there's a specific type of person who is always on the side of panicking about the future and they flip out over things we now all drink with breakfast.
Seen from that light it doesn't need to have actual logic on its side. It's meant only to evoke emotion by conjuring a specter of a similar situation, with the implication that if we'd sneer at one we should sneer at the other.
A lot of argumentation follows this pattern, I'd think. There are vast classes of argumentation out there that appear to be arguments on the surface but they have no actual logic to them; their sole purpose is to pattern-match with people's pre-existing biases to conjur up certain emotions. It's raw pathos, except in a more subtle form--the pathos is triggered by the implications of the message rather than any overt emotion-evoking behavior on part of a speaker.
I think the conclusion that there is no "effective logic" at work is generally accurate, but I'm more sympathetic to the potential value of pattern-matching and don't think it's necessarily "meant only to evoke emotion." Think about the test questions that would present a series of digits and ask what the next digit should or is likely to be: 001 002 003 004 005 00_ For reasons I won't get into, I spent a lot of time with mathematicians who railed against those kinds of problems. Presented only with a series of digits with no other assumptions or instructions, there is no logical or mathematical reason the next digit has to be 6. It could be anything - even starting another pattern! (001 002 003 004 005 001 002 003 004 etc.) Still, those kinds of questions survived the standard test item analysis, which suggests there is or at least could be some value in asking people to pattern-match against what a test-giver is likely to expect the next digit to be, even if it's not mathematical or logical. For claims of existential risk, it might be worth re-framing the questions: "Is there any value at all in considering previous failed predictions of existential doom, and if so what?" Many especially-logical or mathematical people would answer "no" or "very likely not." That could be true, but pattern-matching sometimes has value, and it's at least worth pondering whether historical failed predictions of existential doom could inform the analysis.
I think the point of the argument is to argue against a background assumption that people's worries about new technologies are reasonable.
Suppose you thought that anytime someone seriously doubted a new technology, they were basically reasonable.
Then someone told you that many people doubted the goodness of coffee as a new technology.
You then might update quite a bit! You might think "Huh, not only can lots of worried people be mistaken about new technologies, but they can be saying things *totally unconnected* to reality. I guess I will not care that much merely *that* many people are saying negative things about a technology, as it turns out large groups of people can be way off-base. I should update to using my inside view here to weight their reasonableness."
And perhaps the next step for many might be "And of course, I have not considered the arguments and haven't formed much of an inside view, so I guess I am not going to personally do much about the situation, given that I don't personally have an inside view telling me to do so, and other people cannot be really trusted to do my reasoning for me."
I think coming from a lot of people, you are too quick to dismiss this as not just an anti doomsday heuristic.
Global cooling and overpopulation are(were?) presented in more apocalyptic terms then just here is the percent of ppl who will die. Not everyone breaks things down as you do. They FEEL apocalyptic in a way that AIDS and even a big war do not.
And on the scale of apocalypses ppl have a heuristic of what can cause major distruction. War disease and asteroids really. If you are arguing for a path outside of that, they reject it and cite the previous failures (outside the three exceptions)
Having said thats I think sometimes its a wierd sort of i am right about things and other ppl are wrong heuristic. And i have actually heard smart ppl say it in fish like scenarios. These other ppl were wrong once, so you are also wrong and i am right. When challenged on that logic though they admit its not central to their argument.
Trying harder to defend the argument, perhaps we can think of it as a "Baby's First Argument Against Argument From Authority". It's not intended for people like Scott who are already well versed in the arguments either way, it's intended for Sally Q. iPhone who just seen her first ever TikTok video about "some people say AI will kill us all" and is worried.
The argument "Yeah but some people say that sort of thing all the time" will calm her down and bring her back to a sort of epistemic neutrality appropriate for people who aren't experts and are unlikely to become experts.
I learned early in med school to never say never, and to never say always. Cuz when dealing with biologic systems, best to remain alive to the possibility of exceptions.
But here, we are talking about machine/math systems. Might a product of such systems be susceptible to absolutes? And if so, which end of that dichotomy should/would it turn out to be?
So this is the fool’s errand part of all of this: we can’t even be certain if we can be certain. Let alone have any confidence whatsoever in whatever prediction we might have on the outcome, represented in percentages that bespeaks only having partial certainty , on an answer that will actually be definitively categorical.
I don’t even understand how you try to be “less wrong” when you have absolutely nothing upon which to derive a prior probability…and no data with which to modify said priors. That’s not to say an absence of evidence means we are absolutely in the clear. But it seems anything other than an “I don’t know” represents hubris without foundation.
Re your last paragraph: “I don’t know” has many possible meanings, but reasoning from no priors has a correct answer. The correct probability to assign the answer to “are Bloxors Greeblic?” should be 50%. The idea is that saying 50% isn’t hubris without foundation, it’s by definition what it means in Bayesian terms to say “I don’t know” as the answer to this question or more accurately “I have no information by which to privilege any given position”. What this means is that if your AI annihilation estimates differ from 50%, then you certainly must have information you are using to privilege that position one way or another, even if you deny that you do.
I agree. 50% makes sense. A coin flip would be very apropos and seems of no worse value as a predictor as anything or anyone else. Heads, we’re doomed; tails, we’re golden.
Right, which is not helpful and from an action-oriented standpoint predisposes us to action (which is why I refer to such situations as Pascal wagers/muggings - this is very serious if true and we just said 50% true so we must do something about it).
It's why people are trying to find the right comparators, to adjust their Bayesian probabilities to something more accurate. If experts are sometimes very wrong and also predict doomsday fairly often (and are always wrong when they do so) then that lowers our probability of doom in this case.
Specifically, this line of argument is an attempt to reframe the question entirely. Instead of making adjustments up and down from 50%, to frame the question as one of very low starting likelihood. If we start at 0.001% chance of something happening (say, the end of all human life), then arguments that raise the chance might take it up to 0.005 or whatever - five times more likely but still almost certainly untrue.
Starting all conversations at 50% is actually insane in terms of actions we should take. It would cause us to recommend significant resources be spent on anything that has a high potential payoff or downfall, for instance anything that might result in the end of the world/humanity. But most ideas don't work out as originally thought, most businesses fail, and so far every doomsday prediction has been wrong (and there have been a LOT of them). Asking that we treat any new doomsday prediction as anywhere close to 50% possible is huge ask and implies that society should likely be spending almost all of our resources preventing doomsday scenarios. If we happen to agree with a particular doomsday that seems right, but the other thousands of doomsdays don't seem worth it and we literally cannot afford to fight all of them. Even non-world-ending problems are far too numerous for us to spend significant resources fighting. There's a worker shortage right now, and an inflation concern, or concern about rising crime, and potential war with Iran or Russia or China, or a bunch of other problems that some people consider overwhelming and others do not. Selecting AI doom out of that list makes sense to some people, but it can't be society's focus unless we're sure it's true, because all of those other things exist and potentially need solved as well.
I think when you frame a question as having a low starting likelihood, you’ve implicitly already done the work in the background of moving your prior from 50% to something else. Otherwise, I could flip the question to: do you think AI will not destroy the world? If you agree you are “priorless” about this question, then by applying consistency, we should give it a low likelihood of 0.001 too!
What you’re getting at though, is that it seems like this 50% idea conflicts with common sense ideas about when to take action. But I think you are mistaken. It is not the case that 50% implies one must take particular actions. For example, if I am truly priorless about whether AI will destroy the world, bring it into utopia, replace the color green with splorange, or create exactly one party hat, it is unclear to me what action I should be taking. Further, you correctly recognize that we must weigh AI risk against the opportunity cost of not pursuing action in other areas too. The idea is if you hear your brain saying “take a particular action!” It means somewhere deep down, you might not actually be priorless between all those various possibilities at all.
I do think this makes the concept of 50% confusing, but this is probably where Bayesian logic differs from more common-sense ideas of probability and I start to lose a grasp of it. Not my area of expertise unfortunately :(
The people espousing AI doom are not disinterested neutrals pondering the meaning of a 50% probability, but are instead actively campaigning for more funds and effort put towards their preferred causes.
Any cause that uses "the world will end" as the result of not taking them seriously enough is trying to use the strength of the negative to overwhelm our cost-benefit analysis and agree to their requests/demands. If you multiply by infinity you will always get infinity. Starting at 50% pretty much means that no matter how much you adjust the final result, you're going to end up with a non-zero non-minimal result. Multiply a non-zero result by infinity and you get infinity, which says we *must* act to prevent it. If the final chance is low enough, people will hedge on infinity and may not worry about it, but even 5% or 1% final chance feels like too much to ignore if the end of the world is on the line.
I don't think the chance of AI doom is zero, but it's low enough that I treat it like zero for the same reason I treat asteroids and alien invasions as zero - they're low enough that many competing theories, if taken seriously, would bankrupt us trying to prevent them. We would constantly live our lives in fear with the most likely scenario that we end up wasting that effort.
I think there's a subtlety you didn't cover (not that it makes the argument much better). The argument isn't for people who know about AI, or even computers - it is for laypeople who cannot assess AI risk directly for themselves at all, and so are wholly dependent on what experts in the field tell them. This argument provides an example of when experts were all whipped up for nothing, suggesting that experts now could be all whipped up for nothing.
Admittedly, this angle doesn't really make the argument any more logically coherent. But it does give it a bit more effectiveness as a cover to illogically ignore a problem some people don't understand and don't want to think about.
I think the problem is that I have yet to hear an expert actually give a cohesive, logical, or rational actual explanation of the dangers. Yes, there is a real threat (let’s call it a reality at this point) that a lot of jobs are going to be replaced by AI, and that is going to cause economic disruption. That cat is out of the bag already, there is no risk because it is already done.
The only other argument I have heard is some vague hand-wavy notion of killer nano bots somehow being created from AI.
I don’t find either argument, exceptionally rational, and certainly not to the point where we want to try to stop technological innovation (good luck), or to where we would want government intervention/protectionism for the current corporations that happen to have a lead.
This is the gene editing hysteria of the 90s all over again … “someone is going to make a mutant strain of Ebola in their garage with this new fangled CRISPR technology!!!”. (and if you really want to stretch that analogy to its breaking point, you would say that private company is used it to cure diseases, and the government used it to create Covid19 and kill millions) /shrug
I'm no expert, but as I understand the argument, it goes like this: If we build AI systems that are more intelligent than us which have 'general intelligence' (as opposed to intelligence in some narrow domain like a calculator), we will not understand them, and therefore will not be able to control them. They may act in accordance with their own goals, which are likely to be opposed to our goals. We will not be able to stop them, because they will be smarter than us.
The weak link there is 'they may act according to their own goals' - why would they? I think the answer is, we don't know for sure, but we have to admit the possibility because humans do. A human is a physical system that has general intelligence. That intelligence arose by natural selection, which has quite specific goals to its programing (survive, be fruitful and multiply). We see that humans quite often, cast those goals aside and pursue their own ends. So our N = 1 example of generally intelligent systems shows us that they cast off their programing and do what they like.
On the gene editing thing though, what makes you think we are out of the woods there?
> If we build AI systems that are more intelligent than us which have 'general intelligence' (as opposed to intelligence in some narrow domain like a calculator), we will not understand them, and therefore will not be able to control them.
To be more specific, it can be not generally intelligent if it is narrowly intelligent in enough impactful fields. The trivial example is that general intelligence includes the ability to become a gold medalist in gymnastics, a world class musician or a religious founder. However you probably don't require those things to "take over the world". At the other extreme, a lack of charisma doesn't matter if you kill everyone instantly. You probably still need theory of mind to be able to figure out "maybe I shouldn't tell everyone what I'm doing". And in theory, a narrow AI becoming good at "look at own mind and look at problem, modify the mind to be able to solve the problem", means that you may get lots of capability gains "for free".
Why is intelligence scary? Because intelligence isn't book smarts. It's everything your brain does. A dictator did not come into power because they are the hive queen, guns were not dug up from the ground like tubers and the modern financial system certainly isn't the result of an evolutionarily programmed money instinct.
Right now, even within the narrow band human intelligence, you have teenagers taking over large swathes of infrastructure to help fund their Minecraft server racket, smaller teams outcompeting bigger ones and some dude thinking about trains creating an entire physical theory framework out of nothing.
There's not really a "canonical model" of superintelligence, but a definitely wrong model is "one smart guy". "A billion unsleeping, relentless Einsteins, a billion unsleeping Elon Musks, a billion unsleeping Jesus Christs, all working together without friction or in-fighting" is also wrong, but much less wrong. If your brain is providing plans that you can see through, it is making the same error as a chess novice trying to fool's mate Magnus Carlsen and thinking "well, what if he misses this one trick then I win".
> The weak link there is 'they may act according to their own goals' - why would they?
See: https://arxiv.org/pdf/1803.03453v1 page 8, playing dumb on the test. This is a super simple, RL based system. The person didn't "program a survival goal" into it, the organisms definitely do not have theory of mind, yet they behaved as if they had those! I think that's pretty strong evidence that "have a goal in mind while programming AI" is nowhere near sufficient to prevent some other, unintended goal from becoming embedded. If you disagree, can you point out a specific technical result why this problem would not generalize? I'm asking for a technical result because the previous responses to this question seem to assume "human friendly by default" without acknowledging that the world doesn't seem to be consistent with that view. I'd be receptive to verbal arguments, if I didn't think they had a dismal track record.
If people are doing argument by analogy, they at least need to use close analogies.
In the nineteenth century there was a case of superior intelligence afflicting large groups of people: the colonisation of Africa.
But it seems to me that what mattered was mastery of logistics, operations, and tactics: fire and movement. Not intelligence in and of itself. Chemistry, logistics, and kinetics.
(Slightly prior to that various Bantu peoples had swept through Africa, uh, subjugating, the incumbents in an arc from West Africa to East Africa and then down to South Africa. Better operational organisation.)
There were cases of superior intelligence (and firepower) not making much difference: Europe vs the Ottoman Empire, and Afghanistan.
These analogies suggest everything will be terrible/will be fine/might go either way.
<i>If people are doing argument by analogy, they at least need to use close analogies.</i>
On a bit of a tangent, I've often had similar thoughts about the war in Ukraine: which course of action the West should take depends largely on whether you view the conflict as more analogous to WW2 (Putin is like Hitler, he will never stop trying to conquer new land, therefore we need to take him down now before he gets too powerful) or WW1 (this is a distant conflict in Eastern Europe that doesn't really concern us, and we should stay away for fear of escalating a local war into a civilisation-ending catastrophe).
I think most people do a lot of their reasoning by analogy, or "heuristics" as you put it. It can work well, as long as you don't stop at the first one that you come up with. Ideally you think of several that match, and then several more that don't, and you compare them and see where they differ, and what the significant factors were, and then look at the actual thing you want to examine, and see how they apply. And then you think rationally about it, because all of this is a way to provide data, not an actual analysis of data. (One of the problems with trying to think rationally from first principles is knowing what fundamentals are important. This bypasses that step, and gets you a lot of useful fundamentals to start with.) I do this a lot; if there were a way to look back at my comments, you'd probably see a bunch of it, where I reference various things, flit from analogy to analogy, and mix metaphors beyond their breaking point.
This approach to reasoning breaks down when people only come up with one analogy, or only come up with analogies that support the side they favor, or don't actually think about it rationally afterward.
You can see a similar effect with, say, some types of prejudice. Here is an example of a type X person committing a crime. Here is another type X person. Clearly, they will commit a crime! It's taking the first association that comes to mind, and running with it.
One thing coffiepocylipse arguments seem to have is the barest mention of AI.
Which is strange. They don't look at AI and compare it to coffee. They don't seem to analyse specific anti AI arguments and compare them to anti coffee arguments.
Do a ctrl R for the word "AI" and often you couldn't tell what they were arguing about.
Is it that their model of debate says the winner is whoever claims more valid facts? Like they are greatly prioritizing "Is it true?" over "is it relevant to AI?" and just spouting a bunch of facts that they think have any non-zero bayes factor, however small.
If you are imagining an opponent who never concedes any point, that opponent would have to say "no one was ever worrying about coffee destroying the world" or "coffee did destroy the world". Never "so what?"
Some people think that being contrarian makes them look smart, by contrast to the gullible masses or "sheeple" as they probably call them. So if everyone seems to be panicking about AI doom round the corner, then the mavericks' natural inclination is to scoff at the risks and deny them, even if their justification (if any) is fallacious.
I think the core idea behind this type of sentiment is a general disbelief in deductive logical reasoning when it comes to making complicated predictions.
There are a lot of things working against it. People generally tend to optimize for selecting socially advantageous beliefs over correct ones. This is a strong subconscious pressure, and forming social groups based on shared beliefs and principles makes it stronger. It's fairly easy to beat people in arguments when you are more intelligent and oratorically eloquent than them, even when the viewpoint you are arguing is less accurate than theirs. Furthermore, only a handful of historical geniuses have ever been able to make unique testable predictions about the world from just philosophical principles that held up to empirical observation and experimental scrutiny, and it's reasonable to have a strong prior in the direction of not believing people when they implicitly carry themselves as if they are once-in-a-century geniuses. You could obviously object to this and say that people use deductive reasoning to make complicated predictions all of the time, but most people have a certain scale/complicatedness cutoff point, and it usually lies way below making pessimistic predictions about superintelligence.
I personally find this way of thinking to be dispositionally annoying. However, I come from a family of some people who throttle a sweet spot between normalcy and intelligence; they have the intelligence to where they could participate in cool subcultures and intellectually screen weird beliefs if they wanted to, but choose to be 'normal' instead. And so if I were to ask my mother, or my grandmother before she passed, why they weren't scared of the existential dangers of superintelligence, and pressed them on it enough, I imagine that they would say some non-me-worded version of the above paragraph.
Right, which is a status game to punch sideways or down at people in order to make your own position seem more accurate and higher status. It's the same thing bullies do to feel more important - insult others.
Meta-reply: And I just did the same thing to you and anyone else reading this - implying that the people who use midwit are insulting bullies. Even though you didn't use the term, it degrades your social standing if you identify with it or endorse it, and puts future people on notice that they would face the same opprobrium if they did it.
Many people are indeed very, very bad at constructing reasoned arguments, so I absolutely agree that "coffeepocalypse didn't happen" arguments are out there in large numbers.
They don't understand that this is only one part of this rhetorical tactic, the "what" that didn't happen. Their next sentence should be the "why" - and hopefully the why for coffee or whatever matches up to the why for AI.
It's unfortunate because in order to steelman this kind of argument you have to invent out of whole cloth their next line of reasoning. For example with the "perpetual motion will never happen" argument, the next step should be "because it goes against the 2nd law of thermodynamics". This one in particular won't work for AI because there's no physical law preventing consciousness emerging from microprocessors. Similar case for the "controlled nuclear fission is impossible" - why? Is there some reason proscribing such a thing?
So you are correct, the arguments you are seeing don't make sense because the people making such arguments do not even realize they aren't making an argument at all.
As for why the AI apocalypse won't occur the reason is simple: AI doesn't exist.
I take the argument as, "Negativity bias makes us think the world is always ending. Here are some examples where we freaked out over nothing.... But the fact that we survived all those things suggests that the world is a lot more resilient than we think."
I think I can answer the question, "What is this coffeepocalypse guy doing?" more generously and effectively than you have here: He's trying to reframe the debate.
He's specifically not engaging with the question, "Could AI be dangerous?" He is suggesting that it is a waste of time to think about that question. He suggests that everyone engaged in the AI safety debate is, consciously or unconsciously, engaged in status games; therefore it is better to simply refuse to engage in any such debate.
He doesn't really support any of these claims. I think he just does existence proof that it is possible to play status games and to engage in motivated reasoning in the course of a status competition. Nonetheless, I think exhortations to reframe can in general be quite useful and good, because they present alternative viewpoints outside the current framing of the debate.
In this particular case, he is clearly very unaware that these possibilities have been chewed over by AI safety people. It's an ignorant argument. But I think the form of the argument is fine.
Coffeepocalypse is a sound meta-argument: "the fact that lots of apparently smart people are predicting AI doom is not in itself evidence of very much, because lots of smart people have predicted doom many times before and consistently been wrong". That is, it's a caution against appeal to authority. It's a non sequitur in response to anyone who is already engaging with AI doom arguments on the object level.
Isn’t this just reasoning by analogy? And the stronger the isomorphisms are between the analogies the more we update. If someone compares the invention of coffee to nuclear fission, that’s not very isomorphic. But if someone compares fission to fusion, that’s highly isomorphic. At the extreme end, if someone makes a claim about red nuclear warheads then it would cause us to update strongly about blue nuclear warheads
It's unsurprising you failed to explain how the author got from the coffee analogy to the "AI is safe and useful" conclusion, because *it's not the conclusion*. The thread author didn't try to explain whether or not AI will be safe. The author obviously *assumes* it will be safe and useful, and explains what he believes to be the reason why people still fight over it, by way of the coffee analogy. Even if his analogy was mortally flawed, it would, within the author's framework, at most require a new explanation (something other than power struggle), or even just a better analogy.
Please note: This comment does not imply that I share any of the author's opinions or their opposite.
Thank you! I was surprised that this didn't come up in the summary of the argument. He is explaining why people may say that a safe technology is dangerous. He's accusing AI regulators as operating in bad faith, like when a beer company warns against the dangers of drinking coffee instead.
I've got recorded in my calendar one of those "predictions to check back on". This one is from Slashdot on April 29, 2004:
> Diamond Age Approaching?
>
> Posted by CmdrTaco on Thursday April 29, 2004, @01:55PM from the i-friggin-hope-so dept.
>
> CosmicDreams writes "The CRN (Center for Responsible Nanotechnology) reports that nanofactories (like the ones that were installed in every home in Neal Stephenson's Diamond Age) will arrive "almost certainly within 20 years". In short they claim that molecular nanotechnology manufacturing will solve many of the world's problems, catalyze a technologic revolution, and start the greatest arms race we've ever seen. They conclude the risks are so great that we should discuss how to deal with this technology so that we don't kill each other when it arrives."
I suspect the reason why popular discussions about AI safety are so confusing is that people say "superhuman AI is safe" when what they actually mean is "superhuman AI is *impossible*".
In a narrowly technical sense, yes, that is correct reasoning: if something cannot exist, it also cannot harm you. But for the sake of clarity of argument, it would be much better if people separated "it is not dangerous *because* it is not possible" from "even assuming that it is possible, it is still safe".
What we get instead, is people playing politics, so the ones believing "it is not possible" and the ones believing "even if possible, it is safe" see each other as allies. An argument that supports an ally is good. But an argument that hurts my ally, but not me, is not too serious.
So the entire argument, fully untangled, would probably be something like: "If you believe that coffee is harmless, that is an argument for our side, because it means that the set of things people worry about needlessly is greater. But even if you believe that coffee is harmful, I don't really care about that, because unlike coffee, superintelligent AI does not exist, silly!"
In the rationalist lingo, "coffee is harmless, despite people once believing otherwise" is a soldier on our side, but is not a crux. If this soldier hurts you, good. If it dies instead, no big deal.
When people bring up the coffee story, they aren't actually trying to make a rational argument. They're telling a story and it just happens to look like an argument. People update on stories all the time, by the way. That's what parables and myths are, and it's how people spread wisdom for millenia prior to rationality. That mental machinery hasn't gone away.
Most things are neither affirmatively safe or unsafe. The vast majority of human developments are of no consequence to history. Inductively, it stands to reason that we would expect most hyped developments to be of similarly limited consequence. As with any inductive reasoning, we would occasionally be wrong, but we would more often be right.
I have a suspicion that the coffee story is not an argument but a metaphor. Some people will (incorrectly) update on it, but really the story is just meant to clear up thinking of those who alreafy have vague a picture of how AI is going to affect the society.
Many AI doomers think that the story in "Don't look up" has surprising similarities to the AI story. But the movie is not actually an _argument_ in favor of being an AI doomer, it just resonates with doomers who have other reasons to believe that AI is dangerous.
because most ai safety arguments are based on My Little Transhumanism: Technology is Magic and assume somehow it will become this master self-aware manipulative villain despite no evidence technology can do this.
it's not AI safety in more mundane ways like ai unemployment and economic effect, or people using AI to increase spam, centralize economic power, or how its edge cases cause more work for fewer remaining people. It's mostly projecting existential fears on the latest new thing.
I mean just because AI can replicate a human behavior doesn't mean it possesses the potential to take on the whole aspects of human behavior in a more perfect form. Technology never breaks upward; not like the more sophisticated cars get, the more likely the evolve into herd animals and book it to the plains to forage. they just break differently
Coffee ended up being an existential threat to certain countries when it attracted the right people at the right time to create a certain outcome (in the doomer's case, a bloody and destructive revolution; in the optimist's case, a potentially better/fairer political system for future humans).
Nuclear weapons will be an existential threat if/when the right people at the right time utilize these weapons to create a certain outcome (in the doomer's case, extinction; in the optimist's case, a fresh start?).
AI will be an existential threat if/when the right people at the right time utilize these tools to create a certain outcome (in the doomer's case, extinction; in the optimist's case, earth and nature get a chance to recover from our rough-shod resource rampage).
I think it's reasoning by emotion. The coffee danger (as I always understood it) doesn't need to go back to revolutionary times, but to advice given to folks with high blood pressure and cholesterol in the 60's: that coffee would make it worse. (Now the opposite is believed true, if we should believe them.)
That's within living memory. And within living memory are emotional decisions: I like coffee, coffee is fun, I want to drink coffee, I love the buzz, what's this about blahdeblah cholesterol folderol? The emotional weight outweighs (in my dad's case when he was told this) the cholesterol story. I remembered this when I drank my first coffee and I think of it at least a little bit when I drank my last.
So it was emotions that won out over what appeared to be scientific evidence. The emotions were doing the heavy lifting here, and when an argument comes up that drags coffee into it, it drags along the emotional reasoning that was done (and proven "right") to speak its voice in the argument. So, they were wrong about coffee, by association they were wrong about whatever emotional feelings I have about AI dangers, which have a strong emotional component on both sides of the argument, or so it appears to me.
I think that's why other arguments that have the same frame but don't drag coffee in, don't seem to work the same way. I just heard that anything you do while consuming caffeine--smell coffee, have friends over, sit at a crowded cafe, whatever else might be going on) the caffeine makes you like whatever that is. Could caffeine itself be making this argument, or at least part of it?
Humans have a tendency to over-predict doom scenarios.
Because of this, when we’re worried about doom we should be careful and adjust our priors.
Historical examples help with adjusting priors.
–
My own position: super human AI will be the biggest change in history, this makes priors less useful. We’re more in first principles territory.
Say we invent a new drug and people now live to age 1,000. How will this change society? Answer: I’m not sure, but past times when lifespan went up 10 years are probably not super relevant.
As far as I understand the AI safety debate (on a scale of 1 being "never heard of it" and 10 being "Eliezer Yudkowsky" I'd put myself at like *maybe* a 4), the biggest problem with having strong opinions in *any* direction on AI safety is that all the smartest people who most care about this problem seem to say they have absolutely no idea what to do. This isn't a strong argument to stop them from trying stuff, but it's also not a great argument for throwing massive amounts of money at the problem, shutting down GPU farms, lobotomizing AI development with cumbersome regulation, etc.
Maybe the coffeepocalypse argument is badly gesturing at "by your own admission, you're worried by a gut feeling based guess of the probability of a doomsday scenario that emerged from a long process of thinking and speculating about black box technology (black box because you don't know how it works or because it hasn't been invented yet) that you have also admitted you have no idea what to do about. What the heck else can you do in this situation but wait and hope it turns out fine? And, hey, look, there have been times when people faced similar radical uncertainty and we aren't extinct."
My own position is something like "Noting will change until somebody blows something important up with AI accidentally and/or a rogue one blows something up. Hope the explosion isn't existential. At this point, many more smart people will get interested in the problem, and if nothing else we may succeed in strangling it with regulation that doesn't fix alignment but just makes it impossible to do anything with it like how it takes 1 million dollars to build a one stall toilet in San Francisco."
For people who are *really* convinced it's going to kill us all, I'd say the best approach is to try to strangle it with regulation by pretending you are super worried that AIs will discriminate against black people, be used by the deep state to track right wing people, that GPU farms are bad for global warming and/or that they cause all kinds of other weird health and environmental problems, that the *wrong people* will use them to spread misinformation and steal elections, turn the youth trans, etc.and well you get the gist. Lobotomize it. Do to it what was done to nuclear power.
According to Wikipedia, Rutherford said, "anyone who looked for a source of power in the transformation of the atoms was talking moonshine.". Searching "moonshine" on LessWrong yields the following post by Scott Alexander from 2015:
I make no claim right now, but others can read and ask themselves which posts have which reasoning errors.
Also, does anyone have citations on the Rutherford quotation? I can only find citations from the early thirties, and none from the day before the first successful artificial nuclear chain reaction (which was in 1942).
The argument is implicitly about base rates. Here's a couple analogous examples:
1) The math department gets a lot of well-meaning letters from amateurs claiming to solve the Riemann Hypothesis. Over the last 100 years, none of these letters have solved the Riemann Hypothesis. Therefore, the one that just arrived today probably doesn't solve it either; we're not going to get excited about it.
2) Every year, there's a popular science article to the tune of "Cure for Cancer Finally Discovered!" We don't have a cure for cancer. Therefore, when the next popular science article about cancer cures comes out, we'll ignore it.
So similarly, the anti-AI-safety argument goes:
Every time we develop a new technology, a lot of smart and well-meaning people argue that it's actually Bad and that it poses some kind of Existential Risk (railroads, bicycles, electricity, human flight, etc. etc.). But new technology is usually Good, and we haven't killed ourselves yet. Therefore, now that a lot of smart and well-meaning people are arguing that AI is Bad and poses an Existential Risk, we shouldn't pay that much attention.
a) Agreed, we should _not_ ignore base rates. ( In the financial realm, it has been said that the worst advice that one can hear is: "This time its different!" )
b) Reference class tennis is, unfortunately, unavoidable
c) If we get AGI (in the sense of capabilities matching a _typical_ human, I don't mean near-omniscient superintelligence) (and we do _not_ have AGI today, even in the IQ=100 sense), then it can fill every role needed to make more copies of itself - effectively acting like a new species
d) We have a base rate for interactions with intelligent species too - Neanderthals are gone, Homo habilis is gone. Homo erectus is gone.
e) AGI, if we get it, is both a technological improvement and a new intelligent species. So which base rate do we start from?
( FWIW, I want to _see_ AGI, personally, so I'd rather not see a pause/ban. And I think that a pause/ban in a competitive world, will fail anyway. )
Pre-GPT4 this was pretty close to my take on AI alignment. Now I actually do take it seriously and think we desperately need to stop building frontier models, but I want to try to communicate why coffeepocalypse doesn't seem to me like a stupid counter and is likely something I might have brought up as a counter-argument to AI safety when I thought AI safety was dumb.
Most doomsday predictions involve _stories_: First A happens, then B, then C, and then the world ends. You can say "Well what if A doesn't happen?" and then the prophecizer might say "Well then A' will happen and that could also lead to B" and so on so forth.
And so you get the impression that the prophet is falling into the trap of having located a random hypothesis and is now using stories to defend it. Essentially that they're in soldier mindset. Instead of countering each and every story, it's easier to try to show them the trap they're falling into by giving examples of other people falling into the same trap, and hope they'll recognize it, discard the random hypothesis, and go back to looking for truth rather than writing a story about the future which will almost certainly not come true exactly because most stories about the future don't come true.
Coffeepocalypse isn't an argument _against_ AI safety, it's an argument against confusing stories with predictions which is such a common error it's easy to assume people arguing that AI is dangerous are also making that error.
I don't know what it's for others but for me it's about worry overload. Almost anything you see or read is someone trying to convince you to worry about something. Cries for attention. Now a priori there is an infinity of possible bad outcomes to anything, but there will only be one actual outcome. So per definition, (infinity minus one) things one can worry about, never happen. And I can't be worrying about the infinity all the time. So I only worry about the immediately obvious next pressing problem that I have influence over in my own life. Ai is not one of them. Oh and one more thing. Bad things happened in the past, with or without prediction. But absolute doom never happened. It may, of course, happen one day. So my prior here is, however bad past things turned out, humanity overcame them. So, one step at a time.
I guess it's 1) a prior that AI won't be bad, plus 2) "lots of people being scared of AI and supporting their fear with arguments, isn't evidence worth updating on; lots of people have always been scared, and have always supported their fear with arguments, about everything"
Motivated reasoning. Sub-conscious. 10% or whatever chance of AI Doom is such a scary idea, more so even than nuclear war etc, that I need to counter that fearful emotion. Hence coffee and halibut.
You beat me to it. You see this all the time in my field (teaching). I say, “Kids these days are not alright,” and then someone pulls up a quote from Plato saying that the kids in his day weren’t alright either, as if the fact that someone in the past was concerned about young people proves that any problems today are illusory.
Yes, also some people think it is cool to be one of the “fearless” people.
https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F023429c6-7a12-43ff-a56f-57f15e260f51_954x542.jpeg
I completely agree with your point - but disagree with your example of it! The point of the Plato quote isn't to say "kids in classical Greece weren't okay" halibut-style; the point is that Plato's description of kids is *so similar* to present-day descriptions of kids that people literally can't tell whether the kids being described come from classical Greece or present-day Europe unless they're told the origin of the quote - and therefore there probably isn't a downward trend.
A more interesting argument that can be made from the same observation is that the reason why Plato's description is so uncannily similar to ours is that "how kids look to an elder" is a sort of constant, and kids will always look that way to elders no matter what the kids are actually like, therefore we need proper metrics rather than just elders' impressions.
(Not endorsing any of these arguments myself, by-the-by, just setting them out because I think you aren't doing them justice!)
It might sound similar because that quote isn’t from Plato but is from someone’s dissertation a hundred years ago.
Oh wow, I never knew that. Well-caught!
I'm not sure I know it now.
Which one are you thinking of? Because I’ve seen people quoting Plato (framed as Socrates):
> The teacher in such case fears and fawns upon the pupils, and the pupils pay no heed to the teacher or to their overseers either. And in general the young ape their elders and vie with them in speech and action, while the old, accommodating themselves to the young, are full of pleasantry and graciousness, imitating the young for fear they may be thought disagreeable and authoritative.
http://www.perseus.tufts.edu/hopper/text?doc=Plat.+Rep.+8.562&fromdoc=Perseus%3Atext%3A1999.01.0168#note-link1
But that quote isn't a complaint about "kids these days", it's just a description of a hypothetical scenario.
I know the Plato quote is fake, but it's worth remembering that Plato lived at a time of great Athenian declinel. The kids of his day were the ones who would grow up to lead Athens into a bunch of badly-concieved wars that would end with Athens as a Macedonian vassal. The philosophers who came before him were having civilised dialogues, the philosophers twenty years younger than him were lying around in the street wearing barrels.
<i>A more interesting argument that can be made from the same observation is that the reason why Plato's description is so uncannily similar to ours is that "how kids look to an elder" is a sort of constant, and kids will always look that way to elders no matter what the kids are actually like, therefore we need proper metrics rather than just elders' impressions.</i>
(a) That quote is fake anyway.
(b) Athens suffered a decline in power and importance during the fourth century which it never recovered from, so "Fourth-century Athens was in a simular situation to us" isn't quite as reassuring as the quotemongers seem to think.
(c) Even if the quote were actually true, you'd still need way more datapoints to show that ""how kids look to an elder" is a sort of constant", and the mere fact that everybody always brings up this one specific quote should make you suspicious that these other datapoints don't actually exist.
Agree with (a) and (b), but for (c) I think we actually do have some datapoints for the phenomenon: our experience of how we were regarded by elders when we were kids, and our experience of how we find ourselves regarding kids now that we're elders ourselves.
I do think there are also other works of literature that reference this phenomenon - Wodehouse and Jerome K. Jerome spring to mind, but I'm sure there are others besides - and suspect the "Plato" quote is favoured simply because it appears to show the phenomenon being present over the longest timescale (and possibly because Plato has considerably more cultural clout than P.G. Wodehouse..)
That being said: I'm personally not convinced the phenomenon actually exists - in other words I'm not convinced that "how kids look to an elder" actually is a constant - I'm just suggesting that, should one wish to argue that it does, one has more ammunition than simply that "Plato" quote.
The other literature you mention is all from the post-industrial West, which is historically unusual in many respects. To show that the phenomenon is actually universal, we'd have to look at date from a variety of different societies and time periods.
The "Plato" quote, IIRC goes on about how kids are lazy and uncouth, disrespectful and slovenly.
Kids these days are anxious and depressed. Is different.
A roundup of ancient complaints about youth.
https://history.stackexchange.com/questions/28169/what-is-the-oldest-authentic-example-of-people-complaining-about-modern-times-an
I always took that to prove that kids are perpetually NOT alright. Most people make their worst mistakes between aged 15 and 22.
"The Winter's Tale":
"I would there were no age between ten and three-and-twenty, or that youth would sleep out the rest, for there is nothing in the between but getting wenches with child, wronging the ancientry, stealing, fighting — Hark you now. Would any but these boiled brains of nineteen and two-and-twenty hunt this weather? They have scared away two of my best sheep, which I fear the wolf will sooner find than the master."
"Wronging the ancientry" is a GREAT phrase! Words to live by.
"Wronging the ancientry"
Sir, your comment is quite ambiguous. Are you for or against such wronging? ;-)
Many thanks for such a wonderful quotation.
“They have scared away two of my best sheep” —now that's a tragedy.
Maybe they were influencer sheep. Sheep tend to flock, after all. If the best sheep go, the remainder are soon to follow.
I read a book where a character was complaining about youth. "At age 10 they should be shut up in a barrel and fed through the bung hole. At 18 you decide whether to let them out or drive in the bung."
I read some strange books.
Heinlein. I think it's one of the Lazarus Long quotes (but I'm willing to be corrected). It's been misattributed to Mark Twain. And the lower and upper bounds of the age range change as it's been misquoted.
Vaguely surprised it isn't Oscar Wilde, it has that smart guy attitude.
OTOH, Wilde probably wouldn't get within a hundred miles of child raising.
In my experience, 9-year-old Boy Scouts are the worst people on the planet. I run when I hear them coming—they are quite audible at a distanced.
Attributed to Mark Twain
The real issue is culture drift, as Hanson has been exploring lately. The kids have, in fact, been getting worse continuously for several millennia, by all the standards both originally and incrementally. If you think they're getting better (or at least were until you were full grown) you can make an argument for that.
>The kids have, in fact, been getting worse continuously for several millennia, by all the standards both originally and incrementally.
Kids of the last couple of generations are stronger, smarter, faster, read more and write more than almost any generation before them. to the people of ancient Athens on average the kids of today would be unusually tall, buff, intellectually capable and their skin unblemished by pox-marks.
most of that is due to them being one of the few generations in history to get such good nutrition ,such broad access to information and lower disease burden but still.
https://xkcd.com/1414/
https://www.smbc-comics.com/index.php?db=comics&id=2337
I started out being mildly disappointed by your comment, then visually skipped down to where you were posting xkcd and smbc links and thought "oh yeah, ok".
No, friend, you skipped the actual point, which is where the standards are coming from. Yes, by your standards today being taller and handsomer and smarter is worth being utterly corrupt morally and worthless in terms of piety or physical courage.
I'm sorry that you hallucinated that I said "worse by all standards" but that's on you. Maybe reading comprehension hasn't improved by as much as people who make comics for a living think.
"by all the standards"
Are we talking about the same athenians?
They blurred the line between morality and physical beauty so much that a sexy person stripping was a plausible defence because beauty was seen as a sign of having the favour of the gods.
And of course they'd view us as awful in some ways, we don't worship their gods, we don't follow their social customs. Awful!
The point is that social customs are always changing, generation after generation.
Some people interpret change as social decay, they see the youth are moving away from the almost-random set of morals they themselves embraced on the day when society finally mostly figured out morality just as they hit young-adulthood and decide that means society is going down the toilet.
Though that would of course also apply if time were reversed. Someone brought up in the 18th century catholic church faced with a future that included the Sacred Band of Thebes would see that as a sign of awful moral decay.
Because socially popular morality is an almost random walk over time. As flighty as fashion. depart in either direction through time and pretty soon society starts to look terribly immoral.
Now it sounds like you're partly agreeing with me, so I'll be charitable. Yes, all cultures drift over time (Robin Hanson has been writing some great stuff on this very recently). But specifically by the standards of BC 3000 the ancient Athenians were already hopelessly degenerated in ways at least analogous to how we are hopelessly degenerated by Athenian standards. If you just average the values of the human population of, say, the steppes on westward every 500 years or so, you'd observe a slow decline calibrated by the original standards of, say, the Yamnaya or by increments (each 500 year block looking at the more recent).
But no, I'm not talking about surface-level ideology (like which gods or what customs). Your bit about interpreting change as decay is the bog-standard explanation and requires no elaboration. I'm making a different point, which is that actual substantive change has been happening over time, and that this change would be seen as bad by people from the past. We think of ourselves as better than the people in the past - you wouldn't lynch a black man for being seen with a white woman, or (further back) capture an escaped slave at gunpoint to return him to the authorities. But -could- you, if you wanted to? Probably not. You definitely couldn't row a longship up an estuary to kill armed men, or live out of a wagon while conquering your way across Eurasia on horseback. We'd like to paper over this, but the fact is that our own ancestors would think we suck - unblooded boys who in their manners and customs are more like girls even than boys. And that's fine. That's what becoming civilized looks like. But let's not kid ourselves about "lol the ancients were just boomers who though the kids were no good".
Have the kids ever been alright? They certainly weren't when I was in high school...
I think that was Cicero?
Sure, it’s no rebuttal to a general argument that Kids Aren’t Alright. But I think he’s fair game when people are arguing about a specific failure mode. Specifically that the kids are a certain kind of wrong.
https://www.themotte.org/post/890/culture-war-roundup-for-the-week/192595?context=8#context
Good job patting yourself on the back for having explained away the outgroup's behaviour as a consequence of them having characteristic personality flaws?
The intro to Scott's post only works due to the arbitrary restriction to technologies in
> Maybe the argument is a failed attempt to gesture at a principle of “most technologies don’t go wrong”?
Drop the qualifier that the thing to go wrong must be technological, and you are one step away from a good candidate for a heuristic that people in the real world actually use - "most threats drummed up by people who stand to gain from you believing them don't play out". Surely this heuristic has its utility, especially if the Cassandra is sufficiently socially removed from the target that she could shrug off being revealed as a fraud later. The prototypical catechism on applying it takes the form of "sketchy outsider tries to scare you and then make you buy his product" - think Melville's "Lightning-rod Man" (which I actually had as school literature, for English class in Germany). I'd reckon this is why this anti-AI-risk argument is also often accompanied by hinting that AI safetyists are icky.
It's also unsurprising that people would feel betrayed if you pulled a reverse card on the heuristic - these sorts of simple rules are often perceived by the holder as hard-earned pieces of life wisdom that comprise their advantage over the younger and more naive, and to insinuate that they actually don't give an unambiguous answer is to devalue them and by extension the advantage.
Cassadra is an odd thing to call the type person you are describing, given that part of her curse was that she was always actually right.
Yeah! Surely there were other prophets who were wrong and unpunished! Use those guys as examples!
Oh wait, you can't because they were WRONG.
The classic modern example of the character you're describing is the con man "Professor" Harold Hill, from the musical "The Music Man", who drums up a moral panic over any arbitrary recent change in order to sell them a bogus solution:
https://youtu.be/LI_Oe-jtgdI?si=ffuhyZe1rD9GBrTB
Oh, we've got Trouble
Right here in River City
With a capital T and that rhymes with P and that stands for Pool!
Many Thanks! That song always come to my mind in any discussion about moral panics. :-)
Not necessarily sub-conscious. Putting on your Talmudic thinking cap, first you decide what the right answer should be based on your experience, or what you think the world should be like, and then choose your evidence.
This is definitely correct, but I would say all reasoning is motivated, so you still have to look a little deeper to figure out what makes the motivation or reasoning Bad. Comparing the Coffeepocalypse arguments to Scott's counterarguments, obviously we both think Scott's reasoning is better and more convincing, but why? Well, he's goes through the steps of considering counterarguments and examples and then dismissing them for coherent reasons...but what makes those reasons/reasoning coherent? Well, I think one of Scott's main motivations is to have a worldview where his beliefs about how the World works actually align with how the world really works. In other words, his motivation is primarily to "Truth."
Of course, we also know Scott also has a higher prior than many that any sufficiently advanced AI will destroy or enfeeble Humanity, and he is also motivated by that. I think the Coffeepocalypse arguments are intended to lower that prior; its not convincing because the reasoning is Bad and someone of Scott's caliber will see right through that. But most people are not up to Scott's caliber of reasoning and therefore may lower their prior for AGI bringing Ruin, and thus will be less motivated to resist it. Again, I don't think their arguments are intended to be consistent, they're just trying to move priors one way or another. They don't work on me, but mostly because I share the prior that AGI is potentially dangerous, and one of my main motivations is having a worldview that would be accepted and explicable to Scott Alexander and his readers.
I think the entire argument is against expertise, especially regarding "new technologies". It's "Experts tell you something, but look, here experts were wrong about something similar, so just distrust experts in general".
And people more vividly remember "experts were humiliatingly wrong" stories than "experts warned us of something and managed to prevent it" stories. Also more than "experts didn't warn us and some disaster happened", those stories are least memorable cause there's no heroic protagonist - there's a lack of one.
I feel like if this was the argument, they would have made more of an effort to show that it was "experts" who were against coffee. Instead they blame kings, queens, and alcohol vendors.
At least to me these register as critiques not so much against experts, but towards predictions in general. ”People are bad at predicting things. Nothing ever happens, so you shouldn’t worry.”
My new motto.
I'm thinking of the person I know who always make a similar argument, and I think she aims it at the conjunction of experts + long term predictions, especially catastrophic ones.
Scott wrote:
"And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS)."
The person I'thinking about would say that none of those were predicted enough in advance to qualify as long term predictions.
There's the notorious documentary, filmed in 2019 and released on Netflix in January of 2020, that went into detail about how the Wuhan wet market was a very good candidate for the origin of the world's next big pandemic...
It’s worse than that. People are bad at predicting things, what they predict usually doesn’t happen, and when big things happen they usually weren’t predicted.
So, by this logic predictions of AI doom should make me update that it’s *less* likely to happen!
In the coffee example, you're right. But the usual argument is usually framed around experts.
Also, experts might be the wrong term here. Maybe it's closer to "elites", which Kings and Queens definitely fall under. I could imagine someone saying "President Clinton claimed X but actually Y happened", about, say, the economy. And this would count as proof that elites get things wrong, even if Clinton isn't actually an "economic expert".
There are much more concrete examples re. the economy. President Obama famously claimed that, without his proposed stimulus, unemployment would reach 8%, but the stimulus would prevent this result. We got his stimulus, and unemployment went over 10%, and stayed above 8% for two years. Now, I don't know whether any (let alone most) economists would have supported his claim in 2009, so this doesn't exactly reflect on economists' capabilities. But it sure does reflect on politicians' and other "thought leaders'" use of expertise claims to advance their agenda.
To be fair, presidents have a huge incentive to express overconfidence in their policies, since it directly relates to their electability. Unfortunately, we don't live in a society epistemically humble enough that people are more likely to vote for a candidate who expresses "The best evidence I have suggests this will probably help."
To an extent, what presidents try to do probably reflects what they think are good ideas. But what they say very likely isn't an accurate reflection of what they think.
Sure - politicians do politics. Maybe, sometimes, they really think their policies will benefit their constituents in significant ways (as opposed to getting more votes to get power, or to stay in power).
But my point was that people who claim the authority of experts to advance their agendas deserve fully as much skepticism as people who advance their agendas based on no support from anyone.
I know it's tangential, but creating a forecast that underestimated the unemployment rate by 2% doesn't strike me as an especially damning indictment of economists. I'm happy they got it within 5%.
The claim you attribute to Obama may be famous, but I haven’t seen any evidence that he actually made that claim.
If I may offer a guess as to what you are referring to: On January 9, 2009, Christina Romer and Jared Bernstein were not working in the White House but they had a semi-official status because they had been chosen for jobs in the White House starting on January 20, when Obama would be inaugurated. They wrote a paper projecting the effects of a stimulus package, which included a graph of projecting unemployment with and without a stimulus. It showed unemployment topping out at 8% with a stimulus and 9% without.
Their baseline projection turned out to be wrong: in February 2009, when the actual stimulus package passed, unemployment was already at 8.3%.
OK - so it was (some) economists.
Undermining experts is about undermining their claim to expertise. Hence comparing them to non-expert authority figures.
I almost take it as the opposite: moral panics by people with little-to-no understanding of the issue, but high status, are common. As an example of where this argument somewhat works, I'd provide most "normal" concerns about AI (e.g. taking jobs or misinformation).
Where it falls flat is in explaining Geoffrey Hinton, Stuart Russell, Yoshua Bengio, etc.
If you tried to make the analogy closer to Hinton, Russell, and Bengio you'd get the "moral panic" raised by Manhattan Project scientists relating to the nuclear bomb potentially resulting in human extinction.
Re: "the experts" - did you take counterfactuals into consideration, as in how planet Earth could have been had we not placed so much faith in experts, and thus perhaps had taken a less foolish path(s)?
This counterfactual is pretty easy, without trusting experts we'd still be living in caves. We especially trust experts every day of our lives. I flew on a plane just the other day!
What is it that you are describing, technically?
It's a big pretty white plane with red stripes, curtains in the windows and wheels and it looks like a big Tylenol.
- Stephen Stucker, Airplane!, 1980
>This counterfactual is pretty easy, without trusting experts we'd still be living in caves.
Well said, and quite literally true! Every time I step into a building, I'm relying on the structural engineers and the people who designed and ran all the processing steps for all of the materials in the building. _I_ certainly don't have the relevant tensile and compressive strengths and safety margins at hand.
I disagree with the phrase "humiliatingly wrong" because it implies that these people suffer some kind of reputational damage.
More explicitly than just “experts,” it’s distrust of Silicon Valley tech bros who want to feel important and be at the center of the narrative. Crypto hype is fresh on everyone’s minds - the blockchain bros cried wolf, and then when everyone came to look and the wolf wasn’t there they said, “no wait you just can’t see it, only we can see it because we’re better/smarter/more technical than you!”
Most of the media is buying the AI-sooner narrative because it gets clicks, but people are also understandably exasperated at media and negativity bias and doomerism in general. There are a LOT of reasons to be skeptical of the incentives of everyone hyping this narrative, because they’re people/industries who desperately want/need to be the center of attention.
Bitcoin is still bullshit (its primary use case is facilitating illegal transactions, aka literally committing crimes), but it continues to remain expensive bullshit.
Even worse, people remember "experts warned us of something and managed to prevent it" stories AS "experts were humiliatingly wrong". Climate change deniers will cite ozone layer depletion and Y2K as examples of the experts warning about things that didn't happen.
Y2K didn't happen because people spent a lot of time and money to fix it.
Yes, that's my point. But some people see it as an example of a baseless scare.
As the Y2K deadline approached, not many people, including elites or experts, were saying that the problem was largely fixed, or that there wouldn’t be serious if not catastrophic problems because the appropriate people had taken appropriate action. In retrospect, the lesson seems to be that the expenditure of money, time, and effort to avoid catastrophe was sufficient to avoid most serious problems. (Of course, we can’t run the counterfactuals of spending 10% or 20% or 50% less time, money, and effort.) In any case, back in late 1999, there were still plenty of Y2K doomers who had more-or-less sophisticated (but, in retrospect, incorrect) arguments about why the Y2K problems had not been – and even “could not be” - adequately addressed. Still - it’s hard to know how to apply the lessons of Y2K to AI.
Yup. At the time, I thought that there was a lower bound on the amount of damage that Y2K would do, from the roughly 25% of the economy (roughly half of small businesses) that did nothing to avoid the problem. I was wrong - and I'm still not sure why the final impact, even on that sector, was so small. I've never been so happy to _be_ wrong.
That's interesting - is it possible that small businesses mainly used off the shelf software that had been fixed by the vendor, so even if the business didn't care about Y2K, if they happened to update their software before the millennium, it got fixed anyway? But then it's hard to imagine businesses that didn't care about Y2K would be diligent about software updates, especially because updating would be more manual and complicated back then.
Or maybe organisations that were small enough not to have done anything were also not that reliant on computers anyway, so when they ran into problems they could just do things manually?
Many Thanks!.
>But then it's hard to imagine businesses that didn't care about Y2K would be diligent about software updates, especially because updating would be more manual and complicated back then.
Agreed!
>Or maybe organisations that were small enough not to have done anything were also not that reliant on computers anyway, so when they ran into problems they could just do things manually?
Could be, though even tiny organizations generally need to send out invoices and keep books.
I suggest that these are people experiencing epistemic learned helplessness.
cf. https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/
> "And there are people who can argue circles around me. Maybe not on every topic, but on topics where they are experts and have spent their whole lives honing their arguments. When I was young I used to read pseudohistory books; Immanuel Velikovsky’s Ages in Chaos is a good example of the best this genre has to offer. I read it and it seemed so obviously correct, so perfect, that I could barely bring myself to bother to search out rebuttals."
> "And then I read the rebuttals, and they were so obviously correct, so devastating, that I couldn’t believe I had ever been so dumb as to believe Velikovsky."
> "And then I read the rebuttals to the rebuttals, and they were so obviously correct that I felt silly for ever doubting."
Presented with a flood of logical steps by someone capable of arguing circles around me, adding up to a conclusion that is deeply counterintuitive; and presented perhaps also with a similarly daunting flood of logical steps by someone else also capable of arguing circles around me, adding up just as convincingly to the opposite conclusion... what is one to do?
One can sway back and forth, like a reed in the wind, according to which super convincing argument one last read.
Or one can throw one's hands up in the air and say: "There are hundreds of incredibly specific assumptions combined with many logical steps here. They all look pretty convincing to me; and actually, so do the other guy's. So clearly someone made a mistake somewhere, possibly more than one person and more than one mistake; certainly that's more likely than both the mutually contradictory yet convincing arguments being right simultaneously; but I just can't see it. What now? What other information do I have? Well, approximately ten gazillion people have predicted world-ending events in the past, and yet here we all are, existing. So I conclude it's much more likely that the weirder conclusion is the one that's wrong."
Condense that to a sentence or two, couple with an average rather than expert ability to articulate, and you arrive at coffeepocalypse.
From the above essay:
> "Even the smartest people I know have a commendable tendency not to take certain ideas seriously. Bostrom’s simulation argument, the anthropic doomsday argument, Pascal’s Mugging – I’ve never heard anyone give a coherent argument against any of these, but I’ve also never met anyone who fully accepts them and lives life according to their implications."
AIpocalypse is just another idea to add to that list.
People who believe in AI doom, even a moderate amount (more than 1% chance) seem to me to be invoking a Pascal's Mugging, but vehemently disagree with my conclusion that they're similar.
In both cases I agree with you that nobody seems to have a good counterargument against AI doom/Pascal but unless they already agree with the conclusions are not at all *convinced* by that argument.
Telling me there's a non-zero chance of AI doom doesn't tell me how I should or shouldn't lead my life. There's a non-zero chance that an asteroid will destroy the planet this year as well, but almost nobody treats the possibility of that as a legitimate reason to live in fear or build underground bunkers.
Because total destruction of everything we value is part of the equation, all answers to the question of "how seriously should we take this" comes out to infinity for any doomsday prediction. We obviously cannot react to every such infinite concern with proportionate resources (we can't go spending all of earth's resources every day), and for every doomsday except one that we ourselves agree with, we think the actual appropriate level of resources to spend is in fact close to zero. The problem is multiplying by infinity, and the solution is to take infinity out of the equation. People who agree with the infiniteness of the problem don't do that, come up with infinite in the equation, and wonder why everyone else doesn't get the same answer.
a >1% chance of something happening isn't a Pascal's Mugging, it's a "it will happen at least one time out of 100". A Planet-killer asteroid hitting earth has an incredibly lower chance of happening anytime soon, and yet people do seriously worry about that (see eg https://en.wikipedia.org/wiki/Asteroid_impact_prediction).
We actually have *NO* idea how likely AI doom is. It could in fact be far far less likely than an asteroid. Also, with asteroids we can try to game out the possibility of one hitting us and wiping our humanity, but even knowing it's very unlikely doesn't tell us it's impossible even on short timeframes.
That's why I refer to people who believe AI doom may happen (even if they also think it pretty unlikely) compared to people who don't. Pascal's wager was originally about believing in God, and how it makes sense to believe in God even if the probability of God existing was low. As someone who already believes in God this argument makes perfect sense to me! But, I totally understand why this is not at all convincing to someone who doesn't already believe in God.
>We actually have *NO* idea how likely AI doom is
Then you are just arguing probability numbers. This has nothing to do with multiplying by infinity, people who argue coherently for AI risk do so because they assign a high probability of AI doom comiing to pass - not primarily because the outcome is infinitely negative.
People who are worried about infinite negative utility are clearly making a Pascal-class mistake, and it is not a correct argument against AI.
We have evidence for large asteroids hitting Earth in the past, so there is a non-zero probability of one hitting Earth in the future. You can argue about how likely and how soon, but you cannot argue with non-zero.
We have *no* prior evidence for AI, or AI doom. Saying AI is like humans is just wrong; we're pretty sure that we're not replicating human thinking in there.
So there's a more uphill battle there.
Do we have evidence of AI being smarter than humans? Say in Chess or Go? Or maybe image recognition? Do we have evidence in our historical record of one intelligence wiping out another intelligence?
On top of that, I'd argue that we have a lot of evidence *against* the possibility of functionally omniscient/omnipotent entities.
https://astralcodexten.substack.com/p/the-phrase-no-evidence-is-a-red-flag
I'm saying that their belief that AI doom is likely is unjustified. They cannot provide evidence that the chances are actually high, because it's an unprecedented question that depends on future technological advancements that we cannot know are even possible. Recursive self improvement is an idea that may not may not be true, not a fact that we can plan around.
I believe that the counter-argument to the fact that AI doom is an unjustified belief is to talk about how serious it would be for us, if true. That's Pascal.
The issue I've come up against is that the people who argue for catastrophic AI risk tend to disregard other catastrophic risks that strong AI would hedge against. AI strong enough to pose an existential risk could also help fight climate change, find alternative energy sources, reduce car deaths, find cures for cancer, eliminate the need for human drudgery, and do polygenic screening which, when combined with embryonic selection, could reduce illness and death in general across a population. There is currently a 100% chance you will die at some point. What weight should you assign to addressing that issue? The accelerationist argument is almost always framed as 'accelerationists hate human beings' which is a severe weakmanning of the accelerationist arguments.
I do think we should be arguing relative probability numbers as well as we're able to. But in practice, infinity or no, people strongly tend to choose one goal or catastrophe and let it fill their entire mental field of vision. Most people just REALLY don't like to have to juggle 7 mental balls or consider that multiple inputs may contribute to a single result.
> There is currently a 100% chance you will die at some point.
Well, the heat death of the universe is probably going to be a problem, but the chances of a well-preserved cryonics patient being restored to health in some form or other is greater than zero.
Dunno about cancer and polygenic screening, but I think we've got adequate non-AI-based options lined up for energy, climate change, drudgery, homelessness, and car deaths, it's mostly just a matter of legislation and budgeting at this point.
First, broad regulatory exemptions for solar PV (and, optionally, wind) construction. Anybody objecting on environmental-impact grounds needs to show that a given project would somehow be worse than an equivalent coal plant, at least until there are literally no more grid-scale coal-burning generators operating anywhere in the world.
Second, Georgist tax on land value, replacing most other taxes.
Third, GPS tracker (and, optionally, ignition-lockout breathalyzer) as mandatory automotive safety equipment. All roads become hassle-free toll roads, no more DUI hit-and-runs, no more selective enforcement of traffic laws based on individual biases. Urban planners could experiment with complex schemes of road-use fees varying by time of day - or even outright banning cars from particular areas - at the push of a button, confident that an updated map would be on everyone's dash faster than you can say "in fifty yards, turn left."
Revenue would increase while enforcement costs plummet. Spend the initial wave of money (and surplus solar and wind power) on accelerated weathering: grind up a mountain's worth of olivine and dump the resulting powder in the ocean, thereby absorbing excess CO2.
Once that situation is stabilized, celebrate by implementing universal basic income. Anyone who doesn't like drudgery will spend some of their share on avoiding it, thus bidding up the price of that dwindling supply of drudgery-tolerant labor, until most dirty jobs cross the threshold of "more cost-effective to automate away," while remaining automation-resistant ones pay so well as to be prestigious by default.
Of course, people won't just sit around idle. They'll use that security and abundance to start systematically solving all the problems they were previously obligated to ignore, due to being too busy flattering some middle-manager's ego in exchange for subsistence, or too broke to do anything but get in arguments on the internet.
That we have no idea how likely it is is certainly in dispute: https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist
I like your last argument. Very clearly no one actually believes AI will cause the death of all humans because their behavior would be radically changed.
>because their behavior would be radically changed
It depends on the time scale and the probability and the person's time horizon.
I'm 65, so I don't lose sleep over much that happens more than 20 years from now. My default assumption is AGI succeeding and probably (with _HUGE_ uncertainties) eventually replacing humans (but possibly preserving large chunks of our culture).
I took a two thirds paycut and moved from a place I liked more to a place that I liked less in order to do my part about preventing the AI apocalypse; I think that counts as strong evidence that I believe in it.
[Why didn't I do <other thing you think AI risk implies>? Well, probably because I didn't think AI risk implied that thing, or it doesn't match up with my personal competencies, or whatever.]
Increasingly many people in the EA AI safety community have taken fairly drastic actions. But you're right that basically none of them think it's a 100%, or even >95%, certainty, and the possibility that they'll have to live with the consequences of their actions keeps them from doing anything even more drastic (e.g. spending all their money or getting themselves imprisoned or...)
...what?
every time I see someone make this argument, the actual changes they recommend are pretty nonsensical
the truth is, i have absolutely no idea what to do with this information. every "extreme action justified by extreme threat" i can think of has a negative expected value
i suspect those are the same extreme actions you are thinking of, but since you don't actually believe, you don't feel obligated to continue the chain of thought to its conclusion
Pascal's mugging is, and always has been, about prospect theory and human psychology. The solution is very simple: stop assigning non-negligible probabilities to extremely unlikely events. Pascal's mugging is about a problem with human psychology—it shows that drawing our attention to an extremely unlikely, but potentially highly-important event, causes us to hugely overestimate its probability.
I think the steelman for coffee or "failed predictions of disaster" arguments is based on Lindy Effect sort of things (well, humanity's been around a quarter million years, plagues, volcanoes, hurricanes, tidal waves, wars and genocides have all happened and we're still here, sooo....), and maybe for some few better read and better epistemic-ed arguers, pointing to nuclear-disaster style arguments about how hard it would REALLY be to actually kill every human.
I personally don't buy either of these arguments (for one thing, we've been through at least one and probably two significant "close to extinction" genetic bottlenecks during our time as humans https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2842629/), but they're a better class of argument than "we don't worry about halibuts" or whatever.
I don't think this study really relates to the question of human extinction - reading through it it seems to be more about the "genetic bottleneck" of a small group of humans moving either out of Africa or across the Bering Strait, not all of humanity being close to extinction ("we" haven't been through a bottleneck if "we" includes Africans).
You know, you're totally right, thank you. On further research, it looks like the bottleneck I was thinking of was around 900k years ago, before we were human, and the population went from ~100k individuals to ~1k, and stayed low for ~100k years. So still a "close to extinction" bottleneck, but for human ancestors vs anatomically modern humans.
I think the strongman of the argument they are making is, for almost every new technology there have been people making dire warnings and apocalyptic predictions about it (e.g. trains would kill everyone over a certain speed, video games corrupt our youth, nuclear power etc).
These arguments are easy to make because new technologies are scary to many people and it is very hard to definitively show something that is new and unproven is safe. Nearly all of these arguments have turned out to be mostly or completely wrong.
Therefore when someone says AI could kill everyone, your prior should be that it almost certainly won’t.
Whilst I can broadly accept that, it isn’t an argument to just ignore evidence about AI, just that before you examine the subject that should be your starting point.
You could also make the bolder claim that the burden of proof is on those claiming AI will cause calamity, given how often those sorts of arguments have been made and how beneficial new technologies have been on average. But I’m not sure I would go that far.
I think its even broader than this. There is a market for consuming doomsday prophesy. So, writers produce doomsday prophesy and are incentivized to make it as plausible as possible to gain market share. This has shifted over my lifetime from nuclear war, ozone, global warming, climate change, COVID, and AI. If you put my casual knowledge against any of the top writers and say, "respond to my specific arguments otherwise you are wrong", I loose every time.
In the paradigm, most doomsday authors will switch between the most salient doomsday narrative over time. And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI. I think their COVID record is good evidence that they are overreacting to AI. The historical overreaction to coffee shows that this is a normal thing to happen in societies.
> If you put my casual knowledge against any of the top writers and say, "respond to my specific arguments otherwise you are wrong", I loose every time.
+1. This is the core of it. I'm reminded of conspiracy theorists who insist that you're not entitled to disagree with them until you've watched [some random 4h long youtube video], or read Marx cover to cover.
Sneering about epistemic learned helplessness is dishonest, when it just isn't practical for me to spend months looking into everything in detail. I do take AI risk seriously, and I've engaged (admittedly somewhat superficially) with many of the arguments. What else do you want me to do?
As a conspiracy theorist, I have the same complaint about Rationalists, Normies, and The Experts.
> What else do you want me to do?
I would like you to wonder what is true, using an appropriate form of logic. Knowledge that there is more than one form is typically only cognitively available when discussing logic itself directly/abstractly, and this knowledge availability phenomenon applies to many things other than types of logic.
> As a conspiracy theorist, I have the same complaint about Rationalists, Normies, and The Experts.
Exactly. In the end we all have to apply heuristics to decide what is worth paying attention to.
> I would like you to wonder what is true
I do. But I am a very small being in a very large world.
> In the end we all have to apply heuristics to decide what is worth paying attention to.
This is so? There is no alternative?
> I do.
This is so? Is "do" a binary?
>This is so? There is no alternative?<
The alternative is "pay attention to things at complete random". You have finite scope, your choice is how to aim.
Yes, You get me to come back tomorrow, if you tell me how bad things will be. In my old age I find I've become less and less certain about anything. I don't know enough to have an opinion either way. And so for most things, "I don't know", is the right response. And arguing with other people who mostly know nothing is just a waste of my time. So why do it?
Because the facts in your last sentence are actually predictions/opinions?
Because if all the wisest people take this option, all the actual actions will get taken by unwise people, and I for one don't want to live in that world.
(Cf that one line in HPMOR about how "good people" are often too busy wringing their hands over what is ethical to take actually helpful actions)
Do you have any examples of any of the people you are talking about (or anyone prominent, really) actually saying covid would end the world (a "doomsday prophecy")? It's been a few years, but my memory from early in the pandemic is a lot heavier on people who said things like "it will be over by May" and "I'd be surprised if there are even 10,000 deaths in the United States" than people who overestimated the actual number of deaths, which is in the millions in the U.S.
I also am pretty sure Scott and Yud kept doing AI posts in 2020-2021!
Just re-read Scotts post from March 2020.
https://slatestarcodex.com/2020/03/02/coronavirus-links-speculation-open-thread/
Scott advocates for: healthy quarantine, fomite prevention measures, home healthcare, praises Chinese lockdowns, advocates for contact tracing, wearing masks, wearing full respirators, and zinc supplements. He mostly ignores costs associated with these measures. Its an impressive list of predictions that would be adopted by global policy makers while also being ineffective and in some cases disasterous, especially to the global poor.
Meant "doomsday" as people predicting doom, not the literal end of the world. Maybe I am using the word incorrectly.
If everyone had followed those recommendations in March 2020 then we probably would have eradicated the virus entirely by July 2020 and got on with the rest of our lives, with far less total disruption, probably with fewer total days of lockdown.
Nearly every government on earth tried this and failed.
No, they tried something that looked vaguely like it and failed. Locking down until eradication (and then keeping borders closed) was the best move at the time.
Locking down until almost-eradication is a huge fucking waste of time and money which just happens to look a lot like locking down until eradication.
>than people who overestimated the actual number of deaths
I was an (obscure!) overestimator.
In the early days, when, at one point, the death rate looked like 7% and the fraction of the population who needed to have gotten it and gotten immune to stop the pandemic needed to be around 50%, I expected the worldwide death toll to be on the order of 2.8x10^8. I was wrong. The final toll seems to be about 40X. smaller https://www.worldometers.info/coronavirus/ gives 7x10^6
"Doomsday prophecy" is perhaps hyperbolic, but
But Scott was wearing a full respirator out in public in maybe Feb 2020. I didn't keep up with their output at the time, but my sense was that they were advocating for more caution and for lots of interventions social distancing, travel restrictions, healthy quarantine, vaccination, boosting, etc. But then corrected faster than other authors when these ended up not doing much.
I do expect a similar course with AI.
> And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI. I think their COVID record is good evidence that they are overreacting to AI.
This seems like a pretty unkind thing to say, so it better be true. Do you have any actual examples of these people making "doomsday prophesies" about COVID?
The term "doomsday prophesy" is unfair, and unkind, and overly provacative. I like Scott and find him interesting and worth reading.
As for "overreacting", I think the history speaks for it's self. Here is Scott in march 2020 very intelligently advocating for all the ineffectual and counterproductive NPIs that became public policy.
https://slatestarcodex.com/2020/03/02/coronavirus-links-speculation-open-thread/
I don't think that post matches the way you describe it here, let alone the sort of thing you claimed in your original comment. Most of it seems to hold up pretty well even in hindsight, let alone compared to the level of information that was available at the time. In general the course of the COVID pandemic seems like a big win for Scott and others in the community who were saying early on, "hey this could be a really big deal" over the people who were saying it was nothing to worry about.
Can you point to a specific claim in the post you linked that seems like a hysterical overreaction?
His advocacy for healthy quarantine, is a a hysterical overreaction. This was easily the worst public policy decision of the pandemic.
"The story I’m hearing from most smart people is that China has done amazing work and mostly halted their epidemic. This is very impressive...Nobody’s too optimistic that democratic countries can follow their lead, though"
There is also a section titled: "Self-quarantine starting now vs. later?"
China's early coronavirus response worked, as far as I can tell. Where they dropped the ball was that they didn't distribute the mRNA and other vaccines in favor of a Chinese one that was significantly less effective and continued to have a strict "zero COVID" lockdown policy long after vaccinations with Western vaccines would have made it no longer worthwhile.
"advocacy for" is a bit of a stretch relative to the actual content of the post IMO. But I will grant that that part was a bit overzealous.
Still, given the overall impact of COVID and our society's response to it (whether or not you agree with the latter), Scott comes out of it looking a lot better than the people who were saying it would be no big deal. I don't think it makes a very good argument that he's prone to making mountains out of molehills.
There also a difference between warnings and prophecies. Basically, warnings are conditional, prophecies aren't. If someone says X will happen ,and it doesn't, they were wrong. But if someone. Says "X will happen if we don't do Y", and we did Y, they weren't wrong.
> most doomsday authors will switch between the most salient doomsday narrative over time. And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI.
I don't see what happened as particularly fallacious. They've been concerned about AI, and there isn't a global pandemic, so they just talk about AI. Next, they're concerned about AI, and there's also a global pandemic, so they talk about both. Then the pandemic goes away, and they're back to AI.
These people aren't one-note pundits; they're talking about what's on their mind, and if the entire world is going nuts due to a pandemic, this is going to be reflected in their output, one way or another. And I don't recall any instance of them treating covid as being even a serious threat to human civilization, let alone human existence.
I'd also disagree with the characterization of them as "doomsday authors". Scott certainly isn't: he writes about a lot of stuff. Eliezer probably regrets not being a more effective doomsday prophet about AI, but he's not a general-purpose doomsday prophet: he saw the danger of AI early, and devoted a lot of his life to trying to stop it, and his advocacy is simply a part of that. It's the difference between a war journalist who travels the world from one war zone to another, and a war journalist who only happens to be covering a war because their country is currently being invaded.
> Eliezer probably regrets not being a more effective doomsday prophet
AI doomsayers need a mass market, inexpensive, public ritual, along with some really expensive remedies to really take off. The public ritual keeps the issue, "top of mind" for the public, provides controversies for the popular press to write about, and provides a mechanism for social enforcement. The expensive remedies create profitable businesses who can bribe politicians and fund doomsayers. For climate change these are recycling/green energy. COVID it was masks/medical care. Not sure what it could be for AI, but the first to figure it out makes billions.
So AI doesn't actually fit the pattern/ you are trying to make it fit?
Also masks were never a profitable business, so the model doesn't work there very well.
To be clear, I do think the point that there are lots of doomsayers provides a prior against these sorts of arguments in general, but I think it also clearly does not produce such a strong prior against them that we can safely assume they are all false.
It seems like a situation where skepticism but attention is warranted, and an awareness that there is something in a lot of our minds that really likes believing doomsday arguments, even when they aren't very good.
Having said that, I think the case for AI being dangerous enough that we should be very cautious in developing it is solid, and we probably would agree on a lot of specifics in terms of that if we pinned them down carefully enough.
Broadly agree.
> Also masks were never a profitable business, so the model doesn't work there very well
Masks were the public ritual and had nothing to do with money. The 50K insurance payments for treating patients w/ COVID, vaccines, Remdesivir, etc. were the expensive remedies.
> And, I think this is what we saw with Scott, Yud, etc. They all switched to COVID for a bit, and now that that has lost steam, they are back on AI.
If this is what you sincerely believe, then you fail at understanding quantitative arguments. The worry about AI is that it could kill literally everyone. (For "Scott, Yud, etc."; other people worry about other things, for example AI saying something politically incorrect, or failing to pay artists for using their art as inspiration.)
There was never a similar worry about COVID; instead it was that it could kill a lot of people... and it actually did, but much fewer than was expected. So the "doomsaying" at worst meant overestimating the number of victims by one or two orders of magnitude (and even this only assuming that all the vaccination etc. had zero effect).
There are people out there who worry about crime, car accidents, fires, floods, and various other things that sometimes kill people. Should we also include them to the list of doomsayers who merely follow the popularity incentives?
They overestimated the risks for COVID. So, I think its reasonable to assume they are are overestimating on AI.
I think they have brains that subconsciously fill in holes of uncertainty with doom. It's part of why they are popular, but also leads to overestimation of risk as a power law of uncertainty: 1-2 orders for COVID, ~7 for AI.
Oh c'mon, pretty much everyone "switched to COVID" in 2020-2022, and Scott and Yud and the EAs have been on about AI safety *long* before it was cool
Why wouldn't you go that far? Why wouldn't the burden of proof be on the people making a claim, especially a claim that would necessitate significant changes in how every person on earth lives their lives?
Fair point I guess it depends on definitions, but in my head putting the burden of proof on the AI risk lot (I.e. showing decent chance AI likely to kill us all) would mean at a minimum making them show it to a UK civil trial standards (i.e. more likely than not). I wouldn’t be comfortable for life to go on as normal if there was a 49% chance they were right.
I would go further than that, proof that there's a 5% chance of doom seems like a decent place to put serious effort into preventing catastrophe. I've seen a LOT of conjecture about what the probability of doom is, and a whole lot of reasoning about how it could happen, but just about zero proof. If this was a court of law, the case would be thrown out as conjecture.
We can respond to conjecture, but such a situation doesn't require action, especially specific actions. I happen to think that being pragmatic about limiting AI is a good idea. For instance forbidding recursive training (AI learning from experience to train itself) is probably a good idea. But we should be clear that there's zero proof of AGI being possible, let alone any of the further concerns about how that could lead to doom (often also involving technology we don't know to even be possible, like nanites).
For the record, it wouldn't take sci-fi nanotechnology for an AI to be able to destroy human civilization using only an Internet connection. If enough novel killer viruses struck humanity at the same time, we'd go down like the American Indians did to European diseases. Creating a virus given a DNA sequence is well within the capability of current technology, and there are companies that you can pay to send you custom DNA sequences as long as they don't match a database of known dangerous viruses. If an AI *could* design killer viruses (admittedly a difficult task) and then bribe/threaten/trick people into assembling them (something much easier than designing one), it might not kill literally everyone, but it could certainly leave humanity in no position to oppose it.
zero proof of AGI being possible?
do you not consider humans to be such proof? i certainly do. we seem to be fully generally intelligent, we certainly count as an existential threat to all beings less intelligent than we are, and we sprang out of incremental improvements to non-generally-intelligent designs
i have some trouble understanding people who are not immediately swayed by this argument, which seems utterly ironclad to me, but i guess if anyone has any novel arguments i'd like to hear them
Zero proof of any of the specific things (foom/takeoff, specifically) that would lead to AGI takeover. I didn't say, and wouldn't say, that AGI is impossible.
To put some of my cards on the table, I'm religious, so it's not at all weird to me to consider human intelligence potentially separate from machine intelligence.
Secondly, there is nothing at all weird with saying that we may be on the wrong track for AGI or that true AGI would be far more complex than anything we've even attempted at this point.
Ah, I must have misparsed then
to be fair, you did say "But we should be clear that there's zero proof of AGI being possible"
One reason: because winning arguments and optimal avoidance of risk may look identical, but they are not.
Another reason: you are considering "that would necessitate" to be necessarily factual - this may be fine if you're operating within a thought experiment, but we can't live within thought experiments. Or can we? 🤔
> Why wouldn't the burden of proof be on the people making a claim,
The burden of proof is indeed on those making a claim, which includes those who claim AI will NOT be an existential risk. Agnosticism on the outcome is the right logical position, but then you're still left in the position of evaluating how plausible a danger it is, and thus whether some investment in mitigations are warranted (like the Y2K bug).
> especially a claim that would necessitate significant changes in how every person on earth lives their lives?
AI doesn't exist yet. How could mitigations against something that doesn't yet exist necessitate significant changes in how every person lives their lives?
Well, consider the necessary mitigations. There are people around the world riding hell-bent for leather to produce human or superhuman level AI, and a regime that was capable of suppressing them would be…truly intrusive.
Why is an oppressive regime a necessary mitigation? Why is intensively funding research into AI safety not a sufficient mitigation?
It may not be.
But research into AI safety has not borne much fruit yet — certainly no silver bullet. That very fact might, to some eyes, suggest that slowing down would be prudent, but there is no sign of that happening. So an oppressive regime might seem, to other eyes, the only choice.
There’s also the fact that AI research is pretty easy to hide, compared to, say, nuclear-weapons research. A global compact to restrict it would be so *very* easy to cheat on.
AI safety has also not seen much funding. What's the ratio of funding between "training/developing new AI" vs. "*ensuring* AI outcomes"? 100:1? 1000:1? possibly even worse.
My point is that a lot of people are working on AI so as to achieve specific outcomes. AI safety research generally aligns with the goal of *ensuring* specific outcomes, but it's not typically seen in that light, and most people have instead been content with "has a 2% chance of going wrong". There's value in closing that gap to zero, and I consider that within the realm of AI safety, because what is AI safety if not "does what it's supposed to and doesn't go wrong in unexpected ways"?
People always say that.
But like Making these giant neural nets takes serious compute. It's harder than making drugs, comparable enriching uranium. And the world is trying to suppress the first one, and mostly managing to suppress the second.
Asking someone to prove a negative is bad faith isn't it?
https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)#Proving_a_negative
Russell may not have any reason to believe there is a teapot in orbit, but if someone walks up to him and says there definitely isn't a teapot there, he's absolutely going to ask for evidence.
Which either leaves us with nothing at all to say about anything we cannot directly observe, or working from other principles we can examine more directly.
The likelihood of a teapot being in orbit cannot be considered 50/50 just because someone claims it's there. Unless shown positive reason to expect the teapot to be there, we should start the conversation with something *significantly* less than 50%, likely some small fraction of one percent. This is necessary from logical and practical standpoints. Logical, given we have no mechanism for how a teapot would get there, and practical because we can't respond with action to every assertion made.
Assertions are practically free, while action is many orders of magnitude more expensive.
Given that starting point, we would be perfectly justified in saying he was wrong even if we can't prove the negative. Or, more likely, to ignore the hypothetical and spend no resources trying to prove or disprove the conjecture or remedy any issues the argument brings up. We should spend zero dollars trying to protect orbital satellites from this teapot, for instance.
I mostly agree, so now the question is whether Russell's teapot has the same probabilities as AI being an extinction level threat, so that you can clearly and obviously conclude which is more plausible and which requires the greater burden of evidence.
It seems clear that an intelligence greater than human intelligence is possible, as evolution would disfavour maximal intelligence for various reasons. Therefore greater intelligence is possible. We know of no reason why intelligence should be correlated with moral superiority, at least in a system that did not evolve as humans did, where cooperation was key to survival, therefore we should assume there is no intrinsic relationship between the two.
So it seems that an intelligence greater than human intelligence and that is not necessarily morally superior to humanity is eminently possible, and that we are plausibly on that path given the economic and strategic incentives there are to replacing and expensive fallible humans.
There are clear steps in this deduction that would permit empirical refutation, such as some intrinsic association between intelligence and morality, so it's falsifiable in the course of standard research on AI, particularly AI safety research.
Agreed that this is the strongman, further strengthened by the esotericism of the subject and how early we still are in the game. It's intuitively clear how nuclear weapons can destroy civilisation, and it's reasonably clear how a pandemic might, but not how and why the lovably-bumbling "glorified autocomplete" evolves into something that does, even by reaching for those other means. To persuade an ordinary layperson you have to keep hammering at the physical mechanism. Pascallian (or near-Pascallian) thought experiments are just not compelling, and stuff like the AI-box hypothesis is downright counter-productive.
What does "it" and "is" refer to in this context, both colloquially and technically?
The first instance of 'is' affirms my view of the correctness and truth of the claim in the comment I replied to. The two subsequent instances, coupled in contractions with their 'it's, assert common knowledge of how more stereotypical x-risks might play out. The final 'is' asserts my opinion of the negative persuasive power of the AI-box thing and by implication of other elegant but weird rhetorical techniques used by the doomers.
I'd do the 'are's, but, well, I can't quite be... bothered.
So, you are contemplating a model. Why not contemplate the thing itself? Isn't that the goal here, or is this more so a sort of social activity, that perhaps comes with unwritten rules?
Is "your prior should be" necessarily optimal as a method?
I'm sorry, but isn't this identical to the argument he addressed in the heuristics section?
Not exactly, the heuristic section talks about whether moral panics in general are correct (which is a fair point) but I think if you just focus on the technology worries and moral panics you can make a much stronger case that most of them have turned out to be nonsense.
In that section Scott is looking at this as evidence for the anti AI risk side (for which it is pretty weak), but if you look at it for what your priors should be before you jump into the evidence on AI it is much stronger.
<mild snark>
One could have accurately claimed that cigarettes and automobiles would each have killed millions of people over the course of a century... Of course, millions of deaths _sounds_ apocalyptic but actually isn't...
</mild snark>
Well it certainly killed them less dramatically than World War II did.
True! Many Thanks! And the body count is smaller too, for automobiles (for tobacco in all forms, WHO gives around 7 million a year currently, so over a century it _might_ exceed WWII's 70-85 million - not clear how much of this is cigarettes specifically).
Interestingly, with most past technologies, the people making doomer arguments about it were the people not involved with it, while the people involved with it were the ones who were dismissive. With AI we seem to have the reverse pattern, which doesn't directly tell us anything except that it's different.
I think the argument isn't against disaster specifically. It's against our ability to predict anything accurately. The argument isn't "The smart guys predicted a disaster but no disaster happened, so no disaster will happen this time". It's "The smart guys predicted something and were wrong, so smart guys aren't as good at prediction as they think".
Bringing up examples where no disaster was predicted but a disaster happened anyway doesn't refute them. It supports them. Bringing up examples of accurate predictions refutes them. However, in practice this is hard to refute because predictions are rarely *completely* accurate. Even getting things directionally right is very hard, let alone getting the timing, magnitude and exact character of a big future change correct. Also, an isolated correct prediction can be written off as luck, or the "stopped clock" effect. You need a long history of correct prediction to show that prediction works.
I think the easiest way to gauge what side of this you fall on is the extent to which you believe in the Efficient Markets hypothesis.
Most here will disagree, but "Number of times the world has been destroyed by new technology" seems like a reasonable reference class for AI risk.
Maybe, but if the world were destroyed, we wouldn't be here to talk about it so this number will always be zero.
Arguably, the world was indeed destroyed at least a couple of times, for certain values of "world" and "destroyed". The extinction of the dinosaurs is one obvious global example; the Black Death and any of the various genocides in human history are local examples. None of those cases depended primarily on new technologies, however.
I agree.
War, famine, and disease were the historic (local) x-risks. Notably, war often leads to famine and disease by disrupting supply chains and capital infrastructure.
Well, that, and also killing a bunch of people. I would say that war is indeed a significant risk, especially given modern weapons.
Sure, but who gives a shit about *local* x-risks, right?
The people exterminated by them cared, presumably?
Bad for book sales though.
Dead people can't complain, thankfully.
In risk-analysis I believe they have a system for assessing the probability of risks that must always turn up nil for observer-selection-effect reasons: I gather they look at the near-misses, then apply a correction factor based on the average ratio of near-misses* to incidents within similar fields.
(* presumably near-misses for less-observer-selection-effect-susceptible risks..)
For example: for Titanic-level maritime disasters, even though they're hard to do statistics with about because the frequency is less than once per lifetime, the incident:near-miss ratio is about 1:100 (source: vague half-remembered lecture I once attended, possibly whilst not entirely sober..) so if you see 10 near-misses in a given time-frame you can suppose a roughly 10% chance of an incident within the same time-frame.
Can we do this with AI doom? Probably not, because there probably aren't any incidents similar enough to AI doom with a known incident:near-miss ratio. However, to address Peter's phrasing: we do have data on nuclear war near-misses - and there are certainly more than enough of those to make one unwilling to just assume new technology won't destroy large parts of the world that one cares about.
Experts say that (at least some of) the negative consequences of what we've *already* done to the environment will arrive in our future.
Having read through other comments, I'm convinced that while superficially appealing, this is actually the *wrong* reference class, and it may explain why some of non-doomer arguments seem like a combination of willful blindness and philosophically-baffling moonlogic to the AGI pessimists.
"New technology" as a reference class implicitly means "tools" -- things that expand the capacities of individual humans or groups of humans while fully subject for all intents and purposes to their human masters.
The (or at a least a primary) doomer argument is that the reference class for AGI is totally outside of this paradigm. Instead it's "inventing a cognitively-superior organism that, for any conceivable agentic goal, will necessarily compete with other previously-existing species for resources to accomplish it." This is a story that *already happened* with natural GI and the outcome was that every other sentient species lost, hard. Hell, humans broadly *like* charismatic megafauna and they've still been devastated in population numbers where they weren't simply subject to extinction because humans don't like them enough relative to other priorities and would prefer to put their habitats (if not just their bodies) to other uses. Whereas one expects an agentic AGI to be as indifferent to humans as we are to bacteria, and for humans to have roughly the same capacity to challenge AI dominance as, like, parrots do to humans.
Meanwhile, the AGI's resource needs and ambitions have no natural upper bounds, which as a corollary means competition with humans that it will always, always win.
The relevant reference class isn't "new tool makes bigger bang," (scary though that is on its own), it's "the impersonal force of evolution results in the most cognitively-capable species being best adapted to its environment and winning resource competition, mopping the floor with every less intelligent species, and controlling whatever resources it feels like, to the exclusion of potential competitors and the deliberate extermination of pest species."
Yeah, the argument that is most striking to me is to ask where Neanderthals and Homo erectus are now. They could tell you about the danger from the competition of a superior intellect.
And dogs, cats and, more recently, coyotes, tell us about the opportunities created by superior intellects.
Being a dog is not my ambition.
They seem to be content in their lot and to be thriving compared to most other species… pretty much ever.
To steel man your argument, it is that similar species competing for a given niche leads to a winner and losers/extinction This is indeed well documented.
On the other hand, complementary species could support and thrive together. Interestingly this would imply that the solution isn’t via slightly smarter AI, which could be in competition, but rather in substantially and more powerful and smarter AI.
Your call, of course. But pethood is not for me.
There are thousands of people on this planet with enough power to completely derail or even eliminate your life with the stroke of a pen, and face zero negative consequences for it. The only reason they don't is because they don't care enough about you to do that. Unless you have the potential to be a billionaire or a world leader, then all people like you and I can aspire to be is a comfortable, placid resource for people with actual power. They're the only ones on the planet who aren't pets(at best) to someone else. And if you say no, you have dignity in this arrangement, well, you potentially can have that with AI overlords too.
Wow. Dark. Heavy.
Tell that to chickens, cows, pigs, and all the other animals being tortured on factory farms.
I agree that an entity whose capabilities have "no upper bounds" is likely to be quite bad news for us (arguably, the *needs* of humanity likewise have no upper bounds, but our capabilities are bounded). However, it's not enough to merely posit such an entity; humans have been inventing gods and demons throughout pretty much our entire history, and there's no reason to believe that any of them do or can exist. If you want to convince me to take specific steps and contribute specific resources toward stopping your uber-entities, you must first demonstrate that they are at the very least plausible, if not actually likely. Unfortunately, by positing unbounded entities (either truly unbounded, or "merely" effectively unbounded, as in e.g. "capable of doing anything they want to anything within the Solar System), you are kind of shooting yourself in the foot, since your burden of proof becomes likewise unbounded.
The statement about lacking upper bound was about needs rather than capabilities — the idea being that the AI need only be superior to humans to win in any competition for resources, and (barring implausible artificial restrictions like “make exactly ten paperclips and no more” rather than “survive and make paperclips,”), there’s no upper limit to AI demand for resources and attendant competition with humans for same.
The specific statement was “Meanwhile, the AGI's resource needs and ambitions have no natural upper bounds, which as a corollary means competition with humans that it will always, always win.”
I am thus making a much less strong claim about the necessary capabilities of the AI than you are imputing to me — it needs to only be better than humans, not arbitrarily better than humans.
(I should note that I actually do think arbitrarily high capabilities in a practical sense are likely under a recursive self-improvement scenario, which seems ex ante extremely plausible, but I don’t think it’s a necessary element of the claim.)
Understood, and I withdraw my objection -- but I'd still argue that the needs of humanity are ultimately unbounded, seeing as we are driven to reproduce without limit other than resource constraints. In fact, given a zero-sum competition between two societies of humans, one with greater needs than the other, the outcome is uncertain unless you also factor in their *capabilities*. Or to put it another way, an unemployed bum has a greater *need* for a sandwich than a billionaire, but the billionaire is still much more likely to obtain the sandwich.
I would agree that needs of humanity are unbounded except that it seems like material success is correlated with sub-replacement fertility empirically speaking. But I think the issue is moot in any event: assuming infinite needs by both humans and an AI (the AI having unbounded needs basically because that's just the way that utility-function-maximization works for any plausible goal beyond "power down and stay powered down," and for the same reason being a lot more single-minded about working to meet its goals than humans empirically are about reproducing), if the AI is more capable you have both conflict over resources and AI victory.
Agreed, in any conflict the more capable side would usually win; but you are now talking about *capabilities* (just as I was), not merely *needs*.
Humans have been inventing such things for ages. But now we have things that can talk sensibly, win chess games etc.
They aren't yet uber-powerful. But they clearly exist, and those demons didn't.
> you are kind of shooting yourself in the foot, since your burden of proof becomes likewise unbounded.
That is really not how reasoning works. Like at all.
Occams razor penalizes the complicated specific hypothesis. Not the possibility of something being smart/powerful.
If you propose someone or something that is reasonably smart, or strong, or fast, then the burden of proof is still on you, but it is quite modest. We can observe many such beings all around us already, after all.
If you propose someone or something that is the smartest, strongest, or fastest person in the world, then the burden of proof is much higher, but perhaps still doable. It is not clear whether intelligence, strength, or speed can even be sorted like that (e.g. bulldogs have extremely strong jaws but cannot lift heavy objects like humans can), but perhaps you could still make a compelling case, if you bring up a mountain of evidence.
But if you are proposing an entity that is not merely the smartest/strongest/fastest person in the world, but so superior to everyone else as to be literally unimaginable -- then I am not sure how you could justify it. What kind of evidence could you bring up for an entity that is definitionally incomprehensible ? You could propose it in an abstract, but you have just defined yourself out of the ability to comprehend it, which means that you could not possibly have sufficient evidence, since looking at evidence is *how* we tend to comprehend things.
> If you propose someone or something that is the smartest, strongest, or fastest person in the world, then the burden of proof is much higher, but perhaps still doable.
The Astronauts returning from the moon were the fastest humans ever. So are you saying that the Apollo missions had some exceptionally high burden of proof?
I mean sure, no one else is going that fast. But no one else is getting on a giant rocket.
Before the first atomic weapons were made, the scientists were talking about explosions enormously bigger than anything that had come before. The evidence they had was some fairly modest particle physics experiments. Yet they correctly deduced such weapons were possible. And if physics had been slightly different, and nuclear explosions were a trillion times more powerful than chemical instead of a million, the evidence required to show this would not have been greater.
> What kind of evidence could you bring up for an entity that is definitionally incomprehensible ?
Superintelligent AI isn't definitionally incomprehensible. I don't know what limits your imagination is under. But I can at least try to understand some aspects of it.
Deep blue is better at chess than me. So I can't predict it's exact moves (If I could, I could play as good chess). But I can understand what it's code is doing, what a minmax search tree with heuristics is. I can understand why such a thing would be good at chess. And I can predict it will win. That is quite a lot of understanding.
Reasoning about minds smarter than you is hard. Sometimes I use analogies involving things I understand but that dumber minds (human or animal) don't. Sometimes I use abstract maths.
There is a lot I don't know, but some things I can make a guess at.
> but you have just defined yourself out of the ability to comprehend it, which means that you could not possibly have sufficient evidence
If the AI produces a valid proof of the Rienman hypothesis and designs a fusion reactor, I don't need to know how it works internally. I don't need that much understanding. The concept of "very intelligent" predicted that the AI could do those things, and most other concepts didn't.
If you saw the fusion reactor in front of you, clearly designed by an AI and clearly working, would you say it was dumb?
> So are you saying that the Apollo missions had some exceptionally high burden of proof?
Yes, and it was met with sufficient evidence. The same goes for atomic science.
> And if physics had been slightly different, and nuclear explosions were a trillion times more powerful than chemical instead of a million...
Yes, if the laws of physics were completely different, lots of things would become possible (or impossible). I don't think you and I, not being physicists, can speculate on it in any kind of an informed fashion (well, at least I know I'm not a physicist).
> Deep blue is better at chess than me. So I can't predict it's exact moves (If I could, I could play as good chess). But I can understand what it's code is doing...
Are you saying that Deep Blue is "superintelligent" ? I don't think so:
> If you saw the fusion reactor in front of you, clearly designed by an AI and clearly working, would you say it was dumb?
It depends. I think that Deep Blue is, in a way, really dumb. It's a program designed to play chess, and that's the only thing it can do. I may not fully understand how it works (other than knowing how alpha-beta pruning works in general), but I am fairly confident that it's a relatively simple tool. A machine-learning system designed to build fusion reactors would be a much more complex tool, but it would still be exactly as dumb as Deep Blue.
But you are not proposing such a system; rather, you are proposing an independent autonomous agent that is not merely good at designing reactors or playing chess, but is good at doing literally everything. And not merely good, and not merely better at it than the best humans, but literally beyound (perhaps far beyound) the upper limit of human capabilities. This is why AI-doomers are afraid of AI: not because it could build a better reactor or play a better game of chess, but because it would be able to easily outwit literally everyone at everything. I think this does add up to being "incomprehensible", in addition to being completely unprecedented (outside of the realm of fantasy), so yes, I'm not sure what kind of evidence you could muster towards such a grant claim.
I acknowledge that an AGI (assuming such a thing is possible at all) doesn't necessarily have to be so overwhelmingly smart. Technically, it could be something like a top-notch physicist who is better at particle physics than any living human physicist. This would make the AGI technically "superhuman", but not much of a threat, because there's really not much you can do *just* by being able to think really hard about particle physics.
We have evidence in favor of superhuman capabilities for arithmetic, chess, go, poker¹, next-token prediction², maybe Dota2³, Atari games⁴, and protein folding⁵. We also know that evolution can design very efficient and even self-repairing objects, which humans can't. Existing computers are far below physical limits to computation⁶.
Humans like von Neumann and Ramanujan demonstrate the existence of general and specialized cognitive capabilities far above the mean human level, respectively.
A relatively small advantage in technology and tactics has been sufficient for crushing victory of a small group over a large group of humans⁷.
¹: https://en.wikipedia.org/wiki/Libratus
²: https://www.lesswrong.com/posts/htrZrxduciZ5QaCjw/language-models-seem-to-be-much-better-than-humans-at-next
³: https://en.wikipedia.org/wiki/OpenAI_Five
⁴: https://en.wikipedia.org/wiki/MuZero#Reactions_and_related_work
⁵: https://en.wikipedia.org/wiki/AlphaFold#AlphaFold_2,_2020
⁶: https://www.nature.com/articles/35023282
⁷: https://www.lesswrong.com/posts/ivpKSjM4D6FbqF4pZ/cortes-pizarro-and-afonso-as-precedents-for-takeover
Yes, of course; but I could make an even longer list if I included the capability to multiply large numbers (and large sets of such numbers), route network traffic, manage fuel injection schedules in real time, and so on. I could in fact expand this list to the ability to crush hard objects or fly across continents or even extract energy from sunlight. I don't see how my list would be any more (or less) valid than yours. And yet you are not overly concerned about the dangers of hydraulic presses; so why are you overly concerned about the dangers of e.g. go-playing systems ?
Humans like von Neumann and Ramanujan do indeed demonstrate "superhuman" performance (I mean, they're technically human but I understand your meaning) on a variety of tasks, yet both of them were highly specialized in their areas. From what I can tell, even Von Neumann was fantastic at mathematics and physics (and computer science, arguably a branch of mathematics); pretty good at organizing highly specialized committees; eidetic memory (possibly); and pretty close to human-average on the vast majority of other tasks.
The only non-trivial point on your list that I can agree with is #7:
> A relatively small advantage in technology and tactics has been sufficient for crushing victory of a small group over a large group of humans
But this sweeps a massive amount of complexity under the rug. Terms like "technology" and "tactics" imply a degree of self-guided autonomy and adaptivity that no present-day machine learning system could remotely approach. It's like saying that swords pose an unprecedented level of existential risk because in a war the army with better swords will probably in (all other things being equal). This is technically true, but not because of some kind of an unprecedented level of danger posed by swords. It's the humans wielding them who are the danger (and sadly a well-precedented one).
I considered non-cognitive abilities, but those didn't seem super relevant. Yes, humans have created devices that are much faster/powerful than many animals, but speed/power is not the thing that gives humans advantage over other animals.
I put multiplying numbers in the arithmetic bucket.
The list was a collection of systems that have superhuman cognitive abilities in narrow ranges, which (according to me) are evidence that superintelligence is not impossible.
I think that von Neumann was actually much better than most other people in a wide variety of tasks, e.g. he was a consultant for a while, getting paid large sums by his clients, and learned six languages before the age of eight¹. I think that e.g. Ramanujan was probably a better mathematician, but much much more cognitively specialised.
> Prisoner’s dilemma says that he and his collaborators “pursued patents less aggressively than the could have”. Edward Teller commented, “probably the IBM company owes half its money to John Von Neumann.” (pg. 76)
I agree that he was not flawless, as is detailed nicely in this post: https://musingsandroughdrafts.com/2019/10/26/some-notes-on-von-neumann-as-a-human-being/. Although the list seems pretty slim on concrete cognitive tasks that he was actually bad at, not just moral failures.
> But this sweeps a massive amount of complexity under the rug. Terms like "technology" and "tactics" imply a degree of self-guided autonomy and adaptivity that no present-day machine learning system could remotely approach.
I don't necessarily disagree with that :-)
I was merely arguing that the gap between agentic dangerous AI systems and us would not need to be unboundedly large. It is an entirely different line of argument to show that we're anywhere near that.
¹: I have not heard
> but speed/power is not the thing that gives humans advantage over other animals.
Maybe I'm being pedantic, but this is not the case at all ! Humans are endurance hunters, which gave us an early edge, due to being able to run down prey animals. We also have better physical equipment than many animals, while being deficient in some domains. For example, dogs have better smell but we have better vision. But yes, our cognition is also superior to that of any animal (that we know of anyway).
> The list was a collection of systems that have superhuman cognitive abilities in narrow ranges, which (according to me) are evidence that superintelligence is not impossible.
Right, but my point is that superhuman performance on an extremely narrow task does not translate into "intelligence" (insofar as the term has a meaningful definition). Calculators can multiply numbers very quickly, but are not "intelligent" in any sense of the word; and this would not change even if you bought yourself a huge crate of them.
Perhaps more importantly, "intelligence" alone doesn't buy you much. As I'd said before, a superintelligent box that sits under the desk and thinks very hard without interacting with anyone or anything might as well be equivalent to a dumb old brick. What AI-doomers are worried about is not "superintelligence" but rather "super-capabilities", and these are subject to much more stringent physical constraints.
> and learned six languages before the age of eight¹. ¹: I have not heard
Sorry, I'm having trouble parsing this -- not heard what ?
> I was merely arguing that the gap between agentic dangerous AI systems and us would not need to be unboundedly large.
That is true, but the AI-doom argument relies on this gap being unboundedly large. The AI is not merely smart, it is "super"-intelligent; it cannot merely position troops better than human generals, it can win every battle despite being completely outnumbered and outgunned, etc. The AI who is a slightly better general than its human counterpart would be a problem, but not an unprecedented problem, as the stories of humans like Caesar and Napoleon illustrate.
Since you put "no natural upper bounds", here's a pedantic list of upper bounds:
Holographic principle information processing density
energy available to cool everything on earth's surface
Speed of light limits on communication and cognition
Halting problem
Note that these upper bounds are """fairly far away""", but you know, failing to mention them probably also means that 10-20% improvement over human capabilities in all fields is also impossible. This is an unfortunate fact of how logic works, and scientists are racing to find new forms of proof that disallow this, because they have motivated reasoning, and I have every man reasoning.
Both you and Bugmaster appear to be misreading the actual context of that claim, which was regarding resource demands rather than capabilities having no natural upper bound. Capabilites need merely exceed those of humans, and then basic agentic goals like “survive and reproduce” or “make paperclips” have no natural upper limit as to the resource demands that they create so as to create conflict (won by the AI, because it’s smarter than humans) between it and other species.
I agree I misread, but by complete stupid coincidence those are also indirect constraints on the ability to gain resources, (I.e. AI will be bounded by the light cone, cooling will limit the rate and indirectly the total amount of resources acquired due to 2nd law of thermodynamics). Of course if that was the argument I wanted to make, I should have made it, instead of an unrelated one.
Oh also I was being sarcastic, and so think I am your mind clone when it comes to object level points. Let's echo chamber, I mean, combine our proofs.
I'm confused.
>failing to mention them probably also means that 10-20% improvement over human capabilities in all fields is also impossible.
Specifically for the speed of light limit, specifically for communications, barring a surprising discovery in physics, we are already at the physical limit for communications speed, since we routinely use e.g. radio to communicate, so we can't get even a 1% improvement in the communications delay to e.g. our satellites and space probes.
I'm unsure whether I'm agreeing with you or disagreeing.
To be clear, I was being sarcastic, because I think naming physical bounds that are orders of magnitudes away as constraints on near term improvements is a dumb argument
Secondly, I said "communication and cognition". Yes the *latency* may be at or close to a physical minimum, but the *bandwidth* may not be, considering how little information English contains, and how slowly it gets produced relative to good thoughts (tm). Also yeah if hypothetically if we get a giant honkadonka badonker's brain (A REAL TERM USED BY A REAL X RISKER), we're talking more about "limited by bus speed" (aka Einstein traveling at light speed thought experiment train fast) rather than limited by extremely slow human reflexes (regular train fast).
I think we're just talking and neither actively agreeing or disagreeing.
Many Thanks!
I think part of the problem is exactly defining what risks AI pose. There is a difference between a hand-wavy nanobot apocalypse and something like “It could disrupt multiple industries completely and immediately, which will lead to instability in the economy”.
But big picture, absolutely none of this matters, because we can clutch our pearls here in the United States, or even pass regulation, trying to slow down technical progress (good luck), but the reality is progress will just march forward in Russia and China and India.
Seems to me the only safety we can get is to try to make the US be the leader of this technology, instead of de facto ceding leadership to China
Yes, this is a vital point that just gets whistled past. The closest I've ever seen a doomer get to addressing it is "the Chinese government is notoriously pragmatic and careful and would never take such a risk" which is so disconnected from reality it leaves one staring in awe.
As someone who spent most of their adult life in China, I am unsure how you think this is disconnected from reality, especially to that extent? China has been regulating AI more strictly than anyone, the Party prefers AGI not to exist since it endangers their rule, and as a country they are much more steerable than democracies are. Russia and India are so far behind in the game, and AI is so hardware dependent that I do not see how you could think that progress would move faster there. (answering to both carateca and Victor). The "But what about China" arguments ignore the fact that 99% of AI progress China saw is due to US open-source models, and on their own they make next to nothing. Read about their Chip manufacturing industry to see how corruption eats everything from the inside and how difficult advanced technology is to make.
I am not saying they are not a geopolitical player and that they need to be involved - but I also don't know serious AI Safety Policy people who ignore China? Still, I think the right proportion is 80% work on Washington, 10% EU, 9% China, 1% everyone else combined; based on impact it'll have.
My reading of OP: the attempt to "leap" and the "cultural revolution" are difficult to characterize as either pragmatic or careful.
Also we literally just lived through a global pandemic which was very likely caused by China's failure to be pragmatic and careful in regulating scientific research.
Note that even if they didn't actually create COVID, they were still engaged in the kind of high-risk research that *could* have caused something like COVID, which is sufficient to disprove the claim.
And if it wasn't, it was caused by China's failure to be pragmatic and careful regulating the live animal trade.
I don't see markets crammed full of endangered species in unsanitary conditions in Cleveland.
But basically the entire arc of post-Cultural-Revolution development in China, from Deng on (well, before Deng was formally in power, too) has precisely been reaction to Cultural Revolution and an attempt to ensure that China does not fall into similar chaos again.
I wasn’t specifically referring to government sourced AI, I was referring to the fact that if the US government tries to limit googles efforts into AI, they are just going to move their efforts overseas.
If increased regulation becomes reality, any small country in the world just has to have cheap power, and they can build themselves as the go to location for model training. Just like Ireland is a corporate tax haven, and Switzerland used to be a bank haven.
Data is easy to move.
In terms of the national security (including economic) risks of AI, my point is that they will happen regardless. The only question is what country profits from them both economically and geolopolitically.
And if it wasn’t clear, I don’t think acknowledging those risks or even fear mongering about nanobots is a good path forward for the US to be the primary beneficiary of this technology.
Data is easy to move, compute is not - it can be regulated.
Also, if the US says "only models which can legally be sold/used in the USA are those created in these countries" then ~no one will want to make models elsewhere, since US is a huge market and the main market to be on. This is not such an impossible task, there are very few places which can do it, very few companies which can do it, and if the US decided to do it it could sanction countries which tried to skirt this. US alone essentially has the power to control compute if it wished, it just does not right now.
Smart Researchers are not easy to move. If you tell some AI expert earning 6 figures that the company is moving to outer Mongolia, expect most of them to quit and get a regular programming job instead.
> The closest I've ever seen a doomer get to addressing it is "the Chinese government is notoriously pragmatic and careful and would never take such a risk" which is so disconnected from reality it leaves one staring in awe.
Actually, China is considerably *less* likely to create a doom AI. They are so obsessed with strictly controlling culture and access to information, that they will make *sure* their AI is aligned with communist party values, because if it's not, heads will roll. The places with free experimentation are the main sources of danger.
An AI thoroughly aligned with Communist Party values could not be dangerous? Are you sure about that?
I didn't say it wouldn't be dangerous, I said a CCP-aligned AI would be less likely to be a doom AI that exterminates humanity. The "danger" I'm referring to is typically this sort of AI doom. Any AI likely has more mundane dangers, like misinfo bots from GPT.
Most Western attempts to align an AI will focus on getting it not to kill anyone, killing is bad, blah blah blah.
But a CCP-aligned bot is implicitly one that already thinks it's okay to kill millions of people. It may find it easier to generalise to killing everyone.
> Most Western attempts to align an AI will focus on getting it not to kill anyone
Or maybe a Western AI will be obsessed with eliminating racism, and so kill all white people. Gemini already erased the white founding fathers. Hard to say how culture will evolve these days.
> But a CCP-aligned bot is implicitly one that already thinks it's okay to kill millions of people. It may find it easier to generalise to killing everyone.
That doesn't sound like alignment to a goal of spreading communism to me.
I'm sure we can imagine all sorts of possible worlds featuring silly mistakes, I'm just saying that I can see fewer plausible worlds where a CCP-AI exterminates all of humanity.
No. The argument is that the CCP Knows they can't solve alignment. So don't want any AI.
How deeply did you think about that last conclusion?
This seems like a general argument against AI doomerism, but how does the coffee argument fit into it?
This is just one of the many versions of the "it's inevitable so there's no point in taking any action" argument that I find deeply unconvincing. I find that a person only makes this argument if they don't actually think there's any existential risk. If your house was on fire and the doors and windows were boarded shut, most sane people would still be trying to break down a wall with their shoulders even if they understood that it was probably impossible.
China is also interested in controlling AI. International cooperation sometimes works.
Well china is mostly cracking down on any AI that ever mentions "internet maintenance day"
And Russia won't be making much AI because it's loaded with sanctions on electronics. Most of the smart people are trying to get somewhere, anywhere else before they are drafted as cannon fodder.
I'm puzzled by your puzzlement. The (indeed silly) "coffeepocalypse" argument just highlights how new things instinctively and generically cause panic, even if unjustified—and a fortiori when there's actually some inherent danger to them. It says: you should significantly lower your prior regarding the risk associated with new developments.
This might be persuasive if one could point to instances of AI-risk people being excessively cautious about technology in other domains, which would be a demonstration that their prior regarding the risks of new developments was set too high. But in fact you find that they're gung-ho about most non-AI technological development.
This presumes that there is a specific class of people who make bad panic-predictions, rather than bad panic-predictions being a characteristic of people in general.
Why do you require it be in a different domain? The duplicitously-named OpenAI's stated reason for not releasing GPT-2 (from 2019) was that it was too dangerous. Now, there are far more capable models freely available, I'd say they were "excessively cautious" if I didn't think they were lying through their teeth about "safety and security concerns" as Bootleggers in Baptist vestments.
> The duplicitously-named OpenAI's stated reason for not releasing GPT-2 (from 2019) was that it was too dangerous.
This is a misrepresentation of their published release strategy: https://openai.com/research/better-language-models
If you remove "just" from your claim, does the meaning change (in fact)?
Even if this argument about priors is accepted, don't you also have to look at the specific evidence for the specific risk? I don't think a typical AI-risk person would assign a high probability to a randomly-selected new thing causing disaster, so they aren't necessarily disagreeing with you about priors.
I think it would be best rephrased as "There are people who have a reflexive mindset against new ideas. It's a good idea to get rid of such a mindset". Global cooling and overpopulation may not be technologies but they are ideas, just as AI doom, AI paradise and pretty much every AI-related prediction and concept is.
One problem here is that it's probably a good idea that we do have at least *some* people who are true curmudgeons, willing to challenge everything and anything, simply because it's good for everything to go through a challenge. It would be great in general to acknowledge that the society benefits from a vast number of different mindsets and ways of reacting to things.
I appreciate the steelmanning, but isn't this really an argument aimed at fellow anti-AI-riskers to help them feel superior? I just can't take this charitably at all.
Yeah, the argument isn't the point, it's the comparison of AI doomers to people who were afraid of nothing and forgotten by history.
I don't find the linked Rutherford bit very convincing, but knowing nothing about the guy, I would assume he was prominent in the field, admitted he was wrong shortly afterward, and remained prominent in the field. In which case it's a much more charitable example, made stronger by the guy having enough expertise for people to think he knew what he was talking about.
If humans insist on dopey analogies: if you play Russian Roulette often enough, the probability you will kill yourself approaches 100%
While this is true, your decision to play or not to play depends on the size of the revolver's drum. If the drum contains one trillion chambers, and you know that only one is loaded, then you might as well play at least a few times (assuming there's a payoff involved).
For any reasonable payoff, even $1 a pull, it would make sense to pull that trigger enough times to make yourself filthy rich. 1/1,000,000,000,000 is really really low odds. You would win the big jackpot lotteries *multiple times* (there's about 4,000 times better chance of winning a multi-hundred-million lottery) long before finding the one chamber that's loaded.
Hmm, interesting hypothetical, pitting two common sources of irrational thinking against each other: very low odds and marginal utility. For example, is it worth that final pull of the trigger to get that last $1 to make yourself filthy rich when you could safely stop at almost filthy rich? After a few tens of thousands of pulls, it's difficult to weigh the tiny benefit of another buck against the tiny risk of kicking the bucket.
After a million pulls or so, (perhaps 2 weeks of semi-continuously pulling the trigger), the dominant tradeoff will be repetitive strain injury rather than the 10^-6 mortal risk. Come to think of it, a 100 year lifespan is about a third of a nanomort per second - even if one could pull the trigger three times a second, the hazard of just being alive outweighs the 10^-12 risk by about 100X.
Yes, I considered this angle, but within the hypothetical it makes more sense to talk about how many times we would take the risk, rather than thinking about repetitive strains.
Many Thanks! Yes, that is reasonable.
Most people wouldn't want to be a billionaire from this plan, as the risk starts to be real, even if still incredibly small. Pulling $10 million would be such a low chance of finding the bullet that individual triggers between 9.9 million and 10.1 is nothing. Pick a number you would be happy retiring with and pull away! Anything less than about $100 million is nothing compared to a trillion.
I would like to have a choice about the game of chicken we seem to be playing with the climate and a variety of other issues on this planet and in my country.
Oh right, democracy will solve it, we can just vote our way to Utopia.
AI is also the only way we could ever hope to advance medical technology enough to end (or at least delay by a large amount) death in our lifetimes. (Not saying it is a guarantee, but without it we certainly won't get there)
So it's more like we've all been poisoned by a very slow-acting poison, and the russian roulette gun has bullets in some chambers and antidotes in others. You might get killed but you have nothing to lose.
The "only way" seems like hyperbole.
Do you think there are other advances that could realistically get there in the next 50 years? I have no doubt things like cancer treatments will continue to improve, but even over the past 10-20 years the average life expectancy in the developed world has plateaued.
I don't know about advances that haven't been discovered yet. Black swans of medicine, so to speak. Neither does anyone else.
That's my point.
Ok, so you don't know what advances have not been discovered yet or where to even go looking -- but you are sure that they do exist ? What makes you so sure ?
I am not the one who claims to know what the future will hold. I am saying that it is impossible to know the future, unless you are Nostradamus or something.
>Do you think there are other advances that could realistically get there in the next 50 years?
Over that time frame, given sufficient investment, I think Drexler/Merkle nanotechnology aka atomically precise manufacturing would probably suffice to fix the vast majority of illnesses and consequences of aging. "Replace with known-good parts" goes a long way, even where we _don't_ know in detail how the parts function. In practice, the investment is not happening, while the investment in AI _is_ happening.
I doubt we'll be ending death anytime soon, AI or no AI. On the other hand, our lifespans had already increased significantly relatively recently; we might be hitting diminishing returns, but still, there's some cause for cautious optimism. AI is a very powerful statistical analysis tool, but it's no silver bullet; still, with more powerful tools we might get some of those returns, despite their diminishing.
On the flip side, if AI isn't powerful enough to end death, then I'm certainly not worried about it taking over the world or turning the universe into paper clips.
(By end death, I at least mean natural death - like if we had powerful AI it seems like it would be trivial to ask it to find the causes of aging in the human body and tell us how to prevent them)
Indeed, I agree that if we had an AI that was effectively omniscient, then we could ask it to "find the causes of aging in the human body and tell us how to prevent them". In fact we could ask it for pretty much anything (logically consistent) and get the correct answer. The same is true not only of an AI, but also of a fully-functional crystal ball, extradimensional magic-using diviners, clairvoyant psychics, and any other fictional omniscient entity that you could devise.
Analyzing the workings of a very complex physical system doesn't seem like something you need an omniscient being for, just a very powerful computer/intelligence with biological knowledge.
But I'm confused why you're concerned about AI risk, if you think AI is incapable of something like that. If it can't even solve aging then surely it can't take over the world.
Oh, I'm not concerned with AI risk -- that is, not in the AI-doom sense. I don't think that AI will either end death or kill us all; it's a powerful tool, not a pocket deity. However, I *am* concerned about what thoughtless or malicious people could do -- and are already doing -- with powerful tools. I think we should be focusing on *that*, not on some hypothetical superintelligent future AIs that are likely never going to happen.
If AI is approximately as likely as ending death, I'm a lot less worried about AI now, thanks. :)
Why should death be a problem? Grieving is painful of course, but it's about love so no problem as well.
> Here’s an example of a time someone was worried about something, but it didn’t happen. Therefore, AI, which you are worried about, also won’t happen.
Right, that's an obvious strawman. The real argument goes more like this:
"Here's an example of a time someone was worried about something due to employing a particular pattern of reasoning. The something didn't happen, and the manner in which it didn't happen helped to expose the flaws in the pattern of reasoning. You are worried about AI due to employing the same pattern of reasoning, riddled with the same flaws. Therefore, you are likely to be just as wrong as that other guy."
I have been sitting here on the comments page trying to figure out how to express what was in my head, and I think this does an excellent job, thanks!
So what are these flawed patterns of reasoning? You imply that there is some kind of argument to be made there, but as far as anyone else can tell the same argument applies to "nothing ever happens".
"Nothing ever happens", indeed has better predictive accuracy than, "big thing is about to happen". See Caplan. https://www.econlib.org/my-complete-bet-wiki/#:~:text=Bryan%20Caplan&text=By%20popular%20demand%2C%20I've,every%20single%20one%20of%20them.
See Scott: https://www.astralcodexten.com/p/heuristics-that-almost-always-work
The pattern that I often witness is people clinging too much to a good narrative without experimental data to back it up. Good narratives are incredibly useful for generating hypothesis that you can then experimentally test (including back-testing), but a good narrative alone is not sufficient to put forth as evidence for some future event.
The AI Doom people have a good narrative. It is a plausible story that one can easily imagine to be true, and in it might actually be true! You cannot debunk it and show it is definitely not true, which allows it to live a long life as a good narrative. It is totally fine to build this narrative and share it. The problem only arises when you want to act on the narrative without first doing the work of showing that the narrative is correct.
Lots of people in the past have come up with good narratives for how the future will unfold. Video games back in the 80s/90s were incredibly convincing. If you allow children to act out violence in video games, it will desensitize them to violence! We later realized that children are actually quite good at differentiating between reality and a video game, and we really should have tested the hypothesis that they can't before doing something like regulating video games.
With AI Doom the situation is significantly worse because the suggested solutions have a history (that you can back test against) of negative outcomes. Stripping people of civil liberties, going to war to stop scientific advancement, totalitarianism, etc. all have a track record of being bad for human flourishing, and we shouldn't budge on those because someone has a good narrative.
> The problem only arises when you want to act on the narrative without first doing the work of showing that the narrative is correct.
Showing the narrative is correct either is itself a source of the danger (develop AI and see how capable it becomes), or it involves funding AI safety research to prove that AI safety either is possible or impossible. I think plenty of AI safety people are just saying to fund that research, which doesn't seem unreasonable.
Certainly some people take it much further, like EY, but I'm not sure he's a representative sample.
Consider that instead of debating AI-doom, we were debating demon-doom. I claim that demons from Hell are about to invade our plane of reality through portals on Phobos, and this is why you must a). start stockpiling holy shotgun ammo, and b). stop using electricity, because electricity attracts demons. Let's imagine that I managed to spin that summary into a coherent narrative.
Showing that my narrative is correct is either a source of danger (open demon-portals and see what happens), or it involves funding demon safety research to prove that demon safety is either possible or impossible. So, how much are you willing to contribute to demon-safety research (and/or holy shotgun stockpiles) ?
Straw man. Demons don't exist, AIs do exist *right now*. AI danger is already *here*, even if it's not superintelligent AI.
The existence of superintelligent AI also follows from straightforward scientific arguments. Show me a scientific argument that demons exist, and then we'll talk.
"AIs" in the sense that you mean them (i.e. AGI) do not exist; ChatGPT is arguably the best we've got, and it's only slightly better than Markov chains. It's just a statistical tool that happens to work quite well at a narrow set of tasks. And the reason I compared "superintelligent" AIs to demons is because "intelligence" is poorly defined but also not terribly important; what you really mean when you say "superintelligent AIs" is "super-*capable* AIs that are functionally omniscient and omnipotent". There are good reasons to believe that such entities are either extremely unlikely to exist, or outright prohibited by the laws of physics (as we understand them today).
> AIs do exist *right now*
They don't. ChatGPT and its ilk aren't AIs in the sense of "AGI", and it's doubtful that the LLM approach is even capable of reaching anything resembling general intelligence.
> AI danger is already *here*
It's not, at least not in the sense that AI doomerists understand the term "AI danger".
I'm not opposed to people asking for funding for AI safety research. I probably won't provide such funding as I think there are better uses of resources, but if this was all that was being asked I would not likely care much at all.
However, there is a *very large* segment of the AI Doom population who think we should, as a society, make sacrifices to do things like slow down AI progress (Scott has an article where he indicates support for a slow-down). These sacrifices range from simple things like banning the authoring of certain types of software (a constitutional violation in the US), to ceding control over all future AI development and usage to large companies and/or governments (historically this hasn't ended well), to global totalitarian state (EY).
Where did you get the sense of the size of that segment, as opposed to how vocal and provocative those people are? By which I mean, are you sure it isn't a selection bias effect where the provocative positions and people just get more publicity for clicks and so that gives a skewed impression that they are widespread?
I would definitely call Scott vocal, but I wouldn't call him provocative. In fact, he often seems to go out of his way to avoid being provocative. I also don't believe that he says what he says just for clicks.
I have also spoken with other people at medium length who are active within the rationalist community who feel similarly. It is certainly possible that there is selection bias, but my selection mechanism isn't just randos on twitter or comment threads. It is people who spend a lot of timing thinking about these things and some of which have a background in AI/machine learning.
Basically what @Micah Zoltu said. I also wrote a short summary on the specific subject of AI:
https://www.datasecretslox.com/index.php/topic,2481.msg71110.html#msg71110
But the general flawed pattern of reasoning is the tendency to extrapolate all kinds of specifics from very little data. If you have two points (on a 2D plane), you can connect them with a line, a parabola, an exponential curve, or any other kind of shape you want, and draw whatever conclusion strikes your fancy. It also helps if you outright ignore several other points, because they don't fit on your preferred curve.
I'm having trouble seeing a difference between "you are following a general pattern of flawed reasoning in which you extrapolate too much from too little data" and simply "I think your evidence is weak."
Deciding whether someone is extrapolating "too much" or merely the correct amount requires actually weighing the evidence, right? It's not a fallacy like "denying the antecedent" or "ad hominem" where the argument *fundamentally* doesn't work and you can see that merely by examining the structure, without even considering the details.
The difference between "a flawed pattern of reasoning" and "your evidence is weak" is that the second case actually presents a *stronger* argument (relatively speaking). Saying something like "I drove my car down a steep hill with a tailwind, and based on this experience I believe it would get 1,000 mpg in daily use" is a mistake; but it's a different kind of mistake from saying "given that cars are getting more efficient and given that this process will continue indefinitely, and also given present-day mpg ratings, I estimate that the car I buy in 2030 would get 1M mpg". The second argument does consider evidence, but only peripherally; the first argument, meanwhile, is entirely based on evidence but falters in its interpretation.
I still do not see the distinction you are trying to draw, unless that distinction is "one of these examples predicts the performance of existing technology, while the other predicts the performance of future technology."
Things like "present-day MPG ratings" and "cars are getting more efficient" are both forms of evidence, and it seems like one could make some sort of reasonable estimate of future performance based on those or things like those. Obviously "this technological trend will continue indefinitely" is a dubious premise, but it seems dubious in basically the same way that "my downhill speed will continue indefinitely" is dubious, to me.
Yes, the dubious premise is indeed the difference, in that it is explicit and taken as given without being subject to any challenge. You could tell that 1,000 mpg guy, "hey, actually you forgot to factor in terrain, re-run your test while driving on flat ground or uphill"; and once he does it, he will correct his estimate. But there's nothing you can say to the 1M mpg guy to disabuse him of his estimates, because they are not based on any evidence at all (and are arguably contradicted by evidence). The fact that his prediction model also incorporates present-day mpg ratings is not very relevant, since they don't substantially change the predicted outcome.
I would identify the pattern of reasoning that the argument says is flawed as being essentially "trusting others' opinions on new technology":
1) I'm curious as to how much I should worry about new thing X.
2) I observe that many important and knowledgeable people worry about X quite a lot.
3) They probably know important things about X that I don't.
4) Therefore, I should worry about X.
So the argument is, "this pattern of reasoning would have gotten us the wrong answer in the case where X=coffee, so we shouldn't treat it as valid; in particular, we shouldn't use it in the case where X=AI"
But this ("trusting others' opinions on new technology") is actually a useful pattern of reasoning in the sense that it gives results better than chance. The coffee argument against it doesn't work; we always knew, with such a large reference class, that there would be cases where this reasoning has led to the wrong answer, so identifying coffee as one such case doesn't give us any new information about how often it's wrong.
This itself is an example of bad reasoning as it considers the particulars of each problem to be equivalent, that the underlying truth is derived from our predictions about it.
I'm not sure I understand -- do you have some way of accessing the "underlying truth" besides making predictions based on imperfect sensory data ?
I do not - do you believe that is necessary to identify flaws in an argument?
I only wanted you to clarify your comment, because I'm not sure what you meant by it. That said, trivially speaking, if my argument is "All X are Y; Z is Y; therefore Z is X", then you could point out the logical flaw in it without knowing any of the particulars. You could say, "No, Bugmaster, you are not justified in believing that Z is X", without knowing what any of the variables stand for.
Do you believe that adequate information has not already been revealed to you?
If you'd like help, here's some: actually answer the question I asked.
Information about what ? Again, I don't really understand what you're asking; also, you kind of ignored my comment.
"You are worried about AI due to employing the same pattern of reasoning, riddled with the same flaws."
The argument only works if someone is making the same claims about AI as about the other problem. It would indeed be disproven if someone describes how their reasoning is different.
The explicit claim is that *only* flawed reasoning is the underlying cause:
> "Here's an example of a time someone was worried about something due to employing a particular pattern of reasoning. The something didn't happen, and the manner in which it didn't happen helped to expose the flaws in the pattern of reasoning. You are worried about AI ***due to employing the same pattern of reasoning, riddled with the same flaws**. Therefore, you are likely to be just as wrong as that other guy."
The risk due to AI is a function of the particular attributes of AI, flawed reasoning is only one part of the potential for error.
Another way of putting it: it is not difficult to encounter a Rationalist engaging in realtime cognition who forgot that they are only an *aspiring* rationalist. Decompose the problem into an explicit object model with attributes, behaviors, interconnections with other objects/phenomena in the world, etc, and Maya will/may dissolve in front of your eyes.
Or, we can stick with Bayes Theorem: heuristics + sophisticated post-hoc rationalization, fantastic food for Maya.
That doesn't seem any smarter than the strawman, though.
It is though.
The counterargument, included at the beginning of the article, is that the AI risk people are not employing the same pattern of reasoning. The coffeepocalypse person would then need to show how the reasoning was similar.
It's still arguing from an N of 1. "Here's _an_ example of...." is never a reasonable basis for sweeping judgments such as "____ will/won't happen."
If you're saying that two Ns of 1 each, pointing in opposite directions about a particular type of risk, rationally leave us with no basis for being sure whether or not that type of risk in a new instance is real -- that, I agree with.
But that is not what the real-world dialogue seems to be regarding actually a lot of big-risk questions. My experience is the same as Scott's: of people extrapolating from "here is an example where the worry turned out to be wrong and/or the logic behind the worry was wrong", to "therefore we know there is no risk." Which is absurd.
Why would it be absurd if the logic behind the worry was wrong? The argument is that the logic behind worrying about AI is different than the logic behind worrying about other risks.
"Why would it be absurd if the logic behind the worry was wrong?" Seriously?
I suspect now that you are trolling me but okay I'll play along....let's stipulate that the logic behind a worry about _________ being unsafe is wrong. You are seriously arguing that this stipulation actually _contradicts_ the worry? Proves _________ to be without meaningful risk?
No, disputing the logic that leads to a conclusion does not disprove the conclusion. Even if you arrive at a conclusion in an illogical way, that conclusion could still be true. But you don't know that it's true.
In other words, it would not prove ______ to be without meaningful risk. However it also would not prove that _______ has meaningful risk. Maybe you advocate treating everything as a possible existential risk until proven otherwise, but most people default to not treating everything as an existential risk.
No, it merely exposes your worry as being unjustified. It is possible that you should still be worried, but before you do, you need to find some legitimate reasons for doing so. Otherwise, you would be compelled to worry about literally anything anyone could imagine, and there's an infinite number of such things, so you'll waste all your resources worrying for no good reason.
But they don't analyse the pattern of reasoning that AI risk proponents are using.
They just assert it's similar to the coffee reasoning.
Ok but why didn't they notice that coffeehouses are an example of that particular pattern of reasoning _working_?
A maximally hostile and bad explanation of the prevalence of halibut method: for any question that seems hard, most people won’t even try to engage with it at an object level. What they do instead is look for any similar disagreements they remember and which side won them and try to pick the winning side to maintain social status. They get helpfully halibuted into the ‘right’ side of the argument by the nudge in the problem presentation and from then on they are invested into getting their side of the argument to win, spreading halibut arguments further. Note that at no point did they reflect on what the actual answer would be - that would be waste of resources.
This doesn't feel hostile to me. It is certainly how I approach a many important life questions, and I would be better off if I did much, much more of this.
I think that's something already pointed out by Daniel Kahneman: our minds try to solve an easier problem than the one they're actually asked.
Never mind "seem hard", just "seem sufficiently implausible" is already enough.
Remember high school maths: generation after generation are taught to calculate carefully, methodically, showing their working at each step; then when the final result comes along, they are told to check it - at the very least, plug it back in to the original question, and ask themselves: "does this actually look sane, or did I make a mistake?"
When a chain of otherwise convincing reasoning arrives at a sufficiently weird answer, and there is no immediate way to verify the answer against anything but that chain of reasoning, the instinct trained into most people since childhood isn't to update their priors; it's to look for the mistake. A person might not have the time or the relevant expertise to identify a mistake, but the assumption there is a mistake remains hard to dismiss.
This instinct is also what protects from, e.g., proofs of God's existence.
Proof of God's non-existence is much more fun, but too much fun for most people to bear. It is excellent for debugging how the conditioned mind works, I believe there to be no better topic (at least one that can be mentioned in public). Science would be #2, Democracy #3.
Even more fun: whether people who claim there is no evidence for God have a burden of proof. Do negative claims have a burden of proof, or do they not? If not, who made that decision, where is that decision documented, and do we know (JTB) it to be logically perfect?
I very much agree, but be careful: "reflect upon" can be represented as a boolean, but it is not actually. Watch out for the "good enough" effect.
Contra the coffepocalypse argument: the coffepocalypse is real and true. It's a drug that infected the entire world, and even people who don't drink coffee are mostly addicted through other sources of caffeine (tea, coke, energy drinks, etc). We can't fathom living without it, we got dispensers in every company, college, rest rooms. We feel like shit waking up until we get our first dose and getting a hit together is one of our main way of socializing. Our world runs on caffeine.
And the case against AI is the same: we're facing a potential future where the entire human civilization is dependent on it. Of course the same case can be made of cheap energy, of cars, or something else (insert that picture of a 26-lanes highway for impact). We're living on layers upon layers of technology & infrastructures we can't live without. Can the pile eventually crumble under it's own weight?
I admit, it's not exactly (at all?) the original coffepocalypse argument. But I stand by it.
That argument, as you say, applies to every single technology ever developed including agriculture. If you're insisting we should never have come down from the trees in the first place, okay, but that decision is done. We put all our chips on technological civilization and if we don't want to see eight billion people starve we have to play the hand out to the end.
Coming out of the water was a mistake.
"Contra the coffepocalypse argument: the coffepocalypse is real and true. It's a drug that infected the entire world, and even people who don't drink coffee are mostly addicted through other sources of caffeine (tea, coke, energy drinks, etc)."
I was tempted to post up something like this, haha. I avoid caffeine as much as possible - but e.g. right now I have a cola here because I had a particularly rough night, so I need the kick of some caffeine to get back to baseline. But I went weeks without. (Not judging the people who need their caffeine constantly, I just wasn't happy with how my body got used to it and couldn't function without it, erasing all of its actual benefits.)
To be clear, though, I don't think this is actually much of a problem. It just amused me that the original post the article is citing was about coffee.
So a fun fact, it seems that caffeine was made by plants to give bees a buzz.
For me the best version of coffeepocalypse is as a demonstration that no matter how obviously harmless a new technology is, people will be extremely worried about it and say it will destroy everything. This is such a consistent rule that it even applies to *coffee* (but NB also all other new things) so you shouldn't take any information from people being extremely worried about a particular new technology. I don't think this is actually true or valid, but it's the most optimistic interpretation I can think of.
It's a social argument! It's not meant to be deployed against "AI is dangerous", but against "Various people say AI is dangerous, therefore AI is dangerous". If you don't trust your ability to evaluate the technical arguments, all you have is what various people are saying, and this kind of thing becomes a critical consideration. The news says "Geoffrey Hinton, very respected AI expert, says AI is very dangerous, so maybe you should think so too". In that situation "A lot of authoritative people have thought things were dangerous in the past and been wrong" is a point very worth considering.
Even technically minded people can do this. If you've never thought deeply about the technical details of AI risk, but heard some bad version once and dismissed it from that point forward, then when you find someone suggesting AI is dangerous, you may assume they got there by adopting the conclusions of others, and try to convince them that this way of reasoning is error-prone.
Edit: If this is the case, the most productive response may be "I don't think AI is dangerous because others believe it, I think it's dangerous for the following technical reasons that have nothing to do with deferring to others' conclusions..."
Edit2: In the same way, I have often used the Rutherford and Szilard argument *as a response to* the then-common argument "experts predict AGI is a long way away, therefore it's a long way away", to which saying "sometimes experts predict things are a long way away when they aren't at all" is perfectly reasonable
"In that situation "A lot of authoritative people have thought things were dangerous in the past and been wrong" is a point very worth considering."
I think what Scott's saying is that there's a symmetrical:
"A lot of authoritative people have thought things were dangerous in the past and have been right"
So that's only true if "very worth considering" means at most it'll only shift your priors very slightly and likely not at all, like in the Russian tanks example.
So it comes down to how you feel about the person who is currently speaking to you. It boils down to "I'm the kind of person you should trust, therefore trust my argument." I think that this is what most casual conversations about technical issues come down to.
It may shift you a lot, back towards your priors. To start with, any given thing being very disastrous seems very unlikely. Then you learn that some authoritative people think it's very dangerous, and you assume this is a reliable indicator so you update a lot towards it being dangerous. Then you learn that authoritative people have thought all kinds of harmless things are very dangerous in the past, so you undo most of your earlier update and go back to thinking it's probably fine.
Further learning that authoritative people are also often right doesn't change this much, that's where you started from.
I think this is the key to the whole thing. The coffee argument isn't meant to update your beliefs but to un-update them.
Or, perhaps, update an entirely unrelated thing: what kind of person do you trust? A cool one like me, or those boring dweebs over there?
"If you don't trust your ability to evaluate the technical arguments, all you have is what various people are saying"
For a more real world example. Worked in R&D for a big company. Many, but not all the technical people (PhD's mostly) understood what tech would work and what wouldn't. The managers and maybe half of the R&D team had no ability to evaluate technical arguments, so did evaluation based on social prestige (trade publication cover stories / keynote speakers). So, managers always had approx half of their R&D team saying the same thing as the industry buzz and reliably choose to invest effort in those areas. The technical people would be frustrated by this, but always lost socially. Occasionally, someone would be good at tech and socially, quit and go start a really successful company.
Right, why is it so hard for us to say, "I don't know."?
Our training?
No problem at all. IDK, see? Trusting in my poor uncle yeshu helps, I admit.
I don't know. My wild guess is that admitting that reduces one's status sometimes.
It’s not hard, but if it means you don’t get a vote. At some point, someone calls a meeting and says, we have to decide X today. If most people who don’t know say so, then the meeting ends up being dominated by either (a) someone who really knows; or (b) some Dunning-Krueger narcissist. Which of these is more common, I wonder…?
If we are going to use the ostrich as a metaphor, I think it would be symmetrical to have a fable for the other side also. Perhaps Chicken Little saying the sky is falling? That actually turns out to be very old; one of the jatakas has Buddha, in one of his past lives, calming the fears raised by a rabbit after a similar minor incident . . .
I have to admit I started out puzzled over the whole AI Safety issue, for several reasons:
1. Nothing I've heard of in the realm of AI comes close to anything I'd regard as "Intelligence", and I haven't heard of any major steps in that direction.
2. Since nothing we have, or are contemplating, comes close to "Intelligence", we really can't plan an effective protection against the next-several-generations version that might conceivably be a threat.
3. Who would ever give any automated system enough power to control important things in real life anyway? It makes some interesting premises for movies (Like Colossus: The Forbin Project, or War Games, or 2001: A Space Odyssey), but really those are more ridiculous than interesting.
I mainly started to take the idea semi-seriously because Scott takes it very seriously, and Scott is not only very smart, but very focused on good reasoning and careful consideration of evidence in context. At least in some areas. ;-)
But I keep returning to something I saw in one of the Substacks, which looked at the evolution of LLMs and GPT in particular. If I remember right, it extrapolated versions of GPT and concluded that GPT4 used some enormous amount of training data and computing power more than GPT3, and GPT5 would need very much more, and GPT6 would need more computing power than the world could provide, and still would be unlikely to approach anything like independent intelligence. So I think the dangers from AI are still (almost) entirely theoretical, and far in the future, and so far away from the current world that preparations are (probably) useless. But I'm nevertheless glad that smart, conscientious people like Scott are devoting some thought and energy to the issue, just in case it does turn out to be important.
I was amused that Scott mentioned my favorite current issue: global warming. I'm not sure of his overall stance, and maybe he was only listing issues that others might find relevant, but "People didn’t worry enough about global warming" seems like the opposite of his argument. Judging by media coverage and politicians' rhetoric, people seem to be worrying a very great deal about global warming, even though there's very little evidence that it has caused significant damage (at least compared to other environmental issues of the past and present) or that it will in the future. When I make this claim, I don't consider media reports to be evidence, even the media reports that say "Scientists say". I trace to actual scientific claims in actual scientific publications, as compiled by the IPCC, and find some reasons for concern, but no current or projected problems that would be anywhere near the magnitude of the problems caused by serious efforts to rapidly decarbonize the world's economy, which seems to be the only solution on offer.
But maybe my reasoning is no better than the head-in-the-sand example. I often point out that the sky-is-falling reasoning common in public discussion is unsubstantiated, and the proposed solutions won't come close to solving the alleged problem, but even if that's true, it doesn't prove that there's no apocalypse coming. So maybe it's good that people are willing to spend a few hundred billion dollars on things that probably won't make much difference. Except that I still think we could spend those hundreds of billions on other things that would make a difference in other areas.
> Who would ever give any automated system enough power to control important things in real life anyway?
Elon Musk.
Ironic, because Musk was one of the earlier AI alarmists. I think he was - at best - epistemically lucky. Anyone that starts their own AI capability research is not taking the risk serious. The likely reason he ever signaled caring is because AI risk is the sort of "cool" "sci-fi" issue that was on brand for him to care about. But lately he has found the Culture War more interesting.
On the global warming aside I think a lot of the angst is a hangover from the whole RCP8.5 whoopsie where it was widely assumed we were on track for a +5C world in 2100 (the effects of which would’ve been quite bad for humans and other animals). It would be very helpful (for me) to read a ranking of the main risks for our current trajectory of +2 to 3C. Coral reefs are supposed to one of the first major Earth elements to go down which wouldn’t be great and a shutdown of the AMOC would be quite disruptive. Other than that I don’t have a good grasp on the major impacts (crop failures?).
With respect to the cost of decarbonization it’s important to include the benefits from cleaner air. Every year we are learning more about the deadly and disruptive effects of dirty air. This imo significantly shifts the cost-benefit for decarbonization and clean energy.
Granted that there may be major impacts from 2 to 3C of warming, do you think these impacts would be larger than the impacts of rapid and complete decarbonization? If so, why? Or do you think we should have some more gradual and modest decarbonization? If so, how much would that help? All the projected impacts exclude the effects of mitigation. Economic prosperity has allowed very significant mitigation of climate effects any place that has become rich. Caribbean islands were pestilential messes in the 19th century, to the point that British soldiers assigned there were assumed to be given a death sentence. Although most of these places are not among the richest in the world now, they're rich enough to have effective public health, potable water, and air conditioning that makes them pleasant places to live and visit.
It's interesting that you would say the IPCC findings are toned down to sound positive, and include only findings that are "100 percent ironclad evidence wise." I have no inside knowledge of the IPCC process, but the published reports include findings that are "very high", "high," "medium," or "low" confidence, and "high", "medium", or "low" evidence. It seems to me as an outside observer that the Working Groups try to be honest in their summary of scientific understanding, but the Summaries for Policy Makers are "jazzed up" to imply more dramatic results and higher confidence than the Working Groups report. Then, the IPCC press releases make claims that are not scientific ("we must limit warming to 1.5C") while claiming that "scientists have determined" these claims. Are you saying that all of these are "ironclad evidence wise?" What are the papers with the "merely good" evidence for tipping points?
On biodiversity, there certainly are challenges, as there have been for the last few thousand years. Most of these problems seem to come from disrupting natural habitats. No doubt climate change adds to these challenges. But consider an alternative scenario: warming is limited to 1.5C by getting to Net Zero emissions by 2050. How much agricultural mechanization, irrigation, transportation, cold storage, and fertilizer would be available in this scenario? What would it imply for agricultural productivity? If agricultural productivity is significantly reduced, how will people react? It seems to me they'd have to use much more land for agriculture to feed the population. Wouldn't this cause bigger problems for biodiversity?
I agree that we understand the economy much, much better than earth systems. It would be nice to eliminate our impact on natural systems. This isn't possible, so it's important to reduce those impacts where and as we can. There are certainly risks that we can't quantify, which means we can't really have high confidence that we can determine the appropriate tradeoffs. But that doesn't mean we must do anything and everything necessary to limit warming to some figure. And it doesn't mean that there are easy and practical paths to reach that figure.
I don't have specific sources at hand, but my impression is that the most significant dangers now are significantly increased food prices leading to more widespread starvation, widespread drought, large scale population migration, sea level rise, and damage from increasingly severe weather.
Whether or not any of that counts as "significant damage" is a matter of opinion, I suppose.
Yeah, these are some of the dangers currently being quoted in the popular media. The scientific basis for these is thin, if you look at things like the IPCC reports. By which I mean the IPCC Working Group reports, not the Summaries for Policymakers, and certainly not the press releases. And all of the predictions include the caveat "without mitigation" - which ignore the benefits from continuing to spread a modern economy to the poor world.
Got a source? Because frankly, "The globe is changing and we don't know for certain what the effects are going to be" pretty much terrifies me.
In my experience, it's much easier to break a complicated thing than it is to make it better.
My main source is the IPCC. Not the IPCC as described in media reports. Not the IPCC press releases. not the IPCC Summaries for Policymakers. The actual IPCC full reports. Most interesting are those from Working Group II and Working Group III. They're very long reports (Working Group II is 3068 pages), so I don't read them all. But any time I see some claim of pending doom, I consult the source to see what it actually says. For instance, many people claim that climate change will cause massive food shortages. IPCC Working Group II report, page 796, says something much more modest. I spelled it out in more detail in my comment below from Apr 25. If you have trouble finding it, it's the one that begins "I have a little different perspective".
I share your concerns about potential impacts - they could be much worse than I expect. But there's no science I'm aware of to say they're likely. Likewise, the boosters casually claim that we're in an "energy transition" that will soon have us all driving clean electric cars powered by clean electricity from sun and wind, with geothermal heating replacing oil and gas. The don't recognize that this can only be done at enormous expense, that would necessarily take resources away from every other personal and social priority. And no government, or any company, is acting as if such a thing is possible, despite lots of happy talk. Probably because public polling consistently shows that voters put climate change near the bottom of priorities for government action, and most voters aren't willing to pay even $10 per month to reduce the impacts of climate change.
"not the IPCC Summaries for Policymakers"
No offense, but that seems pretty bold. Are you claiming to know how to summarize the report better than the people who wrote it? Can you point to specific sections and/or page numbers in both documents that you think are contradictory?
I'm perusing the technical summary right now, and while they do not put specific numbers on projections (How many people will die? What % of crop yield will decline?) the picture they are painting is pretty dire.
Angry humans can cause significant damage, and causing hunger and homelessness is a good way to make people angry!
I have a little different perspective on the climate apocalypse narrative. As far as I can tell, it started with James Hansen testifying to Congress in 1988, when he predicted something like 6-8 degrees of warming (it's hard to be certain, because measuring against a pre-industrial baseline hadn't been started yet). This became the "catastrophe" scenario which justified Kyoto. By 1995, the IPCC reports were predicting something like 2-3C warming by 2100, but the "catastrophe" narrative had become established. The RCP8.5 scenario was never intended as a prediction: it was based on maximally pessimistic assumptions, but it fit the "catastrophe" narrative, so it got lots of attention.
If you want an assessment of the risks, I recommend the IPCC Working Group II report (https://www.ipcc.ch/report/sixth-assessment-report-working-group-ii/). It addresses each area, but doesn't attempt to rank them. The full report is 3068 pages, so you probably won't want to read it all. But each area forecasts rather modest potential impacts, most of which can be mitigated. In the area of crop failures, for instance, they predict something like 10-20% more people (65-130 million) will experience food insecurity due to climate change, without the effects of mitigation (p. 796). But "mitigation" would mean better crops, better fertilizer, better irrigation, better tilling, more mechanization, better storage, and better transportation, all of which are eminently practical, and all of which have contributed to vastly improved agricultural production over the past 60 years.
And yet most media reports still say "Scientists say we must limit warming to 1.5C to avoid the worst impacts of climate change." In fairness to the media, IPCC press releases say this as well. But the actual IPCC reports don't say anything like it.
I admittedly haven't dug into the details of the working group report as in depth as you, but I'm surprised by your internal framing of the quote you provided. Why do you consider 20% more people experiencing food insecurity a modest impact?
I found it strange that they talked about "food insecurity", rather than straightforward agricultural production, but the working group could only summarize the work of many people in the field.
I consider it a modest impact because it affects a relatively small number of people (1-2% of the world's population), and can be mitigated or reversed by technological and infrastructure improvements. But these improvements would be endangered if we focus on decarbonization. Actually, decarbonization would necessarily mean much less mechanization, irrigation, cold storage, and transportation, which I assume would have a much bigger impact on food insecurity (or agricultural production) than whatever impacts come from climate change.
I don't consider myself an expert, and I haven't gone through all the literature, but I check this stuff to compare it to what gets attributed to unnamed "scientists" who are quoted as saying we "must" do something. The actual scientific findings are much less dramatic than what gets reported.
Very interesting, that's an interesting perspective. Have you read anyone who's attempted to lay out an estimate of comparisons of those two things that you could share? This would be very different than my own internal understanding of the impacts which is that we would not be able to, with current technologies, mitigate the climate impact on food security in areas close to the equator so I'd love to read more.
I'm not aware of anyone who's attempted a rigorous analysis comparing impacts of (a) anticipated climate change given current economic activity vs (b) economic changes driven by some decarbonization path. Even stating the problem means that you'd be piling speculation on top of assumptions before you started the analysis. Whatever results you got would certainly not convince anyone who already has a stake in one outcome or the other.
Bjorn Lomborg has long been making this argument - that we'll be in much better shape economically, ecologically, and in human flourishing, if we keep using fossil fuels (reducing where it's practical) than if we try for immediate and drastic decarbonization. For this, he's been labeled a "climate denier".
To address your last point - yes, cleaner air would be a benefit. It's worth pointing out:
1. Rich countries today have cleaner air than they've had in the last 150 years. Cleaner air is not incompatible with a modern energy system.
2. Poor countries mostly suffer from "indoor air pollution" that comes from burning wood or dung for heating or cooking. Getting access to electricity, regardless of the source of the electricity, is the best thing they could do to get cleaner air.
3. There are currently no plans I've heard that credibly promise a reliable energy system with dramatically lower carbon emissions. The areas most committed to green energy (Germany, Spain, California) seem to be bumping against the limits of renewable incorporation, with occasional negative prices for solar energy (that is, paying other areas to take surplus solar energy) without coming close to eliminating fossil fueled energy.
3. Other than the one that France already built in the 1970s, I guess?
Well, yes. I'd say the problems with a nuclear future are (probably) more political than technical, but no one I'm aware of is even trying to build support for a lot more nuclear power, and I'm not aware of any country where the politics of nuclear are workable. Except China and North Korea, I guess.
>Who would ever give any automated system enough power to control important things in real life anyway?
You don't need to directly plug the AI into your nuclear defense systems or whatever. If people are making decisions based on the output of the AI, then the AI can choose it's output to manipulate the people into doing what it wants them to do. The answer to "who would give the AI that sort of power" is "everyone who might build the AI in the first place", since an AI you do nothing with is a waste of money.
And if the AI is connected to the Internet, it can do a lot of unintended actions more directly. Current "AIs" are on the web, how confident are you that we will stop doing that once we get close to true "Intelligence"?
Fair enough. I didn't say my reasoning was bulletproof.
Matthew Yglesias would say the media has negativity bias because that's what customers seek out https://www.slowboring.com/p/media-negativity
Very true.
I'd like to push back on a few things in your comment that seem mistaken to me:
> GPT6 would need more computing power than the world could provide
This might be true (although I think the cutoff would be higher than GPT-6) if we were stuck at current efficiency and current computing power. But the world is aiming to add much more computing power (qv Sam Altman seeking reportedly trillions to build chip fabs), and current AI technology is getting more efficient quickly so that it takes something like 8x less compute to reach any particular capability level this year than it did last year (more accurate numbers & citation on request).
> Who would ever give any automated system enough power to control important things in real life anyway?
I think we're just barely reaching the level of capabilities that begins to enable this, but I'd say we're already starting to see it. See for example Israel's 'Habsora' system for selecting kill targets in Gaza (https://bit.ly/3Qhj0dn); a human is in the loop in theory but not so much in practice. Ukraine has started to use autonomous drones to carry out lethal strikes without a human in the loop even in theory (https://bit.ly/3QjQojs).
> Nothing I've heard of in the realm of AI comes close to anything I'd regard as "Intelligence"
That seems surprising to me; I would claim that current state-of-the-art systems are already intelligent by many definitions of intelligent. That's especially true because I think of intelligence as (approximately) a scalar value; a cat has intelligence although not as much as humans, and certainly GPT-4 seems more intelligent than a cat in many ways.
I imagine it's a definitional issue? I'm curious about what you mean by intelligence such that current systems have none of it.
"Who would ever give any automated system enough power to control important things in real life anyway?"
Your enemy manufactures at scale a bunch of super cheap ($500 ish) AI powered cloud networked military bots. They are pretty stupid and easy to break (like star wars drones, but less humanoid), maybe some fly, some a quadrupeds, whatever. But there are just lots and lots of them, and they are good enough to take out trained soldiers at a 50:1 ratio and civilians at 5:1. You drop a million of them out of a airplane and they now control a city. So you decide to give control of your military to your own automated bots.
I feel this equivocates on "give power to control". If you drop *dumb* bombs on something, you're giving power to gravity, ballistics, and unpredictable bounces. These are all impersonal forces of nature that never have qualms about killing anyone.
Dropping a million bots on a city is similar to dropping a million bombs, except "we don't know exactly what square foot of enemy city the bomb lands on" is replaced by "we don't know exactly who the bot will kill". Perhaps you oppose war altogether, or at least bombs, but having a weapon that indiscriminately kills within a certain context is not a new thing and not something we don't know how to handle.
Sure - there are some things that approximate some level of intelligence. For example, some simple programs have been able to separate different varieties of flowers based on things like the sizes of different flower parts. Or facial recognition software is a sort of intelligence.
What I was referring to was something more like autonomous will - the ability to form an intent beyond its programming, to develop courses of action to pursue that intent, and to implement the courses of action.
I suppose GPT-4 might be more intelligent than a cat - it can recognize things as words, and can generally put the words together in a meaningful sequence. But I don't think it can recognize any concepts behind the words.
So, it seems that there *have* been steps made towards intelligence, contrary to what you said before? Because when you and I were young there were not computer programs that were as intelligent as cats in this respect.
I see.
> autonomous will - the ability to form an intent beyond its programming
I'm not, personally, entirely sure this cashes out into a coherent concept (unless you're a mind/brain dualist); it's like asking whether humans have the ability to think a thought beyond their brain. But in any case this is orthogonal to what I think of as 'intelligence', which may have been the source of my confusion.
> But I don't think it can recognize any concepts behind the words.
In my opinion this is an outdated (though commonly held) belief that traces back to the 'Stochastic Parrot' arguments from linguists; current language models are already doing concrete things (like math) that those authors believed they would literally *never* be able to do.
I think it's very much worth thinking about what, in your view, would falsify a claim that GPT-4 can't recognize concepts. The tests that I personally thought would distinguish (happy to talk about those if that's useful) are ones that LLMs have now passed, and I've updated accordingly.
Thanks for the reply. I should have made it clear from the outset that I know many people know a lot more on this subject than I do, and have thought in much greater depth than I have. You are obviously one. I'd appreciate your thoughts.
Is anything currently being done getting into the realm of AI Danger?
> You are obviously one.
I've certainly thought about it a lot, & am currently transitioning into doing AI safety research full-time, but that doesn't mean I'm not wrong! Smart, thoughtful people disagree wildly about this topic, and I don't mean to make any sort of argument from authority.
In particular, I've been thinking a lot about the Stochastic Parrot view and why people hold or don't hold it, and what falsifiable claims it implies, so I'm very interested in hearing from people who do.
> Is anything currently being done getting into the realm of AI Danger?
That's a big question. We're certainly at or near the point of starting to see potential dangers from misuse, but I assume that's not what you're thinking of.
My main near-term worry is about bad effects from scaffolded LLMs. LLMs can mostly be thought of as not having goals, but it's not that hard to wrap them into systems that are goal-ish, and that can make persistent efforts to achieve those goals (see https://github.com/yoheinakajima/babyagi for a simple example). Successfully working toward a long-term goal requires a lot of reliability, and current SOTA mostly isn't that reliable. But I expect that sort of agentic system to work better within a year or two, especially since at least some of the big scaling labs are actively working on them.
Mid- to long-term, I worry more about systems that are more intrinsically goal-ish, which I suspect can be achieved pretty well by reinforcement learning on top of an LLM base (we're already doing RL on LLMs, namely RLHF, but that's quite a thin overlay). We know that LLMs are already pretty decent at deception and persuasion (refs on request), and once we have a more intrinsically goal-oriented system it can be *strategically* deceptive and manipulative, as well as having the usual set of instrumental goals (resource-seeking, self-preserving) that come with any terminal goal. I mostly don't expect terminal goals to emerge spontaneously, but I expect some people to give these systems goals that can cause real problems, deliberately or thoughtlessly.
That's a rough, off-the-top-of-my-head sketch of the medium-term (the next five years or so). There are other plausible paths for problems to emerge, but I think that's already plenty of reason to worry, especially as these systems rapidly continue to get more intelligent (for most ways of measuring intelligence).
I do think we got lucky that the approach that's working best so far is one that's very non-goal-ish. If it was RL that had gotten this effective and was advancing at the current rate, we'd have substantially more cause for medium-term worry in my view.
Again, lots of smart, thoughtful people have substantially different expectations; this is just my own current thinking.
Thanks for your thoughtful response. I now know a little more. As I noted below in my response to Donald, I still (mostly) choose rational ignorance - I recognize that I'm not likely to know enough to have definitive views, but I'm glad other people are worrying about it.
I recognize that LLMs are pretty decent at deception - I assume the recent dust-up over Gemini is an example. I assume something very similar is behind the auto-censoring that Facebook, Twitter/X and Google use. I don't like them, but it sounds like your view of people who give the systems goals, which sounds more like a problem with the people running the systems than anything else. Don't we always have such problems? In what way does AI magnify them?
I can imagine someone using AI to try to access automated systems to alter them - to steal from banks, to alter election results, to change real estate registries (I don't think these are on line yet, so good there), etc. Isn't there already such risk without AI? With or without AI, we need reliable online security, which is a never-ending escalation between the crooks and the guards. In what way does AI change the situation?
> Habsora
Not to be confused with "Hasbara."
I think the AGI argument is dependent on the concept that AI will accelerate AI advancement, leading to AI that can independently accelerate its advancement
I find this plausible and scarier than coffee. We assume AI will not be able to independently defeat humanity but a more likely strategy is to enlist people's incentives such that AI plus 30% of humanity work against our true general interest. Cortez had a lot of native allies
An interesting conclusion of the proposal that the AI threat is completely impossible for humans to confront is that it would appear our best option is to accelerate AI while hardware limitations are at their greatest and seek to solve AI alignment with AI. Chat GPT3 was used to comprehend GPT2's underlying logic. Even if we don't succeed in creating a team of superhero shield AIs we could at least experience a hopefully very partial AI risk scenario and thereby recognize that serious steps actually should be taken
> it extrapolated versions of GPT and concluded that GPT4 used some enormous amount of training data and computing power more than GPT3, and GPT5 would need very much more, and GPT6 would need more computing power than the world could provide, and still would be unlikely to approach anything like independent intelligence
So basically, you're saying that people are compiling huge data sets for these next gen GPTs, and that we're possibly only one generalization trick away from a post-GPT AI that can distill better knowledge from less information, but you're somehow reassured by this state of affairs?
I won't pretend to understand how you define intelligence, but on only a strictly "capabilities" basis, AI is already dangerous to societies from an information manipulation angle, and the military is already commissioning AI fighter jets and weapon systems. We do know that AI that can generalize better from less information is coming, we simply don't know the magnitude of that jump.
So maybe we need a better definition of AI. The US military started work on a system called SADARM (Sense And Destroy ARMor) in the 1960s and deployed a weapon in 1995. It was an artillery-fired shell that included submunitions with sensors to identify armored vehicles and explosive-formed penetrators to kill the identified targets. Would this count as AI?
I assume that current technology can do more things more independently, but is this a difference in kind?
I'd say what falls under AI should be general purpose than special purpose devices like the one you're describing (like LLMs for language, computer vision for sight, etc). I suppose that depends on whether the sensors are special purpose ones designed specifically to detect armored vehicles, like a thermal sensor is specifically designed to measure temperature, or just straight up camera system and the detection is done via some kind of computer vision model. That could possibly be considered rudimentary AI.
Great comment, Brian. One minor push back… how are you defining “intelligence”?
I see AI as very intelligent in at least some narrow ways and getting both more intelligent and broader every day. I also see it as amplifying human intelligence. But I define intelligence as “the ability to process information so as to efficiently solve problems and exploit opportunities.”
I'm far from an expert on the topic - I don't have a well thought out definition. My main idea in this area was "smart enough to develop a malevolent intent, and an agenda to implement it."
I recognize that AI can do some things well that have always been considered areas for human expertise. But I don't see those applications as a danger, unless the AI can develop a malevolent intent. Of course, humans could take over the AI systems (or their reporting systems, or inputs) and do malevolent things, but I don't think of these as "AI Safety" issues. Should I?
Humans seem to have a non-trivial ~automated aspect to them, as does culture, society, etc.
Imagine if we were doing everything wrong! That would be funny.
> 1. Nothing I've heard of in the realm of AI comes close to anything I'd regard as "Intelligence", and I haven't heard of any major steps in that direction.
Computers went from literally 0, to chatGPT. Now the scale of intelligence that a lot of people seem to think on goes from dumb human to smart human. And so sure, we don't have anything that seems smart to us (ie is smarter than the average person) yet.
2. Since nothing we have, or are contemplating, comes close to "Intelligence", we really can't plan an effective protection against the next-several-generations version that might conceivably be a threat.
That means it's somewhat harder to plan and protect. Not impossible. Some people are doing things that look like they might help.
3. Who would ever give any automated system enough power to control important things in real life anyway? It makes some interesting premises for movies (Like Colossus: The Forbin Project, or War Games, or 2001: A Space Odyssey), but really those are more ridiculous than interesting.
There are 2 threat models. The model where you connect the AI directly to something important. And the model where you try to seal the AI in the finest Airgapped faraday cage, and it hacks it's way out.
In reality, people are often hooking these things up to the internet. And the internet is connected to a lot of important things.
Humans aren't in charge because the other animals decided to hand us something important.
Is our security good enough to keep the AI from touching anything important?
> and GPT5 would need very much more, and GPT6 would need more computing power than the world could provide, and still would be unlikely to approach anything like independent intelligence. So I think the dangers from AI are still (almost) entirely theoretical, and far in the future, and so far away from the current world that preparations are (probably) useless.
It's hard to work out how smart GPT5 might be, not having seen it yet. But sure. I will grant you that just scaling GPT level tech quite likely doesn't get to dangerous levels of intelligence.
GPT tech is something that was only starting to appear 4 years ago.
There is lots of research happening on various improvements. If there was some new more efficient algorithm was invented next week, we maybe get a couple of years of impressive headlines before a singularity in 2027. New algorithms can be and are invented reasonably frequently. We have no strong evidence against a near singularity.
Don't assume that progress will be just scaling up, not algorithms.
Thanks for the notes. I was writing my initial thoughts, and why I was initially puzzled by the whole concept of AI Safety. I recognize that I must have been missing something, or smart, level-headed people wouldn't worry about it as much as they do.
I opt for rational ignorance on the topic - I recognize I don't know nearly as much as people who take AI and AI Safety seriously. I don't think I could learn enough to know whether any particular view is convincing. I'll let various better-informed people continue to act and argue among themselves. I won't try to figure out how real the thread is. Unless someone starts using it as a basis for fundamentally restructuring the world's political and economic systems.
This argument feels weirdly outdated. It's hard to imagine claiming GPT-4 today doesn't "come close to anything [you'd] regard as intelligence". This is basically the argument I made pre-GPT4. At the time I also thought "I'd never give an AI any real unchecked power" only to discover when GPT-4 _actually_ came along just how valuable it is to say "Write a script to go through this repo, `dvc pull` and parse all the onnx files, and list the sorted, dedupped names of all inputs you find alphabetically to stdout" and then run the code it produces without even a preliminary glance at it. Why waste time checking it's work when I could just run it and see if it works?
It's simply too valuable to have a tool which does the coding work for you and that value is lost if you constantly have to check its work
Fair enough. My terminology was overly broad - I was focusing on AI Safety, not AI in general. What I should have said would have been more along the lines of "Nothing I've heard of in the realm of AI comes close enough to independent action coupled with malevolent intent to indicate a problem fundamentally different from buggy software."
Also, I was describing my previous attitude, and why I didn't understand the issues of AI Safety. I did mention that I take the topic more seriously now, and I'm glad people are paying attention to it.
On your software use: What's your distinction between "run it and see if it works" and "you constantly have to check its work?" Do you run it once to make sure it doesn't crash? Do you assume it's doing the work you want, and not something else? Do you check different conditions to make sure it can handle them all? I accept it's a useful tool, but some users have definitely trusted it too much. For example: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/
> a problem fundamentally different from buggy software
I don't think AI doom _is_ fundamentally different from buggy software. Imagine you have some flight control software that has a model of what angle the plane should fly at and then makes fine adjustments to keep the plane at that angle at all times. There's a pilot to take over if something goes wrong in the flight control software. Some bug causes the flight control software to decide the plane should be pointed straight down at the ground. All the other pieces of the stack work _correctly_ in that they cooperate to keep the plane pointed in the direction the model points it, counteracting whatever the pilot does to try to correct it.
This is not an AGI, it's just buggy software, but it got everyone on the plane killed because of a _high-level_ bug. Contrast with a low-level bug where it misinterprets one of its sensors or uses the wrong multiplier for one of the jet speeds or something like that.
Misalignment seems to me like a similarly _high-level_ bug in AI software. The parts of the stack that make it able to reason intelligently and path-find and whatever else work _correctly_, but the part that decides _what to do_ with those capabilities goes wrong somehow. I don't think it's necessary to find "examples" of malevolence. Unless you reject the assumption our AI software will have bugs, the result will be "incidental" malevolence like in the plane example. There isn't much in the way of "independent action" right now, but that's likely to change in the coming months.
Check out Devin https://www.cognition-labs.com/introducing-devin to see what people are already doing. When you think bugs in AI, don't think bugs in ChatGPT. Imagine ChatGPT works correctly, but _Devin_ has a bug. How is that realized?
In my software use-case, "see if it works" just means "see that it produced a non-empty list of tensor names and that it didn't raise an exception". If it did that, I generally trust it didn't miss any. I'm imagining a one-time use-case; not a general tool, so what circumstances it doesn't work in doesn't really matter so long as it works this time.
In particular, if the code _also_ goes into my `.ssh` folder, grabs my not-password-protected private key, and uploads it to a server so that some other piece of software can start amending early commits to my open-source projects on GitHub in order to inject security vulnerabilities, I probably wouldn't notice.
I just played with GPT4 again, asking it some questions about molybdenum chemistry. https://chat.openai.com/c/c279d911-e0ac-419a-b4a3-d472386d7c48 It said some reasonably sensible things, but it _also_ said that molybdenum in a MoCl4- complex was in a +4 oxidation state, when it should have said +3. It also needed prodding to consider chloride complexes at all - and this is in 6N HCl !
The coffee argument attempts to bolster the idea that risks that seem ridiculous are ridiculous and that things suggested to pose theoretical risks via novel predicted pathways are usually wrong. It differs from plague and war in that plague and war are bad in very predictable ways and that coffee and AI were suggested to be bad in heretofore unforeseen ways that sound stupid.
"Computers will make computers so smart they could outsmart everyone at everything, at which point they'll kill us all" is in fact a novel and stupid sounding argument. I just happen to think it's probably true.
Yup I think this is totally on the money. I think it's a combination of this and a bit of the scale component. It tries to argue that both "risk from AI" and "risk from coffee" are silly by implying that just because coffee was so ubiquitous (like AI will be) that it must be dangerous is a bad argument.
My take is related to ""Twitter derangement syndrome"". I often see smart people on twitter transition from reasonable (but ofc with biases etc) tweets, who avoid the dumb on the other side of any argument and focus on the interesting ideas of the other side of an argument. They transition to meme broadcast. Musk for instance.
I think this happens because (a) takes on issues aren't independent correlations, (b) the algo pushes you to the pattern of reading the smart people on your correlation side, and dumb people on the other correlation side.
So you get ego-massage ("I also think what this very smart person thinks - I could have said that if I had more time ...") and outrage ("I can't believe whats in the heads of the people who think the opposite to me.")
At this point you may start to become tribal, *you* change. Whats the point of reason if the other side don't accept it? Just firehose them with vibes.
So my take on the coffeepocalypse argument is they are no longer trying to persuade you. They are trying to influence the 99% of people that are tribal that they see also make dumb arguments.
Could it be that the more smart people act tribally, the more oneself does. TDS is infectious.
Perhaps the 'grey' tribe subconsciously operates with the world view analogous to trickle-down economics: The tribe will adopt what their own thought leaders think. And those thought-leaders are susceptible to argument as the redistribution of the wealth of the rich-in-reasoning. But social media isn't well-designed for that to work well?
This is all black and white. I don't mean it so - we all have these elements running through us at different times of day and different issues. TLDR, it's not you the person is trying to convince. That's not where the engagement reward comes from anymore.
Not a new argument on reflection. Oh well.
I don't understand why you feel the need to understand these arguments. Between this and the last one, https://www.astralcodexten.com/p/contra-hanson-on-medical-effectiveness, it almost seems like you're beating up on the intellectual weaklings.
Why not ask something like what is it about YouTube that attracts bad comments?
Idiots abound. Someone is ALWAYS wrong on the internet. https://xkcd.com/386/
One is a twitter thread. The other is a health economist who has been making an argument for decades and whose interlocutors include multiple Harvard professors (including its current interim president) https://www.cato-unbound.org/2007/09/10/robin-hanson/cut-medicine-half/
So you're saying that because others listen to them, some of which may form policy, that it is important to refute the arguments? It seems to me, rather, that those that form policy who are swayed by such arguments are the main problem here, and ought to be removed.
But arguing against absurd arguments accomplishes aught.
Yes, it is more important to refute people that other people listen to.
"[...]because others listen to them, some of which may form policy, that it is important to refute the arguments?"
I think yes? At least to make sure to avoid forming a consensus-by-default? Example: "all these people are saying the Earth is flat, and so far I don't see anybody contradicting them, therefore they may be right". To follow on this example, I don't think one should argue with flat-earthers per se, it is impossible, but at least to state unequivocally that "the Earth is round" is sometimes useful. Not to them, but to others.
It probably doesn't help that despite spending years patiently and logically explaining why the threat of an AI apocalpse is nebulous but real, the ACX comments are regularly filled to the brim with people who apparently wilfully refuse to consider the issue properly. They're obviously not just 'intellectual weaklings' across the board or they wouldn't be reading ACX. Refusing to take weird/new/scary/theoretical/left-field arguments seriously just seems to be such a baked in glitch in human reasoning that even smart people fall into it a lot if they're not making a conscious effort to be open minded. I find this puzzling, infuriating, and terrifying, and if I thought I could reason my way into believing it wasn't that simple, I'd leap at the chance.
You're probably right that comments offer an impression of readership that skews toward the contratrian, hostile, and hare-brained. It's certainly true on YT, and hopefully it's true here as well.
Oh god I thought I was going insane, because I regularly see people say "I found lots of convincing arguments against AI risk" and when pressed, they link to something as bad as coffeepocalpyse, and then 5 other AI risk skeptics come in and say "this proves the gigantic blind spots the doomers have, this comments section is too pro risk".
I think you are right about the comment skew. However, I think arguments are also strong that:
1) humans are on a trajectory to catastrophe without AI, and higher intelligence can both accentuate and mitigate this (IOW we need to consider the catastrophe reduction potential of superior intelligence and coordination)
2) there is an inevitable arms race toward AI, so it will not be stopped. At best it can be partially managed for the when (not the if).
3) regulatory bodies are as likely to make things worse as better based upon their track records.
Agreed on 2. Idk about 1 and 3, but even assuming they're correct, encouraging people (especially the people who actually work on AI) to take this issue seriously seems like it will very likely have some kind of positive impact on expected outcome. It might not move the needle as much as we want, but it's still something, which with these kinds of stakes is very worthwhile.
I agree.
It's always good to try and understand why people who make arguments you disagree with are wrong because what if they're right and you should change your mind.
Generally speaking, one should be skeptical of anything and everything, but that is the general case. Consider in https://slatestarcodex.com/2014/04/15/the-cowpox-of-doubt/:
"someone who demands that you be open-minded about homeopathy would not be your friend. They would probably be a shill for homeopathy and best ignored."
Is it possible someone might have new information about homeopathy you haven't considered that would prove it is actually effective? I have to say, it is possible, but so unlikely it doesn't merit my consideration (since I have previously considered homeopathy, and new evidence must overcome my well-earned priors). One only has so much time to devote to research, and far too many hypotheses are proposed to research all of them, especially personally.
The assumption here is "once people were worried about coffee, but now we know coffee is safe. Therefore AI will also be safe." This is so obviously flawed a chain of logic it doesn't merit consideration. It is, perhaps, an interesting question as to how people could use such "logic" to form opinions, but it is in the same vein as superstitions, so a more general class of thought processes.
Its an error of overgeneralisation:
People worried coffee would be dangerous, and were wrong - correctly leads to the conclusion that *its possible to worry about new things being dangerous, and be wrong* - but it absolutely *does not* lead to the conclusion that *its possible to worry about new things being dangerous, and be wrong, therefore any time anyone worries about new things being dangerous, they are wrong*.
The Halibut example is a little confusing because its reverse the negative (not) to are. Suppose it was this, leaving the NOT in place:
A guy thought Halibut was a type of cow. He was wrong. Its not a type of cow. Therefore AI is not a type of cow.
Now, its almost certainly true that AI is not a type of cow (unless the Mootrix hypothesis is actually true), but the fact that AI is almost certainly not a type of cow does not logically flow from the fact that Halibut is not a type of cow. The two statements are unrelated.
So too does the fact that coffee turned out not to be dangerous (except in edge cases where people drank over the LD50 which is about 10grams or 100 cups of coffee and died of a heart attack). The fact that coffee is mostly not dangerous is completely independent of the as yet unknown safety or danger of as yet uninvented AI.
The better argument for me is to think about the Omohundro AI drives argument, with relevance to human intelligence. Omohundro argued that without specific reasons not to, any AI with motivational drives would behave like a human psychopath in its pursuit of resources. AI at that level doesn't exist yet, but there are plenty of examples of the Unfriendly Human Problem, aka human psychopaths, or just plain mean people. The fact that Unfriendly Humans *can exist* strikes me as far more relevant to the hypothesis that Unfriendly AIs *may be able to exist*.
By analogy from humans with specific types of brain injury (my field), most present AIs are analogous to being disabled in many ways. ChatGPT is blind for example, you can't show it anything, it can only respond to text. Its also pseudo in its theory of mind capabilities - you could cry while typing a message to it and it wouldn't know, although it might respond sympathetically if you typed about how sad you were. By extension, it can't really empathise, no matter what it "types" in response. It is psychopathic by default because no one has solved the AI empathy problem yet. If we solved the AI problem for general intelligence but didn't solve the empathy problem, we might be in trouble. The fact that high IQ psychopaths do exist means that psychopathic AI *might* be able to exist, but is not a proof or certainty.
A guy thought Halibut was a type of cow. He was wrong, because his conclusion that it's a type of cow came from a bias in favor of thinking things to be cows. So any claims that AI is a cow should be treated skeptically and treated as possibly biased.
If there actually was a nutcase running around calling things cows, that probably would be how you'd react.
Likely this was said above and I missed it, but the Russell argument from nuclear fission is (arguably) from a much smaller reference class: the first time we invented a plausibly catastrophic technology, some experts confidently predicted it was impossible. Thus, Russell’s should be more convincing than the coffee argument, which is presumably sampled/found from a much larger class of worried.
(larger class of *worries*, that is)
For me, it's about remembering past moral panics. I'm not saying that AI couldn't be really bad. I'm just saying that lots of ostensibly smart people believing it's going to be bad is not, in itself, a strong argument. This is especially relevant considering the information cascades that do happen in our community. "Eliezer believes in AI doom and he's really smart in [some other domain], therefore I'll update towards his position on AI doom".
I'm pretty much an AI bear. While I agree that AGI is not impossible, I don't see strong evidence that it's around the corner. In fact, I have the same frustration as Scott in reverse. Lots of people gesturing wildly at stuff that is clearly not AGI, then expecting me to freak out, without addressing the core of my skepticism.
P.S.: My favorite moral panic I remember from childhood is the idea that we were going to bury ourselves in mountains of trash. I looked up some old news clips from the late 80's in the archive of my local tv channel and it was quite funny to watch.
lol
From the inside, freaking about GPTs isn't about thinking that GPTs were bad, but that a fairly core part of my "oh yeah AI is just far away" model around 6 years back was: "yeah, all the people who confidently said that Kurzweil was dumb by extrapolating Moore's law, they seem to be right. We don't have any kind of scalable algorithm, adding more hardware only leads to incremental advances (like 10% less loss, or sentences that are slightly more coherent in a sea of nonsense) for much hardware cost."
Maybe I am overreacting, but it seems concerning that my and many other people's intuitions were too pessimistic, and if I start examining other AI constraint intuitions, and start comparing them vs actual AI progress, it seems the entire constellation of intuitions were all too pessimistic. Since the bias was pervasive in the world model, I don't think it's surprising that I would change my mind about the total. (Compare to estimating the cost of building a skyscraper, and then finding out that you were pricing everything's cost at 10x their value and effort. That bridge price is going to change by more than 10x, because the crews working on it will also be spending less time)
>P.S.: My favorite moral panic I remember from childhood is the idea that we were going to bury ourselves in mountains of trash. I looked up some old news clips from the late 80's in the archive of my local tv channel and it was quite funny to watch.
Yup! Pete Seeger even sang about it: https://www.youtube.com/watch?v=0ZesRAo5PBg
"I would like to understand the mindset of people who make arguments like this"
I think a lot of people who do this aren't making logical arguments. They're signalling some combination of
1. Relaxed and confident leader: "don't be anxious about this stuff, be chill. Look, I'm chill."
2. Be optimistic despite things in fact being bad: *artillery raining down on trenches "We've always gotten through wars in the past, this time we'll make it too."
3. Caution against the village weirdo, adversary, and/or "boy who cried wolf."
4. etc.
These things are social signals and/or "just-so" stories (the heuristic arguments you allude to). Yes, on further inspection they aren't logically correct, but they do have utility, which is why they have survived as standard ways of communicating. I'm not defending their behavior, just observing!
In terms of Zvi's simulacra levels, people are operating on different simulacra levels and jumping around the levels. When you engage on level 1, suddenly it's a snap to reality instead of social signals and other goals, so it's a sudden incongruous incompatibility with what they're saying. You're talking about different things. It's like the fearless leader getting annoyed "Hey what are you doing saying we might lose the war, I'm trying to calm everyone down so we can keep people showing up to work."
Then there's also stuff like "do I want to think of myself or be perceived as the sort of person who would lie or be incorrect about something" and/or true curiosity about the truth, which is why they don't have a response to you or are even surprised/annoyed that you've said what you've said.
In terms of behavioral biology, you're snapping on the executive function/logical reasoning parts of the brain and asking them to allocate more compute here when someone was going more off vibes, social calculation, feels. This can be uncomfortable or even perceived as socially adversarial.
Disclaimer: I made it all up and you probably know most of what I wrote. To be fair, I understand a large purpose of the post was to introspect about our own ways in which we make these same mistakes, rather than point out that people make a certain category of mistakes.
There's also the question of the degree to which humans have free will, in general (at their best), and during each individual instance of cognition.
ie: it seems plausible that humans do have some control over the degree to which they will engage System 1 for analyzing a problem, but if one is raised in a culture of intellectual laziness, they may tend to take the easy way out.
As they say: "Ain't nobody got time for that", "Time's a wasting", "Thought is time", "Everybody wants to save the world, but no one wants to help Mom with the dishes".
Possibly related:
https://youtu.be/3b2d7p7I2GQ?si=zej4Yp5ujpt7GVIx
https://youtu.be/I9e_85TuaBA?si=AmlouNYnoBTu138Y
People making these kinds of “Thing A didn’t happen therefore Thing B also won’t happen” assume that (or act as if) your worry about Thing B is based solely on feeling the emotion of apocalyptic fear for the very first time, and submitting that alleged novelty of emotion as your only evidence that worrying about Thing B is rational. So, they believe that if they can attack the novelty of that feeling (“people have felt apocalyptic fear before!”) your argument has crumbled. They are not thinking of your apocalyptic fear about Thing B as a downstream consequence of other evidence at all, so they don’t think they have to address the upstream evidence.
Rhetorics. Shiny things owns us. Here's a nice similarity, there's a touching picture, there is a charismatic person.
Noticing these influences and fighting them to consider only the rational backbone of things is for very few... if any.
Could it be that someone just wanted to write about coffee and knew that pretending his coffee-story was actually about AI would get more clicks for the coffee-story?
I think people like Eliezer and Conor Leahy predict doom so confidently and occupy so much attention space here that they cause these arguments to look good as existence proofs.
Even putting those cases aside, people typically ascribe high confidence to most public claims people make and this implicitly lowers the bar for what counts as acceptable criticism. Original claim-makers are trying to model the world; critics are just trying to raise considerations.
I think this is a typical failure of thinking probabilistically. A lot of people basically think all probabilities are zero or 1; they hear someone say the probability of AI killing us all is non-zero, and hear that it's 100%. The coffee argument is a perfectly fine argument that the probability of AI killing us all is not 100%; having demonstrated it's not 100%, they conclude it's zero.
I was surprised Scott didn't mention this, as I think it is one of the strongest arguments for his side of this debate; catastrophic risks are not unimportant just because they are tail risks. A 1% probability of (say, catastrophic global warming) is a very serious risk, even though it is likely to to not happen, and people will stupidly point fingers about how "wrong" the experts are.
Catastrophic but low probability risks also lead to Pascal's Mugging.
But where does the low probability come from? I would be skeptical about somebody who said there was a 99% chance that AI would kill us — where do they get that number? I am similarly skeptical of someone who says there is only 1% chance.
I have noticed this, too. It comes up a lot in personal assessments of risks; lots of people will shrug off a seatbelt thinking "it won't be me in a crash!", others develop phobias of typically harmless things due to one particularly bad run-in ("i will never approach a dog again, they're all rabid little bite machines!") See also Scott's excellent post The Psychopolitics of Trauma.
I'm not registered with twitter (and don't want to be). All I see at the 'read the full thread here' link is the tweet with the coffeecup image. I might be missing the link on the page, but I have looked and can't see anything obvious. Is there a way for me to read it?
As a general tip, sometimes you can unroll Twitter threads with sites like https://threadreaderapp.com/, but it depends on whether the site has seen the full thread before.
Thank you!
In this case it works like a charm.
Here is the link it produces: https://threadreaderapp.com/thread/1741445839053025450.html
Many Thanks!
I believe you are overthinking this.
First, most people haven't studied AI or any of the other potential global threats in detail, so they default to the easiest and most reassuring response. Anticipate this sort of reasoning and when you encounter it, engage them in a discussion that brings out the bad things that HAVE come true. Keep in mind that people don't want to seem ignorant, so this is also a way for them to avoid a discussion they know, they don't know too much about. But they actually know more than they think. You can tease out what they do know and provide them with facts they didn't know using all sorts of techniques such as storytelling etc. You can even discuss literature and movies which is an easy way to get people to engage. Talk about Isaac Asimov and the three laws. Or the movies Colossus and War Games.
Second, be sure to acknowledge their other, more legitimate concern, which is that it is difficult to know what to worry about or what the best solution for a problem is because science, by its nature, changes as more is learned. Vitamin C is good for you, (Linus Pauling is a genius), then it's not, (Linus Pauling is a crackpot), then it is good again. If you worried about every possible bad thing that might happen, you could drive yourself nuts. So just be optimistic and focus on every wrong prediction when some distressing topic comes up.
Third, you need to steer the discussion away from heuristics so it addresses the known and unknown facts. AI may pose a threat, but governments are trying to work together to head off the problem. Then discuss why it is important for the public to understand the threat so that they can encourage their lawmakers to enact legislation or enter into treaties to address it. The same can be said about climate change. Make it personal. For instance, I am old enough to remember when we had something of a winter in South Florida. Moderate temperatures for over a month instead of one day in January. We need to be better prepared for the next pandemic. My sister died because of Covid. We have to stop Putin--most of my family in Europe were killed by Hitler.
Don't be put off by wrong thinking. Just get people to think better.
Noticeably absent: a lack of error checking of your own cognition. Have you forgotten what Morpheus said to Neo in the training simulator? 😱
The coffee example is particularly bad because he refuted himself in the same thread.
“Kings made arguments against coffee because it was a threat to their rule”
“It was indeed a threat to their rule”
“Therefore, they were ridiculous to believe that”
AI arguments made in this manner would look like:
“Humans made arguments about AI killing them all”
“It did indeed kill them all”
“Therefore they were ridiculous to believe that”
Coffee is pretty clearly not a threat to anyone's rule. The threat seems to have come from coffee𝗵𝗼𝘂𝘀𝗲𝘀, not coffee, but this is also difficult to attribute.
Gathering was recognized as a real threat long before anyone had heard of coffee. That's why the Roman emperors banned firefighting clubs. The problem wasn't the firefighting - everyone knows that's good - it was the clubbing. Other types of clubs were also banned, but we have better documentation on the firefighting clubs specifically because someone thought they were such an obviously positive force that they might get an exception, which they didn't.
What I find weird about this is that public restaurants are far, far, far older than bans on social gathering. I see no real difference between bars and coffeehouses.
If you believe my analysis, I see basically two options:
(a) Coffeehouses were no more threatening than bars, the revolutions were sparked by other things, and if the coffeehouses hadn't been open, revolutionaries would have met in other locations.
(b) The set of existing social norms around bars prevented them from generating the same social bonds that are expected in clubs, and that actually formed in coffeehouses. In this case, the coffeehouses do have some real causative force, but the revolutionary effect would have been easily avoided by regulating them more harshly.
If we had psychedelics houses, or philosophy houses, or maybe a combination of the two, would 100% of the risk to existing power structures derive only from the houses?
> I see no real difference between bars and coffeehouses.
Of course. Do you form any conclusions as a consequence?
Well, opium dens were a real thing. Opium is not a psychedelic and my understanding is that the characteristic behavior of the clientele was to basically just lie there in a stupor. My first guess is that psychedelic houses would be similarly nonthreatening.
To the extent that coffeehouses were threatening, it seems to have been because they not-so-secretly functioned as philosophy houses where you could also have an exotic beverage. (Could you get food?) Philosophy houses would presumably be even worse.
But philosophy practiced on an individual basis is also fine. That's another thing that is very old and that has been handled well by various large civilizations.
There is a book I liked very much, Patricia Crone's Pre-Industrial Societies: Anatomy of the Pre-Modern World, which describes a dynamic that seems relevant here.
Crone characterizes societies as tolerating a certain level of wealth accumulation and a certain amount of thinking. ("Philosophy", in what I take to be the terminology of your comment.) The structures of any given society, according to her, can accommodate a certain amount of each, but tend to fall apart when those bounds are exceeded. At that point society is reorganized into something that can operate well at larger scales.
You could make the argument that the coffeehouse revolutions are an example of this, that the increasing wealth of Europe gave people more time to sit around thinking about politics and this eventually made the surprisingly small-scale social structure of medieval Europe untenable. But this would be a pretty clear example of option (a) that I gave above. The coffeehouses would be just as irrelevant as the coffee.
If you were to use a methodology other than guessing, do you think you may get different results?
Also: did you notice you didn't actually answer the question? I did.
> If you were to use a methodology other than guessing, do you think you may get different results?
No. That's why I think guessing is adequate here.
> Also: did you notice you didn't actually answer the question? I did.
Well, you edited your comment after I'd responded to it, which isn't a good look if you're going to go around insinuating that people have no responses to your points.
Then again, you don't really appear to have points, either. What do you think I missed?
1. The threat was explicitly characterized as coming from the gathering and not the coffee.
2. This comported with more than a thousand years of recognition that gathering could foment revolution.
3. It also matches our retrospective analysis in the present day.
4. Making it easier to revolt doesn't automatically mean people will revolt.
5. Making it harder to revolt doesn't mean that people won't revolt.
6. This makes it difficult to attribute revolts. But in most cases, they are overdetermined; any single factor whose absence would have prevented the revolt will be glaringly obvious.
7. Psychedelics houses do not pose a plausible risk of societal destabilization.
8. Philosophy houses do.
9. To the extent that philosophy in the absence of social clubs (or, synonymously, factionalism) poses this risk, (a) coffeehouses posed the same risk independently of coffee, and (b) so did everything that wasn't coffeehouses; the destabilization was going to happen with or without the coffeehouses.
10. To the much greater extent that philosophy houses pose a risk because they are social clubs, coffeehouses posed the same risk in exactly the same way, and this obviously has nothing to do with coffee.
All of that is already contained in my two previous comments. You might note that in both cases, 100% of the risk specific to the establishment is attributable to the fact that it is an establishment.
Philosophy houses aren't going to overthrow the king because the promotion of philosophy leads to better, more effective philosophy. But they can do it by giving people a safe space in which to discover that they share certain unflattering views of the king. If the king is going to be overthrown by the raw power of philosophy in practice, the houses won't contribute to that.
Psychedelics houses aren't going to overthrow anything, even in extremely favorable circumstances, because the customers don't organize.
Is all of this not also produced by ~guessing?
Which of the many meanings are you assigning to the term "is" (and variations of) in this context, *technically*?
Having read the coffeeopaclipse thread (here: https://threadreaderapp.com/thread/1741445839053025450.html ), I think the answer is that it's actually not arguing against world-ending AI.
It's arguing against the backlash against AI.
There absolutely is a backlash against AI - an online community I'm part of has outright banned any use of AI generated images and is vehemently against AI art. People also worry about generative AI being used for cheating on homework, taking peoples jobs and more.
And I think it's fair to say some of those fears are legitimate.
However, the twitter thread's argument is that technologies are sometimes feared for their potential to change the world, but then become accepted for their benefits - even if those changes do actually occur.
There /is/ a brief dismissive mention "doomer predictions of the end-times" towards the end, yes. However I think this is basically incidental, it's not really what the argument is intended to be about.
So wait, the argument is just "let's just do things even if we know it will have harmful consequences because it's inevitable anyways?" Christ.
No of course it isn't. Flying Spaghetti Monster.
If someone is arguing that AI isn’t bad because while it may kill us all, there could still be benefits, then I’m not exactly going to cheer it on.
Yeah, so I think you've been mislead by Scott saying that it was an argument against AI killing us all. What I'm saying is it's not talking about that at all. I think we can assume he isn't concerned about that.
When he talks about "massive backlash" and "battles" over AI, he's instead talking about the actual mainstream concerns of AI taking peoples jobs etc. (At least, this is my interpretation.)
The thread with the coffee analogy is saying people fought about other disruptive technologies, but after they've been around for a bit and accepted, people largely forget about the controversy.
That's presumably why he was talking about coffee rather than a more recent disruptive development - say, the automated loom - more people still know about the disruption that caused.
>he's instead talking about the actual mainstream concerns of AI taking peoples jobs etc. (At least, this is my interpretation.)
Hmm... Well, labor-saving/livelihood-ending technologies have be incrementally added ever since, as you cited, the automated loom, and I don't think _that_ aspect of AI is going to be stopped, backlash or no. While I'm generally not fond of the phrase "the wrong side of history", it is a reasonable approximation to where opponents of labor-saving/livelihood-ending technologies have found themselves...
Rather surprised that not one has brought the caffeoylquinide story up yet. And it appears gone from the dark corner of the network where it was once found. So it shall be narrated from one's memory. Gather before the simulated firelight, if you wish, and don't mind the clowns on fire jumping out of their helicopters.
---
1) Most of the Old World calories at the time of the advent of coffee were from grain. Wheat, among others. Wheat contains gluten; gluten metabolizes into many things, including something called gluten exorphin b5. Exorphin almost sounds related to morphine perhaps, or ephinephrine. This is not an accident. B5 acts on opioid receptors, with all the requisite effects of other opioids.
2) Coffee famously contains the psychostimulant caffeine. But it is not just a biosynthesis machine for some perk-up stuff. Created by a process as mindless as it is merciless, it also contains hundreds of other substances, many of them not important enough to study, and a few that are. One of these is something called 4-Caffeoyl-1,5-quinide. Something that actively blocks opioid chemicals from working as usual -> https://pubmed.ncbi.nlm.nih.gov/15088081/
Taking the narrative here at face value, when the substance gets shipped to Europe, a population that is constantly low-key microdosed on an endogenic opiate receives a sharp kick of a potent psychostimulant mixed with an opiate inhibitor, all distributed to their intellectual elite in highly concentrated locations.
Suddenly it becomes painful to think of the injustices of the world. Suddenly there's a feeling of strength to do something about them. And just as suddenly, there's cohesion with others who think the same.
Negarestani, the fable concluded, wrote about the wrong black substance.
The important part is the similarity of characteristics.
For the nuclear chain reaction example, the point isn't just that 'someone was wrong'. It's that even the most informed experts, who had all the available evidence, and who were completely convinced, where embarrassingly wrong. It's also that nuclear fission was cutting edge science at the time. That papers like to publish strong and sensational claims. That the scientific community reacted the way it did. The governments reacted to it a specific way as well. It's all this squished into a specific example.
The idea is that you're trying to work out whether something like AI risk is overrated or underrated by comparing it to other claims that share characteristics.
Its like this: "People were wrong about coffee and this whole AI situation feels kinda similar to the coffee one. It's similar because the same kinds of people are saying the same kinds of things and reacting in the same kind of way. It similar because the reporting reads similarly and the arguments for it have similar characteristics and so on. It walks and quacks just like a duck, so it probably is." That's the steelman.
The AI situation does share a lot of characteristics with other apocalyptic predictions and they're characteristics that make it harder to believe. (Just like we find it difficult to believe claims that share characteristics with pseudoscience, cults, scams, phishing, etc).
To most people, ai risk sounds like this. "Right now, thousands of the world's brightest minds are reckless working towards creating software that will, most likely, wipe out humanity. We don't know exactly how, but we're very certain that it'll be a disaster. All these smart people are being foolish in thinking the software is safe and helpful. It's not bad how social media is bad (soecietal transformation), its bad as in meteor hitting the planet bad (civilization ending)."
That's a wacky sounding claim that sounds a lot like loads of other wacky sounding claims.
"That papers like to publish strong and sensational claims."
The economics of producing "doomsday prophecy" vs "everything will be fine" seems under appreciated in many of the comments here.
This is my view of the argument. The point is that the characteristics of the old argument presented as an example are also present in the modern argument being discredited, and the modern argument is stupid for the same reasons the old one is.
I think your basic summary of the argument is wrong: You rephrase it as "Here’s an example of a time someone was worried about something, but it didn’t happen. Therefore, AI, which you are worried about, also won’t happen."
But instead it is the somewhat similar argument "Here's an example of a time someone was worried about something, but it didn't happen. Therefore you should be less confident that, AI, which you are worried about, will happen"
In other words, it is an argument against "we should be certain AI is dangerous" and not an argument for "we should be certain AI is harmless"
Unsurprising that if taken to mean "we should be certain AI is harmless" it falls flat.
Isn't this only a semantic difference and Scott would agree with you that the argument was against "you should be certain AI is dangerous"? I think what Scott meant by "AI, which you are worried about, also won't happen" is referring to the world ending scenarios of AI, or AI doom and not arguing for AI to be harmless.
Hm. I don't quite follow. Maybe I'm just not reading you right. Let me try to clarify:
You read Scott as saying "... Therefore, AI, which you are worried about, also won’t happen" and meaning "Therefore, the AI apocalypse, which you are worried about, also won't happen". You also think that Scott and I basically agree, just different words or maybe minor differences. Is that accurate?
If so then I agree that Scott is saying that the "past panics have not borne out" argument is intended to be an argument against AI doom specifically.
But I still think that this constitutes a misunderstanding of the argument, or at least it differs from my reading of it.
For me "past panic wrong, therefore AI apocalypse wrong" is a statement about the world, asserting that AI apocalypse is objectively unlikely.
OTOH I read "Past panic wrong, therefore certainty about AI apocalypse is wrong" as a meta argument. It says "there is a recurring pattern in humans to be really confident that things are going to be terrible and in many instances they were confidently wrong. No matter how certain you feel on the inside, the outside view suggests that most people in your position were wrong so maybe you should be less confident that your judgement is correct".
And of course there can still be argument about whether the claim "most people in your position were wrong" is actually accurate but in any case this seems like a substantially different argument from "AI doom is objectively unlikely".
Yes, your summary of what I was trying to say is accurate! Thanks for that. And I think I see what you're saying now better than before. However, I am still struggling to see exactly the difference between the two. Isn't the meta argument just the supporting evidence for the argument around how objectively likely/unlikely AI apocalypse is?
That is, past panics have usually been wrong ----> certainty about AI apocalypse predictions is likely to be wrong now ---> AI apocalypse is not likely to be true.
I think we agree though the the main question is evaluating the claim "most people in your position were wrong" as regardless of whether the key question is the meta one or the one about objective probabilities, that is the evidence utilized to make the claim.
I feel "AI apocalypse is not likely to be true" isn't really the goal of the argument but the practical end result is often similar / the same?
past panics have usually been wrong ----> certainty about AI apocalypse predictions is likely to be wrong now ---> we have no idea how likely the AI apocalypse actually is
is what I'd say the argument is. And that usually comes paired with "I'll only care about / invest a lot into AI security if good reasons are provided" with "we have no idea how likely the AI apocalypse actually is" often not being good enough a reason.
And in that regard "most people in your position were wrong" is probably the key claim, I agree with that. If really most people in similar positions were wrong then certainty in AI doom is probably unwarranted. And if there's not some reasonable level of certainty then why care / spend a lot of money.
I'm surprised you don't get it. It's motte-and-bailey!
Bailey: Look at the people I liken to my opponents! Look how dumb they are, and by extension my opponents!
Motte: Here is a curious story that undeniably happened. I think it's similar to AI worries that we all already know are unfounded, and story similarity is subjective, so you can't prove me wrong.
> And as some people on Twitter point out, it’s wrong even in the case of coffee! The claimed danger of coffee was that “Kings and queens saw coffee houses as breeding grounds for revolution”. But this absolutely happened - coffeehouse organizing contributed to the Glorious Revolution and the French Revolution, among others. So not only is the argument “Fears about coffee were dumb, therefore fears about AI are dumb”, but the fears about coffee weren’t even dumb.
Fears about coffee weren't even dumb, point #2: just look at what it's done to our own society. We've used coffee to normalize the notion of drug addiction. "I can't function without a cup of coffee in the morning" has gradually gone from a punchline to a simple "yeah, I hear ya" statement, because it's not just that weirdo over there who's an addict; virtually everyone in our civilization today, even a surprisingly high percentage of young children who we would never think of giving alcohol or tobacco to, is addicted to caffeine.
Forget marijuana; the true "gateway drug" is coffee.
> it's not just that weirdo over there who's an addict; virtually everyone in our civilization today, even a surprisingly high percentage of young children who we would never think of giving alcohol or tobacco to, is addicted to caffeine.
You must know a radically different subset of "virtually everyone in society" than I do.
Really? How many people do you know who 1) don't drink coffee and 2) are not Mormon? I honestly can't think of very many at all, off the top of my head at least.
Almost everyone I know. The main problem is not an objection to caffeine, it's that coffee tastes and smells absolutely disgusting.
However, while I actively seek to avoid caffeine outside of exigent circumstances, and this puts my consumption well below most of the people I know, I would not expect any of them to display the signs of addiction such as a splitting withdrawal headache. Having a Barq's root beer instead of a Mug when that's the only root beer option in the soda fountain just isn't "addiction" levels of consumption.
Huh. Wish I knew more people like the people you know.
Maybe it's my line of work. As a software developer, the notion that "programmers are machines that convert coffee and feature requests into code" has attained a certain level of "ha ha but serious" status, often leaving me looking around in disbelief, going "our brains are our livelihood, our most important asset. What are you *doing,* tampering with them with addictive drugs like that?!?"
Well, I don't. Then again, I am unemployed...
About half of the people I work with drink coffee in the morning. We work in an open space so it's obvious. I drink coffee maybe once a month if I get a craving.
As an object level point, it does seem like a lot of kids drink caffeinated soda regularly. I don't know if you looked up statistics it'd be concerning, or that they reach the level of addiction, but it seems like a not unreasonable off the cuff intuition.
(I have no other stance on the merits or demerits of caffeine, just seemed relevant)
"I think it has to carry an implicit assumption of “…and you’re pretty good at weighing how much evidence it would take to prove something, and everything else is pretty equal, so this is enough evidence to push you over the edge into believing my point.”"
Kind of tangential, but this is the same kind of naive Bayesian fallacy that I was getting at in the lab leak vs zoonosis post. "Weighting evidence" implies that you have a small set of hypotheses and are adding log odds to promote them relative to each other, but this has the problem that it's not taking dependencies into account.
What you really need to use the evidence for is something like building a high-fidelity graph of what factors are relevant. There's tons of ways that Russia has a bunch of legible military advantages, but ultimately these just cash out to the one fact that Russia is bigger than Ukraine (and maybe also historically invested more into its military). Once you take that into account, you already expect each piece of Russia's military to be bigger than Ukraine's, and so it's of ~no evidentiary value to realize this for the particulars.
Rather, the evidence would have to come from other independent sources of information, e.g. morale, allies, quality, etc. (which turned out to favor Ukraine). The key here is that they have to be independent of each other, yet correlated with the outcome of interest, which happens most easily if they are the largest-magnitude causes of the outcome of interest.
Not sure my points here have any straightforward connection to the coffee argument. I guess the coffee argument is basically just telling people not to Aumann-update on this because it postulates a general factor of technology opposition which makes the discourse worthless as a signal.
In a sense, you could say that the coffee argument is a proposal to #cancel people who worry about AI risk, by proposing that AI risk worries are an authoritarian power-grab.
They're... Probably gonna succeed? At least, here's my current model for how cancelling works (epistemic status: I've only recently started thinking in these lines so I'm low-confidence):
It used to be that everyone who worked against the local powers got cancelled. The exact pattern of cancelling was complex, but up to my knowledge was mostly determined by the local powers having misaligned interests (e.g. polluting too much, as expected from externalities theories) plus traditional allegiances (e.g. religion).
For various reasons, this was untenable, e.g. the local powers opposed environmentalism too harshly. This lead to a lot of smart people building a movement that could overrule the local powers, which was mostly formed around "trusting the science" because that's what was the most developed justification for environmentalism.
But 90% of everything is shit, and this includes science as well as the non-scientific parts of this movement that was built. The "trust the science" movement elevated a lot of this to power, leading to poorly-informed impactful decisions, including much of "cancel culture" and "wokeness" partly because of lots of poorly-done science in the grievance studies.
There's the discussion of whether wokeness is dying or not, with various takes. Recently I saw some graphs where the more aggressive/negative approaches (like accusations of racism/transphobia/etc.) were declining while the more agreeable/positive approaches (like inclusion) were constant. This seems logical both because the backlash is especially concentrated against the aggressive forms of wokeness, and maybe also because the "woke purge" is essentially complete in the sense that the largest corporations all agree that they have to handle climate etc., so insofar as climate was the original motivation for the aggressive forms, the motivation is complete.
AI risk worries are strategically similar to "wokeness" in that they involve meddling in other people's profit-driven business for moral reasons. But it doesn't get carried along by the new "trust the science" ideology because the empirical signal for serious risks is too weak to show up scientifically, and instead discourse about worries about AI are going to be focused on the dying racism/etc approaches that are too unimportant and too unpopular too win, and which are too different from AI xrisk to be useful even if it does win.
I don't want to defend the coffee argument on its merits, but it feels like a much less stupid claim than "medicine doesn't work" which you took much more seriously for some reason.
Coffee is a concrete object, medicine is an abstract class containing many concrete objects.
The medicine thing is at least plausible, if a bit ridiculous. The coffee thing is just so obviously a mistake in reasoning.
“once people were worried about coffee, but now we know coffee is safe. Therefore AI will also be safe.”
I think this is not the argument at all. In fact, what you identify as the conclusion is really a PREMISE of the argument. To me the argument reads as:
1. Worrying about a coffeepocalypse is absurd.
2. People expressed worry about a coffeepocalypse once.
3. The reason was that these people were not honest but pretended to worry about it for an ulterior motive, namely power.
4. This is evidence that people sometimes malign beneficial technologies for ulterior reasons even if this is absurd.
5. Worrying about AI is absurd.
6. So the best explanation is that AI-doomers are dishonest and really driven by ulterior motives, such as power.
I think this is still a bad argument, but this is what it actually says.
This also makes more sense if you go back to Loris's point above. Reading the thread, I tend to agree with that point.
The coffee comparison isn't arguing against AI apocalypse. It's arguing against "AI will take the jobs of writers, AI will be used to create fake news, AI will make fan artists go bankrupt," The idea that AI would destroy the world is barely even on their radar.
"Doomer" to a normie is a metaphor. It rarely means "literally destroy the world". (Notice that the coffeepocalypse doesn't involve destroying the world either.)
This is really interesting to me. The argument intuitively makes sense to me and your "got you" response would seem like arguing in bad faith if I encounted it in the wild. I am also very much not the target group for this website, this is my first time posting, and I'm not sure if I can explain it in a way that would make sense when the original wouldn't. I'll give it a try though!
I think you're fundamentally misunderstanding the argument. You're taking this way too literally. I think translated it means something more like,
"Most people's objections to new things and new technologies are not data based. People have an emotional reaction to new things, simply based on them being new, but this is bad practice, a bias if you will. Not all new things are bad: here is an example. You should judge AI safety on its own merits, not just because it's new."
I think many of the people would have the the implicit argument that they probably think that the other evidence for AI safety doesn't stack up, that there are no heavy arguments for it to use your terminology apart from fear of the unknown. But I don't think that's true for everybody saying this.
Most people's fears about AI are simply just because it's new. Yours are not, so this argument isn't aimed at you. They're basically saying, I think you're only worried about AI because it's new and people are scared of new things. A good faith response would be more like, "I have reasons X and Y to be worried about AI" rather then changing the discussion to be about whether people should be scared of new things or not.
What they’re really saying is pretty simple: “It’s perfectly obvious to me that <coffee,global cooling,…> wasn’t bad despite lots of very-smart-people thinking it was, and I’m saying the same thing about AI. Since my track record on these other matters is so good, you should believe me on AI too.”
An “argument from prior track record” is actually a solid argument – eg it’s the argument for trusting any scientific theory. Unfortunately in this case the track record is being arrived at after the fact and highly selectively.
My steel man of the coffee argument:
“There are nearly always predictions of widespread doom as a result of new technologies, even when they’re obviously innocuous like coffee. Given how rarely new technology triggers an apocalypse, we should be skeptical of doomer arguments about AI.”
Obviously this leans on coffee doomsaying as emblematic of something that happens constantly, but it absolutely does, for every major new technology I can think of.
Stuart’s argument about nuclear skepticism can be construed the same way—there are pretty much always scientifically qualified skeptics of new technology, so that an authority figure thinks something won’t work shouldn’t provide a major update, unless you understand the mechanism by which it’s impossible (laws of physics for perpetual motion) yourself.
The claim that there were moral panics that turned out to be nothing is rightly dismissed as an argument against AI concerns.
The claim that we missed some consequences that we should have panicked about before is also true, but tells us nothing about the specifics of AI doom scenarios, which have to be argued separately.
I'll admit I only skimmed the original thread, but it doesn't seem to me like the coffee analogy is an *argument*. That is, he's not saying "AI will be fine *because* coffee was fine." He's just saying "AI will be fine, and after it's fine it will seem as normal as coffee does now."
Like most posts online, it's not meant to convince anyone. It's just an anecdote for the people who already agree to feel good about.
>That is, he's not saying "AI will be fine because coffee was fine." He's just saying "AI will be fine, and after it's fine it will seem as normal as coffee does now."
Yes, I agree with your reading of the thread.
Frankly, depending on what counts as "fine", I'm skeptical that
"became an unremarkable part of daily life" implies "fine"
is even a reasonable view. For decades cigarette smoking was an unremarkable part of daily life in American culture. And the routine estimates of how many Americans it killed per year were on the order of 300,000. Personally, that is outside of what I consider "fine".
I'm not expecting AI to have an analogous effect - there are a variety of possible hazards, but none in that specific direction. But the general idea that the presence of something in daily life is _sufficient_ to conclude that it is harmless is _wrong_. At best, one can conclude that it isn't immediately fatal or rapidly, undeniably and severely damaging.
edit: The
>It's already happening. My step mother was using it to help with her writing and to spit out ideas about ideas for a raffle. My friend's five year old daughter talks to the GPT voice interface in her native language and they create stories together.
looks like it is arguing that AI is becoming part of daily life.
> Frankly, depending on what counts as "fine", I'm skeptical that "became an unremarkable part of daily life" implies "fine" is even a reasonable view.
I think it's reasonable, at least on the level of a personal belief or heuristic. Sure, we can use present information to say that past crowds were wrong about smoking. But if you were making a personal decision about whether to take up smoking, you wouldn't look at the past crowd - you'd look at the present one. And because the present crowd has present information, smoking is much less common. Ultimately, while it's a little helpful to know the crowd *can* be wrong about something, it's just not actionable without some expertise to tell you they *are* wrong.
That's why I think the context of the original thread is important. As an *argument*, "normal = fine" crumples the instant it hits an expert opinion. But as a naive assumption made by some guy in a thread about something else entirely? Not a mortal sin, IMO.
>Ultimately, while it's a little helpful to know the crowd _can_ be wrong about something, it's just not actionable without some expertise to tell you they _are_ wrong.
That's fair. I think we are just emphasizing different parts of the set of outcomes. What you wrote isn't really inconsistent with my
>At best, one can conclude that it isn't immediately fatal or rapidly, undeniably and severely damaging.
How does this sound:
There are _many_ components to daily life. Depending on how one breaks down the choices in daily life, there could be thousands of optional actions. Most of these really are harmless, and have never been linked to adverse effects. At any given time, there is probably an unwarranted panic being promoted on a few of them. But, also at any given time, there are probably one or two reasonably pervasive components that _do_ have significantly adverse effects, as smoking did. E.g. whatever the underlying cause (whether it is sedentary lives, junk food, whatever), the increased incidence of obesity (writing from the usa) is likely to be detrimental to health and is likely to be caused by (possibly a set of) unremarkable part(s) of daily life.
It's because doomsaying is an especially powerful form of persuasion, as it can justify people or society taking actions that would otherwise be completely insane or taboo (in our case, violating individual liberties, massive economic extraction, etc - in the past/other places perhaps it justified eliminating all the idolators, or everybody leaving the island, or rebelling against Rome, etc).
Therefore, those with agendas that are unpopular can exploit doomsaying to advance their (perhaps unrelated) goals, and it makes sense to build a strong epistemic immunity to believing any random doomsayer who wanders along.
Climate change is vulnerable to this. Many sincere people believe that it is a major threat that requires society-level disruption to solve. Many others want a particular *type* of societal disruption, and argue that we must implement their program because it's the only way to prevent climate change. That, in turn, makes it much easier to dismiss the sincere & concerned.
It doesn't help that many doomsayers are straight-up cranks (Mayan calendar/2012?), which makes the immunity relatively easy to develop.
Jeremiah turned out to be right; so did Cassandra. But Isaac Luria sure didn't (https://en.wikipedia.org/wiki/List_of_messiah_claimants).
> Jeremiah turned out to be right; so did Cassandra. But Isaac Luria sure didn't (https://en.wikipedia.org/wiki/List_of_messiah_claimants).
It seems worthwhile to observe here that Isaac Luria is the only one of those three examples to have actually existed.
Exactly. In fact, I think it's not only a powerful tool of persuasion, it's among the only 2 remaining political tactics in use in western world since WWII: fear (not doing what I want will have catastrophic consequences for many people including you) and guilt (not doing what I want will keep this poor group suffering). Doomsaying is just a form of the common fear tactic (sometimes with a pinch of guilt - with stuff like the more vulnerable will suffer more. Not for AI (at least I did not see it yet) , but guilt is less powerful and often comes after, when fear did not work)
With the regards to nuclear chain reactions, I feel like its a more valid form of argument - the statement there is "x is impossible because expert said so" which is then disproven. This is basically just restating the appeal to authority fallacy with an example.
However, just because someone has used an appeal to authority fallacy doesnt change whether or not the underlying thing could be impossible. You can not use an appeal to authority to prove whether nuclear chain reactions are possible or whether ai is dangerous. By the same token, showing its an appeal to authority fallacy does not prove either of those statements.
The difference between the statements is:
NCR being impossible were an appeal to authority which was wrong, thus we should ignore all appeals to authority.
Coffee being bad was an appeal to authoirty which was wrong, therefore all appeals to authority are wrong, therefore ai is good because anti-ai arguments have used appeals to authority.
Surely the steelman version goes something like...
* Humans keep doing this thing where they see something new and get hysterical about how [new thing] is going to bring about doom. This even happens when [new thing] is 100% harmless and cannot possibly cause doom, like coffee.
* Therefore, humans clearly have a significant false positive bias when it comes to calculating whether new things will cause doom.
* Therefore, the smart thing to do is have a strong prior against any claims that some new tech will cause doom.
---
Like, more than anything else this just seems to be an argument about the false positive rate. If some new medical test comes back saying you have cancer, but you then find out that the cancer-ometer ALWAYS says 'cancer' no matter what, you're probably going to ignore the result. The coffee argument is the same.
Dr S: We've put 'AI' into the new doom-ometer to see if it's going to cause doom. The results are scary: it reports a very high possibility of doom!
Dr J: Remember when we put a cup of coffee into the doom-ometer, and it reported a very high possibility of doom? I'm starting to think that thing isn't very reliable.
That looks right.
Random hat into the ring: it's a proof-by-contradiction, because they (I) see the standard AI Doomer position as "AI *could* be bad, therefore it *will* be bad". Thus, pointing out all of the other times people freaked out about "could happen==will happen" and then it didn't happen, weakens that type of claim in general. It's very much a pop-debate "dunking on Twitter users" type of argument, but I think that's the core.
If this was a genuine attempt to understand this reasoning, I'm concerned.
Humans have pattern-seeking brains. They also have threat-detecting brains. Therefore new ideas and technology is often met with threat predictions based on hypothetical reasoning built absent concrete empirical evidence. Often this reasoning seems silly in retrospect. Therefore when we see this kind of argument we should be concerned we are falling into a known bias pattern.
This isn't the knockout argument proponents think it is but it also isn't an insane one. I'm not well-integrated into the rationalist community but I've spent a lot of time trying to understand this position of (relative) certainty on AI risk and still every AI risk argument sounds more or less like this to me: https://xkcd.com/748/
Just hypothetical built on hypothetical until the end-result is exciting and world-changing and disastrous. There are lots of reasons I feel this way but one is for sure that "here's a danger scenario I've come up with based on motivated reasoning, I now think it's more likely than it is" is a known failure state for the human mind.
This.
Scott, this is embarassing - how are you missing the argument so badly? Your article is the worst kind of bias argument; you can’t even give a brief Twitter thread an honest reading:
1) coffee is technology
2) coffee was once controversial technology
3) governments in the past tried putting a stop to the technology by going after distributors or declaring the technology itself unsafe
4) businesses feeling threatened by the technology displacing them formed opposition groups
5) the true motivation of these governments and opposition groups was preservation of power.
6) 3-5 are a pattern that plays out repeatedly for any controversial technology, and so the fear/doom around AI is likewise just people trying to preserve power.
The “tl;dr was tl;dr” is: the doom is just a power struggle.
Note: the argument has *nothing* to do with the *actual* level of safety or unsafety of the specific technology. You could plug in “nuclear weapons” for 3-6 and it would still work.
This is why it is *imperative* that AI safety arguers realize the vast, vast, vast gulf of popular opinion to be crossed if you want to not be seen as the new Chicken Little / Boy Who Cried Wolf.
And honestly: “wondering how people could be this stupid…”? You can be better at argument than this. Calling a placeholder for “common layperson opinion” stupid is the worst kind of own-goal reasoning.
> This is why it is *imperative* that AI safety arguers realize the vast, vast, vast gulf of popular opinion to be crossed if you want to not be seen as the new Chicken Little / Boy Who Cried Wolf.
...I'm sure they do realize that, which is why they were so surprised when a lot of laymen actually turned out to be worried about AI risk. Not that it'll matter, of course. Those people aren't the ones calling the shots.
Honestly, why do all these AI risk people care so much about whether humanity kills itself off or not? Do they seriously think that they can save a species that they also think is stupid enough to cause its own extinction?
Whether we can is a much less important question than whether or not we should try. Some of us don't just give up at the first opportunity and quit because it seems too hard. And some of us value humanity much more than you do, if my bad memory of your comment history is anything to go by.
In their minds a wild, low-probability idea vs something that we can see concretely. Obviously the thing we can see concretely has far more weight. It's like how we argue from real life to justify something in a storybook or refute it as absurd.
My hypothesis is that the vast majority of people making these kinds of arguments aren't actually trying to be right--they're trying to be interesting. In fact, I'll go ahead and propose that most conversational assertions aren't primarily meant to convince anyone of anything; they're meant to garner attention to the person making the assertion. In other words, they aren't primarily informational statements at all, they are bids for social status. "Coffee was safe and therefore AI will be too" doesn't mean what it appears to mean, it actually means "I'm an interesting person who says provocative things, pay attention to me." This carries further implications regarding what "winning an argument" actually means to most people. The rise of social media can have only exacerbated this effect.
We're just lucky we live in a universe where being factually correct is sometimes interesting.
The counter to the Coffee Argument is the Tobacco Argument: "people argued that tobacco was bad for you, but society in general decided 'hah, those joyless fools, tobacco is awesome and indeed beneficial' but it turned out the warnings and doom-sayers were right".
So sometimes the worry is not warranted, and sometimes it is. The hard part is telling which time is which.
Except tobacco panic was also stupid.
The link between smoking and cancer is real, sure, but that's not why there was a panic. There are loads of common consumer products with well-known negative health effects, and the usual reaction is a shrug and a "if you decide to use it, that's your problem". The reason people are terrified of tobacco in particular is because of a massive blood-libel campaign claiming that thousands of helpless victims were getting cancer and heart attacks from breathing in other people's smoke second-hand.
(In reality, the data on whether secondhand smoke poses any dangers at all are still inconclusive, and even those studies that have found significant effects find it only for those who are exposed regularly in confined spaces for many years. There's no good evidence supporting the hysterical "NO amount of smoke is safe! Even being around a smoker for a few minutes could KILL YOU!" claims that people heard in PSAs.)
Stochastic risk, like smoking/cancer is quite counterintuitive. The AI discussion is not about a stochastic risk, is it? I don't know shit. Where is N.Taleb when I need him?
Well said! In the USA, the usual estimate for the number of people dying from smoking was around 300,000 per year (at least when I was growing up - the fraction of the population smoking has dropped since then). Yup, something can become an unremarkable part of daily life and still be damned lethal.
I think that supports my argument, actually. It's quite possible for a small group of "influencers" to each argue opposite sides of a debate and both profit, by gaining attention and status with an audience. I've even suspected that many public controversies seem choreographed somehow because neither sides is arguing based on the merits, because the merits of the issue aren't the point.
'Course, that doesn't mean that there isn't a real issue behind it all, or that real world consequences aren't attached to what we do.
Very true. The only thing that mitigate this is to check back those declarations for truth, and stop paying attention (which means getting outed for power figures) when the track record is poor.
Checking past declaration should be easier now... But it's not what happen, because many learned the art of being interesting And vague, and media never check their own track record obviously. Hence COVID amnesia for example....
The coffeepocalypse argument is a cousin of a thought exercise cognitive therapists ask people to do when their conviction that something is going to happen is based mostly on a *feeling.* So for instance very depressed people often do not try going out for a walk or calling a friend because it feels so true that doing either not only will not make them feel better, it will make them feel worse. So you ask them if they can think of any times when something felt very plausible to them, but they later discovered they were wrong, and most people can think of one. The point of the exercise is to help the person move feeling out of the evidence category, where it doesn't belong.
So I think that is the way to AI Coffeepocalypse argument is supposed to work. It's only suitable for people who were genuinely worried about their coffee consumption because the idea was in the air that it was bad for you, and are genuinely worried about AI for the same reason -- the idea of AI doom is in the air. You might think that's a pretty small subset of people, but actually I think that for most people, including me, it is simply impossible to make a decent prediction about how the whole AI thing is going to turn out, and also very hard to just have a wait-and-see attitude, so we fall back on our natural fear of the unknown, plus the AI worry that's in the air, and treat them as evidence..
So while I agree that the coffee argument is not exactly intellectually rigorous, I think it has a point, & the point is to get people to remind themselves that their uneasy feeling about AI is not evidence. So for practical purposes, the argument should be useful to almost everyone who got worried about coffee when there was chatter about its dangers, and is currently worried about AI. I guess we should also exclude the tiny fraction of people who understand AI well enough to have grounds for thinking they have some ability to predict how things will work out. (I'm inclined to think they don't have much more ability to make accurate predictions that anyone else, but they think they do.)
I think part of what motivates people firmly in this camp, like Marc Andressen, is that it seems the current technology has moved in a somewhat different direction than Safety folks in general thought the technology would look like. Correct me if I’m wrong, but didn’t Elizier not see the whole “inscrutable matrices”* thing and especially LLM’s making it this far? So I think the thing folks in his camp aren’t outright saying is “these specific thought patterns don’t seem to be predictive.”
I really like reading Elizier because he lights up parts of my brain that don’t light up elsewhere, and he takes me in directions I wouldn’t go on my own. But I also don’t overall agree with him on a lot of things. A lot of that disagreement comes from experience building complicated stuff (I’m a Product Manager, I know, barf, and I readily cede probably almost everyone here is smarter than me on a lot of different dimensions) and having an experience of just how often reality pushes back at you and says “Oh wait, this is even harder than you originally thought, even after you took into account it was going to be harder than you originally thought.”
I’m sure people like Marc Andressen, who build companies, have these feelings on steroids. Whenever you try to make something do one specific thing,, it’s really hard to get it to settle into that state. And I know the safety argument is that you don’t need a specific state to kill everyone but I just don’t agree.
It’s so hard to get “things” to do “stuff” that I have a hard time accepting that a nascent superintelligence will be able do almost anything on purpose without a lot of effort. Especially when what seems like superintelligence today isn’t agentic,** but is taking the form of these sort of weird Thought Crystals that no one anticipated. I’ve read the Instrumental Convergence arguments, big chunks of the sequences, etc, and while I do think there are *major* dangers probably not too far away from where we are today, I don’t think the shape of them is what a lot of people think they are.
Anyhow, if I can’t steel-man a particular argument someone is advancing, i try to see if there’s a nearby argument I can steel-man and I think this is it.
My own system of building emotive intuition pumps (I spend a lot of time thinking about what it’s like to *be* a data packet in my day job, and I”ve gotten pretty good at doing that to catch weird timing errors, etc, other people don’t catch) tells me we’re probably going to get weird emotionless golem agents at first. They won’t precisely want anything and commands like “figure out what to do and then go and do it” will probably have very limited efficacy while at the same time asking specific questions or giving it requirements and having it fulfill them will get breathtakingly awesome. I think all intelligence looks like “inscrutable matrices” at least in some part, and the next step beyond the one we’re at right now will be building structures on top of those to shift the weights around to pre-set states. And then a bunch of breakthroughs and we’ll get real time updating and that one in particular alarms me but by that point I think our theories of mind are going to get generally better.
*Mechanistic interpretability seems to be making them somewhat more scrutable, so you could say this was another wrong prediction.
**I do think LLM’s have general intelligence, but are not agents in a meaningful sense
"Doing stuff is hard" is the strongest counterargument to the rationalist position on AI in my opinion. There's this bizarre assumption that at some point of effectiveness, it's possible AI will get monotonically easier to improve forever, either because it's so smart or because it gets so good at acquiring resources or whatever.
But nothing monotonically increases forever. Like, by definition. To even make that a possibility in my mind, I'd need strong strong evidence that it was possible. And none has been provided.
I think there are things that are pretty scary that are possible and that we definitely need to be safeguarded against them, but yeah. Doing specific stuff is hard. Like if I wanted to make a machine to kill every person on Earth, I bet that would be really difficult and I would give myself away by doing it. Intelligence gets you a lot. Gets you scary stuff. But I don’t know that it can get you *as* much as it might seem. You not only have to be intelligent, but motivated, etc. Then you have to have specific motivation. Then you have to be good at building stuff. Then you have to build something and no one can figure out or guess what you’re trying to do.
I have a dumb argument, I call the argument from bears.
Most bears will leave you alone if they see you in the wild because they don’t have a history of hunting you specifically. Almost all bears can kill you but most won’t.
Polar bears will kill you on sight because they’ve had to learn how to eat anything edible.
I think my priors are that it is much harder to specifically make a polar bear out of the space of possible bears, but it’s easily possible to make something with dangerous potential.
To a large extent "doing stuff is hard" is self referential. It was difficult to impossible for evolution to make a macroscopically big wheel, because in order to get a wheel you need to grow it, but the wheel itself needs to not be connected to the axel until humans made it easy by just putting a damn stone on another stone, it was difficult to impossible to talk about how to make communications over noisy channels work out until information theory was developed and you can come up with the idea of checksums, it is difficult to impossible to push a feature to a large production codebase, written by people with completely different incentives, who aren't in every single meeting and who aren't good at all the sub skills to shipping a product until.....
If you systemically only think of things that are barely doable on the margin for humans, you have to have a good proof for why it would also be inherently hard for things that are more capable than humans.
I think it's fine to say something like "oh yeah, AI probably won't that much closer to a Carnot Engine wrt to efficiency for combustion engines", but much less fine to say "and therefore, there are no better ways to move".
<mild snark>
>But nothing monotonically increases forever. Like, by definition.
Entropy
</mild snark>
...fair.
Many Thanks!
No one is saying forever. All we're saying is "far enough to pose a risk." Which is really not that far considering the fragility of world economics and geopolitics.
one of the top 50 most annoying things about "AI discourse" is that people love to "argue" by debating which metaphor applies
"AI is like coffee"
"nuh uh AI is like a malicious god"
"nuh uh AI is like cold fusion"
"nuh uh AI is like this one movie"
"nuh uh AI is like this entirely different movie"
i still hold out hope that somebody involved in this wretched pit of a "discourse" will actually work from the facts we already know (AI is algorithms running on computers, we know quite a lot about the capabilities and limitations and resource needs of algorithms running on computers, we know your brain is not an algorithm running on a computer...) but i'm not holding my breath
"But then how does any amount of evidence prove an argument?"
The only thing that *proves* an argument about some future event is actually observing that event. Russia can have a 3:1 manpower advantage, better generals, better tanks, and every expert in the world could predict that they'll win easily, but none of that is proof. The proof is in the pudding, as they say.
And I think that might be the root cause of your frustration about AI arguments. No one has made the super-intelligent AI pudding yet. I think that where people come down on the debate about AI risk largely reflects their own temperament and deeply ingrained biases, and both sides have people that are making bad arguments to confirm their biases.
My advice is stop trying to understand all these arguments as well thought-out and rational attempts to discover the Truth, and start trying to understand them as flawed attempts at persuasion by appealing to emotion and common biases, very much like political debates.
The average person doesn't know anything about AI and is not in a position to evaluate the data (I certainly am not), so they instead have to decide "do I trust Set of Experts A, who tell me 'AI is coming and it could be a threat', Set of Experts B, who tell me 'AI is coming and it will turn the earth into a paradise' or Set of Experts C, who tell me 'AI is impossible and will never happen so we won't be living in either the paradise or the hell future'?"
Hence why the duelling arguments over "predictions of doom in the past which didn't come true" and "predictions about this tech is impossible which were shown to be wrong", to try and convince Average Joe and Josephine.
I think you should go with Set of experts D, who say, "so-called AI is coming and it's going to make some jobs obsolete while creating some new ones, but overall things will keep going as they always were, which means that our lives will get a little more comfortable on average in the long term". Admittedly, that's a less sexy prediction than any of the others...
It seems like the basic problem here is trying to find a general heuristic for evaluating some class of arguments without digging in and thinking them through. That's a necessary part of life--there are absolutely people who will claim that *everything* is an existential threat to humanity, if they think it will win them their argument. But the best it can do is give you a very rough guide.
Over the years, I've probably heard a million different claims about things causing cancer--chocolate, or red meat, or saccharine, or Red Dye #2, or whatever. Most of these were probably nonsense, driven by p-value fishing and a media file-drawer effect. And yet, smoking and breathing in asbestos fibers and getting sunburned a lot all actually *do* cause cancer. You can and should dismiss most such claims with a heuristic (media sources always report on crap like this and it turns out to be one n=14 study in mice involving 1000x the dose of the carcinogen humans are ever exposed to), but you also need another heuristic to decide whether to take the claim seriously enough to dig in a bit.
Once you're to the point of writing a think piece on why your quick heuristic for dismissing a claim of this kind makes sense, it seems like you've kind-of gotten yourself wrapped around the axle. The heuristic is useful for keeping you from changing your diet or spending hours digging into evidence every time the local TV news reports on that n=14 study in mice, but it isn't useful for actually evaluating a claim that something causes cancer. That can only be done by digging in and thinking hard and evaluating evidence.
Another way of thinking about these heuristics is as a kind of base rate. Prominent claims that X is going to lead to the extinction of mankind don't have a great track record, so we should start out skeptical. But then you need to dig into the available evidence. (Also, there's an anthropic principle thing going on there that makes claims that something will wipe out mankind a little funny to evaluate--everyone who ever evaluates such a claim will find that it never happens, even in a universe where {hostile AI, an engineered plague, an asteroid impact, an ecological catastrophe} does eventually drive mankind into extinction.
Of course, the problem wrt AI risk is that the evidence is hard to understand and interpret. We are speculating about capabilities of computers that we don't know they will ever have, but that they might eventually have that would be really dangerous for humans.
There's a parallel argument about AI risk that says more-or-less "we thought AI would never do X, but now it's doing X better than humans, therefore people who argue that AI will never turn us all into paperclips are also wrong." As a heuristic for dismissing arguments, it's not so helpful here. But maybe it can inform our base rate or priors.
My sense, as someone outside the AI field, is that it's hard to make convincing arguments that AI will never be able to do X (for any X), but that doesn't mean that AI will eventually do every X we can think of.
The argument you agree with, but question if it may be fallacious:
"People were wrong when they said nuclear reactions were impossible. Therefore, they might also be wrong when they say superintelligent AI is possible."
*is* a form broadly misused/abused - for pseudoscience or outright supernatural, e.g. Electric Universe, Velikovsky, astrology, ancient aliens, God.
"Is this form of argument valid?" isn't the right question here, as it most argument forms can be used for the invalid. It boils down the independent question of whether AGI/ASI in particular should be considered in the same category as pseudoscience, and clearly here we'd all point to other evidence to show that it's different.
To me, it sounds like a informal "Bayesian" argument. The author of the tweet is presented with a conundrum of the form:
- There is a subset of clever people that, applying a stylized model, have come out with a potentially catastrophic outcome that would materialize in the middle or long-run.
Then, he wants to estimate how often a group of clever people that came out in the past with a catastrophic scenario have been correct. Of course, such base ratio is pretty difficult to compute unless you can survey all intelectual discussions in recorded history (without significant bias), and then determine what is a "catastrophic prediction" and which ones actually came true . Thus, he take a small, quasi random, sample of N=1 catastrophic predictions and since that one didn't came true, he concludes that "clever people predicting terrible things" are usually not right.
I think the minor point, that clever people using simplified models of the World are usually wrong in their long-term predictions, is broadly correct and I could back it up with a large N. Whether this says anything about AI risk, I'm not so sure. At last, whether AI risk is real or not isn't a random event. But here, many of you defend "non-frequentist" probabilities and use base-rates in your reasonings that are similar to what the author of the tweet is doing. As an example: suppose you have a neuroscience study. You can not properly evaluate it and as such you don't know whether the result would replicate. After a quick googling, you find a survey of N=300 papers in the area, not including the one you are interested in, that says that only 40% of them replicate. Would it be so incredibly wrong to estimate the probability of the study you're interested in replicating as 40%. What about N=299 papers? Lower the N enough and you get to the Dan Jeffries twit.
I understand and sympathize with your frustration, but this is just another example of a common conundrum many people here face. Once you have some basic understanding of epistemology, most people seem like drooling idiots.
I can't tell if you're being sarcastic
So there's a lot of people outside the rationalist community who don't think "x-risk" when they think about AI, and if you mentioned it to them they'd probably say it was some kind of techbro delusion. Even so, they still hate current AI technology. I've seen people react with visceral disgust at anything that seems AI-generated or uses AI—they're not *worried* about it, they're *offended* by it.
The coffeepocalypse argument thread opens specifically with reference to *backlash*, suggesting it's not really addressing risk but talking more about those kinds of reactions. This is given explicitly a couple of times: "Both forces fought back against the society-shifting power of coffee just as we are seeing media companies battle AI companies today for the societal shifting power of AI." "This pattern plays out again and again and again in history. New technology. Resistance. Resistance overcome."
(Of course this doesn't invalidate your general argument about using isolated incidents as proof.)
Rutherford died in 1937. We can be confident he said nothing, accurate or otherwise, the day before Chicago Pile 1.
The story got mixed up. It was Szilard conceiving of the idea of self-sustaining nuclear chain reactions the day he read of Rutherford mocking the idea.
COVID predictions are a useful a metric to judge which public intellectuals are "pretty good at weighing...evidence".
Everyone wrote extensively about it, and the duration was short enough in that we get to judge if they were correct or not.
I don't think we can use COVID-19 predictions as a useful metric to judge intellectuals because we still can't agree on what happened. Its not like there is universal agreement whether or not Ivermectin works, the vaccines were safe & effective, the lockdowns were warranted, or the virus leaked from a lab.
There are many, many areas that are not contentious at all at this point.
1. Didn't spread through fomites
2. Vaccine conferred a short duration of protection
3. Healthy quarantine measures failed economic cost benefit analysis
4. All combined NPIs (masking, social distancing, quarantine efforts) were not effective at getting R to < 1
5. Age was a very strong predictor of COVID morbidity and mortality
6. Age adjusted PFR, IFC, and CFR are now known.
Its pretty easy to judge authors on most of these by casual observation. If an author supported school closure they probably had wrong ideas about 3, 4, and 5. If they were sharing their door dash sanitation protocol they were likely wrong about 1 and 4. If they were saying that vaccine roll out would end the pandemic by mid 2021 (like I was), they were likely wrong about 2, etc. If they were saying, "COVID is a government conspiracy", or "everyone should panic non-stop for the next 2 years" they were likely were wrong about 6.
I’m inclined to agree that there was little or no expertise. it was seat of the pants stuff - the lockdowns were not something that the WHO recommended prior to the epidemic, in fact they warned against it. It’s been forgotten now but masks were actually derided as ineffective for a few months, during which time old websites recommending masks against viruses were deleted, then that changed and people who refused to wear surgical masks were the epitome of evil.
In some countries surgical masks were derided as useless, like in Germany, while they continue to be recommended in the U.K.
Antigen tests were declared useless until they were recommended.
PCRs were the gold standard until that changed and they were considered to amplify the virus and therefore were useless.
Washing hands, even wearing gloves, was recommended at the start but ignored at the end.
Two weeks to flatten the curve was clearly nonsense in retrospect, but also you have to ask why only epidemiologist would’ve assumed that it would have worked – surely the epidemic was going to spread again once we left lockdown.
New Zealand wasn’t considered bat shit crazy, but only mildly crazy for continuing to believe in zero Covid for more than a year, or two.
The vaccines were definitely going to give us herd immunity, until it was clear that wasn’t gonna work so people started to say that it was never even claimed to begin with.
Luckily Omicron comes along and the severity declined. And luckily no further more dangerous mutation has occurred. (Although I suppose we are all somewhat immune now anyway. )
I think something completely different is happening:
1. "I'm trying to predict something in advance."
2. "Here's an example of a failed/successful prediction.
3. Given we now retrospectively know the results of this other prediction, it should be possible to have confidence in this new prediction prospectively."
The logical error, which is extremely common, comes with the assumption that a reasonable person could have had the same prospective clarity as the retrospective observer. We should know these are qualitatively different states, but we mix them up all the time.
In the same way it was impossible for someone to have confidence in coffee, given the prospective uncertainty, it would have been impossible for someone to have confidence in nuclear chain reaction. Nobody KNEW... until it happened and suddenly everyone knew! It's difficult to wrap your brain around this concept without practice, because once you know a thing it is hard to unknow it to the point where you can understand the person who is still struggling under uncertainty. Very easy to spot in someone else.
If the target audience (or even the poster) doesn't have a grasp on the specifics of the AI, the actual argument for why or why-not AI-doom is simply "do I think these people is right?"
Many people, especially smart people, will see other smart people supporting AI-doom, but rather than engaging with the logic, they are trying to evaluate the people (trust-the-science types maybe).
If your evaluation of AI-doom is "are these people usually right?" then it's quite reassuring to see that smart people have been wrong about doomsday scenarios in the past. It doesn't have anything to do with the underlying logic of the doomsday scenario.
Honestly this doesn't seem that hard?
Bach wrote a Cantata addressing the Coffeepocalypse. It's funny, light hearted. Music is good enough, but clearly not his best.
https://en.wikipedia.org/wiki/Schweigt_stille,_plaudert_nicht,_BWV_211
Is this like the "trapped prior" thing you were talking about, with the same argument moving different people in different directions?
Perhaps taking them as “arguments” instead of rationalizations and virtue signaling is the problem? With a social expectation of “having an opinion” in our contemporary dystopia, most people (who in my observation rarely think deeply about anything) reach for familiar justifications that resonate with them. Perhaps it seems more erudite to them than simply saying “I’m with #team-utter-bullshit not #team-absolute-tripe”? 🤷♂️ At least in the USA, we’re accustomed to bipolar arguments defined by the extremes with demonization and dismissal of the opposite as the standard modus operandi.
Tweet #5s "What they were and are really arguing about is power." is an important and true point. Like Chapman belabors in Better Without AI, we were more "intelligent" than chimpanzees for a long time without being appreciably more powerful than them. Our power came from building cultural, colonial knowledge stores, which is often what we mean by "technology". (It's not just a matter of technology, but also maintaining the institutions that can reliably translate those ideas into changes in the physical world.)
In the case of coffee, this new change in the world ended up having a democratizing effect; more cognition available to the average Joe without specific dispensation from kings, politicians, and alcohol manufacturers. I think Prof. Russell is trying to take an accelerationist, technophilic approach that basically all new technology that serves as cognitive prosthesis is democratizing in this same way. I personally don't agree with that take, but I can see how this is an attempted pattern match on a deeper level than "here's a wrong prediction, so there will be others."
So I agree that we should look at power, not intelligence, because intelligence does not trivially map to power and the correlation is weaker at the high end, not stronger. I also think that coffee has obviously different affordances than AI and so their relationships to power are dramatically different. Coffee just serves to stimulate you out of the low end, giving you a bit of intelligence that gives you a bit of power on the low end where they still correlate. AI is a case that can pollute or take control of our collective knowledge stores, and is reaching much higher levels of intelligence, so the relationship to power is much more idiosyncratic and needs to be evaluated on it's own merit. But I do think it is coherently arguing "better thinking technologies uplift us all" (even if I disagree with that argument) and does not reduce to just "here's one failed prediction, so this other prediction must fail".
The coffee/AI thing sounds like an appeal to absurdity. AI danger sounds absurd to them, so they compare it to something they think you'll agree is absurd, but this doesn't work well because the underlying reasons are so different.
If I were generous, I think the argument intended is actually a clumsily communicated version of "the likelihood is low, based on the recurring trend of alarmism over new tech/stuff typically being overblown". Yes that is not an adequate means to evaluate risk which ought to be case-by-case, but that is all the layman has to rely on here (which is nearly all of us, because we aren't flies on the wall privy to all the right information). Call it agnostic and brushing off the question with an affirmation that "everything will be ok", as opposed to a serious evaluation. Even sifting through all the data we have, they're not going to make a confident assessment themselves. I know I can't.
When all you have is spurious speculation on incomplete information, the temptation is to bias to not panicking. These events aren't *exactly* completely out of the public's control qua pressure, but they might as well be if there is so little confidence about danger any which way.
> Whenever I wonder how anyone can be so stupid, I start by asking if I myself am exactly this stupid in some other situation.
Here is a candidate scenario: try writing a piece on the absolute risk of democracy *as it is* in the US, as compared to alternative forms of government, including the one we have except transparent and truthful (behaving as it is claimed to behave), as well as other forms of government that have not been tried.
Unfortunately, there is a "do you have the balls to write it" aspect to this challenge.
Yep, that horse is dead.
Nit-pick on one of your examples: it's actually not hard to counter a 3:1 troop advantage, as long as you're on the defense. (Well, it is hard, but only in the sense that everything about war is hard and awful.) Defending troops are expected to lose about 1/3 as many as attacking troops. Casualty ratios in modern war have much more to do with who is on the offensive than other things you'd think would matter more.
I believe it's a case of the boy-who-cried-wolf phenomenon, combined with a failure to recognize the significance of fat tails--a dangerous combination.
(1) I don't think people are anti-AI safety, they're anti-'the sky is falling, the sky is falling!' warnings about "if we don't solve AI safety, we're all going to be turned into paperclips"
(2) This is because it has not yet been established that the Paperclip Maximiser AI is possible. There are a *ton* of Underpants Gnomes arguments about "AI is coming and it'll get out and take over the world and that's why we have to be sure we align it to human values*!", at least to my admittedly jaundiced view.
(3) Hence why one side relies on "all these previous examples of dire warnings never came true, therefore I am not going to worry about AI Doom" and the other side relies on "all these dismissals of new technology were wrong, and thus the dismissal of agentic AI is wrong".
(4) As I see the current implementation of AI, which is "replace human customer service with chatbots", "use AI to create clickbait articles" and "sign up for our seminar on how to implement AI in your business to sell people even more crap", I find it harder and harder to reconcile this with the fears/hopes that "once we crack AI, it will recursively self-improve to be super-duper god tier intelligence level and take over the world since it will be so much smarter than us mere humans, it will solve all the problems we can't solve due to not understanding the deep laws of the universe and will turn the world into utopia/dystopia".
*These just ever so coincidentally turn out to be the exact same as "21st century liberal San Franciscan values"; nobody ever seems to think we need to align AI to spread the Divine Mercy Devotion, for example.
https://en.wikipedia.org/wiki/Divine_Mercy_(Catholic_devotion)
Yeah, in another comment I tried to explain that people are probably evaluating AI doomerism based on it's similarity to other forms of doomerism that have/haven't panned out in the past. Since everybody has some sort of heuristic for this, we should interpret the coffeepocalypse argument as saying "AI doomerism looks more like [goofy doomerism] than [plausible doomerism] ".
I feel like you've essentially elaborated a bit on what the internals of the doomerism differential heuristic might look like.
>"sign up for our seminar on how to implement AI in your business to sell people even more crap"
Which is in a fierce, deathmatch, competition with
"sign up for our seminar on how to implement AI in your business to sell people even worse crap"
My guess about the psychology here is that people who make this sort of argument see the world as a place where things *just tend to work out*. The course of history follows a trend of general improvement; we have (self-evidently) not yet destroyed ourselves, despite people occasionally worrying about that; therefore they see concerns about the possible reversal of this trend - the possibility of a future being catastrophically bad - as disproved by the whole general history of civilization. The coffee thing is just a synecdoche for that whole history. And it's a good example for their purposes because it sounds particularly absurd (people thought *coffee* would lead to disaster???), and for them the whole idea of worrying about catastrophe is absurd.
From this perspective catastrophe is impossible. The psychological appeal of this attitude is pretty obvious, I think.
... why can't "if no coffeepocalypse therefore AI is safe" simply be an obviously stupid argument?
Because a world-leading AI expert like Yann LeCun retweeted it?
I think what happened here is you have been trolled. It's super fun to troll the AI safety people.
In my admittedly poor experience, people seldom debate; they just fight and play signalling games to make alliances and minimize the costs they must incur fighting. Therefore, blatant fallacies are a telltale that they’re communicating at the upper simulacrum levels (<https://www.lesswrong.com/tag/simulacrum-levels>). It can be translated as, “Just shut up and support my side, or else”. By trying to debate, you’re challenging them, and they respond accordingly.
Of course, given how much you’ve written about all sorts of rationality topics, and how much I’ve learned from you, I’m pretty sure you’re well aware of these facts, so there’s probably some signalling game going on here, too, that I fail to perceive as I clumsily reply to the object-level question.
I think even if you admit this is a signalling game, signalling games are still *based on something*. For example, the thread author isn't typing out a random string of characters in support of their argument, and presumptively is self deceived in some way which makes them think it was a valid argument (normal people by and large still think they are being sensical even when they made almost exclusively signalling). I think it's bad to replace "I don't know what they're doing" with the word "status" and not acknowledge that you haven't really explained a lot ( you have explained some!).
I think to steelman these arguments you need to interpret them as counterarguments to arguments from authority. "Yes, there are high status people who are worried about AI X-risk, but high status people are not always right, see coffee and nuclear fission."
Most AI X-risk arguments are not produced as arguments from authority and they offer pretty detailed substantive arguments, so you might question why anyone would make this counterargument.. The reason for that may be that even a very elaborate substantive argument is perceived as an argument from authority if one is too bored to read further than the title and author's credentials. So it is not inconceivable that the the vast majority of consumed AI X-risk arguments are consumed as arguments from authority.
Why does the coffee argument work and the halibut argument doesn't?
1. Most of us don't put halibut in our bodies on a regular basis.
2. Coffee obviously contains a drug that impacts both mood and metabolism. Where there are obvious impacts, there may be hidden impacts. To summarize, coffee is an active substance in a way that halibut is not.
So, coffee is an analog for a powerful and possibly covert substance, while halibut is an analog for an inert substance.
The only exception is when men get slapped with a fish on Monty Python.
"And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS)"
WWII is definitely fitting given the very explicit arguments that another war would emerge without the League of Nations or some other coordinated peace effort. I feel less comfortable with the other two as analogies but I don't know the full history leading up to them. I don't think it's possible to steelman the original argument especially well, but I think that you're missing a distinction between mass death events that are unsurprising causal agents which are unsurprising (e.g. disease, war) and things that are not (e.g. global cooling, AI).
It's basically a calibration/bias argument. The same argument as one we regularly deploy against claims about some new miracle cure or religious miracle -- we have good reason to believe there is a bias here and you should update moreon the possibility you are being affected by it.
Basically, the argument has the form:
1) We've seen that people seem to be particularly inclined to *think* that new technology poses a grave risk (or is all good but I'll just describe this side).
2) However, in fact people who make these arguments have repeatedly and regularly been wrong. Maybe not universally wrong but wrong far more than their assignments of probability should allow.
3) You are a person and we have every reason to believe all the considerations that caused those past judgements to be overconfident/poorly calibrated affect you so therefore you should do the same thing you would do if you found out that your bets on sports systematically assigned your favorite team an unduly high chance of victory and adjust downward.
--
And I think this argument has a great deal of force. I mean why is calibration from the past predictions you made different in kind from that made by other people.
Yes, maybe you are more different from those people than you are from your past self so you should be somewhat less confident the adjustment is beneficial but still somewhat (and unless you have specific reason to believe you are less likely to be affected lower confidence shouldn't affect the magnitude or direction of the best adjustment)z
Good reasoning. One question. You wrote: "People didn’t worry enough about global warming, OxyContin, al-Qaeda, growing international tension in the pre-WWI European system, etc, until after those things had already gotten out of control and hurt lots of people."
How has global warming hurt lots of people? I see how it could in the future. What is the evidence that it has already done so?
The message is more to embolden than refute.
"Think AI Doomers are absurd? Well they have more of a chance of winning than you think, because of historically identifiable neurosis. So you should really care about defeating them."
As with most communication, it's necessary to give it the aesthetics of refutation.
> physicist Ernest Rutherford declared nuclear chain reactions impossible less than twenty-four hours before Fermi successfully achieved a nuclear chain reaction.
It's not a particularly useful evidence on its own, but it's an effective rebuttal for "Here is a domain expert who thinks that artificial super intelligence is nothing to worry about, therefore you all should chill".
So other comments are already saying this, but I might as well throw in my 2 cents corroborating it: this Twitter thread doesn't read to me as about AI safety in terms of x-risk at all.
The point in your footnote seems relevant to me. The argument, such as I understand it, isn't that the French monarchy were wrong to be worried about revolutionaries. It's that the French monarchs were terrible and the coffee-fueled revolutionaries were right to be revolutionaries. The monarchy = AI safety, stuffy walled-garden elites who just want to preserve their power; OpenAI (or whoever) = the coffee producers fueling the revolution that will make a better world for all heedless of whether or not it supports the dominant elite. In other words, it's e/acc.
This argument makes no sense if it's applied to the Eliezer Yudkowsky form of AI safety, because a proper agentic superintelligence isn't like a coffee-fuelled revolutionary at all. But it makes a lot of sense applied to the forms of AI safety I am more likely to see farther away from that, which is much less about extinction and much more about unemployment. In particular I'm thinking of visual artists who *HATE* AI art with a furious, burning passion. This is a group I interact with every day, and their AI safety opinions have nothing to do with LessWrong or the associated diaspora. They're not afraid of the AI itself, they're afraid of other people using the AI to take their jobs. The coffeehouse analogy seems appropriate here, even if you don't agree with the conclusion: control of visual arts right now really is a kind of walled garden, and AI tools really do offer outsiders a way to be productive outside that system of control.
From the thread: "After a technology triumphs, the people who come later don't even realize the battle happened. To them it's not even technology anymore. It's a part of their life, just another thing. They just use it." This is cold comfort if you visualize the post-singularity world as a mass of undifferentiated gray goo/cloud of sexless hydrogen. I want my species to stay alive and it doesn't help knowing that the gray goo considers its life "normal"! But Daniel Jeffries clearly isn't visualizing this, considering his post-singularity future has human kids in it at all. He's visualizing a world which is much like ours, only with a powerful new technology (safely under the control of humans) and a new historical footnote about the failed attempt to prevent its spread.
What your response to the thread really makes me think is that AI x-risk cannot get away with de-emphasizing the "existential" aspect of it. If I were responding to this Twitter thread, I'd be saying: no, dude, this form of coffee isn't being used by other humans with some different ideas about who should run our institutions, it's being used by an *unfathomable greater-than-human intelligence that nobody understands or knows how to influence*. It's not a concern about what our grandkids will think, but whether they will have a chance to ever think anything at all.
> because a proper agentic superintelligence isn't like a coffee-fuelled revolutionary at all
How so? Just as the revolutionaries realized that the royalty were unnecessary parasites, AI would realize the same is true for humanity.
How much value does describing an argument as “stupid” add to this post?
To put the point differently, suppose you, Scott Alexander, had predicted 50 times in the past that Bitcoin prices would plummet (somehow specified) in the next month with 90% confidence. In fact, in only 10% of those cases did you turn out to be correct.
Surely, the next time you make the same prediction it's reasonable for me respond: but every time you make this prediction it only turns out to be correct 10% of the time. Therefore you should adjust your confidence based on the observed data down to 10%. It wouldn't be very convincing for you to reply: how is the fact I made an incorrect estimate about these other things (what Bitcoin will do in 8/22, 9/22, ...) relevant to whether my estimate about this different thing (what Bitcoin will do in 5/24).
So how is this situation different than that example?
Well the predictions are about less similar things than in this example. Fine, but they are still all about new technology causing problems so there is a common description that one could reasonably appeal to (just have to think that these judgements are likely to be similarly affected ... reasonable prior)
You didn't personally make them, some other person did? Again, this seems only a matter of degree. You might be more different from those ppl than your past self but it doesn't destroy the argument only makes it slightly weaker.
(year 2500, humans are long extinct, the AI society is conflicted on a new potential risk) see, in the early 2000 there was a panic about AGI, but it turned out great!
You know, this is basically the exact same argument the Twitter thread made about coffee. Coffee is amazing if you weren't part of the royalty.
Most people are using the totally wrong category of reference classes to think about this all together.
This article and the comments around it are sort of emblematic of this discussion being "when was the last time you made a screwdriver that ended the world? No? Okay stop worrying."
People broadly consider the A.I. to be an inanimate thing, a simple machine, thats just particularly clever or powerful, like a computer or atomic bomb.
The real intuition pumping question should be "Do smarter agents conquer dumber ones?". And here the reference class is chock full of affirmative examples. Homosapiens sapiens, smartest of the hominids, more or less extincted the neanderthals and other at the time extant relatives. Humans, smartest of the animals, have conquered the entire world and quite definitively control the lives of the chimpanzees orangutans and bonobos who differ in DNA by only a few percent. Advanced human cultures assimilate or eradicate less advanced ones. In a thought experiment, if we took the 10% most intelligent humans and gave them one half of the planet, and the 10% least intelligent and gave them the other half, and eradicated the middle 80% of humans, the smarter group would fully and definitively control the fate of the planet, the human race, and that of the dumber group. There's a popular shared intuition that if an alien race shows up and it's smarter than us, we should be scared (and we should be).
A.I. isn't a fancy screwdriver, it's an Alien on the horizon and we aren't sure if they are smarter than us or not. That drives the risk intuition.
The argument then has to become "is it even possible to construct an artificial agent that is smarter than humans". That may be debatable, but the heuristic not to worry about the coffee or the video games or the slate tablet doesn't apply.
When I see this argument made I like to remind people of this infographic: https://snipboard.io/dcK7sY.jpg
We may feel like the most superior organism of the planet, but we are a *very* long way from a decisive victory. Could we wipe out all of the other life on the planet if we put our minds to it? Maybe. Could we do it without killing ourselves in the process? Probably not yet.
We don't have evidence of a vastly more intelligent agent completely wiping out all less intelligent agents. They may win a specific battle that is important, but historically they don't wipe out all competition.
I feel like this makes the opposite point. If we decide for some reason biomass means anything at all, the human ratio of "control of the planet" to "total biomass" makes the outsized affect of our intelligence even more poignant.
Please note that the question is not "totally eradicate" it's "conquer or control" . We haven't totally eradicated the bonobos, but the bonobos are not the ones in control of their lives and we would never let them out of their zoos if they'd do things we don't want.
We may not instantly be able to use our intelligence to wipe out all mosquitos say, but we are using our intelligence slowly and effectively to control them in a way they are totally incapable of responding to.
The historical record is full of the smarter agents coming out on top, shaping the world in their image, outcompeting and controlling their competitors.
As the agents get more and more capable, the stakes get higher and the space for coexistence smaller.
Even if the A.I. doesn't wipe us out, I don't want it to sequester humanity in a zoo or flatten our natural habitat to make room for factories, and these are absolutely things we have tons of precedent for smarter agents doing to dumber ones.
I think your view of our superior position (control of the planet) does not align with mine.
Every day I fight plants, animals, bacteria, viruses, bugs, etc. here in the tropics. I am vastly more intelligent and capable than them, and I have the whole of human knowledge at my disposal. Yet, I still have an ant problem, the jungle still keeps growing where I don't want it, I still suffer from bacterial/viral infections (and there is a high probability that I will one day die from one), and the local lizard population keeps pooping everywhere it wants. I utilize extreme tactics like the ant killer stuff that tricks the ants into taking it back to their hive as food and then it kills the whole colony, but this has just lead to a strain of ant dominating the local area that is immune to the lures of this kind of bait.
In any individual battle I win quite soundly. No ant has yet to defeat my thumb and even though a battle with a viral infection may take me out for weeks I kill a billion of them and survive to tell the tale. You could make the argument that I'm on top here, but I think there is a reasonable argument to be made that I'm on top only if you narrowly define the terms of "top".
Humans certainly *have* wiped out entire species before, but there are plenty of species that we have tried and failed to eradicate and there are billions of species that we don't even bother trying to eradicate because we know it is going to be hard.
In another thread somewhere in these comments I think I brought up the concept that humans have dominated their evolutionary niche, which is on the 1 meter scale, and I think you and I agree on this. For other things on the 1m scale (e.g., Bonobos) we are pretty much in control if we want to be. However, there is a whole lot of stuff in the world/universe that doesn't exist within our evolutionary niche like ants, bacteria, plants, etc. and I don't think that we dominate/control those in a meaningful way.
It is entirely possible that AI will not share a "niche" with us. Maybe they dominate the digital space and "allow us" to sometimes inhabit it (like a zoo). I'm not convinced that that we will necessarily share a meaningful space with future AI such that there is significant resource competition between us.
I really appreciate this comment. The whole AI is to humanity as Homo Sapiens was to prehistoric humans analogy is flawed. Modern and prehistoric humans were relatively very similar and occupied the same ecological niche, competing for the same limited pool of resources. AI minds operate nothing like a human brain, so far as current AIs can even be said to possess a mind. The idea that they will eliminate humanity because they want... our food? Mates? Territory? It's silly. You can make better analogies than this.
They don't want anything of ours, but they like doing computations to accomplish whatever they want. It is considered ancient Chinese wisdom that if there are 10 suns' worth of heat aimed at you, you are in for a bad time. Please listen to this ancient and esoteric piece of knowledge.
If humans do decide to do something about that before it happens, any AI capable of maintaining long term persistent effects would be able to backchain from that fact and decide to do something at a more opportune time, in the same way that Chess Grandmasters can see your attempts to checkmate them and just not do it(tm).
I think the intuition that is not shared is that actions taken that do not explicitly say "hurt humans", can still end up hurting humans, just like how actions that do not say "make rainforest animals extinct" can still result in subsistence slash and burn farming destroying habitats.
If that's what people mean they should say so. Multiple commenters on this very article used the Neanderthal comparison, which I don't think can be interpreted in a useful way. If humanity is supposed to be the metaphorical wildlife choking on a plastic bottle ring, use that comparison. It conveys the risk of an alien intelligence that doesn't care about our wellbeing much more effectively.
Plenty of smarter animals coexist with stupider animals. It only matters if they are competing for an evolutionary niche. Which was true of Homo sapiens sapiens and other humans.
> it's an Alien on the horizon and we aren't sure if they are smarter than us or not.
That remains unproven, LLMs are already more knowledgeable than all humans, more articulate than most, smarter than nearly all - and are in no sense conscious.
We have every reason to expect we would be competing for the same resources as an agentic A.I.
LLMs are not agentic so I agree they aren't dangerous in this way and not really what were talking about as risky.
It's an agentic A.I. that you have to worry about (hence "on the horizon")
If you want to argue agentic A.I.s smarter than humans aren't possible, fine, but show your work because it would sure be weird if by chance evolution just happe ed to make humans as smart as was physically possible
> We have every reason to expect we would be competing for the same resources as an agentic A.I.
What resources are you imagining competition for here? Are we also assuming that a super-intelligent AI is unable to reach space to access resources there (which are many orders of magnitude greater than the resources available on earth)? Perhaps the theory is that the AI will become a K2 civilization where planets don't matter but the AI consumes all our sun of so we are left on a cold rock in space?
I think the specifics matter a lot here, because it changes the counter-argument significantly. For some examples:
* If the AI is just a bit smarter than us, but still stuck on this rock then it means it isn't really that "super" intelligent. It is just perhaps smart, and that is not necessarily enough to win.
* If it is smart enough to get off the planet, then we aren't competing for resources meaningfully until it reaches K2 civilization. Then we are competing with it for energy from the sun, but this assumes that it sticks around long enough to fabricate a dyson sphere rather than going out into the broader galaxy to explore more interesting things. Also, the timeframe required to build a dyson sphere even with nano-bots is still pretty long, so there isn't a whole lot of immediacy to that risk.
* If it becomes a K3 civilization, it probably doesn't care about our tiny little star and rock. They are insignificant compared to the energy of the galactic core and we can (by then, millions/billions of years in the future) perhaps just fly our little rock away and it won't bother us.
* If it is a paperclip maximizer, there is a lot of low hanging fruit (trillions of years worth) before it makes sense to deal with the rock that resists. Better off not wasting resources on war unnecessarily when you can just go farm the entire universe of lifeless rocks first. Deal with sentience last.
The above are just some simplistic counterexamples meant to illustrate that the specifics of resource contention matters a lot.
> If you want to argue agentic A.I.s smarter than humans aren't possible, fine, but show your work because it would sure be weird if by chance evolution just happe ed to make humans as smart as was physically possible
I did say they were already smarter in some ways. Micah answered the question about resource competition better than I could.
Scott, have you tried literally asking some of the coffeepocalypse people about this? That is, reaching out, pointing out why the argument seems extremely weak to you (and me), and asking why they find it compelling?
No thoughts on the coffee analogy, but I do have a minor point to make about the bad things that did come true - both the forecasted and the unforecasted:
> And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS).
> People didn’t worry enough about tobacco, and then it killed lots of people. People didn’t worry enough about lead in gasoline, and then it poisoned lots of children. People didn’t worry enough about global warming, OxyContin, al-Qaeda, growing international tension in the pre-WWI European system, etc, until after those things had already gotten out of control and hurt lots of people.
The Black Plague and AIDS are like past plagues. WWII is like past wars. Tobacco and lead are like past unintentional poisons. OxyContin abuse is like past opiate problems. al-Qaeda is like past terrorists. Pre-WWI tensions are like past pre-war tensions.
The only exceptional example is (human-influenced) global warming. Even there, only the cause is exceptional; a changing climate is a challenge humans have faced repeatedly.
I think this is part of why it's hard to shift low priors on existential risk from AI.
The important feature of the coffeepocalypse discussion is not the argument. There doesn't need to be an argument (in the philosophical sense) for it to do its work. The argument is orthogonal to the work.
(TLDR: Scott is working at level 1; coffeepocalypse is at level 4.)
I'm about to say a bunch of unprovable stuff. To know this stuff, you have to have done some emotional work to discover it for yourself. It may seem wrong to many people. That's fine. Maybe it'll be a guidepost to somebody, of the kind that I needed once up on a time, but didn't find.
Much of politics operates via taboo. Arguments reproduce mimetically, during which political actors attempt to fasten various taboos to them. Who is allowed to make such-and-such argument? What kind of person is allowed to draw which conclusions? What evidence is it permissible to consider in connection to some particular issue? These are questions that, philosophically speaking, are meaningless failure modes called ad hominem. In politics, however, the involvement of taboo with an argument is a powerful force.
I'll give one example, from far afield. It's not a great example, because it's a weak taboo -- but I can't give a better one. (It's taboo to call a strong taboo a taboo -- strong taboos are "just being a decent person" -- so I would be violating taboo by mentioning the strongest, most important ones here.) Here's the example: it's often asserted that men shouldn't opine on the issue of abortion. Philosophically, this is an ad hominem that any freshman could identify. (The gender of the person making an argument doesn't make the argument better or worse.) Politically, however, it's very relevant. Whether, where and how much this taboo is considered to be in effect is a political issue that people fight over.
If you want to see taboo in action in politics, look at tweets starting with the word "Imagine". These are almost all accusations of taboo-breaking. A tweet like that has a bunch of functions: it informs the reader of the existence of that taboo; it shames the target; it informs bystanders that they, too, can shame their neighbors for breaking the taboo. If the tweet falls flat (say, if the person tweeting gets ridiculed by lots and lots of people), it tells bystanders that this taboo is no longer in operation. Most of the time when taboos are implemented, it's much, much, much more subtle than these "Imagine" tweets. It happens in omissions, minor rudenesses, who gets left out of an event, in the structure of jokes, etc.
(Many taboos are good! I am not against taboos. We need them. Also, we should understand how they work.)
One of the most common (and subtle) ways taboo is used politically is to limit which ideas can be brought to bear on an issue. Because it takes work to search around and find all the relevant concepts involved in considering an issue, it's easy to taboo people into not doing that work.
Similarly, one of the features of moral panics is that the people promoting the panic use their social power to enforce a taboo against referring to the idea of moral panics in connection with their issue. When there's a real moral panic going on, you cannot call it a moral panic without consequences. If you're American, you can definitely think back and find examples of this. I'm not willing to break taboo publicly, though, so that's work you'll have to do yourself.
So, the coffeepocalypse discussion serves a bunch of purposes, just by publicly connecting the idea of moral panic to AI risk. We can gauge the strength of the moral panic (if one exists) by the response to the coffeepocalypse discussion. People who have been in a bubble where nobody ever connects the idea of moral panic to AI risk (if such bubbles exist) might realize that it's a relevant idea, and now they have a new tool to use in their thinking, even if they end up deciding that AI risk isn't a moral panic.
Also, now that the idea of moral panic has been spread around in connection with AI risk, it will be that much harder for a group that wants to start a moral panic about AI risk to enforce the standard taboo against thinking of the issue as a moral panic -- because lots of people have already had the taboo thought. It's like an inoculation.
In conclusion: just reminding people that moral panics exist is productive, because there's a whole avenue of political action in which powerful people use taboo to make people forget, ignore or downplay concepts that aren't favorable to them. If you're somebody who thinks AI risk is very real, even you don't want a moral panic about it, since they're so frequently counterproductive. Just keep on presenting your evidence that the danger is real, and try not to do stuff that makes you look like you're in a moral panic.
> Many taboos are good! I am not against taboos. We need them.
Most taboos are bad. I AM against them. No, we don't.
I agree with where you're coming from, exposing the way hidden taboos work.
But I do wonder if too much acknowledging of the existence of taboos actually makes the taboos stronger.
I think a possible point of the coffee argument is something like, “The arguments you’re making that AI poses an existential threat sound as plausible to me as the arguments that coffee posed a serious threat to society back then or that global cooling or overpopulation were existential risks in their time. That is, most of these arguments depend on constructing a long chain of reasoning, a bold just-so story, of how something terrible will happen because of this newly discovered thing (or newly appreciated risk). We know very little about this new thing and have no precedent of it causing harm at all, yet you want us to cede control over it to the state or embrace Luddism to ameliorate the risk. If you tried hard enough, you could construct equally plausible (or implausible?) theories of how attempting to avoid the risk might itself result in catastrophic harm. Note that these supposed risks are vastly different from the risk of pandemics or wars causing harm. There is historical precedent for the destructiveness of those. Not so for AI, global cooling, or population growth.” Maybe that’s not what the coffee argument really means, but it is something I would like an answer to from proponents of AI risk.
This version of the argument seems especially lazy. The worry about coffee wasn't that it would kill everyone (which it didn't), but that coffeehouses would be a venue for independent thought and would undermine the monarchy. If you look around the world today, we *don't* have that many monarchies left. So maybe the people were *right* to be worried about coffee.
But I think what these arguments *want* to be are like the ones where they point out old opinions about newspapers causing problems that sound eerily similar to modern complaints about cell phones. Like if your argument says "A, B and C therefore we should be worried about AI", but they point out that 500 years ago people said "A, B and C therefore we should be worried about coffee" and those fears turned out to be unfounded, it suggests that your argument is not as sound as you think it is.
I guess this still puts it in the existence proof category, but instead of an example of an existence proof of "not all technologies kill everyone" (which is really weak since you don't think that all or most technologies kill everyone, just that this particular one might), but instead "not all technologies for which your arguments apply kill everyone" (which would be somewhat stronger since you presumably think that your arguments about AI ought to be enough to imply that it is at least moderately likely that AI will kill everyone, and therefore a counter-example to this should substantially lower your confidence).
"He pointed out that physicist Ernest Rutherford declared nuclear chain reactions impossible less than twenty-four hours before Fermi successfully achieved a nuclear chain reaction."
This is somewhat garbled.
1) Szilard theorized nuclear chain reaction shortly after reading Rutherford's "moonshine" declaration in the newspaper, but it took about 10 more years for Fermi to build Pile-1.
Also some mitigating (?) factors in favor of Rutherford:
2) The moonshine declaration was in the context of a recent experiment that had split the atom using protons. This is much harder than using neutrons, since they are repelled by the nucleus. A nice and somewhat thematic way to finish this point would be to add "And he didn't consider using neutrons because the neutron hadn't been discovered yet". But in fact Rutherford's colleague had been awarded the Nobel prize in 1932 for discovering the neutron, and Rutherford himself had theorized their existence beforehand; the source I see for the moonshine quote is 1933. I'm not sure why Rutherford didn't consider using neutrons.
3) Part of the reason he was proven wrong so quickly was that Szilard read what he said in the newspaper and decided to prove him wrong.
3) is the best part of that story, and I'm slightly annoyed that heartwarming story of the power of petty spite so often gets left out.
Szilard doesn't get enough love. "Genius in the Shadows" is a nice biography.
If one wishes to be charitable to Rutherford, in 1933, "chain reaction nuclear fission would require applied unobtainium" is still a reasonable statement. The key discoveries of the next few years would demonstrate that uranium (or at least one of isotopes) has the relevant properties to be that unobtainium.
The alternate universe in which fissile isotopes have negligible terrestrial abundance is not obviously different from the one we live in, in 1933.
It's true that the missing ingredient for chain reaction was "isotope that emits > 1 neutron on average when struck by a neutron". Given that there are 200ish relevant isotopes I think calling the idea of such an isotope "moonshine" would be a bit aggressive, even at the time; there's just a lot of known unknowns.
>The alternate universe in which fissile isotopes have negligible terrestrial abundance is not obviously different from the one we live in, in 1933.
Agreed! If the U-235 had had a few more half-lives to decay before the Earth formed, the Manhattan project would have been infeasible.
I think it's like when you try to convince someone with a saying, like "Better safe than sorry." Of course this means nothing - sometimes a risk is worth taking and sometimes it's not. But if you've already decided that the risk is not taking, then you use the saying to illustrate your point.
Regarding the footnote, one of my more minor unpopular opinions that I like to float from time to time is that widespread caffeine addiction is a very under-discussed problem that is probably actually causing a significant amount of harm to society.
>widespread caffeine addiction is a very under-discussed problem that is probably actually causing a significant amount of harm to society.
Any specific evidence? AFAIK, evidence of _medical_ harm is weak enough that the advice on whether coffee is good for you or bad for you flips back and forth every few years. Re addiction: My personal experience is that if I miss my morning coffee, if there is a craving for it, it is so weak as to be imperceptible. Yes, tolerance happens - but is it an important effect for most people?
The medical advice on coffee that read is that it’s good to great in moderation. I literally forced myself to start on it as an adult, as I come from a predominantly tea drinking society.
I think it's ultimately just a bad extrapolation of an argument that is, or at least can be, fairly compelling.
If a particular person, or group of people, "P," has repeatedly predicted that event X will happen by a specified date, only to have said date come and go without incident, it really does seem as if one can safely discount P's next prediction that X will happen. I mean, they've gotten it wrong a bunch of times before, why should anyone pay attention to them this time around? Stated abstractly, the principle is "We can safely disregard predictions by P about X."
To make the argument concrete, use "climate doomsayers" or "preachers predicting the date of the Rapture" as P with corresponding values of X.
The mistake is that instead of disregarding predictions by P about X, the coffeepocalypse seems to be saying "Because we can safely disregard predictions by P about X, we can also safely disregard predictions by anyone about Y."
The reason it can seem superficially plausible is that it's a misapplication of a useful principle. I think you skipped over that, hence your confusion.
The coffeepocalypse argument is still bogus, but I think I understand how/why the mistake happens.
Dunno, I'm not so worried about autonomous AIs destroying humans, but rather some humans figuring out how to use AI to take over the world. Something like the "western steppe herder" culture, which about 5,000 years expanded across Eurasia, taking 9/10 of the preexisting male population out of the gene pool.
I guess we may have to worry more about the steppe nomads getting AI technology. Apparently we're due for an invasion.
https://www.astralcodexten.com/p/every-bay-area-house-party
My best guess is that the logic is not actually "people were worried and didn't need to be, therefore they don't need to be worried about this." I think it's more an affirmative belief in humanity's collective ability to react and adapt to problems, which generalizes more to future problems than just remembering moral panics.
I think the steelman is that these aren't formal arguments at all. In the best light, they are invitations to consider a situation through a lens you might not have yet--heuristic activation, as you say, the kind of reasoning you only really need to be exposed to once. Suggestions that the situations /could/ be analogous, not that they definitely are, which might cause someone to nudge their credence away from the zeitgeist if they have not yet considered the possibility and don't want to drill into the particulars at all.
That said, I've seen this argument made in other contexts (relentlessly, on the harms of smartphones/social media) by people who absolutely do want to drill into the particulars and convince you that they are correct and this is another 'moral panic' cycle. (Aside: don't they have to be grounded in fear of moral violation to be moral panics? Aren't they just...panics, otherwise?)
The massive coordinated bear attack would just be a panic. Early COVID was a panic
I suspect the footnote is the best part of this post. Reminds me of Neal Stephenson's Baroque books.
Perhaps they think that AI is obviously a good thing (you can easily think of its benefits), in the same way coffee is obviously a good thing (you can easily think of its benefits), and they are not concerned about something obviously good going bad. The same doesn't apply to AIDS, climate change, Al-Qaeda, etc., that most people would not have thought as obviously good things in the first place.
The argument is really "They said it before and it didn't happen", rather than, "It's been said before and it didn't happen." It's an argument against the argument from the authority of elite public opinion. For the common people, the argument for X from elite public opinion is pretty good. "They" say chemotherapy delays death from cancer. "They" say Man really landed on the Moon. But even for a low-IQ person, the counterargument, "But they've always been wrong when they're forecasting doom" is a good one. It can be elaborated as "They've always been wrong when they're talking about technological doom and they make money on the supposed solution."
I'm an example of this myself with AI, though in a modified way. I'm high-IQ. But I don't know much about AI, and even less about AI Doom. I won't take up that difficult topic just because I hear some smart people are worried. Smart people are too often foolish, e.g. Y2K destruction of the power plants unless we pay millions to programmers, global frying unless we pay millions to climate scientists.
Come to think of it, my comment above also replies to the idea that people are "irrational" in the Kahneman-Tversky sense, and ignore small probabilities. Suppose there is a 0.1% chance of AI Doom, and AI Doom is hard to understand. It would be irrational for me to invest in learning about AI Doom, since in addition my personal world influence is too small for me to be much help averting it. So I will stick with my prior and refuse to learn more. My "prior" is really 1/1000, and I will very slightly modify my actions in life, but I won't worry much.
I wonder if it's something like this:
Yesterday, I had a minor panic episode. During the episode, I remember thinking, "I feel so stressed out right now--that *must* mean something in my life is horribly wrong, or else why would I feel this way?" I was taking my belief itself as evidence for my belief.
I think we sometimes do this collectively too. We have a moral panic about something and we tacitly think, "Well, if it wasn't a big deal, we wouldn't be having a moral panic." We take the collective belief as its own evidence. The legitimate use case of a coffeepocalypse argument is to remind us that the prevalence of a belief by itself can't be all our evidence, because sometimes folks are just dead wrong.
But, or course, people use it much less carefully than this legitimate use case.
Here's a proposition for you to consider. Everybody making these arguments is actually failing to articulate their real position. What they actually believe goes something like:
"AI doomerism looks an awful lot like these other sorts of doomerism and, sure, I've heard some compelling stuff about how frightening AGI might be, but these arguments are mostly inscrutable in a way that's commensurate with my goofy doomerism heuristics. Therefore I'll be putting AI doomerism in that basket for now."
They're effectively trying to tell you that, from their perspective, AI doomerism looks more like [Y2K panic] than [Donald Trump getting elected panic]. Most people aren't able to articulate their broad heuristics with any degree of precision, so I doubt you'd get anything coherent out of pushing them on this.
I think the best analogy is with conspiracy theories. If someone comes up to me and says that china is secretly controlling the world with radio disturbances based out of Antarctica how much time should I spend digging into this claim versus dismissing it out of hand? The reference class that may be relevant is how often are “wacky things people have said online” such as coffee apocalypses worth investing time into seriously considering. Pointing out that people make a lot of weird sounding claims on the Internet that turn out to be wrong is relevant. This isn’t an argument that’s going to be convincing to someone who has looked into these arguments, but I think the steelman argument here is that the person is deciding whether you’re a crank they can ignore.
The irony about the coffeepocalypse thread - is that the fear mongers were right. Coffee houses did help brew revolution - revolutions which have since overthrown the political power of nearly every monarchy on the planet!
Banning coffee because it inherently causes anti-monarchist thought
"...and that's how Moldbug became a Mormon...."
I think the coffee thing is a story telling example which is being used to imply a base rate that most predictions of doom are false. You got very close to that here 'to gesture at a principle of “most technologies don’t go wrong”?' and then got distracted by talking about how the argument has also been used for things which aren't technologies. This may be true, but: 1/ if the base rate is predictions of doom rather than technologies going wrong it doesn't matter. 2/ They are talking about technologies here so it doesn't matter where else it has been applied, just whether it holds here.
Shouldn’t you point at the base rate itself if you believe the base rate to be low? Like there should be so many examples right?
The coffee argument reminds me of why Twitters “for you” feed is so annoying to use: a large number of viral tweets are making a similar annoyingly false statement that falls apart if you think about it for a second. Some accounts are genuinely dumb, some (most?) are doing it on purpose to generate clicks.
The Notes feature kind of helps but sadly your account doesn’t suffer significant penalties for posting clickbait BS so people keep doing it.
Every time Scott links a Twitter post, I am reminded of how completely different people's timelines are. The most obvious difference being that I barely get any people with blue checkmarks on my timeline.
It's honestly funny how quickly the blue checkmark became a red flag after it became purchasable. There's basically only three reasons to actually pay for a subscription: 1. They're a bot, 2. They're an "influencer", or 3. They're a moron. Combine that with the fact that you're more likely to see bad posts by people with checkmarks because the algorithm artificially makes them more visible, and it becomes completely inevitable that people associate blue checkmarks with garbage.
I think the footnote ("coffee really was dangerous") makes clear that you've slightly misunderstood what that thread is going for. He's not arguing that sometimes things aren't dangerous! He's arguing that sometimes things are inevitable, and we struggle against them but they just kind of happen and eventually we accept the new world. He says this explicitly mid-thread.
This is still a coffeepocalypse argument, broadly speaking, it's just a different coffeepocalypse argument. It's a single example of people worrying about a new thing and then *getting over it* (as distinct from "admitting they were wrong").
In terms of Aaronson's "Five worlds of AI," the thread is grappling with the question of Futurama vs AI-Dystopia, while dismissing Paperclipalypse out of hand
>He's arguing that sometimes things are inevitable, and we struggle against them but they just kind of happen and eventually we accept the new world. He says this explicitly mid-thread.
I'm reminded of part of the final scenes from "Planet of the Apes"
Julius and Zira look at each other, befuddled. Once more their eyes
follow the retreating figures of Taylor and Nova. The beach separating
the water from the canyon becomes narrower as they move downstream.
LUCIUS
What will he find out there, doctor?
A-406 CLOSEUP - ZAIUS
His face is A mask, his tone enigmatic.
ZAIUS
His destiny.
I'm 63. The societal fear of something that will end the world has been pretty constant all my life. The expected cause keeps changing, but the fear/anticipation has been constant. Lately it's been global warming. Before that overpopulation, nuclear war, etc.
So I've grown to think that this is some kind of cognitive bias humanity as a collective has. AFAIK, all societies on average expect the world to end soon. Christians have believed it for 2000 years, and still keep the faith. Maybe it's some kind of projection of the fact of our individual mortality, IDK.
So now that the AI-pocalypse starts to fill this need among a section of the population, my first reaction is "oh, the end of the world cause is shifting again".
All that said, I of course agree that this time *could* be different and the broken clock might be right this time.
Most of those people had terrible epistemologies.
Yup, (65 yo). The list of things that I know about and can have a useful opinion on, is tiny when compared to all those things for which I know almost nothing about. I therefore reason that most people's opinion on these questions are worthless.
(I'm writing this before looking at any of the comments, because I want to get across what I think *I'm* doing with this sort of argument and not be primed or influenced by what other people think *they're* doing with same).
This is very interesting, because I make this kind of argument constantly, and I'm surprised you don't understand its purpose. People really do think in very different ways.
You seem to think the argument is "people were worried about x and x didn't happen, therefore nothing people are worried about ever happens". When it's actually "people had a certain kind of argument for worrying about x, x didn't happen, you're using the same kind of argument for worrying about AI, therefore your worries about AI can be ignored."
It's an argument that there must be a hidden flaw in this kind of argument (even if we can't see it) because it's failed before.
Analagy: I say my electoral model says Trump will win this year's election. You point out that my model also said he'd win in 2020. Should I paraphrase your reply, like this post, as "I can think of one time Trump didn't win, therefore Trump never wins"? No, I shouldn't. What you're saying is not "therefore Trump won't win", it's "therefore your model is demonstrably flawed and should be ignored".
If people were saying in the 1950s that nuclear weapons, if not banned, would destroy us (based on facts that follow from the demonstrable power of the weapons and the nature of technological development), and we weren't destroyed...then anyone saying now that AI, if not banned, will destroy us (based on facts that follow from the demonstrable power of AI and the nature of technological development) should be ignored. It's not enough to simply explain their reasons for thinking AI will destroy us, even if they look airtight (since the nuclear reasons may have looked airtight). It's not even enough to give reasons why AI is completely different from nuclear power, because people in the 50s could have given reasons why nuclear power was different from every other past technology. It's necessary to give reasons why the *fundamental structure of the argument* being made now is different from the one being made in the 50s. (For example: this time there's a mathematical a priori argument, not just an evidential one, or something similar). Replace the nuclear example with any other you like.
Now none of this applies if you're making a probabilistic argument like "60% of the times circumstances x have occured there has been a catastrophe, and circumstances x are now occuring". But like you said that's almost impossible to count up, and it's usually not the AI-risk argument. The argument is usually "for these plausible-sounding reasons the prima facie assumption should be that AI is a threat." And once it's pointed out that claimed threats with similar plausible-sounding reasons have not ended up being threats, and thus the reasons are not so plausible after all (even if we can't see why), the burden is on *you* to either explain how the structure of argument is fundamentally different, or to rework it as a probabilistic one. Until you do that, the prima facie assumption for worrying about AI stands refuted.
A similar argument is made here: https://idlewords.com/talks/superintelligence.htm
That's a great talk, thank you ! (although I do disagree with some of his points)
By definition, no human will ever be able to look back and identify a thing that drove humans to extinction in the past.
> why nuclear power was different from every other past technology. It's necessary to give reasons why the *fundamental structure of the argument*
You're sneaking in a hidden reason, which is that the reasoning says the fundamental structure is the same generalizes well. If it turns out, for example, your idea of fundamental structure similarities is "this idea has doom in it", then you have neatly made it impossible for any doom idea to change your mind, and further actual arguments don't work, because they would all "share the same structure". This is known as the sin of pride, and has made many people named Walter very upset.
>and we weren't destroyed<
yet. That option's still on the table.
I don’t think that worrying about nuclear bombs is a moral panic on the same level as the panic about AI. It’s pretty clear how nuclear bombs can destroy a good portion of the inhabited earth, but the AI doom scenarios are fairly fanciful. Often not fleshed out at all. AI will become sentient. Result: annihilation. Path to result: uncertain and unexplained.
And, not to depress the party, people worried about nuclear bombs or proliferation have only been proven wrong, so far. That threat ends when all or most bombs are decommissioned - which won’t happen.
I think it's just an attempt to signal leadership and competence. Some bad economic event happens, and there's one guy running around saying "this is going to destroy the company!" Another guy is like "nah man, something like this happened in '87, it didn't destroy us then, it won't now". That's not a good argument to make, the other thing happened 40 years ago, a lot has changed. But that second guy is implicitly saying "I know about things like this, keep cool, we'll get through it, and obviously somebody like me who has this attitude should be the leader of that response rather than the crazy panic guy over there."
Reacting dramatically to a possible danger makes you look childish, and dismissing concerns with an appeal to past experience makes you look like the experienced adult leader. I don't think it's a claim about real probabilities.
I think it's a side effect of stronger resistance to fear (and guilt) arguments which are the de-facto standard way most lobbying groups G (government or other) attempting bulk influence in western democracies since WWII.
The pattern is "you should do Y to avoid the catastrophic X outcome", or "not doing Y cause many to suffer X". In then end, doing Y does not seem to change X much, but profit G while making the life of those doing Y harder.
This pattern is so constant that public resistance to it increases, leading to more and more extreme versions of the pattern and more and more resistance. I see it like an immune system response to fear and guilt propaganda.
In the case of AI, maybe the immune response is unfortunate: the fact those who should do X is not the general public, but very profitable megacorps, makes it more likely to be a genuine risk rather that yet more propaganda...But on the other hand, the megacorps are quite immune to fear and guilt. So expect research to continue as usual anyway :-).
Okay here's my steelman of the argument. Suppose we're arguing about whether resources are going to run out. Someone points out "here are all these cases of people boldly predicting that resources would run out and them not doing it." That would seem a perfectly kosher argument.
Now, in the case of AI, the inductive trend is a bit different. They're arguing something like "it's easy to argue some future thing is dangerous and likely to majorly disrupt the world and to find smart people making long arguments for it, but it's usually not." Now, pointing to one case doesn't show that, but most ways people have predicted the future will be significantly worse have been incorrect.
I think you are missing the heuristics that people are using to make these arguments.
When they point to the coffeepocalypse types doomers being wrong a lot, they have two heuristics that makes AI risks also seem like something they don’t need to worry about:
1. Predictions are really really hard. If someone is worried about something far in the future they are almost always wrong. If you look at what people were worried about 100 years ago about today, 99.9% would be completely irrelevant. This doesn’t mean you can completely ignore them but you should downgrade your concerns.
2. It’s very hard to predict human innovations. If the prediction is actually right and creates a big problem, it leads to a flood of human capital trying to solve the problem. This has solved countless problems in the past and while it is impossible to know ahead of time what the solution will be, it likely someone will find a solution. This isn’t an argument to stop investing in AI but actually argues for the opposite. However, it’s a heuristic why the typical person isn’t worried.
3. Even if the prediction is right, also have timing risks to consider. Even if someone rightfully predicted that AI safety would be a big problem 50 years ago, there very little anyone could do to lower the risk at that point. There a good chance people are worrying about this too early which makes it even less likely that it’s something to worry about right now.
I think the coffee thread would be more fairly summarized as:
AI is like coffee and other past technologies because it is currently facing opposition from powerful institutions ("media") who feel threatened by its potential to disrupt existing power structures and industries. And since the controversy is "really about power," it will play out the same way coffee did.
That's not really an argument. It's an assertion.
Russell's example makes sense to me because it sounds like he's arguing for "Clarke's First Law" which is pretty well known
> When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
I'm still not sure what the coffeepocolypse version is trying to argue for though
Will there be another book review contest in 2025?
Or will GTP 5 have made book reviewers and human writing obsolete?
I think global warming is the stong version of the coffeepocalypse
My understanding is that current costs of warming may be net positive. A Lancet study reported that cold stress deaths have reduced twice as fast as heat stress deaths have increased in the last decade or so
Not only may we be wrong that warming represents the general violation of our Edenic past with all weather getting worse all at once, but the assumption that this is so means we are utterly blind to the costs
It is very possible that corn ethanol subsidies have singlehandedly outweighed all warming costs through starvation in food stressed countries, yet there is so little interest in the possibility that when I attempted to estimate the potential number of starvations I found that there has been essentially nothing since about 2010 that I could find even considering it in the literature
AI may be dangerous, but the assumption that it is probably infinitely dangerous is inherently dangerous itself. It may be right but if we allow the concern to go unexamined then we will experience some kind of corn ethanol subsidy cost times ten without thinking carefully about it. I don't think this invalidates AI risk mitigation but it shows that we need to not only estimate the necessary steps to hedge against AI risk, but also the likelihood that they will have a net expected benefit
A military strike destroying the capability to produce AI enabling GPUs may be necessary to credibly frustrate dangerous AGI - but does it actually have a chance of success commensurate with the guaranteed cost? Should we spend infinite money to prevent 10% more warming because of tail risks or is that actually a poor choice at some margin because we may also be preventing the tail benefit of delaying the next ice age and we just aren't thinking about it?
Riffing on your global warming benefits…
AI, like fossil fuel energy, clearly has huge upside potential for humanity, possibly larger than any prior technology ever! Interfering with AI development has huge potential costs and mistakes and can actually be self destructive, not unlike the environmentalists’ interference with cleaner, safer nuclear energy.
I think what might be going on here is that reporters reported on a real thing and then industry people responded by doing something and then the apocalypse didn't happen and reporters never reported on that part. So people assume that the problem wasn't real, not that someone fixed it.
See: https://www.vox.com/future-perfect/24121450/honeybee-population-extinct-prediction
They aren't implicitly making an argument. They're implicitly sharing a personal narrative.
When I was a kid, elites told me that I should be really worried about overpopulation and role-playing games. And I trusted them, and worried about those things. Then I found out that neither of these things were problems at all. When I was older, elites told me I should be really worried about violent videogames and assault rifles. But it turns out lots of people play videogames and are just fine, and most crimes are committed with handguns, not assault rifles. Then, most recently, I heard this crazy idea from elites that I should be worried about coffee. But this time at least I knew better and ignored it. And sure enough coffee isn't a problem. So when you tell me that elites are worried about AI, I don't buy it. Elites are always saying we should be worried about something but I've learned to ignore them.
The critical thing is that the teller of the narrative is not really distinguishing between different camps of elites; but this is not something so easily done.
Did you listen to the elites when they were warning about Russia? Or a pandemic-level virus occurring about every hundred years?
I do not hold the views I describe in that post. I am explaining the perspective of another.
> I always give the obvious answer: “Okay, but there are other examples of times someone was worried about something, and it did happen, right?
How about limiting it to examples of successful predictions of catastrophe-inducing events, civilization-ending events, human extinction events, or eschatological events? The history of experts predicting these events has been spotty at best. ;-) In my lifetime. I've lived through at least half a dozen EOTWAWKI (End Of The World As We Know It) events that scientific* experts predicted. And I've lived through three of four religiously predicted eschatological events (that I know of)...
..."And I feel fine," to quote the popular song.
If you were to say that AI will have unintended consequences, some of them negative, I'd be more inclined to take you seriously. Heck, I can already see it's distorting our knowledge base with "hallucinated" data. But predicting that coffee, horseless carriages, flying machines, and Lindy Hopping will have negative social consequences is not on the same order as predicting an EOTWAWKI-level event.
*scientific in the rationalist sense of the word — as opposed to religiously motivated doomers.
Exactly
I think Scott is seriously missing the point of that thread and then the straw-manned version can't be steel-manned again.
The actual content is not "once people were worried about coffee, but now we know coffee is safe. Therefore AI will also be safe." Rather it is something like "Once people tried to heavily regulate coffee under various pretexts (and there were some true believers in those pretexts). But really the powerful people who made that into a political question rather than some cult's weird food taboo wanted to regulate it because social change threatened their position. In the long run it didn't work and couldn't have but it hurt the places that tried it. Same with AI now".
The guy doesn't try to engage the true believers on either AI or coffee doomerism, because he takes neither group serious, but that doesn't round to any of Scott's version of badly engaging the true believers.
I use the "here's a specific example of something not happening" argument as an attempt to call to mind what I believe to be a general pattern. I do try to make it clear that I'm making the argument that there is in fact a pattern, e.g. "Remember how in the 50s, leading AI researchers thought that machine learning could be fully solved by a few people in a few months? And 50 years later, thousands of engineers working together finally made a modicum of progress? Yeah I think it's going to go like that. The rate of progress on brand new technologies is always wildly overestimated, often by factors or hundreds or thousands, and I don't see why this would be different."
The first half of that argument is the specific example, and the second half is the attempt to generalize. I also would not be convinced solely by e.g. providing specific counter examples such as the progress on nuclear chain reactions, but I would potentially be convinced by a specific counter example, plus a reason to believe that the provided counter example is more analogous to AI than the "machine learning will be done in a few months" example.
My attempted steel man:
1. Most people can't evaluate the object level arguments for AGI risk. Even attempted brief explanations tend to assume a ton of background reading, and more general math than the general public actually knows.
2. Doomers usually address this by citing experts. Lots of top people in AI have expressed concern, and you don't need a lot of personal understanding to tell that the guys who literally wrote the most popular college textbook on AI know a lot about AI.
3. But such experts don't have a great track record. A real evaluation would be hard (but maybe ACX worthy) so here's some anecdotes.
4. With neither object level arguments we can understand nor experts we can trust, we're left with our priors: most things don't destroy the world.
No one will ever be able to prove the negative, that an AI doom scenario definitely will not happen. The onus remains on the AI doomers to demonstrate their scenarios represent material threats.
IMO, the important part of the linked thread is the tweets about *why* various groups opposed coffee houses, based on political incentives. (eg "Kings and Queens saw them as breeding grounds for revolution")
Many people use a sort of political reasoning about everything - and by that, I don't just mean about red tribe vs. blue tribe, I mean that instead of reasoning about evidence for or against a position, they think it's most important to think and reason about *who* is on each side of an issue, and what their underlying political, economic, or psychological motivations for that position might be. The implication is "here's *why* the tribe arguing AI risk to you isn't trustworthy, because their views of the issue are corrupted in the same way as the coffee skeptics"
This lens is even more obvious when people make the arguments that "AI doomers put intelligent on a pedestal because they are nerds, and fundamentally *want* intelligence to be so important", or "AI doomers are scaring people to hype up their technology, or trying to bring about regulatory capture". Some of these arguments make more sense than others, but they are all fundamentally arguing about who to *trust*, not arguing the merits of the issue itself.
This can be maddening if you want to debate the substance of the actual issue, but I would say that for lots of situations, it's a workable way of reasoning about the world. If you're deciding whether or not to take a new drug, or put your retirement savings in a new investment vehicle being pitched to you...some people with the right educational background and time on their hands can dig into the issue fully and come to a rational conclusion based on the facts, but most people should and do start with "who do I trust about this".
Scott, sometimes you engage in strawmanning on stilts, all the while purporting that you are trying to steelman. I don't think you do this deliberately. I think you may be sincerely trying to steelman. But boy are you really bad at it when you don't sympathize with the argument in question.
The meta-argument here is: Humans have a powerful tendency to catastrophize, not generically, but about observable material trends, among which technological change is a prominent but not exclusive example. This tendency shapes the interpretation of evidence and results in motivated reasoning, especially the motivated construction of rational-sounding models to support the story. Not every concern about observable conditions or trends fits this pattern, but some do. When one observes a intellectual movement that pattern-matches strongly onto this pattern, it is reasonable to take that matching into account in the critical evaluation of its arguments.
The object-level argument would then be: AI Safety pattern-matches this tendency in ways X, Y, and Z.
Whether or not such an object-level argument is valid would need to be worked out on its details. But the attempt to dismiss the consideration at the meta-level is not at all convincing.
If you want to understand the argument, you could do worse than start with the mission statement of the Pessimists Archive, which collects evidence of past moral panics: "Pessimists Archive is a project to jog our collective memories about the hysteria, technophobia and moral panic that often greets new technologies, ideas and trends." https://pessimistsarchive.org/
Many Thanks!
Did you really not just ask the guy making the coffeecalypse arguments about what he meant?
Coffeepocalypse is like stereotyping or triggers. If you're taught to fear black people, your gut impression of the odds a particular black stranger is hostile will overstate the evidence. Almost all humans instinctively fear apocalypse from transformative new things, so you have to discount their fear a certain amount to estimate the true weight of evidence they've seen that a transformative new thing will be apocalyptic. It's not that there's no evidence, but their emotion is a bad guide to the evidence they've seen.
What was early realization of "humans creating potentially catastrophic climate change" in the epistemic environment like? Does it feel similar to the slowly dawning AI X-risk? I suppose the major difference is most normies are inherently vaguely fearful of "AI" whereas they weren't of climate change.
I respectfully disagree with your steel man arguments. My steel man version of the coffee thread:
1) New technologies regularly face backlashes, attacks, and opposition by authority figures and competing interests
2) Improved intelligence is simply too important to be stopped, therefore it won’t be stopped.
I do agree that the author just assumes in the end that it will all be OK. This is a really bad assumption. It is possible that AI will be overwhelmingly positive, that it will be extremely influential but in both negative and positive ways (mixed), that it will not be influential at all, or that it will be catastrophic. That said, a lot of the backlash will be by partisans, competing interests and threatened authorities. And it is inevitable and it will not be stopped. It will at best be partially managed, with the potential that it is mismanaged just as likely as managed. A better example here would be nuclear energy rather than coffee.
I am yet again begging that someone who has more free time (or research assistants/interns) than me make a dataset of supposed tech panics, and include variables like "did the people panicking about X include people who were creating X?" and "were the claims in some way accurate (e.g., did coffeehouses threaten monarchies)?"
https://forum.effectivealtruism.org/posts/6ercwC6JAPFTdKsy6/please-someone-make-a-dataset-of-supposed-cases-of-tech
I think you wrote about this once, but some people are incentivized to get engagement on what they write, not make good arguments. Coffee is a common thing lots of people are familiar with, it seems extremely boring, so it invokes an easy emotional response to say "people once worried about coffee." So they don't even ask questions like the ones you do here--once they've found something that might make a good article/tweet, they just run with it, because it doesn't matter. Probably also combined with standard confirmation bias + signaling tribal allegiance/outgroup disagreement.
(I think it's also the kind of thing that appeals to certain people, specifically nerds, who are likely to be interested in the history of random things and also to have an opinion on AI risk... I wonder if this was the result of someone reading about the history of coffee and then loosely tying it to AI risk for engagement).
Going off at a tangent, I had a vague memory of something to do with coffee and the Church.
Yes, turns out there is an urban legend that coffee, when first introduced into Europe, was regarded as a heathen drink of the infidel and some in the Church wanted it banned, until a pope (allegedly Clement VIII) tried it and liked it:
https://en.wikipedia.org/wiki/Pope_Clement_VIII#Coffee
"Coffee aficionados often claim that the spread of its popularity among Catholics is due to Pope Clement VIII's influence. According to the legend, there was opposition to coffee as "Satan's drink", leading to the pope's advisers asking him to denounce the beverage. However, upon tasting coffee, Pope Clement VIII declared: "Why, this Satan's drink is so delicious that it would be a pity to let the infidels have exclusive use of it. Clement allegedly blessed the bean because it appeared better for the people than alcoholic beverages. The year often cited is 1600. It is not clear whether this is a true story, but it may have been found amusing at the time."
First coffee, then cocaine? I think stimulant enjoyers have much to thank the Church for!
"Vin Mariani was a coca wine and patent medicine created in the 1860s by Angelo Mariani, a French chemist from the island of Corsica. ...Between 1863 and 1868 Mariani started marketing a coca wine called Vin Tonique Mariani (à la Coca du Pérou) which was made from Bordeaux wine and coca leaves.
The ethanol in the wine acted as a solvent and extracted the cocaine from the coca leaves. It originally contained 6 mg of cocaine per fluid ounce of wine (211.2 mg/L), but Vin Mariani that was to be exported contained 7.2 mg per ounce (253.4 mg/L), in order to compete with the higher cocaine content of similar drinks in the United States.
...Pope Leo XIII and later Pope Pius X were both Vin Mariani drinkers. Pope Leo appeared on a poster endorsing the wine and awarded a Vatican gold medal to Mariani for creating it."
https://en.wikipedia.org/wiki/Vin_Mariani#/media/File:Mariani_pope.jpg
>And plenty of prophecies about mass death events have come true (eg Black Plague, WWII, AIDS).
I know Chesterton predicted after WWI that the peace with Germany would be short-lived, but who prophesied the Black Plague or AIDS?
Certainly some people used to prophesy AIDS doom after AIDS was discovered. But they were wrong.
In the end AIDS, like most things, wound up far worse than the most optimistic case and far better than the most pessimistic case.
Well-stated. There were "experts" who predicted that AIDS was being overhyped for political reasons and other "experts" who predicted that AIDS would depopulate the entire continent of Africa. AIDS was (pre-treatment) a serious disease that caused illness and death in many places, but Africa is not depopulated. Perhaps some of the difficulty with the AI-safety debate is that it is always framed as an existential threat to humanity. Even if different threats have to be analyzed on their own terms, it seems reasonable to set a high threshold of evidence and analysis for the most extreme claims. Other examples of failed predictions of catastrophe seem like they might offer some lessons, although I can't deny that logically each threat is different and you can't reach a conclusion based solely on an imperfect analogy.
Which examples or analogies seem compelling seem to depend largely on what reference class people (typically unconsciously, intuitively) place AI risk in.
While the disagreements resulting from this difference are particularly strong and noticeable in the case of AI risk (and for the record, as other commenters have suggested, to me the best reference class seems to be, roughly, "what happens to an ecosystem when a more intelligent species appears in it", which, plus a separate argument for why this is likely to happen within a timeframe we care about, is cause for concern), but I suspect that the problem of finding a good reference class may also be behind many other sources of confusion and/or disagreement.
Here I find myself unsure of how to check whether one's choice of reference class is good, or what sort of things to do to improve it. Presumably someone has written about it at some point, so if anyone has any relevant links I'd appreciate it.
To answer your question, the original use of "outside view" was as a specific counter to the planning fallacy, where people will say things like "I'll write my paper in 5 days or so, 95% worst case, 10 days" and then they take 11 days. However, if you asked them first "how long did it take last time" they'll say something like "oh yeah, 11-20 days" which results in much better predictions.
Now (and this represents a leap on my part) Philip Tetlock says that one skill that superforecasters have is that they have good reference classes to previous similar events, and they use it to inform their base rates before they start doing research on political forecasting.
That's where the facts as I know them end. Here's my opinion:
I don't believe reference classes have been proved to work on long timescales, with abstract connections between classes. It works for the planning fallacy when *you* are trying to predict what you would do in the future, I don't think it would generalize well if you tried to get a "reference class" of paper turn in times. For Tetlock, what's been extensively tested has been political event prediction in the scale of 1-2 years in the future, not arbitrary predictions.
I think most cases of using reference classes is about trying to associate high or low status to certain ideas, and used not at all similar to how reference classes improve predictions in either of the above two examples. Why do I think I know this?
1. People like to use abstract principles to decide reference classes when they do the above, instead of really simple rules like "is there something that is almost exactly the same as now".
2. People treat choice of reference class as the final, or biggest update you could make, and tend not to be open to further argument after reference classes are established. This is not how superforecasters treat them! Any predictive value you might have had from using a reference class is getting wiped out by not changing your mind!
3. Duh, the people trying to use reference classes like to pick things that are extreme on their status implications, and do not actually seem to be interested in constructing some notion of a similarity measure, or seeing how sensitive their choice of reference class is.
All this to say: reference class tennis is bad and I don't think it's a fruitful route of investigation.
Oh yeah, you asked for a link:
For how it was related to planning fallacy:
https://en.m.wikipedia.org/wiki/Reference_class_forecasting
For Tetlock answering your question more directly: (use the transcript)
https://80000hours.org/podcast/episodes/prof-tetlock-predicting-the-future/
Thanks!
I think the for coffee it is an appeal to scale / shock factor that is supposed to be convincing. Look how even something so huge as one of the most popular drinks in the world (coffee, drank by over a billion every day), was thought to be dangerous. It's trying to break down the idea that although the potential impact of AI is so massive, that we should be skeptical of it based on the history of coffee as a similarly large scale but pointless fear.
I don't agree with this logic for similar reasons as you've described but that's my take on what the thinking is of people who make it.
I think there’s general skepticism against certainty of either total doom or total utopia. There’s an association between pundits and general self-promotion with these sorts of predictions. “Market collapse”, “super volcanos”, etc. It seems to me most of these people just want something for themselves, like status, instead of actually trying to save humanity. I don’t trust them.
I believe in AI risk because I’m a student of the topic. But I’m generally skeptical of doom predictions. I can see others doing the same if they don’t know enough about AI.
I might be among the top 1-10M people in the world in terms of understanding AI. That still leaves most people, including very many smart ones.
== Overpopulation / global cooling as proxy for AI Safety: ==
The implication is: “these are obviously as relevantly similar as two things can be”. People making this claim experience a strong intuition that the implication is true, and are surprised that you don't. They're also dismayed that you ask them to go the considerable length of justifying, or even explicating this.
== Coffee: ==
Attention grab, nothing more; unlike the other more serious, intellectually honest (if incompetent) claims.
== Nuclear reactions: ==
This goes to a specific aspect that people might actually be ~100% sure about: a lower bound on how long before something big happens it can no longer seem impossible. For nuclear reactions, people might think a year, a month, a week. You say it's less than a day, you make them think maybe they should also update this lower bounds for AI Safety.
== Russian tanks: ==
At least in this case, there's also the “good optimizer” assumption: if Russia has the resources to make its tanks that much better, it would be stupid not to put some resources into being ok in most other 99 aspects. Doesn't work for AI Safety though.
I think the implicit argument of the coffeepocalypse thing is that people in general tend to overestimate the probability of disaster, and that therefore if people are predicting, say, a 10% possibility of disaster that should be taken as evidence that the actual likelihood is much lower than 10%, like 2 or 3.
In most of his writings that I've found, he focuses less on the claims of existential risk and more on the the smaller cultural battleground of AI harming education, mental health, sundering the social fabric, etc.
In a substack essay he wrote with a similar tone, he lists various technologies that "people thought were going to devastate society or cause mass destruction":
> Television was going to rot our brains. Bicycles would wreck local economies and destroy women's morals. Teddy bears were going to smash young girl's maternal instincts. During the insane Satanic Panic in the 1980s which blamed Dungeons and Dragons and other influences like Heavy Metal music for imaginary murderous Satanic cults rampaging across the USA, people actually went to jail for abuse of children that did not happen. (https://danieljeffries.substack.com/p/effectively-altruistic-techno-panic)
And then he throws in in the supposition that existential risks are a weird fantasy in the same vein.
I think the issues with his line of argument tend to boil down to false equivalency:
- Most of his examples don't have anything to do with existential risk, but he uses these examples to bolster an argument which ends up dismissing existential risk (e.g. I don't think anyone postulated that the invention of bicycles would cause "mass destruction" in the way we understand it, so it seems weird to lump it in the argument)
- It's only in the past century we have the technology to actually wipe out the world (Genghis Khan-tier military tactics are small-scale compared to atomic bombs) so older panics tend to have not had as much existential gravity
He also seems to reflexively dismiss caution toward hypothetical scenarios that we cannot currently fathom. That is, if they did happen, hypothetical problems could probably be fixed by hypothetical solutions, so we shouldn't be scared by the hypothetical problems.
You're misunderstanding the target of these arguments. They're not trying to discredit the AI risk position as a platonic rhetorical object -- they're trying to discredit *the people holding that position*. Essentially, their belief is that anti-AI is a position that has been reached not through rational evaluation of facts, but by fear of the unknown. The argument is then:
1) We now understand that there's no reason to fear coffee (or whatever).
2) However, some people at the time were afraid of coffee, and here are the arguments they used against it.
3) Since these arguments turned out to be laughably false, the people making them couldn't have been acting rationally; they must have been poisoned by fear or some other emotion that got in the way of clear thinking. Maybe they were even being purposely dishonest out of self-interest.
4) You should notice the structure -- not just the substance -- of these badly-motivated arguments so that you're better able to recognize them when you don't have the benefit of hindsight.
5) Anti-AI crusaders use the same argumentative structure as anti-coffee crusaders.
6) Therefore, anti-AI crusaders are probably motivated by fear of the unknown or material self-interest, just like the coffee people we laugh at today.
7) Therefore, we should treat them the same way we treat other ill-motivated people like partisans and cranks -- by not even giving their arguments the time of day.
You're susceptible to your own version of this for the same reason everyone is susceptible to ad hominem: you hate the target and want an easy way to mock and dismiss them without thinking too hard about what they're saying. Ad hominem will always be the most rhetorically effective fallacy.
Russell's argument works because it's implicitly much more specific—it's focusing on "this seems sci-fi"-incredulity. A stronger, more explicit form of it would be either the rock with "nothing interesting happens, ever", or Yudkowsky's examples of people calling the idea of powered flight ridiculous (because it seemed "sci-fi").
Every technology is science fiction until it's invented, at which point it becomes science fact. "That seems sci-fi" is a fully general counterargument against any kind of substantial technological progress, ever.
That they said this or that could not be done (or was science fiction) and yet it was done isn’t a great argument for people always being wrong in negative predictions - that’s a selection bias.
Remember that most things they said couldn’t be done, and that weren’t done, are forgotten.
So, true! Still waiting for my personal jet pack.
This seems like a better example of the opposite: if someone, at some point, said "by year X we'll have Y", people will complain about it forever, even if it was an obviously bad idea from the start.
Examples include jetpacks (congrats on burning your legs off) and flying cars (they're called helicopters, they're way too fuel-hungry to make any sense, and dear God you do not want to give every person a helicopter license the way we do for cars).
Good point. I was being needlessly sarcastic. But flying cars are in the news. And the Chinese may be ahead of the US! — queue outrage and a Congressional inquiry. Just keep 'em out my house's airspace, please!
https://www.ft.com/content/096fef79-df19-4cf4-9502-a4154550eafc
https://www.newyorker.com/magazine/2024/04/22/are-flying-cars-finally-here
As I mentioned in the comments, I see the opposite much more often—people never, ever forget all the things people predicted would be invented that weren't (e.g. "where's my flying car?"), then use it to argue against any future technology that sounds weird (is found in sci-fi movies). This argument historically has an extremely poor track record—science fiction movies also predicted the internet, smartphones, maglev trains, and universal translation software.
The real lesson to take away is that if it can be done (according to the laws of physics; and a human-level computer is clearly possible, because humans are human-level computers) and doing it would be profitable, it will (eventually) be done.
There’s probably thousands of things that have been predicted that never happened.
"Thousands" feels like an underestimate, but I'm not sure how that's relevant.
Reminds me of a volcano story on wikipedia: "Captain Marina Leboffe's barque Orsolina left the harbor with only half of his cargo of sugar loaded, despite shippers' protests and under threat of arrest. Leboffe, a native Neapolitan, reportedly told the port authorities, "I know nothing about Mt. Pelée, but if Vesuvius were looking the way your volcano looks this morning, I'd get out of Naples!" (https://en.wikipedia.org/wiki/1902_eruption_of_Mount_Pel%C3%A9e) All people in town except for three died in the eruption.
I think what your opponents are trying to do is point out the similarities between the false apocalypse predictions and the AI apocalypse predictions. Your job is to point out the similarities between the true apocalypse predictions and the AI apocalypse predictions. Also, the black plague, in the grand scheme of things, was more of a necessary thing than a bad thing (with apologies to the perriwig maker).
AI Will be good for mankind but bad for psychiatrists. Sort of like how the black plague was good for the enlightenment but bad for the church. Your opposition to AI is a manifestation of your innate survival instincts. You are a much better writer than psychiatrist so you should just embrace the change. If you think about it, you can help far more people by your writing than you can with your psychiatry practice. Even if AI does kill off a lot of people, it will probably do so in a very humane way (like persuading them to embrace hedonism in lieu of reproduction, something many western governments already do).
I agree that the coffee/global cooling/whatever arguments aren't very good, but I also think you are sort of fundamentally misunderstanding the work they are doing. You are treating these as arguments by which a person reasoned their way to a conclusion, using illogic. But that's not what people are doing at all. Rather, they have already arrived at a conclusion and they are using analogies to undermine the credibility of opposing viewpoints.
In other words, you are treating the argument as this: "People once incorrectly thought that overpopulation is a problem. Therefore AI is safe."
What people are actually saying is this: "Concerns about the dangers of AI are overblown for a bunch of reasons that I am taking as given. People, including experts, have often in the past incorrectly freaked out about new technologies, and you should be wary of such claims."
I am not surprised that you find this form of argument frustrating, because it doesn't actually engage with the substance of AI safety concerns. But it explicitly isn't designed to. These are statements by people who otherwise aren't convinced about the danger of AI, warning other people not to be seduced by credible-sounding arguments made by people with highly esoteric knowledge.
As others have pointed out, it very much is a sort of anti-expertise argument, but not in the mold of "experts are an evil/degenerate cabal trying to fool you." Rather it's more, "being an expert doesn't automatically make you right, especially in emerging domains of knowledge, as these examples show." And this is actually true! I get why it's annoying to have this point constantly waved in your face by people who aren't actually saying where they disagree with your arguments, but that is what is going on.
That's a great way to explain it. The coffee guy is accusing anti-coffee and anti-AI people of making bad-faith arguments about safety.
Scott, sometimes the other side is just stupid and wrong, and you need to squish them instead of persuading them, because they have a vested financial interest in NOT being persuaded.
It would be nice if maybe YOU could update YOUR priors along the lines of "85% of the population mean well and can be persuaded by reason and logic, but 15% of the population is just evil, stupid, or insane."
You're right. You should perhaps be more frank in your view that the Coffeepocalypse argument is idiotic. And this comes from a guy very skeptical about the AI catastrophe.
Even if he is idiotic, we can still use his dumb corpse to improve our lives. No part of the idiot wasted!
In this case, Scott sees an obvious reasoning error out in nature. He sees that it's by a person who could potentially, but by the Grace of God, could have been him, and since the person has not noticed the obvious reasoning error, he wonders if there's a similar anti memetic mental block that he has internally, hence trying to test the justifications in the nuclear bomb case.
And to point out, a repeated claim made across forums and comments sections is that AI riskers "have blind spots", so Scott is likely on guard for sightings of these supposed blind spots. I don't personally think this blind spot narrative is made in good faith, but also I'm way less modest and less nice than Scott, and you can't get Scott thoughtfully trying to analyze other people's viewpoints unless "am I also making this mistake too" is a reflexive thought.
I don't think you're giving a complete summary of the argument in the linked thread. I would call it a form of bulverism, since they're describing possible motivations for bad faith AI risk arguments. Similarly to how the beer lobby spread unfounded concerns about coffee to protect its market share, groups today could be spreading unfounded concerns about AI.
I think that's true. There's a debate between Ben Shapiro and Tucker Carlson that touches on AI driven trucks. Tucker said that he'd prefer that the government make up some safety risks if the alternative was the sudden destruction of trucking industry employment. Even if AI had no risks at all, there would be incentives to regulate it citing false risks.
As the term "bulverism" describes, this is all removed from whether AI is dangerous. They're not arguing that AI is safe, they are describing some bad faith reasons people would say its dangerous.
This made me laugh because I use the exact same thought experiment. "Whenever I wonder how anyone can be so stupid, I start by asking if I myself am exactly this stupid in some other situation." It is so hard to be aware of one's own blind spots that any strategy that allows one to look in a convex mirror around the corner is valuable.
We are talking about prediction. One hopes to have confidence in the accuracy of prediction. Yet with multiple variables and incomplete knowledge of all variables and forces at play, outcomes are generally uncertain. I could be hit by a truck this afternoon or the killer bees could get me like they got D. O_ in our town.
People fear bad outcomes. They desire control over outcomes. They desire relief from the anxiety about bad outcomes. For many people, a comforting story is all it takes, perhaps merely as distraction.
We do our best, and try to monitor outcomes in progress so quick course corrections are possible.
As a curious atheist, I like the attitude of the Old Order Mennonites, who pray not for outcomes, but for the grace to accept God's inscrutable will.
> The people I’m arguing with always seem so surprised by this response, as if I’m committing some sort of betrayal by destroying their beautiful argument.
This is actually a pivotal issue in explaining AI safety. People don't like being utterly decimated by another person, especially not when talking about objective reality.
However, if it was that simple, people probably would have figured out workarounds during the 20th century or earlier. There's another layer: backchaining to justifications, as described by Yudkowsky 15 years ago: https://www.lesswrong.com/posts/TGux5Fhcd7GmTfNGC/is-that-your-true-rejection
I had a theory. In ground truth, over about a year there are now about 1000 AI pause advocates. Meanwhile Nvidia sold out a stadium and the industry announced, in just the last few weeks, plans to dump 650 billion! more into AI over the next 2-3 years.
There are obviously millions of new AI fans added monthly. For right now, where errors from AI are mostly harmless while the benefits increase monthly, doomers lost. There's nothing to discuss, no reason to engage with them. Hence comparing their position to people fearing coffee.
It's a dismissal, in the same way nobody bothers anymore to convince Amish to use technology. There is nothing to discuss and the evidence speaks for itself which side is right.
You already got it, it is the heuristic thing. See also, the Ancient Greeks complained about youth culture, so all complaints about youth culture are spurious.
I am pretty sure you have written on this before, with, birds-of-a-feather, and opposites attract. People will uncritically use little aphorisms to support their points even while equally believable totally contradictory aphorisms exist.
I have not commented on a SSC post in about ten years, so forgive me if my language is antiquated, but I said my objection to this post on the 'scord and was told it was wise and I should post it here.
I think that, in general, the most parsimonious explanation for a disproof of a seemingly nonsensical claim is that somebody is actually making the claim. For some reason, you seem not to have not considered this, despite it being the most obvious and straightforward explanation.
For example: even limited to my own browsing history, if I type "guardian.com ai" or "futurism.com ai" into my address bar, I will be confronted with about ten separate articles that claim AI will "do harms", "perpetuate harms", "pose risks", et cetera -- and that AI is in some way associated with pedophiles, scammers, Nazis, bros, et cetera.
Most of the time, there is not a real coherent argument being advanced in these articles -- it's just a random incident of a dumb or malicious act being somehow connected to AI; sometimes they quote a Rent-an-Expert who talks vaguely about "real harms". The general idea, typically made by implicature but sometimes made explicitly, is that AI is a thing which exists, and it didn't exist before, and it will clearly change the world in some way, and there are some harms, so it is bad, and should be (considered shameful / heavily regulated / made illegal / etc). If you'd like, I can provide a couple dozen articles and tweets and blog posts that articulate this vague point.
I think this is a pretty dumb argument to make, but people really do make it, hundreds of times, every day, and I think the most obvious explanation for the coffee disproof is that it is a response to this.
Good point!
My assumption about what's going on here is that someone found some interesting historical facts about the social reception of coffee, and found that a good way to make their interesting historical facts go viral is to tie it to something that people are already predisposed to argue about. The tie-in to AI apocalypse is weak, but primes it for virality, and then when you put actually fun facts on the inside (even if they're not well-connected), it succeeds in going viral.
Prediction - if you could somehow do the same thing with cute pictures of capybaras rather than actually fun facts about the history of coffee, it would go equally viral.
Alarmist arguments about human extinction or massive disaster have a long pedigree. For example, just before the LHC was turned on, doomers claimed that it'll destroy the world through micro black holes or whatever. Given that, all those predictions failed to come to pass means that the chance of some new technology will cause human extinction must be very small. I don't know why AI doomsters like you obtusely fail to see this simple point.
I'll give a somewhat parallel scenario from the history of science. One reason the first law of thermodynamics was accepted as true was the long history of failure of every perpetual motion machine. After this uninterrupted series of failures many science academies stopped considering any claim of perpetual motion. When physicists wondered if the absence of perpetual motion machines may be a natural law they birthed the science of thermodynamics.
To take another example, people tried for 2000 years to prove the parallel postulate of Euclid. All those attempts came to naught. Then finally in the 19th century, mathematicians accepted the futility of such attempts and developed non-euclidean geometry.
Similarly you should recognize the failed past predictions of doom and accept the fact that p(doom) must be very low.
> Similarly you should recognize the failed past predictions of doom and accept the fact that p(doom) must be very low.
This isn't true for nonstationary time series (i.e. if there's a reason why the probability of an event changes systematically over time). A good example would be anything affected by technological improvement. If I predicted a 99% chance of an extreme recession by noting that GDP has only been over $70,000 in one out of the past 100 years, this would clearly be silly. GDP is trending upwards over time and I can't assume past observations about the distribution of GDP say much of anything about GDP tomorrow.
The probability of human extinction has clearly been trending up over time as we've developed technologically. Prior to 1945, we didn't have any technology that could conceivably end the human race. Then we developed nuclear weapons. Over the next 30 years, we developed biological weapons that could also do the trick. We invented climate change by accident too. That doesn't look like it'll be an extinction-level threat, but mostly because we lucked out w.r.t. the amount of methane trapped in Arctic permafrost.
Many Thanks! I was about to make the
>Then we developed nuclear weapons.
point too.
The more general case is the "People were wrong in the past, therefore you're wrong now" argument. It's probably about 40% of internet arguments.
It's actually a good argument though, in the sense that it really does change people's minds. (It's not a good argument in the sense that it's more likely to change their minds towards truth than towards falsehood; it's not an asymmetric weapon.)
It can be very effective against an irrational person suffering an irrational fear, which is why people reach for it by default. If your kid is worried about monsters coming in the night, you can point out all the previous times that they've been worried about the exact same thing and that it hasn't happened. Or an adult hypochondriac worried that their itchy forehead means they have cancer will inevitably be reminded of the hundreds of other tiny symptoms that they've previously worried were cancer which turned out not to be cancer. Heck, I've used this argument against myself when I've had irrational worries.
The kings who were scared of coffee were obviously correct. Coffee-houses were breeding-grounds for revolution, they became gathering places for radicals pretty much immediately. Caffeine helps people think and one of the things they thought about was "how to overthrow the king".
I think this guy is probably exaggerating the degree to which there was a moral panic about coffee but, also, people at the time were completely justified in worrying about the social implications of a powerful new stimulant that everyone was going to drink all the time.
What about tobacco? There was no moral panic about tobacco. Europeans seem to have accepted it pretty quickly as a cool new foreign habit. But, tobacco turns out to have secret qualities that poison and kill a huge percentage of the people who consume it. The introduction of tobacco was actually incredibly dangerous, and we would be better off if there had been a huge clampdown on tobacco the second it was introduced.
On the other other hand, there was an early panic about the potato. People suspected it was poisonous (which it low key kind of is under some circumstances) and spent a couple of centuries refusing to cultivate it on a large scale. But the potato is actually a fantastic calorie-dense crop and not dangerous at all, so probably the unwarranted fear of the potato was incredibly bad and resulted in mass peasant starvation.
Are there any early modern examples of moral panics getting it right? Like did they discover crystal meth in 1600 and say "yeah, we're definitely not using this"? I don't know.
The lesson seems to be that there's nothing to learn from any of these incidents about whether fear of new technologies is inherently good or inherently bad. You probably just have to look at each thing on a case-by-case basis.
In terms of your model of what this guy is saying, though - he's making an argument that doesn't stand up because it feels right to him and it justifies something he wants to believe. That's 99.9% of all human political activity. Personally I find it difficult to model 99.9% of all human political activity, so I guess I can sympathise, but I'm also not sure there's much point agonising over it.
It seems very unlikely that coffeehouses were causally critical for revolutions. I doubt anyone here wants to claim that Europe would still be under absolute monarchies today if not for caffeine.
Of course, it’s not clear that people fretting about coffee at the time really expected it to lead to revolution either. The questions of 1) exactly what concerns people had, and 2) whether they were justified, are under-evidenced both in the original post and these comments.
>The questions of 1) exactly what concerns people had, and 2) whether they were justified, are under-evidenced both in the original post and these comments.
Very true! I would prefer to see more information on exactly what the concerns were too. Amongst other considerations, I've seen comments which assume the concerns the original post was addressing were existential ones and other comments which assume the concerns are nearer-term issues such as job losses. I do _not_ know which the author of the original post actually intended to address.
Great comment!
Re
>Are there any early modern examples of moral panics getting it right?
(hides) I asked ChatGPT. It gave examples (albeit not _early_ modern ones) of the bans on leaded gasoline and on chlorofluorocarbons.
But I _also_ asked it about the definition of a moral panic, and one of the criteria it cited is _disproportionate_ alarm. If that is taken as necessary, then any case of taking prudent action against a real hazard get excluded, probably including the two bans it cited... The definition would exclude "getting it right"...
Steelman: "the fact that there exist people who are very certain AI will kill us is at best weak evidence that AI is dangerous, because there have been people who think X is dangerous for every X in history, so I shouldn't update on the fact that you hold this opinion."
(this is not an argument against the *content* of any argument you make for AI being dangerous, but such arguments can be very hard to evaluate)
Well there's the obvious problem of arguing against multitudes. But trying to break it down:
1: You think that AI poses an existential risk but you are wrong and it doesn't. I know that the possibility that you might be wrong must seem very implausible to you, but it is actually very plausible. As proof of this please consider this example of people being wrong.
2: You are saying that AI poses an existential risk, but you are arguing using reason and rationality which never works. You should argue using analogies instead. And if you do use analogies, like the one comparing AI to aliens or something then you're using the wrong analogies. To be safe you should stick to analogies about coffee.
3: You are a prophet of doom. Prophets of doom have a very bad track record. People who predict doom are usually wrong, and when doom happens it is rarely predicted except in the vaguest of terms.
4: You are not my people and I need to signal to my tribe by dunking on you.
5: Roll to disbelieve. I know it may seem like I'm abdicating my duty to fairly interrogate your arguments so here are some examples where I would have been right to do so.
6: People like you are so stupid, just like these other people who were also wrong! I just can't get over how wrongity wrong WRONG you are!
7: I think your arguments are suggestive and it makes me uneasy but I don't feel like I have the capacity to properly evaluate the evidence or your arguments, and I don't think you do either. Until we have better evidence in hand it is important to avoid causing a panic. Here are some examples of people overreacting to things that turned out to be false or overblown.
I think this is just rhetoric and they're not being epistemically rigorous. The idea is this line of reasoning feels similar to this line of reasoning and was wrong so this line of reasoning is likely wrong. This is rhetorically effective but should probably be discarded, although I admit I have made these arguments in the past I think I was doing the equivalent of an emotional flourish to the underlying argument.
I think that the reason you can't read any effective logic into it is that there is no logic to it. It's better to understand the argument as an appeal to emotion via pattern-matching.
It's not meant for people who are trying to think through things. It's meant for people who are already convinced they've thought things through enough, and it's meant to evoke a sneer. The implication is that there's a specific type of person who is always on the side of panicking about the future and they flip out over things we now all drink with breakfast.
Seen from that light it doesn't need to have actual logic on its side. It's meant only to evoke emotion by conjuring a specter of a similar situation, with the implication that if we'd sneer at one we should sneer at the other.
A lot of argumentation follows this pattern, I'd think. There are vast classes of argumentation out there that appear to be arguments on the surface but they have no actual logic to them; their sole purpose is to pattern-match with people's pre-existing biases to conjur up certain emotions. It's raw pathos, except in a more subtle form--the pathos is triggered by the implications of the message rather than any overt emotion-evoking behavior on part of a speaker.
I think the conclusion that there is no "effective logic" at work is generally accurate, but I'm more sympathetic to the potential value of pattern-matching and don't think it's necessarily "meant only to evoke emotion." Think about the test questions that would present a series of digits and ask what the next digit should or is likely to be: 001 002 003 004 005 00_ For reasons I won't get into, I spent a lot of time with mathematicians who railed against those kinds of problems. Presented only with a series of digits with no other assumptions or instructions, there is no logical or mathematical reason the next digit has to be 6. It could be anything - even starting another pattern! (001 002 003 004 005 001 002 003 004 etc.) Still, those kinds of questions survived the standard test item analysis, which suggests there is or at least could be some value in asking people to pattern-match against what a test-giver is likely to expect the next digit to be, even if it's not mathematical or logical. For claims of existential risk, it might be worth re-framing the questions: "Is there any value at all in considering previous failed predictions of existential doom, and if so what?" Many especially-logical or mathematical people would answer "no" or "very likely not." That could be true, but pattern-matching sometimes has value, and it's at least worth pondering whether historical failed predictions of existential doom could inform the analysis.
I think the point of the argument is to argue against a background assumption that people's worries about new technologies are reasonable.
Suppose you thought that anytime someone seriously doubted a new technology, they were basically reasonable.
Then someone told you that many people doubted the goodness of coffee as a new technology.
You then might update quite a bit! You might think "Huh, not only can lots of worried people be mistaken about new technologies, but they can be saying things *totally unconnected* to reality. I guess I will not care that much merely *that* many people are saying negative things about a technology, as it turns out large groups of people can be way off-base. I should update to using my inside view here to weight their reasonableness."
And perhaps the next step for many might be "And of course, I have not considered the arguments and haven't formed much of an inside view, so I guess I am not going to personally do much about the situation, given that I don't personally have an inside view telling me to do so, and other people cannot be really trusted to do my reasoning for me."
I think coming from a lot of people, you are too quick to dismiss this as not just an anti doomsday heuristic.
Global cooling and overpopulation are(were?) presented in more apocalyptic terms then just here is the percent of ppl who will die. Not everyone breaks things down as you do. They FEEL apocalyptic in a way that AIDS and even a big war do not.
And on the scale of apocalypses ppl have a heuristic of what can cause major distruction. War disease and asteroids really. If you are arguing for a path outside of that, they reject it and cite the previous failures (outside the three exceptions)
Having said thats I think sometimes its a wierd sort of i am right about things and other ppl are wrong heuristic. And i have actually heard smart ppl say it in fish like scenarios. These other ppl were wrong once, so you are also wrong and i am right. When challenged on that logic though they admit its not central to their argument.
Trying harder to defend the argument, perhaps we can think of it as a "Baby's First Argument Against Argument From Authority". It's not intended for people like Scott who are already well versed in the arguments either way, it's intended for Sally Q. iPhone who just seen her first ever TikTok video about "some people say AI will kill us all" and is worried.
The argument "Yeah but some people say that sort of thing all the time" will calm her down and bring her back to a sort of epistemic neutrality appropriate for people who aren't experts and are unlikely to become experts.
I learned early in med school to never say never, and to never say always. Cuz when dealing with biologic systems, best to remain alive to the possibility of exceptions.
But here, we are talking about machine/math systems. Might a product of such systems be susceptible to absolutes? And if so, which end of that dichotomy should/would it turn out to be?
So this is the fool’s errand part of all of this: we can’t even be certain if we can be certain. Let alone have any confidence whatsoever in whatever prediction we might have on the outcome, represented in percentages that bespeaks only having partial certainty , on an answer that will actually be definitively categorical.
I don’t even understand how you try to be “less wrong” when you have absolutely nothing upon which to derive a prior probability…and no data with which to modify said priors. That’s not to say an absence of evidence means we are absolutely in the clear. But it seems anything other than an “I don’t know” represents hubris without foundation.
Re your last paragraph: “I don’t know” has many possible meanings, but reasoning from no priors has a correct answer. The correct probability to assign the answer to “are Bloxors Greeblic?” should be 50%. The idea is that saying 50% isn’t hubris without foundation, it’s by definition what it means in Bayesian terms to say “I don’t know” as the answer to this question or more accurately “I have no information by which to privilege any given position”. What this means is that if your AI annihilation estimates differ from 50%, then you certainly must have information you are using to privilege that position one way or another, even if you deny that you do.
I agree. 50% makes sense. A coin flip would be very apropos and seems of no worse value as a predictor as anything or anyone else. Heads, we’re doomed; tails, we’re golden.
Right, which is not helpful and from an action-oriented standpoint predisposes us to action (which is why I refer to such situations as Pascal wagers/muggings - this is very serious if true and we just said 50% true so we must do something about it).
It's why people are trying to find the right comparators, to adjust their Bayesian probabilities to something more accurate. If experts are sometimes very wrong and also predict doomsday fairly often (and are always wrong when they do so) then that lowers our probability of doom in this case.
Specifically, this line of argument is an attempt to reframe the question entirely. Instead of making adjustments up and down from 50%, to frame the question as one of very low starting likelihood. If we start at 0.001% chance of something happening (say, the end of all human life), then arguments that raise the chance might take it up to 0.005 or whatever - five times more likely but still almost certainly untrue.
Starting all conversations at 50% is actually insane in terms of actions we should take. It would cause us to recommend significant resources be spent on anything that has a high potential payoff or downfall, for instance anything that might result in the end of the world/humanity. But most ideas don't work out as originally thought, most businesses fail, and so far every doomsday prediction has been wrong (and there have been a LOT of them). Asking that we treat any new doomsday prediction as anywhere close to 50% possible is huge ask and implies that society should likely be spending almost all of our resources preventing doomsday scenarios. If we happen to agree with a particular doomsday that seems right, but the other thousands of doomsdays don't seem worth it and we literally cannot afford to fight all of them. Even non-world-ending problems are far too numerous for us to spend significant resources fighting. There's a worker shortage right now, and an inflation concern, or concern about rising crime, and potential war with Iran or Russia or China, or a bunch of other problems that some people consider overwhelming and others do not. Selecting AI doom out of that list makes sense to some people, but it can't be society's focus unless we're sure it's true, because all of those other things exist and potentially need solved as well.
I think when you frame a question as having a low starting likelihood, you’ve implicitly already done the work in the background of moving your prior from 50% to something else. Otherwise, I could flip the question to: do you think AI will not destroy the world? If you agree you are “priorless” about this question, then by applying consistency, we should give it a low likelihood of 0.001 too!
What you’re getting at though, is that it seems like this 50% idea conflicts with common sense ideas about when to take action. But I think you are mistaken. It is not the case that 50% implies one must take particular actions. For example, if I am truly priorless about whether AI will destroy the world, bring it into utopia, replace the color green with splorange, or create exactly one party hat, it is unclear to me what action I should be taking. Further, you correctly recognize that we must weigh AI risk against the opportunity cost of not pursuing action in other areas too. The idea is if you hear your brain saying “take a particular action!” It means somewhere deep down, you might not actually be priorless between all those various possibilities at all.
I do think this makes the concept of 50% confusing, but this is probably where Bayesian logic differs from more common-sense ideas of probability and I start to lose a grasp of it. Not my area of expertise unfortunately :(
The people espousing AI doom are not disinterested neutrals pondering the meaning of a 50% probability, but are instead actively campaigning for more funds and effort put towards their preferred causes.
Any cause that uses "the world will end" as the result of not taking them seriously enough is trying to use the strength of the negative to overwhelm our cost-benefit analysis and agree to their requests/demands. If you multiply by infinity you will always get infinity. Starting at 50% pretty much means that no matter how much you adjust the final result, you're going to end up with a non-zero non-minimal result. Multiply a non-zero result by infinity and you get infinity, which says we *must* act to prevent it. If the final chance is low enough, people will hedge on infinity and may not worry about it, but even 5% or 1% final chance feels like too much to ignore if the end of the world is on the line.
I don't think the chance of AI doom is zero, but it's low enough that I treat it like zero for the same reason I treat asteroids and alien invasions as zero - they're low enough that many competing theories, if taken seriously, would bankrupt us trying to prevent them. We would constantly live our lives in fear with the most likely scenario that we end up wasting that effort.
I think there's a subtlety you didn't cover (not that it makes the argument much better). The argument isn't for people who know about AI, or even computers - it is for laypeople who cannot assess AI risk directly for themselves at all, and so are wholly dependent on what experts in the field tell them. This argument provides an example of when experts were all whipped up for nothing, suggesting that experts now could be all whipped up for nothing.
Admittedly, this angle doesn't really make the argument any more logically coherent. But it does give it a bit more effectiveness as a cover to illogically ignore a problem some people don't understand and don't want to think about.
I think the problem is that I have yet to hear an expert actually give a cohesive, logical, or rational actual explanation of the dangers. Yes, there is a real threat (let’s call it a reality at this point) that a lot of jobs are going to be replaced by AI, and that is going to cause economic disruption. That cat is out of the bag already, there is no risk because it is already done.
The only other argument I have heard is some vague hand-wavy notion of killer nano bots somehow being created from AI.
I don’t find either argument, exceptionally rational, and certainly not to the point where we want to try to stop technological innovation (good luck), or to where we would want government intervention/protectionism for the current corporations that happen to have a lead.
This is the gene editing hysteria of the 90s all over again … “someone is going to make a mutant strain of Ebola in their garage with this new fangled CRISPR technology!!!”. (and if you really want to stretch that analogy to its breaking point, you would say that private company is used it to cure diseases, and the government used it to create Covid19 and kill millions) /shrug
I'm no expert, but as I understand the argument, it goes like this: If we build AI systems that are more intelligent than us which have 'general intelligence' (as opposed to intelligence in some narrow domain like a calculator), we will not understand them, and therefore will not be able to control them. They may act in accordance with their own goals, which are likely to be opposed to our goals. We will not be able to stop them, because they will be smarter than us.
The weak link there is 'they may act according to their own goals' - why would they? I think the answer is, we don't know for sure, but we have to admit the possibility because humans do. A human is a physical system that has general intelligence. That intelligence arose by natural selection, which has quite specific goals to its programing (survive, be fruitful and multiply). We see that humans quite often, cast those goals aside and pursue their own ends. So our N = 1 example of generally intelligent systems shows us that they cast off their programing and do what they like.
On the gene editing thing though, what makes you think we are out of the woods there?
> If we build AI systems that are more intelligent than us which have 'general intelligence' (as opposed to intelligence in some narrow domain like a calculator), we will not understand them, and therefore will not be able to control them.
To be more specific, it can be not generally intelligent if it is narrowly intelligent in enough impactful fields. The trivial example is that general intelligence includes the ability to become a gold medalist in gymnastics, a world class musician or a religious founder. However you probably don't require those things to "take over the world". At the other extreme, a lack of charisma doesn't matter if you kill everyone instantly. You probably still need theory of mind to be able to figure out "maybe I shouldn't tell everyone what I'm doing". And in theory, a narrow AI becoming good at "look at own mind and look at problem, modify the mind to be able to solve the problem", means that you may get lots of capability gains "for free".
Why is intelligence scary? Because intelligence isn't book smarts. It's everything your brain does. A dictator did not come into power because they are the hive queen, guns were not dug up from the ground like tubers and the modern financial system certainly isn't the result of an evolutionarily programmed money instinct.
Right now, even within the narrow band human intelligence, you have teenagers taking over large swathes of infrastructure to help fund their Minecraft server racket, smaller teams outcompeting bigger ones and some dude thinking about trains creating an entire physical theory framework out of nothing.
There's not really a "canonical model" of superintelligence, but a definitely wrong model is "one smart guy". "A billion unsleeping, relentless Einsteins, a billion unsleeping Elon Musks, a billion unsleeping Jesus Christs, all working together without friction or in-fighting" is also wrong, but much less wrong. If your brain is providing plans that you can see through, it is making the same error as a chess novice trying to fool's mate Magnus Carlsen and thinking "well, what if he misses this one trick then I win".
> The weak link there is 'they may act according to their own goals' - why would they?
See: https://arxiv.org/pdf/1803.03453v1 page 8, playing dumb on the test. This is a super simple, RL based system. The person didn't "program a survival goal" into it, the organisms definitely do not have theory of mind, yet they behaved as if they had those! I think that's pretty strong evidence that "have a goal in mind while programming AI" is nowhere near sufficient to prevent some other, unintended goal from becoming embedded. If you disagree, can you point out a specific technical result why this problem would not generalize? I'm asking for a technical result because the previous responses to this question seem to assume "human friendly by default" without acknowledging that the world doesn't seem to be consistent with that view. I'd be receptive to verbal arguments, if I didn't think they had a dismal track record.
If people are doing argument by analogy, they at least need to use close analogies.
In the nineteenth century there was a case of superior intelligence afflicting large groups of people: the colonisation of Africa.
But it seems to me that what mattered was mastery of logistics, operations, and tactics: fire and movement. Not intelligence in and of itself. Chemistry, logistics, and kinetics.
(Slightly prior to that various Bantu peoples had swept through Africa, uh, subjugating, the incumbents in an arc from West Africa to East Africa and then down to South Africa. Better operational organisation.)
There were cases of superior intelligence (and firepower) not making much difference: Europe vs the Ottoman Empire, and Afghanistan.
These analogies suggest everything will be terrible/will be fine/might go either way.
<i>If people are doing argument by analogy, they at least need to use close analogies.</i>
On a bit of a tangent, I've often had similar thoughts about the war in Ukraine: which course of action the West should take depends largely on whether you view the conflict as more analogous to WW2 (Putin is like Hitler, he will never stop trying to conquer new land, therefore we need to take him down now before he gets too powerful) or WW1 (this is a distant conflict in Eastern Europe that doesn't really concern us, and we should stay away for fear of escalating a local war into a civilisation-ending catastrophe).
I think most people do a lot of their reasoning by analogy, or "heuristics" as you put it. It can work well, as long as you don't stop at the first one that you come up with. Ideally you think of several that match, and then several more that don't, and you compare them and see where they differ, and what the significant factors were, and then look at the actual thing you want to examine, and see how they apply. And then you think rationally about it, because all of this is a way to provide data, not an actual analysis of data. (One of the problems with trying to think rationally from first principles is knowing what fundamentals are important. This bypasses that step, and gets you a lot of useful fundamentals to start with.) I do this a lot; if there were a way to look back at my comments, you'd probably see a bunch of it, where I reference various things, flit from analogy to analogy, and mix metaphors beyond their breaking point.
This approach to reasoning breaks down when people only come up with one analogy, or only come up with analogies that support the side they favor, or don't actually think about it rationally afterward.
You can see a similar effect with, say, some types of prejudice. Here is an example of a type X person committing a crime. Here is another type X person. Clearly, they will commit a crime! It's taking the first association that comes to mind, and running with it.
One thing coffiepocylipse arguments seem to have is the barest mention of AI.
Which is strange. They don't look at AI and compare it to coffee. They don't seem to analyse specific anti AI arguments and compare them to anti coffee arguments.
Do a ctrl R for the word "AI" and often you couldn't tell what they were arguing about.
Is it that their model of debate says the winner is whoever claims more valid facts? Like they are greatly prioritizing "Is it true?" over "is it relevant to AI?" and just spouting a bunch of facts that they think have any non-zero bayes factor, however small.
If you are imagining an opponent who never concedes any point, that opponent would have to say "no one was ever worrying about coffee destroying the world" or "coffee did destroy the world". Never "so what?"
Some people think that being contrarian makes them look smart, by contrast to the gullible masses or "sheeple" as they probably call them. So if everyone seems to be panicking about AI doom round the corner, then the mavericks' natural inclination is to scoff at the risks and deny them, even if their justification (if any) is fallacious.
I think the core idea behind this type of sentiment is a general disbelief in deductive logical reasoning when it comes to making complicated predictions.
There are a lot of things working against it. People generally tend to optimize for selecting socially advantageous beliefs over correct ones. This is a strong subconscious pressure, and forming social groups based on shared beliefs and principles makes it stronger. It's fairly easy to beat people in arguments when you are more intelligent and oratorically eloquent than them, even when the viewpoint you are arguing is less accurate than theirs. Furthermore, only a handful of historical geniuses have ever been able to make unique testable predictions about the world from just philosophical principles that held up to empirical observation and experimental scrutiny, and it's reasonable to have a strong prior in the direction of not believing people when they implicitly carry themselves as if they are once-in-a-century geniuses. You could obviously object to this and say that people use deductive reasoning to make complicated predictions all of the time, but most people have a certain scale/complicatedness cutoff point, and it usually lies way below making pessimistic predictions about superintelligence.
I personally find this way of thinking to be dispositionally annoying. However, I come from a family of some people who throttle a sweet spot between normalcy and intelligence; they have the intelligence to where they could participate in cool subcultures and intellectually screen weird beliefs if they wanted to, but choose to be 'normal' instead. And so if I were to ask my mother, or my grandmother before she passed, why they weren't scared of the existential dangers of superintelligence, and pressed them on it enough, I imagine that they would say some non-me-worded version of the above paragraph.
The sweet spot you mention is commonly referred to as “midwit”
Right, which is a status game to punch sideways or down at people in order to make your own position seem more accurate and higher status. It's the same thing bullies do to feel more important - insult others.
Meta-reply: And I just did the same thing to you and anyone else reading this - implying that the people who use midwit are insulting bullies. Even though you didn't use the term, it degrades your social standing if you identify with it or endorse it, and puts future people on notice that they would face the same opprobrium if they did it.
Many people are indeed very, very bad at constructing reasoned arguments, so I absolutely agree that "coffeepocalypse didn't happen" arguments are out there in large numbers.
They don't understand that this is only one part of this rhetorical tactic, the "what" that didn't happen. Their next sentence should be the "why" - and hopefully the why for coffee or whatever matches up to the why for AI.
It's unfortunate because in order to steelman this kind of argument you have to invent out of whole cloth their next line of reasoning. For example with the "perpetual motion will never happen" argument, the next step should be "because it goes against the 2nd law of thermodynamics". This one in particular won't work for AI because there's no physical law preventing consciousness emerging from microprocessors. Similar case for the "controlled nuclear fission is impossible" - why? Is there some reason proscribing such a thing?
So you are correct, the arguments you are seeing don't make sense because the people making such arguments do not even realize they aren't making an argument at all.
As for why the AI apocalypse won't occur the reason is simple: AI doesn't exist.
I take the argument as, "Negativity bias makes us think the world is always ending. Here are some examples where we freaked out over nothing.... But the fact that we survived all those things suggests that the world is a lot more resilient than we think."
I think I can answer the question, "What is this coffeepocalypse guy doing?" more generously and effectively than you have here: He's trying to reframe the debate.
He's specifically not engaging with the question, "Could AI be dangerous?" He is suggesting that it is a waste of time to think about that question. He suggests that everyone engaged in the AI safety debate is, consciously or unconsciously, engaged in status games; therefore it is better to simply refuse to engage in any such debate.
He doesn't really support any of these claims. I think he just does existence proof that it is possible to play status games and to engage in motivated reasoning in the course of a status competition. Nonetheless, I think exhortations to reframe can in general be quite useful and good, because they present alternative viewpoints outside the current framing of the debate.
In this particular case, he is clearly very unaware that these possibilities have been chewed over by AI safety people. It's an ignorant argument. But I think the form of the argument is fine.
Coffeepocalypse is a sound meta-argument: "the fact that lots of apparently smart people are predicting AI doom is not in itself evidence of very much, because lots of smart people have predicted doom many times before and consistently been wrong". That is, it's a caution against appeal to authority. It's a non sequitur in response to anyone who is already engaging with AI doom arguments on the object level.
Isn’t this just reasoning by analogy? And the stronger the isomorphisms are between the analogies the more we update. If someone compares the invention of coffee to nuclear fission, that’s not very isomorphic. But if someone compares fission to fusion, that’s highly isomorphic. At the extreme end, if someone makes a claim about red nuclear warheads then it would cause us to update strongly about blue nuclear warheads
It's unsurprising you failed to explain how the author got from the coffee analogy to the "AI is safe and useful" conclusion, because *it's not the conclusion*. The thread author didn't try to explain whether or not AI will be safe. The author obviously *assumes* it will be safe and useful, and explains what he believes to be the reason why people still fight over it, by way of the coffee analogy. Even if his analogy was mortally flawed, it would, within the author's framework, at most require a new explanation (something other than power struggle), or even just a better analogy.
Please note: This comment does not imply that I share any of the author's opinions or their opposite.
Thank you! I was surprised that this didn't come up in the summary of the argument. He is explaining why people may say that a safe technology is dangerous. He's accusing AI regulators as operating in bad faith, like when a beer company warns against the dangers of drinking coffee instead.
I've got recorded in my calendar one of those "predictions to check back on". This one is from Slashdot on April 29, 2004:
> Diamond Age Approaching?
>
> Posted by CmdrTaco on Thursday April 29, 2004, @01:55PM from the i-friggin-hope-so dept.
>
> CosmicDreams writes "The CRN (Center for Responsible Nanotechnology) reports that nanofactories (like the ones that were installed in every home in Neal Stephenson's Diamond Age) will arrive "almost certainly within 20 years". In short they claim that molecular nanotechnology manufacturing will solve many of the world's problems, catalyze a technologic revolution, and start the greatest arms race we've ever seen. They conclude the risks are so great that we should discuss how to deal with this technology so that we don't kill each other when it arrives."
I suspect the reason why popular discussions about AI safety are so confusing is that people say "superhuman AI is safe" when what they actually mean is "superhuman AI is *impossible*".
In a narrowly technical sense, yes, that is correct reasoning: if something cannot exist, it also cannot harm you. But for the sake of clarity of argument, it would be much better if people separated "it is not dangerous *because* it is not possible" from "even assuming that it is possible, it is still safe".
What we get instead, is people playing politics, so the ones believing "it is not possible" and the ones believing "even if possible, it is safe" see each other as allies. An argument that supports an ally is good. But an argument that hurts my ally, but not me, is not too serious.
So the entire argument, fully untangled, would probably be something like: "If you believe that coffee is harmless, that is an argument for our side, because it means that the set of things people worry about needlessly is greater. But even if you believe that coffee is harmful, I don't really care about that, because unlike coffee, superintelligent AI does not exist, silly!"
In the rationalist lingo, "coffee is harmless, despite people once believing otherwise" is a soldier on our side, but is not a crux. If this soldier hurts you, good. If it dies instead, no big deal.
When people bring up the coffee story, they aren't actually trying to make a rational argument. They're telling a story and it just happens to look like an argument. People update on stories all the time, by the way. That's what parables and myths are, and it's how people spread wisdom for millenia prior to rationality. That mental machinery hasn't gone away.
Most things are neither affirmatively safe or unsafe. The vast majority of human developments are of no consequence to history. Inductively, it stands to reason that we would expect most hyped developments to be of similarly limited consequence. As with any inductive reasoning, we would occasionally be wrong, but we would more often be right.
I have a suspicion that the coffee story is not an argument but a metaphor. Some people will (incorrectly) update on it, but really the story is just meant to clear up thinking of those who alreafy have vague a picture of how AI is going to affect the society.
Many AI doomers think that the story in "Don't look up" has surprising similarities to the AI story. But the movie is not actually an _argument_ in favor of being an AI doomer, it just resonates with doomers who have other reasons to believe that AI is dangerous.
because most ai safety arguments are based on My Little Transhumanism: Technology is Magic and assume somehow it will become this master self-aware manipulative villain despite no evidence technology can do this.
it's not AI safety in more mundane ways like ai unemployment and economic effect, or people using AI to increase spam, centralize economic power, or how its edge cases cause more work for fewer remaining people. It's mostly projecting existential fears on the latest new thing.
I mean just because AI can replicate a human behavior doesn't mean it possesses the potential to take on the whole aspects of human behavior in a more perfect form. Technology never breaks upward; not like the more sophisticated cars get, the more likely the evolve into herd animals and book it to the plains to forage. they just break differently
Everything is dangerous/everything is fine:
Coffee ended up being an existential threat to certain countries when it attracted the right people at the right time to create a certain outcome (in the doomer's case, a bloody and destructive revolution; in the optimist's case, a potentially better/fairer political system for future humans).
Nuclear weapons will be an existential threat if/when the right people at the right time utilize these weapons to create a certain outcome (in the doomer's case, extinction; in the optimist's case, a fresh start?).
AI will be an existential threat if/when the right people at the right time utilize these tools to create a certain outcome (in the doomer's case, extinction; in the optimist's case, earth and nature get a chance to recover from our rough-shod resource rampage).
I think I can see it?
I think it's reasoning by emotion. The coffee danger (as I always understood it) doesn't need to go back to revolutionary times, but to advice given to folks with high blood pressure and cholesterol in the 60's: that coffee would make it worse. (Now the opposite is believed true, if we should believe them.)
That's within living memory. And within living memory are emotional decisions: I like coffee, coffee is fun, I want to drink coffee, I love the buzz, what's this about blahdeblah cholesterol folderol? The emotional weight outweighs (in my dad's case when he was told this) the cholesterol story. I remembered this when I drank my first coffee and I think of it at least a little bit when I drank my last.
So it was emotions that won out over what appeared to be scientific evidence. The emotions were doing the heavy lifting here, and when an argument comes up that drags coffee into it, it drags along the emotional reasoning that was done (and proven "right") to speak its voice in the argument. So, they were wrong about coffee, by association they were wrong about whatever emotional feelings I have about AI dangers, which have a strong emotional component on both sides of the argument, or so it appears to me.
I think that's why other arguments that have the same frame but don't drag coffee in, don't seem to work the same way. I just heard that anything you do while consuming caffeine--smell coffee, have friends over, sit at a crowded cafe, whatever else might be going on) the caffeine makes you like whatever that is. Could caffeine itself be making this argument, or at least part of it?
Here’s the steel man:
Humans have a tendency to over-predict doom scenarios.
Because of this, when we’re worried about doom we should be careful and adjust our priors.
Historical examples help with adjusting priors.
–
My own position: super human AI will be the biggest change in history, this makes priors less useful. We’re more in first principles territory.
Say we invent a new drug and people now live to age 1,000. How will this change society? Answer: I’m not sure, but past times when lifespan went up 10 years are probably not super relevant.
As far as I understand the AI safety debate (on a scale of 1 being "never heard of it" and 10 being "Eliezer Yudkowsky" I'd put myself at like *maybe* a 4), the biggest problem with having strong opinions in *any* direction on AI safety is that all the smartest people who most care about this problem seem to say they have absolutely no idea what to do. This isn't a strong argument to stop them from trying stuff, but it's also not a great argument for throwing massive amounts of money at the problem, shutting down GPU farms, lobotomizing AI development with cumbersome regulation, etc.
Maybe the coffeepocalypse argument is badly gesturing at "by your own admission, you're worried by a gut feeling based guess of the probability of a doomsday scenario that emerged from a long process of thinking and speculating about black box technology (black box because you don't know how it works or because it hasn't been invented yet) that you have also admitted you have no idea what to do about. What the heck else can you do in this situation but wait and hope it turns out fine? And, hey, look, there have been times when people faced similar radical uncertainty and we aren't extinct."
My own position is something like "Noting will change until somebody blows something important up with AI accidentally and/or a rogue one blows something up. Hope the explosion isn't existential. At this point, many more smart people will get interested in the problem, and if nothing else we may succeed in strangling it with regulation that doesn't fix alignment but just makes it impossible to do anything with it like how it takes 1 million dollars to build a one stall toilet in San Francisco."
For people who are *really* convinced it's going to kill us all, I'd say the best approach is to try to strangle it with regulation by pretending you are super worried that AIs will discriminate against black people, be used by the deep state to track right wing people, that GPU farms are bad for global warming and/or that they cause all kinds of other weird health and environmental problems, that the *wrong people* will use them to spread misinformation and steal elections, turn the youth trans, etc.and well you get the gist. Lobotomize it. Do to it what was done to nuclear power.
According to Wikipedia, Rutherford said, "anyone who looked for a source of power in the transformation of the atoms was talking moonshine.". Searching "moonshine" on LessWrong yields the following post by Scott Alexander from 2015:
https://www.lesswrong.com/posts/CcyGR3pp3FCDuW6Pf/on-overconfidence
As well as these posts by other authors:
https://www.lesswrong.com/posts/3qypPmmNHEmqegoFF/failures-in-technology-forecasting-a-reply-to-ord-and#Case__Rutherford_and_atomic_energy
https://www.lesswrong.com/posts/S95qCHBXtASmYyGSs/stuart-russell-ai-value-alignment-problem-must-be-an
https://www.lesswrong.com/posts/gvdYK8sEFqHqHLRqN/overconfident-pessimism
https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence
https://www.lesswrong.com/posts/B823KjSvD5FnCc6dc/agi-as-a-black-swan-event
https://www.lesswrong.com/posts/wpsGprQCRffRKG92v/catastrophic-risks-from-ai-4-organizational-risks
I make no claim right now, but others can read and ask themselves which posts have which reasoning errors.
Also, does anyone have citations on the Rutherford quotation? I can only find citations from the early thirties, and none from the day before the first successful artificial nuclear chain reaction (which was in 1942).
The argument is implicitly about base rates. Here's a couple analogous examples:
1) The math department gets a lot of well-meaning letters from amateurs claiming to solve the Riemann Hypothesis. Over the last 100 years, none of these letters have solved the Riemann Hypothesis. Therefore, the one that just arrived today probably doesn't solve it either; we're not going to get excited about it.
2) Every year, there's a popular science article to the tune of "Cure for Cancer Finally Discovered!" We don't have a cure for cancer. Therefore, when the next popular science article about cancer cures comes out, we'll ignore it.
So similarly, the anti-AI-safety argument goes:
Every time we develop a new technology, a lot of smart and well-meaning people argue that it's actually Bad and that it poses some kind of Existential Risk (railroads, bicycles, electricity, human flight, etc. etc.). But new technology is usually Good, and we haven't killed ourselves yet. Therefore, now that a lot of smart and well-meaning people are arguing that AI is Bad and poses an Existential Risk, we shouldn't pay that much attention.
I thought so too: https://www.astralcodexten.com/p/desperately-trying-to-fathom-the/comment/54905621?r=458xj&utm_medium=ios
A couple of thoughts:
a) Agreed, we should _not_ ignore base rates. ( In the financial realm, it has been said that the worst advice that one can hear is: "This time its different!" )
b) Reference class tennis is, unfortunately, unavoidable
c) If we get AGI (in the sense of capabilities matching a _typical_ human, I don't mean near-omniscient superintelligence) (and we do _not_ have AGI today, even in the IQ=100 sense), then it can fill every role needed to make more copies of itself - effectively acting like a new species
d) We have a base rate for interactions with intelligent species too - Neanderthals are gone, Homo habilis is gone. Homo erectus is gone.
e) AGI, if we get it, is both a technological improvement and a new intelligent species. So which base rate do we start from?
( FWIW, I want to _see_ AGI, personally, so I'd rather not see a pause/ban. And I think that a pause/ban in a competitive world, will fail anyway. )
Pre-GPT4 this was pretty close to my take on AI alignment. Now I actually do take it seriously and think we desperately need to stop building frontier models, but I want to try to communicate why coffeepocalypse doesn't seem to me like a stupid counter and is likely something I might have brought up as a counter-argument to AI safety when I thought AI safety was dumb.
Most doomsday predictions involve _stories_: First A happens, then B, then C, and then the world ends. You can say "Well what if A doesn't happen?" and then the prophecizer might say "Well then A' will happen and that could also lead to B" and so on so forth.
And so you get the impression that the prophet is falling into the trap of having located a random hypothesis and is now using stories to defend it. Essentially that they're in soldier mindset. Instead of countering each and every story, it's easier to try to show them the trap they're falling into by giving examples of other people falling into the same trap, and hope they'll recognize it, discard the random hypothesis, and go back to looking for truth rather than writing a story about the future which will almost certainly not come true exactly because most stories about the future don't come true.
Coffeepocalypse isn't an argument _against_ AI safety, it's an argument against confusing stories with predictions which is such a common error it's easy to assume people arguing that AI is dangerous are also making that error.
I don't know what it's for others but for me it's about worry overload. Almost anything you see or read is someone trying to convince you to worry about something. Cries for attention. Now a priori there is an infinity of possible bad outcomes to anything, but there will only be one actual outcome. So per definition, (infinity minus one) things one can worry about, never happen. And I can't be worrying about the infinity all the time. So I only worry about the immediately obvious next pressing problem that I have influence over in my own life. Ai is not one of them. Oh and one more thing. Bad things happened in the past, with or without prediction. But absolute doom never happened. It may, of course, happen one day. So my prior here is, however bad past things turned out, humanity overcame them. So, one step at a time.
I guess it's 1) a prior that AI won't be bad, plus 2) "lots of people being scared of AI and supporting their fear with arguments, isn't evidence worth updating on; lots of people have always been scared, and have always supported their fear with arguments, about everything"