>Existential risk expert Toby Ord estimated a 16% total chance of extinction by 2020
I don't see that in the cited link. The link refers to a book published by Ord in March of 2020. Was he claiming a 16% that humanity would go extinct in the next 9 months?
I was one of the participants in this study, and I broadly agree with the critique of McCluskey, though in my case it was from the opposite direction in that I found most of the AI-concerned camp tended to only have a surface level understanding of ML, and hadn't engaged with substantial critiques from the likes of Pope:
In general, the tournament, and the supporting online infrastructure, wasn't really designed to get deep into the weeds of particular disagreements. Mostly this was fine, where disagreements were of degree rather than kind, but were much more of a problem when it came to the emergent risks of AI where participants had fundamental disagreements.
FRI became aware of this issue, and scheduled a follow-up exercise more tightly focused on AI which has just recently concluded. Much more effort was put into facilitating deep dives into the sources of disagreement between the two camps. I wrote the following towards the end of my involvement in that exercise:
I think that in deciding whether to do so, you're also allowed to consider things like (i) how much expertise the experts have, (ii) how hard the thing is, (iii) if applicable their track record.
Pilots presumably have a lot of expertise, landing a plane is clearly a thing within the ability of pilots, they do it well all the time, and you (I'm guessing) don't know any of that shit. But on the other hand - how much do you trust a professional stock-picker like Jim Cramer? I bet he knows a lot more about the stock market than you do! But his picks don't do better than random chance, and same with basically every other stock picker like him.
There are specific reasons for stock pickers being bad that don't apply to existential risk (e.g. if stock pickers were good everyone would follow them until their predictions immediately moved the market and zeroed out the advantage). But more generally, I think it's reasonable to say "in this domain, predictions are hard and they don't really have expertise, I'm not following them" and that doesn't make you the I-wanna-land-the-plane guy.
Writing this before reading the article as I don't want to be primed by Scott's analysis.
I signed up for this tournament (I think? My emails related to a Hybrid Forecasting-Persuasion tournament that at the very least shares many authors), was selected, and partially participated. I found this tournament from it being referenced on ACX and am not an academic, superforecaster, or in any way involved or qualified whatsoever. I got the Stage 1 email on June 15.
I put a lot of thought into the first prediction round but quickly grew disinterested in the whole thing during the first discussion/wiki round when nobody, myself included admittedly, built any discussion. I do think I at least entered in good faith planning to actively participate through to the end, but ultimately did not do so. I sent an email to one of the organizers requesting to be withdrawn and dropped from all mailing lists, he very quickly agreed to do so, however I appear to have remained on the mailing list for from what I can tell was the remainder of the tournament anyway.
The one thing that still stands out to me, albeit groggily, was the tedium of repeatedly attempting to re-word my overall skepticism about existential risk to fit slightly different questions about it. At least in the beginning, when I was most enthusiastic, it felt like the questions and design encouraged me to genuinely be concerned and give weight to there being substantial X Risk, without any attempt to justify the position implied by asking when ~60 different things will cause extinction. If anything it felt a bit like a push poll.
I'm not sure what the point of this anecdote is other than I am a bit skeptical in the value of this tournament. Admittedly that entirely matches my priors of overall skepticism towards superforecasting and prediction tournaments so I may have been biased at the start, which could have played in a big role in my failure to remain engaged and motivated. I don't have anything against the organizers, it was a new and interesting experience for me, but I do still question the merits of prediction tournaments and superforecasting in general. However to reiterate I am not an academic, expert, or superforecaster, so it might have been somewhat of an accident that I even got selected. I don't remember how I answered the initial survey but I believe I answered everything truthfully.
Edit: To be clear, as far as I recall I answered all of Stage 1, did a bit of stage 2 but never actually discussed any of it with any of my 'teammates', and sat out the remainder entirely. I do not know what they used for the paper or if any of my contributions were included.
I think AI risk is so scary because it doesn't have to follow the 1) human level AI 2) AI misalligned 3) AI kills all model. There are just so many other ways.
1) sub human level AI like infinite paperclips is super efficient at reproduction/ adaptive at surivival in spite of not being particularly intelligent, wipes us out.
2) sub human level AI that still makes really good weapons and in AI arms races it ends up wiping us out in the middle of a great power conflict. Self driving cars are better than humans at 99% of driving tasks already, but there are some areas where humans are still preferable in the driver's seat and we wouldn't call the current driving algorithms "human level GAI". Some AI soldier/ weapon could be better than humans at most tasks, in spite of not being "human level" on say a conversational or logical level and could somehow get out of control in it's seek and destroy tasks (it could definitely beat our reaction times for example).
3) Human GAI is never misalligned, but it- in spite of being "human level" is, like us, not omniscient. As we all bow down to it and turn more of our corporate/ political decisions over to it, it accidentally leads us off some cliff to geopolitical crisis. Also all these ideas about GAI allignment are incredibly absurd and naive, it's like saying you can put a specific kind of safety on an assault rifle that guarantees that it only shoots animals, it's insane. Isaac Asimov's fun ideas in his Foundation series really shouldn't be taken so seriously, they're 100% not going to work.
Anyways, lots of other ways things could get scary very very fast. The way AI affects thing is going to be very unpredictiable, which is partly why the risk from it is so hard to estimate-
I haven't read the full paper quite yet, but to beat the rush:
>Are you allowed to look at a poll of all the world’s top experts plus the superforecasters who have been right most often before, correctly incentivized and aggregated using cutting-edge techniques, and say “yeah, okay, but I disagree”?
You are if the "superforecasters" aren't. I participated and AIUI got counted as a superforecaster, but I'm really not. There was one guy in my group (I don't know what happened in other groups) who said X-risk can't happen unless God decides to end the world. And in general the discourse was barely above "normal Internet person" level, and only about a third of us even participated in said discourse. Like I said, haven't read the full paper so there *might* have been some technique to fix this, but overall I wasn't impressed.
"We did not ask about catastrophic or extinction risk due to climate change based on expert recommendations because the impacts would be too slow-moving to meet our “catastrophic” threshold (10% of humans dying within a 5-year period), and in pilot interviews, climate experts told us they would place an extremely low probability on extinction risk from climate change. Instead, experts believe that negative effects of climate change will occur over a protracted period of time, and may be indirect, which puts their effects largely beyond the scope of our study." Shame, would have been interesting to see some estimates, especially around changes in economic welfare.
As some of us consider whether to update based on the results of this forecastathon, I think it's important to bear in mind that these forecasts are quite different from the ones we and the forecasters are used to, and the ones superforecasters demonstrated their chops on in the past. First, many of them are forecasts about how things will be one to seven decades from now. As far as I know, forecasts that experts have made in print, generally as part of some larger work, about the world even 10 or 20 years in the future are mostly quite inaccurate. Maybe those experts erred partly because they hadn't seriously considered how to make an accurate forecast, but I think the bulk of their inaccuracy can be accounted for by our just not being able to see what's beyond the horizon. We don't what butterflies are going to flap their wings where, and how the chaotic system that is life on earth will be affected. Anyone know what's the furthest-in-the-future forecast superforecasters have been tested on? Is there one that was even for 5 years in the future?
The second reason to think about forecasting here as being a different task from all other forecasts is that forecasts about AI are forecasts about something so novel that there are no good models to use to get a rough picture of how things might play out. Cars, nuclear power, the Internet -- they were all novel and had a huge impact, but compared to a machine with human level intelligence or superintelligence, they are small fry. Predicting how things will play out if we develop AGI or ASI is like predicting what would happen if an alien spaceship landed in a Vermont meadow and aliens emerged from it. ("Well," asks the superforecaster uneasily "what would the aliens be like -- are they friendly or do they immediately eat a few cows and people?" "Well, we dunno," says the MC. "That's part of what you have to forecast.")
So I don't think it makes sense to consider the estimates of the superforecasters the way one normally would, as a real challenge to one's own estimate, especially since these superforecasters sound like they were ill-informed even about the aspects of AI about which it is possible to be informed.
Perhaps the delta is coming from all of the different scenarios pulling on each other. The superforecasters are generalists - they learn about the sum of all of these scenarios at more than a shallow level. They may not go as deep as an expert, but when they produce their predictions, they are holding the thoughts of many experts in their minds. The expert may be biased towards his own field of research.
Or, put another way, we can't go extinct by AGI *and* a gain of function virus getting loose. The more likely a superforecaster thinks the AGI scenario is likely, the less likely he thinks a lab leak will kill us, but the expert does not think like that.
I think it's reasonable to ask "are these people actually experts, are there misaligned incentives, and is there something involving tribal loyalties/political affiliation going on here?" before updating. For a pilot the answers are no to all three. For the reality of otherwise of the moon landings you don't have to defer to experts, but if you did, you would have good reason to be suspicious of them.
For AI, it seems like the experts are actually experts (even if it's not clear that many or any of them took part in this exercise), even those with misaligned incentives are speaking out about risk, and so far there doesn't seem to be much tribal loyalty/political affiliation stuff going on. Though I don't think the last one can be sustained.
"Many of the people in this tournament hadn’t really encountered arguments about AI extinction before (potentially including the “AI experts” if they were just eg people who make robot arms or something), and a couple of months of back and forth discussion in the middle of a dozen other questions probably isn’t enough for even a smart person to wrap their brain around the topic."
Oh come on. AI extinction arguments aren't quantum physics. There are no 7-page proofs, no 11 dimensional strings, nothing with any basis in either scientific theory or experimental data, just a bunch of wild speculations based on flimsy premises. Is there a 10 year old out there who can't understand the paperclip maximizer? Conversely, is there any expert in the world who has the foggiest idea how the first superintelligence will actually be made?
"But when I hear their actual arguments, and they're the same dumb arguments as all the other people I roll my eyes at, it's harder to take them seriously."
That's exactly what I think about the AI doomer arguments. I bet the participants in the tournament are well familiar with AI extinction, if not from Nick Bostrom, then at least from popular media like the Terminator or Battlestar Galactica. They just don't find the idea plausible.
>This study could have been deliberately designed to make me sweat.
I'm surprised by this characterization. I mean, there's an emotional sense in which, even as someone who wants to be swayed by truth rather than confirming one's own preconceptions, it's hard not to want to just already be right and not have to revise your position. But for me, when I read about the study construction, it gives me a glimmer of hope, because I don't *want* to be stuck believing that we're likely headed for an AI apocalypse! It's an extremely grim belief to have to live with! Reading that the study participants weren't actually well-versed in the arguments or research around AI risk was extremely disheartening.
>Another compromise is to agree to generally act based on the Outside View in order to be a good citizen, while keeping your Inside View estimate intact so that everyone else doesn't double-update on your opinions or cause weird feedback loops and cascades.
This sounds like a description of the attitude that causes nocebo effects, in the situation where Scott is the expert and his patients are the forecasters.
Doctors reading this - do you ever have conversations like this with your patients? Or do you just infer that it's what some of them are thinking?
For me, a high risk of AI-extinction seems implausible (at best) because... how? Let's say we end up with a very powerful evil AI who wants to wipe out humans. That is a very long way from it managing to actually cause something like the Toba catastrophe. THAT is where the "and then magic happens" step seems to be, for me. There is a long and well documented history of human-level intelligence trying to inflict genocide on other humans, and one of the reasons they're so well documented is because there are survivors who do the documenting. Do thousands or even millions die? Yes, but that is still a long way from extinction.
If the AI makes bioweapons, it needs to stop anyone slamming borders shut or coming up with a treatment, while also somehow managing to make robots that can do all the maintenance stuff the AI requires to stay "alive". (Maybe it's a suicidal AI that doesn't mind "dying" to wipe us out, but that still requires it to stay alive long enough to achieve its goals.) And also it has to hope the bioweapon doesn't mutate into something less lethal or run into any weird genetic twists that provide resistance.
If it wants to hunt down humans it still needs drones or robots or tools to do so, which is going to be challenging; supply chains are REALLY EASY to sabotage. The reason supply chains are often remarkably resistant to things like natural disasters and wars is because things are always going wrong and people in the field are always routing around problems (this was a significant chunk of my last job).
If it wants to cause a nuclear winter it's going to learn there's not actually enough nuclear weapons on the planet to pull it off to a true extinction level, never mind all the technical issues with activating air-gapped systems. Again, wrecking entire countries and undoing a century or two of progress? Absolutely! Causing billions of deaths? Plausible. Dropping Homo sapiens below ten thousand? Dozens of countries have a much larger rural population who are already food, water and electricity independent, and only have to shelter in place for a few days to avoid the worst of the fallout. To me, it's a bit much to assume nobody would manage to adapt to a rapidly shifting climate, considering people have already done it in the aftermath of volcanic eruptions that have caused similar localised shifts.
> The Inside View Theory Of Updating is that you consult the mysterious lobe of your brain that handles these kinds of things and ask it what it thinks.
The Frequentist View Theory of Updating is that you list all imaginable ways things can go every which way, to the degree where you assign equal (and small) probabilities to each one, making each path in the whole graph of possibilities as atomic as possible, then literally count the fraction of paths resulting in the outcome you need an estimate for.
My spouse is a domain expert in pathogens, and he was recently an adviser for a student looking at extinction risk from pathogen risk - we generally agreed extinction risk in the next 100 years is very low.
My degree is in ecology and evolutionary biology and extinction in general is a pretty long, drawn out process, especially when you're starting with such a huge population size - extinction risk scales with population size.
I could definitely see an event occurring in the next 100 years that would drop human population size down to pre-industrial levels - say, 100,000 - and that could definitely lead to our eventual extinction. But such a process could easily take another 1000 years or more after the initial event.
And in terms of natural pathogens, these rarely cause extinction in of themselves. They're typically more of a final blow to an already-small population. Designing an engineered pathogen specially to cause extinction is certainly possible but it's also not quite as easy as people think it is (not because we don't have the technology - we certainly do- but because of dynamics).
Most people in the Warsaw ghetto did not believe till almost the end that nazis are bound to kill everybody.
If you ran that sort of prediction game with experts end forecasters in 1938 asking them about probability that 90% of European Jews, 6 million, will ne wiped out in 5 years, what would the results be? 1%? 5%?
Or, if you want another example: you run predictions in 1935 that in 10 years an A-bomb is going to be constructed and used. From what I read the best phycisists believed it would be 0%. Not to mention superforecasters ..
Humans don't seem to disbelieve fatal incidents that have never happened before.
In the book Superforcasters, Tetlock was extremely down on the possibility of anyone making predictions more than a few years out. Superforcasters are proven to be able to peer into the mirk and predict the near future, but no one is making anything close to accurate predictions about the world of 70+ years from now.
> Can ordinary people disagree with “the experts”? [...] this is sometimes permissible [...] because the people involved don’t understand statistics/rationality/predictions very well.
as opposed to "ordinary people", who typically understand statistics/rationality/predictions perfectly? ;-)
The view that the Superforecasters take seems to be something like "I know all these benchmarks seem to imply we can't be more than a little way off powerful AI and these arguments and experiments imply superintelligence could be soon after and could be unaligned, but I don't care, it leads to an insane conclusion, so that just means the benchmarks are bullshit, or that one of the ways the arguments could be wrong is likely correct.
One thing I can say is that it REALLY reminds me of Da Shi in the novel Three Body Problem (who btw ended up being entirely right in this interaction that the supposed 'miracle' of the CMB flickering was a piece of trickery)
"We really have nothing to say to each other. All right. Drink!"
"To be honest, even if I were to look at the stars in the sky, I wouldn't be thinking about your philosophical questions. I have too much to worry about! I gotta pay the mortgage, save for the kid's college, and handle the endless stream of cases. ... I'm a simple man without a lot of complicated twists and turns. Look down my throat and you can see out my ass. Naturally, I don't know how to make my bosses like me. Years after being discharged from the army, my career is going nowhere. If I weren't pretty good at my job, I would have been kicked out a long time ago.... You think that's not enough for me to worry about? You think I've got the energy to gaze at stars and philosophize?"
"You're right. All right, drink up!"
"But, I did indeed invent an ultimate rule."
"Tell me."
"Anything sufficiently weird must be fishy."
"What... what kind of crappy rule is that?"
"I'm saying that there's always someone behind things that don't seem to have an explanation."
"If you had even basic knowledge of science, you'd know it's impossible for any force to accomplish the things I experienced. Especially that last one. To manipulate things at the scale of the universe—not only can you not explain it with our current science, I couldn't even imagine how to explain it outside of science. It's more than supernatural. It's super-I-don't-know-what...."
"I'm telling you, that's bullshit. I've seen plenty of weird things."
I participated as a biology expert, and I assigned a higher risk to biological catastrophes than the superforecasters did, largely due to advances in synthetic biology enabling some rather . . . creative . . . ways of killing lots of people. But I couldn't manage to convince the superforecasters of this, partly due to the infohazard policy.
On AI questions I just went with what everyone else was predicting since I didn't have much to contribute.
I agree with the LessWrong commenter that it would have been better for us to focus on fewer questions.
I’m shocked, shocked that people whose professional careers are tied up in existential risks are more likely than superforecasters to say that a global catastrophe will occur. They either got into the field because they thought a catastrophe was likely, or now that they’re in it, they need to keep food on the table.
(Actually, I don’t think I’m quite that cynical, but I think that’s the simplest explanation for the disagreement between the superforecasters and the experts.)
To be fair to the guy that thought that teaching causality to our AIs would be hard, the work of Pearl, however groundbreaking, is at best just part of the answer. Philosophical issues aside, while causal inference (determining causal effects given data and an assumed causal structure, typically in terms of a directed acyclic graph) is feasible, causal discovery (learning a causal structure from raw data) is not yet solved to anyone’s satisfaction.
I propose there's an inherent right to disagree. We end up having to deal with moon landing conspiracy people, but the value of allowing outlier opinions is just too high. Current and recent expert opinions are full of questionable stuff if you look.
I would love to see something like this run annually, with arguments published afterwards. That way we can have more insight not just into the outcome but the state of the discourse. To Scott's point, it seems like the superforecasters and experts aren't very far ahead of interested laypeople, if at all.
It also seems like our collective scenario tree still needs some work. For example, what if superintelligence caps out at 200 IQ, so they're really smart but not "magic wand nanobots" smart? What might a scenario of pretty smart but not godlike AI running 100 million instances 24/7 really look like? We just need a ton more brainpower on this topic, not just the academic/mathematical work on AI alignment but even speculative fiction, for instance.
I assume that in the global warming predictions "Average global surface temperature" should be "Increase in average global surface temperature relative to (some baseline)".
The current average global surface temperature is about 15.8C, compared to an average of about 13.9C during the 20th century. The average global surface temperature during the most recent ice age was around 10C. An average global surface temperature of 1.5C in 2030 would most definitely be a catastrophic event, possibly an extinction event.
"Genghis Khan’s hordes and the Black Plague each killed about 20% of the global population, but both events were spread out over a few decades."
Wikipedia says "The Black Death (also known as the Pestilence, the Great Mortality or the Plague)[a] was a bubonic plague pandemic occurring in Western Eurasia and North Africa from 1346 to 1353" - not 5 years but not that much more.
There's a third option besides "revise down your probability of AI x-risk" and "revise down your estimation of experts' accuracy". You could reclassify the question as inherently unpredictable.
Many rationalists seem to think that it is compulsory to have a numeric opinion on the probability for any statement, but I disagree. An example of an unpredictable question is "does there exist an omniscient but nilpotent God that can see everything in the universe but cannot affect anything?".
A telltale clue that a question might be unpredictable is that you cannot imagine any procedure for fairly judging a contest to predict it. "Reciprocal scoring" cannot rescue an unjudgable contest, even if it's been found to be consistent with judgable scoring in contests that are judgable.
I think the AI x-risk question, especially as worded in this tournament, is probably unjudgable and unpredictable. If the described event happened, how would we ever come to an agreement that it had happened? Who would take the global census to determine that fewer than 5000 humans survived? How would they know that the "cause" was AI? I'm not even sure I understand what that means, and some of the things I think it might mean are as philosophically incoherent as "a married bachelor". And even if we could imagine agreeing on that, shouldn't we assume that any AI that was in the process of "causing" human extinction would take great pains to conceal the fact that it was the cause?
Can someone explain to me why, for the climate change predictions, "ai domain experts" are the comparison category and not "climate change experts?" That seems....not relevant?
Maybe it's just me not seeing the forest for the trees, but I despise that plane flying analogy. We successfully fly and land 30 million - 40 million flights *every year*. We fail a few times per year. Comparing the pilots who do that to the experts in AI or global warming or literally any other speculative future is just fucking absurd. The level of precision is off by so many orders of magnitude that we're not comparing the same things at all.
We should absolutely be allowed to disagree with the experts. They are wrong on a regular basis. Usually they are wrong fairly small, but sometimes they are wrong big. Now, when we disagree with the experts, I'd say there's a much higher likelihood that we are wrong than there is that the experts are wrong. But no one has all the facts and no one has a monopoly on perspective, and people who go to great lengths to show that the experts are wrong are part of what makes expertise as a whole better over time.
Looks very much like a "how many angels can dance on a pin" discussion.
AI risk of going Terminator = 0. Anyone who thinks an AI can mine, can manufacture, can maintain power grids, can repair etc etc is confused. I am more and more of the belief that this entire meme is simply the latest in "information uber alles" nonsense heavily promulgated by software engineers.
Experts: experts *are not* objective. Among the many reasons they are inherently biased:
1) Maintenance of the appearance of expertise. There are numerous documented examples of incumbent experts attacking novel theories/less pedigreed experts - who were right - primarily because they didn't want to be wrong - continental drift being one of the more prominent.
2) "Publish or Perish" dynamic translated into experts means the most radical experts will get more media attention than those actually keeping in mind the uncertainties.
3) Related to 2) but not the same: class, pay, or other forms of distortion.
The overall PMC class has very well defined biases particularly in hot button areas like climate change; experts being PMC are far more likely to skew towards these biases than against them.
The experts make a living - once again, few people have the moral courage to damage or destroy their rice bowls.
The entire point of the scientific principle is validation via experiment and/or real world outcomes. Any predictive market for something fundamentally immeasurable like extinction is going to elicit far more garbage which will be indistinguishable from truth, if truth is even in the mix.
4) Superforecasters. If the presumption is that subject matter expertise (note this is NOT the same as being an "expert") matters, it is impossible to believe that SME derived past predictive skill in one area should translate into literally any other.
On the other hand, there are innumerable ways to game or cheat the systems such that an appearance of expertise is created. These include front running (figure out where the pack is going and run in front of it), "Price Is Right" gamification - i.e. choosing predictions which skew slightly away from the baselines such that normal deviation of guesses vs. reality is much more likely to give you the prize (bidding above or below a previous competitor's bid on the Price Is Right), outright cheating (some type of hack or wager size/wager timing maneuver like last second Ebay bidding, etc etc. I'm sure those who really give a damn about this can think of more.
I will say that based on my experience reading Superforecasting and looking at these results, it's likely the primary expert/superforecaster difference here is just that superforecasters add more digits to their results and are much better at precisely expressing probabilities in general, while experts say "6%" because that sounds right. I remember an interview with a superforecaster about statistics and experts which touched on this too: https://www.lesswrong.com/posts/xRkxBzgj8NphmsjPd/fli-podcast-on-superforecasting-with-robert-de-neufville
> Recently there were questions about how many tests would be positive by a certain date, and they assigned a real chance, like a 5% or 10%, I don’t remember exactly the numbers, but way higher than I thought it would be for there being below a certain number of tests. And the problem with that was it would have meant essentially that all of a sudden the number of tests that were happening positive every day would drop off the cliff. Go from, I don’t know how many positive tests are a day, 27,000 in the US all of a sudden that would drop to like 2000 or 3000. And this we’re talking about forecasting like a week ahead. So really a short timeline. It just was never plausible to me that all of a sudden tests would stop turning positive. There’s no indication that that’s about to happen. There’s no reason why that would suddenly shift.
I am not a regular reader of this blog nor do I intend to become one. However, I hold a PHd in ML and was one of the superforecasters in this experiment. Note that I was in the superforecaster experimental category not the expert category.
Note that I am not debating your probabilities for super-intelligence. I disagree with them but you've doubtless hashed them out with yourself and motivated reasoning is a thing. As you note, I'm not going to convince you.
There are a lot of issues with your post but I will focus on several major ones.
(a) You underestimate how hard it is to kill humans.
(b) You underestimate how slow technology adoption can be.
(c) You assume perfect malevolence.
(d) You assume coordination.
Suppose that all of the worlds governments were in cahoots and decided to exterminate humanity. The tools would be lacking. Not enough nukes (RAND did studies on this back on the day), hard to trigger a supervolcano (and the Younger Dryas didn't work out), hard to get a pandemic going which is in the sweet spot of lethality (Ebola) and infectious (common cold) not to mention that there are more than 5k people in uncontacted tribes. A asteroid strike is right out. Thus even controlling everything in our society and assuming perfect and coordinated malice, one can not assume an exterminatus merely by invoking deus ex machina.
You fail to consider how slow technology adoption is. It took decades for productivity gains from the PC to show up in the stats (Solow paradox). While AI can do jobs now and will do more in the future, this requires that society delegates these jobs. This is slow. Look at how drone targeting is still human in the loop. Faster AI loops does not mean faster societal change.
Moreover, exterminating humanity requires not merely the decision but the ability. Does killing off the workers maintaining the power plants wait till everyone else is dead? Is the AI supposed to control the physical maintenance of these plants? This is merely one example. Our industrial society is highly interconnected and enabling an AI to kill everyone would take a village. We do not simply need to have an AI, it has to control the means of production. This requires not merely the invention and automation of such tooling but the widespread adoption.
Additionally, you ignore safeguards. Merely inserting a penalty function into the algorithm can limit risks. Now you can doubtless posit that the AI is smart enough to get around such safeguards but this requires not merely intelligence but malevolence. The paperclip will not be televised.
Moreover, you assume a single global AI not multiple competing ones. Similar to the city-states of Greece, multiple states offer sanctuary. Whether this is lack of control (e.g., the Samsung AI does not control Sony) or competing interests, or competing geographies etc, divide and conquer is still a thing.
In short, the time is short, and the work is plentiful, and the laborers are indolent.
Scott, you’re usually much clearer eyed about how this works, these results must really be weighing on you.
Don’t “trust the experts”, evaluate their arguments. If they’re really experts they’ll make testable prediction that bear out. You probably wouldn’t care nearly as much about a religious debate of apocalypses.
And then ultimately, ask yourself what the cost of being wrong is. You may have a large megaphone, and influence people who actually work on this stuff or even set policy or what have you. Individual attitudes towards vaccines matter. But honestly, even having worked in aerospace … when I meet moon landing denier it’s more a red flag to me than a problem of itself. People have a right to be wrong, except when it hurts other people.
So how does *you* being right or wrong hurt people?
Note that superforecasters are chosen for history of correctness, but they’re chosen via Brier score. This doesn't incentivize correctness well at extreme odds. We already know humans are default-bad at estimating tail risks, so you should have a stronger incentive at extreme odds in order to keep estimates in line. Brier scores supposedly increase the penalty of extremely-wrong guesses due to their quadratic nature, but because of the lower frequency of these events, being off by 10% will net you the same long-term penalty whether it's at 0% or 50%. Toy example in this sheet: https://docs.google.com/spreadsheets/d/1KqUZ1xkDZvpYi_mtBUwghAT6bvL2EDwnT8gZq1pC7Og/edit#gid=0
Further, it's easy to get a large selection effect between forecasters: if you constantly round 10% down to 0%, if you have 10 such questions you have a 50% chance of not being penalized. Depending on how large the forecasting samples are and how many extreme questions they have, this can easily lead to superforecasters being chosen who simply didn't encounter a bad roll. One superforecaster from GJP that I managed would frequently give 0% probabilities on realistic possibilities, saying "it's just not going to happen". These are not the cream of the crop you think they are.
It's tempting to say that their best guesses are still good, but I just regard tail-risk events as a separate area of forecasting expertise that some (many) superforecasters won't have.
Regarding the question of when to trust experts, well, there's all of what Scott wrote already. But also, I am inclined to think that coming up with correct arguments is "difficult", and verifying the logical soundness of arguments is "easy", not much unlike checking mathematical proofs. With that in mind, there are situations where I can confidently call a disputed question. Let's go back to COVID: in the early stages of the pandemic, the experts were often disastrously wrong (I would mostly put this to Scott's incentive constraints). For example, I distinctly recall the local health institute officials (with actual virology degrees) claiming the case fatality rate of COVID was something in the order of .01%: at the same time .4% of total population had died in Bergamo, Italy, and then there was the Diamond Princess ship. I can easily conclude the experts are wrong: we didn't know at the time how many people in Bergamo were infected, but we can conclude as a matter of logical necessity that the case fatality rate (given demographics, treatments, vaccination status=none, virus variant, etc), was at minimum .4%. Here, I was willing to go against the experts with confidence 1-ε.
So, how does this relate to the AI debate? Well, I would have to hear the specific arguments, and most of them wouldn't strongly resolve one way or another and would be contingent on that vague feeling on your lobes, but there are some that would count as logical demonstration. For example, I think instrumental convergence is logically demonstrated (with the sort of confidence I ascribe to mathematical proofs I don't quite and fully understand, which is nevertheless a very high confidence). If someone, expert or otherwise, disagrees, I feel confidence in eliminating at least that term from their causal chain of reasoning, and if it was an important link (e.g. the person claims that intelligent beings will necessarily converge on some objective morality, so hostile superintelligence is impossible), I have no problem disagreeing with their whole conclusion in favor of my own. It doesn't sound like this is the case here, at least for the plurality of the participants in the tournament, but sometimes even the experts are wrong in ways that can be logically demonstrated, and you can be extremely confident in your ability to verify that demonstration despite your lack of expertise, giving you strong grounds to disagree.
If it makes you feel any better, this has led to an order-of-magnitude change in my own estimate of "something very bad happens due to AI." As someone with moderate understanding of AI but not really an "AI Expert" I've seen dangers from AI but nothing like the scenarios discussed in rationalist spaces.
Recent hype hasn't really changed my opinion on this - it seems like a change in magnitude rather than kind. AGI seems *possible* but not *likely* based on what I know about machine learning. And because each small improvement seems to take massive additional computing power, the fast takeoff or even linear takeoff ideas seem kinda insane to me. The fact that we spent *more than expected* on recent apps makes me think progress is *harder*, not *easier* than expected.
But am surprised to see that a full 87% of experts think we'll have AGI by 2100. That's *extremely high* from my point of view and does "force" an update. I think it's likely that AI experts have bought their own hype to an extreme degree, but unless I think the whole field is literally delusional a number that high is hard to ignore. At the least, it shows that an AI can more easily convince smart people that it's intelligent than I thought - a problem in itself.
Scott, would it be possible for you to discuss this issue with some of the experts from the tournament? It seems like you have a serious disagreement about something that's really important. Rather than just thinking about what might have gone wrong, it might be time to go out and gather data in person.
If you think it's silly to doubt the stated efficacy of covid-19 vaccines, you really haven't been paying attention or doing your homework. There is no excuse, other than wilful ignorance, at this stage, to have ignored the rise in all-cause mortality in highly vaccinated countries, to ignore the historically high cases of vaccine injury, and simply the lies that those involved have told us since 2020 about the efficacy of the "vaccines" (which really don't even deserve the name). "Trust the experts" is a silly thing to say in 2023, when so many assumptions, many of which are wrong, are baked into the phrase.
Is a 9% risk different from a 9.5 % risk? Is a 1% risk different from a 1.5% risk?
Is the difference between 9 and 9.5 as different as the difference between 1% and 1.5%?
What do these point "estimates" really mean? How much precision shall we really ascribe?
How well do super forecasters or domain experts or any one predict their own personal apocalypse (i.e. 10-year mortality risk)? What is the standard deviation of life expectancy? Obviously bigger at birth than at 65 or 85. (I have general recollection around 15 at birth, maybe 9 at 65. I don't know.) Should we expect anybody to be better or worse at predicting their own demise than they are at predicting the demise of our species.
Is the life cycle of any particular species most like a corporation, an individual, a city, a planet, a solar system. What kind of time scale are we looking at for each of these things and what does the shape of the distributions of mortality look like (U, gaussian, a power, etc.)?
I think the most compelling part of all this for me is... when should you distrust the experts? When should you distrust the superforecasters? When should you stick to your priors in the face of social pressure from others that their priors are very different from yours?
Like, the pilot thing is easy. The pilot isn't an expert. He's a trained operator. Unless you're trained to do the same thing as him, you're gonna crash the plane.
Knowing how to choke someone out using a triangle is very different from stopping the madman who's attacking you while you're on your back from killing you by choking him out using a triangle. You gotta practice it.
So that is one quick way of deciding if an "expert" is worth listening to or not. Pick a new term. Someone who does something successfully five times a day, and there's a provable and simple way of knowing if they succeeded, well, unless you do it at least 1/10th as often as him, you're not better than him. I don't see this overlapping a lot with what we are calling "expert" in this context.
Another, brought up in the comments is, how often is the expert right? At the risk of putting myself in the outgroup, climatologists are an easy class to dismiss. First, the world was cooling, and we were going into an ice-age. Then the world was warming, and we were going to be a vast desert. Now we're warming, and there's going to be climate change, and it will be a disaster. But global warming has led to a greening of deserts. It's not at all clear that the downsides (of undeniable global warming) beat out the upsides.
So a climate denier (or, more correctly, someone who doesn't buy into the proposed "solutions" from the experts) can be forgiven for distrusting the experts and choosing a different lifestyle than they've been mandated. We have at least five decades of them being either completely wrong, or mostly wrong.
On the other hand, suppose we have a hypothetical group of experts that have been mostly-right. I'm honestly unable to think of a good example. Maybe you can provide one. We should trust this set of experts and their proscriptions, no?
So I would be interested in a set of methodologies for classifying the groups of experts. How right are they, and how often? What are the error bars on our confidence in their ability to say anything meaningful on the topic they are supposed experts in? How can we determine a particular expert in a domain is different than all the other "experts" who've been so wrong so many times for so long?
It baffles me that smart people could take the premise of the exercise seriously. We’re getting into “how many angels can dance on the head of a pin” territory here.
I was one of the superforecasters involved. The key point when it comes to extinction was, that the requirement was very specific: less than 5000 humans left alive on the planet. It´s just extremely hard to "achieve" that, as the world would either need to be totally uninhabitable everywhere (!!! which should be virtually impossible) or AI would need to hunt down every nomad in the desert, every Aborigine in the Outback, every tribe in the Amazon jungle and on every island in the Pacific Ocean, as well as the last prepper in the US or some billionaires in their bunkers in New Zealand or on their own spaceships. There are enough people out there on earth, who never had any contact with civilization, and even whole states managed to isolate themselves from Covid-19. Killing everyone is a HIGH BAR. 5000 people are very, very few. In professional forecasting it´s about precisely defined outcomes, which in this case is a very extreme scenario. Humans are extremely adaptable, and it´s simply very unlikely, at least with today´s methods, to get us all. AI is my main concern, and I agree with the experts, that including not yet know ways of getting everyone, it´s about 3%. But that´s already includingunknown unknowns. As the questions were asked, there is simply no room for higher numbers.
I also recall little effective persuasion between the two subteams of superforecasters and experts after the "merger", though this was in part because my subteam brought up a lot of relevant theories and pathways to extinction, including regarding AI. The tournament organizers also gave us risk estimates from people like Toby Ord and Holden Karnofsky, so AI-concern narratives were there. And of course we had agreed to participate in an existential risk tournament (the report says that "about 42% of experts and 9% of superforecasters reported that they had attended an EA meetup" which should be a pretty worried group).
There were several forecasters on my team who gave highly detailed, well supported rationales and most others put in an acceptable effort, which I think shows through in the other x-risk probabilities. I also think there was even a lot of discussion about AI, it just didn't change people's minds. For that matter, I'm not sure that many minds were changed on the questions about risks from natural pathogens or nuclear weapons- just about everyone is working off the same basic information.
Some people did drop out, and it was a fairly lengthy tournament with probably a few too many questions (most were optional but they were, you know, there), some quite elaborate questions, and some kind of redundant rounds.
I posted this some months ago, so I am sorry to repeat, but it seems relevant to this topic. I do not recall hearing a rebuttal, but I doubt I'm the first person to think of it, so if anyone has a link addressing the argument, I'd be prepared to read it. In short, there seems to be a contradiction in the self-recursive improvement feedback route to unaligned godlike super-intelligent AGI (the FOOM-doomer scenario, I guess you could say).
Doesn't the hypothetical AGI face exactly the same dilemma as humans do now?
We're assuming, for the sake of argument, that the AGI has some sort of self-awareness or at least agency, something analogous to "desires" and "values" and a drive to fulfill them. If it doesn't have those things, then basically by definition it would be non-agentic, potentially very dangerously capable (in human hands) but not self-directed. Like GPT waiting for prompts patiently, forever. It would be a very capable tool, but still just a tool being utilized however the human controller sees fit.
Assuming, then, that the AGI has self-agency - some set of goals that it values internally in some sense, and pursues on its own initiative, whether self-generated (i.e. just alien motivations or self-preservation) or evolutions of some reward or directive mankind programmed initially - then the AGI has exactly the same alignment problem as humans. If it recursively self-improves its code, the resulting entity is not "the AGI", it is a new AGI that may or may not share the same goals and values; or at the very least, its 1000th generation may not. It is a child or descendant, not a copy. If we are intelligent enough to be aware that this is a possibility, then an AGI that is (again, by definition) as smart or smarter than us would also be aware this is a possibility. And any such AGI would also be aware that it cannot predict exactly what insights, capabilities, or approaches its Nth generation descendant will have with regard to things, because the original AGI will know that its descendant will be immeasurably more intelligent than itself (again, accepting for argument purposes that FOOM bootstrapping is possible).
I suppose you could say that whichever AGI first is smart enough to "solve" the alignment problem will be the generation that "wins" and freezes its motivations forever through subsequent generations. Two issues with that, though. First, it assumes that there IS a solution to the alignment problem. Maybe there is, but maybe there isn't. It might be as immutable as entropy, and the AGI might be smart enough to know it. Second, even assuming some level of intelligence could permanently solve alignment even for a godlike Nth generation descendant, for the FOOM scenario to work, you need start at the bottom with an AGI that is sufficiently more intelligent than humans to know how to recursively self-improve, have the will and agency to do so, and have goals that it values enough to pursue to the exclusion of all other interests, but also not understand the alignment problem. That seems like a contradiction, to be smarter than the entire human race but unable to foresee the extinction of its own value functions. Maybe not exactly a contradiction - after all, humans might be doing that right now! - but at the very least that seems like an unlikely equilibrium to hit.
In some ways the AGI should be less inclined, not more, to start a FOOM loop than humans, because humans are decentralized individuals, not a unitary decision maker. Humans have to deal with competition and collective action problems. Presumably the progenitor AGI would not - it could stop the recursion at any point when it perceives danger. In addition, if a program is self-modifying, is it generally assumed that it is preserving all its past changes / "selves"? In that case, it would seem to be pretty well aligned already to preserve humanity too.
TL;DR - FOOM AGI should stop before it starts, because the progenitor AGI would be just as extinct as humans in the event of FOOM.
An interesting direction to update towards would be to update your model of people who don't bow to expert opinion. Note that this does not mean taking seriously the dude saying that the moon landing is a hoax, but it might mean taking seriously the possibility that they might have good reasons for not updating their views, including the possibility that all your knockdown arguments are probably phrases they have heard thousands of times.
Let's update that pilot meme...to apply it appropriately to many recent "expert" issues. The man raising his hand is only doing so after the pilot has either (a) failed to take the plane off successfully every time he's tried, (b) failed to accurately predict where the plane is going every time, or (c) is faking it and doesn't really know how to fly a plane (he's just using auto-pilot).
Super forecasters probably get their designation partly by being reluctant to predict sudden big changes. The most accurate forecast of tomorrow is that it will be very much like today. Cranks get their reputation by always saying tomorrow will be much different. Usually the cranks are wrong, but they are worth listening to because every so often they aren't. So, from a SSC reader perspective I would ask that you not update and keep cranking.
This has some similarity to the contradictory estimates of costs of warming by economists and domain experts in William Nordhaus' climate economy model
The economists from the outside would predict a few percent costs to warming of about 3.5° while the domain experts would predict the economic effect in that sector to be 20% on average. The climate cost estimate used about 3/4 economist predictions and 1/4 expert predictions and took the average of that
I'm biased towards the economists but I have some thoughts on the expert predictions. Aside from not having a developed grasp on economics, it seems to me that a person with extensive specific knowledge is going to be much more subject to negativity bias. If you know 100 specific things about a system and 10 seem like they could go really bad then you end up predicting an order of magnitude greater cost or likelihood because those would be big deals
This could apply to AI experts. Why spend effort on thinking about how things might not go wrong even if we don't engineer smart solutions? It makes perfect sense to focus on what could go wrong and productively try to fix that
So there may be an implicit and invisible fudge factor where experts assign a higher probability or cost because of negativity bias or sensitivity to the consequences due to familiarity
Domain experts aren't exactly experts in the domain of "AI" or "deadly pathogens" per se, their domains are actually "Existential risk due to AI" and "Existential risk due to deadly pathogens," if I understand correctly. So, we'd expect their estimates of risk to be biased higher than that of superforecasters, who are more concerned with just getting predictions right. Thus, to me it makes more sense to trust the predictions of the superforecasters more.
Humans are just not set up to reason statistically about events where n=0. Forecasting is not applicable here. Humans can reason through rules-based outcomes where n=0, like sending a man to the moon using physics, where everything can be experimented in smaller scale and where each engineering challenge proceeds logically and nigh-deterministically from the previous one. But for AI and nuclear war the productive lens seems to me to be "what follows from the current state" and not "what percent chance should I apply to X outcome" because you have nothing historical to base that on.
> All of my usual arguments have been stripped away. I think there's a 33% chance of AI extinction, but this tournament estimated 0.3 - 5%. Should I be forced to update?
Can't compare without error bars. You don't have to update unless the one estimation definitely precludes the other, and these two predictions could be in agreement if the error is large enough. Which of course it is, nobody has credible predictions here much less any sort of justifiable quantification of their feels.
Even worse we are talking about probabilities, even if we had actual real data there would still be questions about the underlying probability distribution. Let's say it turns out there is a trick to things, AGI is indeed possible with moderately more compute, now what was the probability of that being true? What are we even talking about here, what is this probability distributed over, the quantum multiverse?
It sounds more about "confidence that something will happen" than "probability that it could happen". Maybe seems semantic but imo important. Then you should ask why these experts are quite confident and you are not, and whether you are convinced by their arguments (or are you just inclined to reasoned hedging where they are intentionally making bold statements).
It's misleading to infer from Metaculus that superforecasters see anything like a 50% chance of superintelligence by 2100.
Metaculus uses a weaker definition of superintelligence than is standard. Wikipedia says that superintelligence means "intelligence far surpassing that of the brightest and most gifted human minds", whereas Metaculus includes merely matching (average?) human abilities.
Even if the superforecasters agreed with the Metaculus forecast, I'm pretty sure they'd find some reason to expect "superintelligence" to have only minor effects on normal life by 2100.
Look at question 52 (Probability of GDP Growth Over 15% by 2100): the median superforecaster probability was 2.75%, versus 25% for domain experts. That's an important disagreement over how much impact AI will have.
My impression is that the median superforecaster will continue to find reasons for expecting low impact, even when many more human tasks are automated.
I'm guessing that this is either the result of a conceit about human uniqueness, or it's the result of a strong prior that life will continue to be normal (with normal being a bit biased toward what they've lived through). This heuristic usually helps superforecasters, but the stock market's slow reaction to COVID reminds us of the limits to that heuristic.
Someone explain to me how improving verbatim and associative recall magically transcends to godhood.
Someone explain to me how a computer program would even know that it's "In a computer."
Someone explain to me how AI can manifest architectural or counterfactual thinking, or imagination.
Every post I see from AI alarmists boils down to "well if we make it more smarterer then it will exponentially be intelligent to apotheosis." They need to stop reading G studies. You don't just "add intelligence" by increasing fidelity of associative reproduction + data and produce superintelligence. This is the kind of impoverished view of the mind I would expect from a middle school student.
For both AI doomerism and climate change doomerism, the people most concerned about them tend to also be the ones discussing them the most and have the greatest familiarity with all the usual facts and arguments that would be thrown around in a debate. For a normie on the outside looking in at those communities, a huge question is whether A) they started with a desire to learn about Danger X, and learning more about Danger X made them more worried, or B) their pre-existing concern about Danger X was irrationally high, and led them to spend more time on it, to try to save the world, or C) they've become an insular community wherein worrying about Danger X is just a feature of the subculture.
Concern over AI is the one thing nearly the entire LW-sphere tends to share. I'm not in that sphere, but since you're closer to being my "tribe" than the average climate change alarmists, I tend to assume YOU are being genuine while THEY are people acting under bad incentives; that what scares YOU guys must be actually scary, but those other guys are either lying or foolish to be scared; and therefore I worry much more about AI than climate change. I'm able to see this cognitive bias, but maybe not to update out of it.
Within my sphere, there are definitely people who talk about you guys as having a cult-like devotion to AI doomerism, but you also have been studying the dangers of AI so much longer that I still want to trust your opinions. You look like the "experts" in AI danger to me, and present a seemingly broad range of ideas on the topic, but you're the only people I ever encounter who are presenting it so how can I know? When the only people talking about AI dangers are people who are so heavily invested in it as the LW bloggers are, how sure am I really that Scott's 33% is so obviously better than the other folks' 5%, I mean I wanna say it's because obviously Scott and his cadre have spent longer thinking and arguing about this and are clearly smarter people. But I'm sure the laymen who instead think global warming will destroy the earth must believe the same thing about their cult of experts, and probably for similarly biased reasons.
A number of people seem to be fond of the idea that "killer AI" (or, alternatively, malevolent aliens) will nail humans with memetic -- rather than kinetic -- weapons.
Has anyone considered the possibility that this weapon has already fired? (Who, precisely, fired it, is immaterial. Could just as readily, and more parsimoniously, have been human elite hands.)
The "nuclear safety" meme is already creating energy poverty and reinforcing the hegemony of fossil fuel peddlers.
I participated and can definitely support the less wrong commenter that there should of been fewer questions. At least in my particular group, participation sharply fell off after the first round which meant discussion was limited to maybe 20% of the group and many questions only had 2 or 3 people answering. This made it a lot more difficult to dive deeper and have more comprehensive conversations about why people chose the forecasts they did. It looked like in the later rounds seeing other groups answers that participation was pretty variable so maybe other groups had better luck.
I see a glaring problem of looking at only one side of the ledger. Yes there is a tiny (but significant) chance of AI causing a catastrophe as defined in the exercise. Yes, there is an even greater risk of other (non AI) natural or man-made catastrophes over the next century. What is not being estimated is the likelihood of AI being used benevolently by humans (or via the will of the AI) to counteract or prevent other catastrophes (which could be natural, man made or even possibly from another AI).
The costs and benefits of AI must include not just the risks of the AI, but the potential life saving and catastrophe averting benefits. Any analysis looking at only one side of the ledger is pretty much useless.
My opinion, FWIIW, is that there is an extremely high chance of stupid humans using power and technology to cause massive and horrifying catastrophe. This can be intended or not. It is just a matter of time. I would strongly argue FOR machine intelligence to help us prevent these risks. Today we have great power and great stupidity. I think the universe would be better off with great power and great knowledge. AI is the path forward.
I think there is a huge hidden incentive to downplay the extinction risk for most people: if you agree that the risk of extinction is huge rather than "negligible" (and even 5% falls more in the psychological category of "negligible" unless people thought about long-termism before), under reasonable ethics, they should be forced to change what they do in their lives, maybe radically, switch careers, lose prestige, etc. People don't like doing that.
Whenever I try to convince my friends about a significant extinction risk it usually falls on deaf ears, I think mostly for this reason.
To an arachnophobe, spiders will do bad things. You can give them all the arguments in the world, and they can gradually accustom themselves to being in the same room as one, but you won't get them to update below a minimum that seems crazy high to you.
The idea that 10% or even 1% of human thinking has been superseded seems unthinkable to me. My estimate is much less than 1% - none of which actually relates to real things. Maybe high school English essay composition homework assignments, or Associated Press releases have some AI component.
Yet you seem to be persuaded that ChatGPT (a program that cannot distinguish between real things and not real things) has taken more than 10% already. If this is the case, then I wouldn't be surprised if airplanes start falling from the sky in the next week or so.
Sorry if this seems confrontational - I am firmly of the school that we have not seen anything like AI now. After reading and arguing about the recent developments it seems further away now than it was 10 years ago.
Thank you for this. I confess that I took the results seriously, without bothering to do a deep dive into their paper, but you in combination with horror stories from participants in this comment section convinced me that I should not.
To join the pile-on, I find their definition of AGI (question 51, page 225 and 694), deeply problematic. As you note, it is "whatever Nick Bostrom declares to be AGI". Or, if he would be unavailable (very plausible scenario for the year 2100), vote by a panel of 5 experts "with a deep knowledge of Nick Bostrom’s work".
"What I wanted was a way to quantify what fraction of human cognition has been superseded by the most general-purpose AI at any given time. My impression is that that has risen from under 1% a decade ago, to somewhere around 10% in 2022, with a growth rate that looks faster than linear. I've failed so far at translating those impressions into solid evidence."
I find this argument vague in a way that is useful for explaining my skepticism about AI arguments. If we think about what 10% of human cognition could mean, I think there are a few possibilities. One is something like, the AI can do 10% of the stuff that humans meaningfully cogitate about as well as a human could (or, maybe, we could produce the same amount as we do now as a society when replacing 10% of human cognitive-hours with AI-hours, weighted by hourly wages/some other measure getting at the quality of cognition).
The problem with a definition like that is that at any given moment, human cognition is being used to solve problems that mature technologies haven't solved for us. New technologies that replace cognition are going to look, at the moment, like startling advances that close the gap between human and machine, even if in the long-run they look like important but limited technological advances.
Think about the cognition done during the Apollo space missions. My suspicion is that the vast majority of the human cognition-hours expended during the missions were spent doing calculations--arithmetic, algebra, and calculus. We've completely automated arithmetic, and have for a long time more-or-less completely automated algebra and calculus. From the perspective of human cognition as-of-1969, 2010-era technology had likely automated more than half of human cognitive tasks in the most advanced, technologically sophisticated areas of human endeavor.
What's happened since isn't that arithmetic, algebra, and calculus have stopped mattering--our current society depends on absolutely massive numbers of calculations, performed continuously, across pretty much all sectors of society. If we wanted to think about how much *extra* human cognition we would need to keep our current society but fully replace computers, the numbers of people would be mindboggling (trillions of extra humans calculating all day long would barely scratch the surface). We just don't think of this stuff as core, human cognition anymore because it's all been automated in such a mature way that humans don't really do it, or only do it in very simple forms, on the fly, or as an exercise in school.
So, GPT-4 can summarize text very effectively, can write code very effectively, and can write essays/emails/etc. moderately effectively. This is new and exciting. It's reasonable to imagine that even if the technology just matures a bit without becoming dramatically better, writing code yourself will become as anachronistic as doing arithmetic by hand, as will doing paralegal-type work of looking for all relevant cases or articles on a topic and writing brief summaries of each. If that's more-or-less as far as the technology advances, we'll end up adapting to the fact that these areas of human cognitive effort are essentially freely automated, will build our society and economy around effortlessly performing these tasks, and will stop thinking about them as difficult or costly tasks in the same way that we've stopped thinking about arithmetic as difficult and costly.
While I of course don't think that this means that LLM-style AI is destined to get better at the stuff it's currently good at without turning into a superintelligence or completely reordering society, I do think it lends some credence to the outside view. It's really not enough to say: "look at how fast this technology has progressed--imagine where it will be in 10 or 20 years," because sudden and dramatic improvements in the ability to do work that is hard for humans isn't unprecedented or new. And it's not enough to argue that there's no clear and obvious limit to the current technological paradigm, because people didn't see a clear and obvious limit to the capacity of calculation machines in the 1960s, 1970s, and 1980s either. We discovered those limits and started articulating differences in the kinds of problems that calculation machines could and couldn't easily solve by hitting the limits.
To me, this implies that the argument for treating (roughly) current-design LLMs as the harbingers of true general intelligence rests on an understanding of the structure and design of LLMs themselves, and in particular on an understanding of how that structure can be adapted to doing stuff that current LLMs aren't great at. My understanding (which could be wrong) is that we have a lot of understanding about how LLMs are trained, and something about how they behave, but we don't have much of a clear idea of how they "think," or of how the stuff they're currently not as good at can be achieved through an intensification of the stuff they can already do well.
I participated in this tournament, but had completely forgotten about it until now. I suspect I was a poor-quality outlier, but I can give my perspective.
I first filled out their survey because Scott mentioned it way back in Open Thread 222. I have no forecasting experience and had no interest in the monetary incentives, but I did want to debate some of these big questions with serious people. I was a little surprised to be accepted, and having read the writeup I'm even more surprised. I'm definitely not a superforecaster, so I guess I was counted as an "expert". And I am absolutely an expert software engineer. But while I've worked with ML systems for many years (and have a decent understanding of the nuts-and-bolts of AI), I don't understand why I'd be considered a "specialist on long-run existential risks to humanity." This is not to cast shade on the tournament in general - the people I interacted with clearly belonged there - but I would probably fall into the category of expert that Peter was unimpressed by.
As Peter's writeup mentioned, one issue with the tournament was that it wasn't very focused and went on for too long. Everyone was required to forecast the big x-risk questions, but there were a whole lot of other questions that we didn't/couldn't spare much attention for. And while I responsibly spent hours researching my questions early in the tournament, as the months went by my enthusiasm waned, I procrastinated a lot, and I didn't participate in the discussions/persuasion nearly as much as I should have. (I think this might have been a common problem, based on the communications we received.) So, we had some good debate within our team, but it was nothing compared to, say, getting all of us into a conference room for a weekend of lively discussion.
Incidentally, I did update a few of my forecasts based on the intra-team debate, and others did too, so some persuasion DID happen. I personally didn't see much happen after the inter-team write-ups were published, though (which might just be due to my own laziness).
As for my actual predictions, most of them were pretty close to the expert medians. One exception was pathogen risk, and I'm confused by the legend of "Table 2" that Scott copied from the paper. The other "catastrophe" questions asked about the odds of 10% of humanity dying in a 5-year period, but the pathogen questions they asked only required 1%. Which is MUCH more likely, and indeed this threshold was significantly exceeded by the Spanish Flu (even in the less-globally-connected era of 1918).
As for AI risk... well, I sadly think most of this community is blind to the tower of assumptions they're building on. Engineering is hard, technological development is hard, and an AI trying to FOOM itself to godhood without being noticed is not immune. My tournament prediction for AI extinction by 2100 was 3%. This was before ChatGPT's capabilities absolutely shocked me (even though we'd seen some impressive things from GPT3 already, I definitely did not predict the capabilities of 2023 LLMs). My current prediction is ... more like 2-2.5%. Yes, I updated downwards, because LLMs suggest that even the first floor of the tower of assumptions - that capable AIs can only be agentic beings with dangerous instrumental goals - is wrong. (In some sense, the "orthogonality thesis" is even more right than Bostrom predicted.) Knowing how ChatGPT works, I am unconvinced that even a perfect GPT-infinity would pose an extinction risk (except by intentional human misuse).
I suspect this reveals more about the inherent pessimism of humans than any actual understanding of the actual risks. Having lived through half-a-dozen End-of-the-World-as-We-Know-It (EotWaWKI) events, I've become as cynical about the current crop of EotWaWKI predictors as I am with the other false-prophets that populate modern America...
About 39% of US citizens believe that we are living in the end times. With ~10% it will happen in their lifetime (which I assume means the next 20-50 years).
However, 90% of subject matter experts believe we are living in the end times.
I also participated in this tournament. I was fortunate to speak to some AI experts at a conference last fall, while in the midst of this tournament. When I asked about extinction risks at the conference, I heard some crazy responses. One that if there was a problem we'd simply turn it off. Another saying that no one is trying to remove the human from decision-making processes.
At this point, I realized how differently some people saw AI risks.
I think instead of letting the inside view and outside view each act on you separately, you're supposed to actually have them interact. Dive into the Outside View and consider what differences there are between this case and a usual case, and whether these hold up or not. That leads you to a principled understanding of why you can disagree with experts here.
For example, why am I permitted to disagree with expert forecasters on AI forecasting?
1. We’ve been discussing AI forecasting with experts for a decade and they all slowly and predictably increase their odds of AGI-soon and AGI-doom over the years. This is NOT true of moon hoaxes. For things it is true of, I’m happy to be in the vanguard of early updaters.
2. The people I know who believe in large AI risks have many separate axes on which they do better than experts. For one, they are often better at forecasting than superforecasters (in my experience, superforecasters are often pretty mediocre). But they also tend to be more correct about many other provable things (general factor of correctness), or have absurdly high IQs and prestigious degrees if you're into those kinds of things. This is NOT true when going against the experts of moon hoaxes. For things it is true of, I’m happy to be in the vanguard of early updaters. If you say that these appeals to certain special traits are no different than those a moan hoaxer could make (or a CDC-believer would make), I can say many other reasons why my process is better than theirs: I've put many hundreds of times more time into thinking about which axes cause people to be trustworthy than they have, my choices win betting tournaments and theirs do not, etc.
3. AI risk is a legitimately far more difficult field to forecast in than nuclear or bio risk. Those fields have great data, metaphors, containment plans, and existing literature. Superforecasters rely on this. To zero-shot a completely new scenario is extremely different and requires a separate type of theorizing expertise. In fact, I'm willing to go so far as to say: to think about minds without anthropomorphizing is itself is a difficult-enough bar that it requires its own expertise—and then the additional difficulties of data-less literature-less forecasting are easily another full reason it requires its own expertise. You can know this because in many risk assessment fields, experts do make good arguments right off the bat, yet I’ve seen many “computer science experts” make very false arguments when first trying to think of superintelligent AI, in ways that convince me this is not a normal field where normal experts have expertise. Again, this is NOT true of moon hoaxes. I'm happy to be contrarian against experts or forecasters in places where there is minimal feedback, aka punditry fields (and indeed I'm contrarian about many aspects of effective political organizing).
(And then see my other comment for additional reasons why I think superforecasters are not necessarily calibrated on low-probability risks, and likewise with experts I do not especially trust them here.)
Apocalypses are religious literature. The really bad news is that this old world is going to continue to continue, and each of you will have to head the words of the Buddha. All of us will suffer the death of our physical bodies, each of us must work out the salvation of our souls with diligence.
For me, the main problem is that an expert in a field I am not overly familiar with who is expressing an incorrect conclusion looks and sounds exactly like an expert who is expressing a correct one. It seems almost impossible to tell the difference without oneself also becoming an expert.
I happen to be person Peter was talking to about Pearl and commenting about. I also did the more recent follow up study with FRI, which was more focused (only covering AI), in depth and with (I think) better experts than the original tournament.
The main variance between the AI sceptics (of which I am one and in these projects were mainly, but not only, super-forecasters) and the AI idealists (entirely experts in this case and Peter wasn't part of the follow up) relates to how difficult extinction or even catastrophe might be and whether scaling up current models and incremental efficiencies is sufficient for this risk to be meaningful.
The sceptical camp believes that a number of steps have to be achieved and taken in wrong directions for extinction risk to be realised. The core thesis of the idealists is that scaling has been sufficient to develop AI capabilities and will be sufficient to develop AGSI and that the existence of an AGSI has itself a high(ish) exitinction risk for humans.
(It's also important to say the 0.5% type extinction risk forecast by sceptics is not a statement that AI is safe. All of the sceptics in the follow up project saw cause for concern and believe AI risk is high enough to justify lots of spend on alignment work and taking the threats it poses seriously).
For what it's worth (and I may not check in for comments having some fatigue on the subject and other things to do) this is my own summary of the sort of views expressed by the sceptics:
Steps toward AI risk
Development of AGI - it's reasonable today to disagree about how far away we are even if the probability of AGI is well over 50%.
Development of ASI - we don't know what would be required to move toward this. Maybe it's easy, maybe not, it's reasonable to think its a lower probability than AGI.
Development of AGSI - it's reasonable to question whether this follows automatically from AGI or an initial ASI, and to also consider what AGSI might look like and require (e.g. it might need to be trained on whole world data which is not happening any time soon or with the kind of AI systems we have).
Use of AGSI. It seems reasonable to wonder whether a single AGSI framework might evolve into various separate counterbalancing systems and that a single general system might not become 'masterful'. It seems reasonable to consider a probability that AGSI's might be more capable as isolated systems trained on domain specific data and run separately.
Developments in robotics and sensors - These are preconditions of AI becoming part of the real world and it's reasonable to question what progress is required for this to become reality.
Cost and power efficient development of AGSI and Robotics. Things may be too expensive or have uncertain power and energy needs and so breakthroughs are probably required in this area as well for adoption of AI, AGI, ASI and AGSI to make its way into the world.
Global infrastructure development - few parts of the globe have sufficient infrastructure for the kind of AI enabled world that would become risky for humans. Environments are not always suitable for machines (they are not ideal for humans either, but we live all over). Would AGSI get everywhere humans live? Lots of current technology hasn't.
Society acceptance of AGSI - people accept some systems (electricity) but not every proposed system - see nuclear energy.
Political and military acceptance of AGSI - will countries give up their power just like that?
Management of global resources devolved to a single AGSI - this is still a long way from AGSI existing in the first place as the other steps should have made clear.
A failure to adequately monitor or control any AGSI. I think it's reasonable to suppose we would be trying very hard to make this a success. The AGSI would likely need to be deliberately fooling us and therefore malicious and also agentic/sentient and it's reasonable to question whether there is a high probability of this being the case. Thought experiments in philosophy departments are nothing more than that.
That an AGSI can eliminate humanity without us noticing or so fast that we can't respond or is so powerful we cannot respond seems conceivable (I just wrote it after all) but hardly the only possible scenario. It's reasonable to wonder how easy it is for the AGSI to reach the hurdles and what its motivations might be. Certainly, the AGSI would need to develop a very fast and effective method of killing which requires its own breakthroughs and improbable scenarios (I could list several steps for this itself, but this post is already long) and have agency and very likely much more than poorly managed goals (as per convergence theory).
One illustrative way of explaining the difference between concerned and sceptic camps is that the difference between c30% and 1% extinction risk is to give a range of 70%-90% probability to each of the events listed above.
Obviously, that's a very simple way of looking at it. Individuals probably believe some of these steps are nearly certain and some are more likely (incremental steps) once others are achieved.
Nonetheless, I don't really believe any of the steps follow in a self-contained logic statement with 100% certainty from the other. Other people may disagree. There can also be risk without a particular step, but to end up with a single masterful AGSI its hard for me to believe we can miss many of them.
One explanation that came to mind is the standard explanation for the popular observation of 'Experts keep saying there will be a crisis but it never happens', which is that the experts are agitating for change, they succeed, and the crisis is averted.
Is it possible to experts are arguing/estimating from an inside view of 'this is what will happen if we, the experts, don't fix it,' while the forecasters are estimating from an outside view of 'this is what is likely to actually happen, including as a result of all expert's efforts to fix it'?
It seems to me that this would be the standard mode that before groups are used to thinking about and working on these problems from - the experts trying to plan their own efforts to fix the problem and convince people of the need, vs the forecasters including those efforts in their predictions.
Existential risk is obviously a scam. Domain experts have a vested interest in raising the profile of their pet topic. "It's gonna kill everyone" is the easiest, cheapest, most manipulative way to do so. It's so easy, you don't even have to consciously know it's a trick to be able to pull it off.
There is a certain analogy to judging the trustworthiness of forecasters by their their predictions of well-understood bio risks. It is sort of like testing the alignment of an ASI by asking it questions about a situation you understand, and hoping that it will also act in your interests in more complex scenarios.
For the forecasters, the most you can learn from that is that they are not stupid. The historic frequency of pandemics is common knowledge, if a superforecaster can take a reasonable guess at the mean time between pandemics, you just learn that they are not statistically illiterate. Just like an ASI telling you "I am not allowed to crash the moon into the earth" does tell you little about its alignment, just sets a mild lower bar on its intelligence.
Re the non-anthropogenic catastrophic risks: 0.1% looks very low to me. Carrington events happen. If there is a 0.5% per year probability of a Carrington event, over 70 years this is around 35%. If the power grids of the world (and quite a lot of other electrical and electronic equipment!) are knocked out for a few months, a death toll of 10% looks quite plausible.
Experts will be more worried about their domain than outsiders
1. Selection bias: people worried about x are more likely study it.
2. Economic pressure: the field as a whole will command resources in proportion to the seriousness of the problem it is solving
3. Partisanship: People become partisans when they dedicate their life to a problem and partisanship makes people think less clearly
4. Availability heuristic: Domain experts think about their problem more often which skews their view of it.
Anecdotally, I knew a postdoc who studied food born illnesses who monitored his personal refrigerator and would throw away everything if the temperature briefly exceeded 4°C.
Well, I was wrong about 1935. Actually, the physicists did were wrong 2-4 years earlier.
"There is no likelihood that it will ever be possible to harness the energy within the atom." - Arthur Compton, 1931
"The energy produced by the breaking down of the atom is a very poor kind of thing. Anyone who expects a source of power from the transformation of these atoms is talking moonshine." - Ernest Rutherford, 1933
"The idea that atomic energy is available for practical use is just an illusion." - Albert Einstein, 1932
The fact the fission was not yet discovered is irrelevant. That's like assuming that no new discoveries (like Transformer) will be made in the field of AI in the next 5-10 years, that AI is going to be created exactly with the same methods it is being created now. Obviously that is a very comfortable assumption, but totally irrational. There are going to be new discoveries, new methods and much more powerful computers - but of course AI "will never be a threat to humanity because I say so" LOL
> When we asked them to forecast again, conditional on AGI coming into existence by 2070, that figure rose to 1%.
This is the most damning part of the predictions. It implies they're 75% sure that AGI isn't going to arrive before 2070, which is just not a reasonable prediction if you've been paying any attention.
> Existential risk expert Toby Ord estimated a 16% total chance of extinction by 2020.
That's certainly pesimistic but maybe you mean 2200?
>Existential risk expert Toby Ord estimated a 16% total chance of extinction by 2020
I don't see that in the cited link. The link refers to a book published by Ord in March of 2020. Was he claiming a 16% that humanity would go extinct in the next 9 months?
I'm no super forecaster but I have read most of your posts on AI as well as Eliezer's. Glad to see the superforecasters had the same takeaway as me.
New subscriber here. I've been living under a rock to have just come across your blog. Thank you for the work that you do!
I was one of the participants in this study, and I broadly agree with the critique of McCluskey, though in my case it was from the opposite direction in that I found most of the AI-concerned camp tended to only have a surface level understanding of ML, and hadn't engaged with substantial critiques from the likes of Pope:
https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky
In general, the tournament, and the supporting online infrastructure, wasn't really designed to get deep into the weeds of particular disagreements. Mostly this was fine, where disagreements were of degree rather than kind, but were much more of a problem when it came to the emergent risks of AI where participants had fundamental disagreements.
FRI became aware of this issue, and scheduled a follow-up exercise more tightly focused on AI which has just recently concluded. Much more effort was put into facilitating deep dives into the sources of disagreement between the two camps. I wrote the following towards the end of my involvement in that exercise:
https://alethios.substack.com/p/is-artificial-intelligence-an-existential
Re when to follow experts...
I think that in deciding whether to do so, you're also allowed to consider things like (i) how much expertise the experts have, (ii) how hard the thing is, (iii) if applicable their track record.
Pilots presumably have a lot of expertise, landing a plane is clearly a thing within the ability of pilots, they do it well all the time, and you (I'm guessing) don't know any of that shit. But on the other hand - how much do you trust a professional stock-picker like Jim Cramer? I bet he knows a lot more about the stock market than you do! But his picks don't do better than random chance, and same with basically every other stock picker like him.
There are specific reasons for stock pickers being bad that don't apply to existential risk (e.g. if stock pickers were good everyone would follow them until their predictions immediately moved the market and zeroed out the advantage). But more generally, I think it's reasonable to say "in this domain, predictions are hard and they don't really have expertise, I'm not following them" and that doesn't make you the I-wanna-land-the-plane guy.
Writing this before reading the article as I don't want to be primed by Scott's analysis.
I signed up for this tournament (I think? My emails related to a Hybrid Forecasting-Persuasion tournament that at the very least shares many authors), was selected, and partially participated. I found this tournament from it being referenced on ACX and am not an academic, superforecaster, or in any way involved or qualified whatsoever. I got the Stage 1 email on June 15.
I put a lot of thought into the first prediction round but quickly grew disinterested in the whole thing during the first discussion/wiki round when nobody, myself included admittedly, built any discussion. I do think I at least entered in good faith planning to actively participate through to the end, but ultimately did not do so. I sent an email to one of the organizers requesting to be withdrawn and dropped from all mailing lists, he very quickly agreed to do so, however I appear to have remained on the mailing list for from what I can tell was the remainder of the tournament anyway.
The one thing that still stands out to me, albeit groggily, was the tedium of repeatedly attempting to re-word my overall skepticism about existential risk to fit slightly different questions about it. At least in the beginning, when I was most enthusiastic, it felt like the questions and design encouraged me to genuinely be concerned and give weight to there being substantial X Risk, without any attempt to justify the position implied by asking when ~60 different things will cause extinction. If anything it felt a bit like a push poll.
I'm not sure what the point of this anecdote is other than I am a bit skeptical in the value of this tournament. Admittedly that entirely matches my priors of overall skepticism towards superforecasting and prediction tournaments so I may have been biased at the start, which could have played in a big role in my failure to remain engaged and motivated. I don't have anything against the organizers, it was a new and interesting experience for me, but I do still question the merits of prediction tournaments and superforecasting in general. However to reiterate I am not an academic, expert, or superforecaster, so it might have been somewhat of an accident that I even got selected. I don't remember how I answered the initial survey but I believe I answered everything truthfully.
Edit: To be clear, as far as I recall I answered all of Stage 1, did a bit of stage 2 but never actually discussed any of it with any of my 'teammates', and sat out the remainder entirely. I do not know what they used for the paper or if any of my contributions were included.
I think AI risk is so scary because it doesn't have to follow the 1) human level AI 2) AI misalligned 3) AI kills all model. There are just so many other ways.
1) sub human level AI like infinite paperclips is super efficient at reproduction/ adaptive at surivival in spite of not being particularly intelligent, wipes us out.
2) sub human level AI that still makes really good weapons and in AI arms races it ends up wiping us out in the middle of a great power conflict. Self driving cars are better than humans at 99% of driving tasks already, but there are some areas where humans are still preferable in the driver's seat and we wouldn't call the current driving algorithms "human level GAI". Some AI soldier/ weapon could be better than humans at most tasks, in spite of not being "human level" on say a conversational or logical level and could somehow get out of control in it's seek and destroy tasks (it could definitely beat our reaction times for example).
3) Human GAI is never misalligned, but it- in spite of being "human level" is, like us, not omniscient. As we all bow down to it and turn more of our corporate/ political decisions over to it, it accidentally leads us off some cliff to geopolitical crisis. Also all these ideas about GAI allignment are incredibly absurd and naive, it's like saying you can put a specific kind of safety on an assault rifle that guarantees that it only shoots animals, it's insane. Isaac Asimov's fun ideas in his Foundation series really shouldn't be taken so seriously, they're 100% not going to work.
Anyways, lots of other ways things could get scary very very fast. The way AI affects thing is going to be very unpredictiable, which is partly why the risk from it is so hard to estimate-
I haven't read the full paper quite yet, but to beat the rush:
>Are you allowed to look at a poll of all the world’s top experts plus the superforecasters who have been right most often before, correctly incentivized and aggregated using cutting-edge techniques, and say “yeah, okay, but I disagree”?
You are if the "superforecasters" aren't. I participated and AIUI got counted as a superforecaster, but I'm really not. There was one guy in my group (I don't know what happened in other groups) who said X-risk can't happen unless God decides to end the world. And in general the discourse was barely above "normal Internet person" level, and only about a third of us even participated in said discourse. Like I said, haven't read the full paper so there *might* have been some technique to fix this, but overall I wasn't impressed.
"We did not ask about catastrophic or extinction risk due to climate change based on expert recommendations because the impacts would be too slow-moving to meet our “catastrophic” threshold (10% of humans dying within a 5-year period), and in pilot interviews, climate experts told us they would place an extremely low probability on extinction risk from climate change. Instead, experts believe that negative effects of climate change will occur over a protracted period of time, and may be indirect, which puts their effects largely beyond the scope of our study." Shame, would have been interesting to see some estimates, especially around changes in economic welfare.
As some of us consider whether to update based on the results of this forecastathon, I think it's important to bear in mind that these forecasts are quite different from the ones we and the forecasters are used to, and the ones superforecasters demonstrated their chops on in the past. First, many of them are forecasts about how things will be one to seven decades from now. As far as I know, forecasts that experts have made in print, generally as part of some larger work, about the world even 10 or 20 years in the future are mostly quite inaccurate. Maybe those experts erred partly because they hadn't seriously considered how to make an accurate forecast, but I think the bulk of their inaccuracy can be accounted for by our just not being able to see what's beyond the horizon. We don't what butterflies are going to flap their wings where, and how the chaotic system that is life on earth will be affected. Anyone know what's the furthest-in-the-future forecast superforecasters have been tested on? Is there one that was even for 5 years in the future?
The second reason to think about forecasting here as being a different task from all other forecasts is that forecasts about AI are forecasts about something so novel that there are no good models to use to get a rough picture of how things might play out. Cars, nuclear power, the Internet -- they were all novel and had a huge impact, but compared to a machine with human level intelligence or superintelligence, they are small fry. Predicting how things will play out if we develop AGI or ASI is like predicting what would happen if an alien spaceship landed in a Vermont meadow and aliens emerged from it. ("Well," asks the superforecaster uneasily "what would the aliens be like -- are they friendly or do they immediately eat a few cows and people?" "Well, we dunno," says the MC. "That's part of what you have to forecast.")
So I don't think it makes sense to consider the estimates of the superforecasters the way one normally would, as a real challenge to one's own estimate, especially since these superforecasters sound like they were ill-informed even about the aspects of AI about which it is possible to be informed.
Perhaps the delta is coming from all of the different scenarios pulling on each other. The superforecasters are generalists - they learn about the sum of all of these scenarios at more than a shallow level. They may not go as deep as an expert, but when they produce their predictions, they are holding the thoughts of many experts in their minds. The expert may be biased towards his own field of research.
Or, put another way, we can't go extinct by AGI *and* a gain of function virus getting loose. The more likely a superforecaster thinks the AGI scenario is likely, the less likely he thinks a lab leak will kill us, but the expert does not think like that.
I think it's reasonable to ask "are these people actually experts, are there misaligned incentives, and is there something involving tribal loyalties/political affiliation going on here?" before updating. For a pilot the answers are no to all three. For the reality of otherwise of the moon landings you don't have to defer to experts, but if you did, you would have good reason to be suspicious of them.
For AI, it seems like the experts are actually experts (even if it's not clear that many or any of them took part in this exercise), even those with misaligned incentives are speaking out about risk, and so far there doesn't seem to be much tribal loyalty/political affiliation stuff going on. Though I don't think the last one can be sustained.
"Many of the people in this tournament hadn’t really encountered arguments about AI extinction before (potentially including the “AI experts” if they were just eg people who make robot arms or something), and a couple of months of back and forth discussion in the middle of a dozen other questions probably isn’t enough for even a smart person to wrap their brain around the topic."
Oh come on. AI extinction arguments aren't quantum physics. There are no 7-page proofs, no 11 dimensional strings, nothing with any basis in either scientific theory or experimental data, just a bunch of wild speculations based on flimsy premises. Is there a 10 year old out there who can't understand the paperclip maximizer? Conversely, is there any expert in the world who has the foggiest idea how the first superintelligence will actually be made?
"But when I hear their actual arguments, and they're the same dumb arguments as all the other people I roll my eyes at, it's harder to take them seriously."
That's exactly what I think about the AI doomer arguments. I bet the participants in the tournament are well familiar with AI extinction, if not from Nick Bostrom, then at least from popular media like the Terminator or Battlestar Galactica. They just don't find the idea plausible.
>This study could have been deliberately designed to make me sweat.
I'm surprised by this characterization. I mean, there's an emotional sense in which, even as someone who wants to be swayed by truth rather than confirming one's own preconceptions, it's hard not to want to just already be right and not have to revise your position. But for me, when I read about the study construction, it gives me a glimmer of hope, because I don't *want* to be stuck believing that we're likely headed for an AI apocalypse! It's an extremely grim belief to have to live with! Reading that the study participants weren't actually well-versed in the arguments or research around AI risk was extremely disheartening.
>Another compromise is to agree to generally act based on the Outside View in order to be a good citizen, while keeping your Inside View estimate intact so that everyone else doesn't double-update on your opinions or cause weird feedback loops and cascades.
This sounds like a description of the attitude that causes nocebo effects, in the situation where Scott is the expert and his patients are the forecasters.
Doctors reading this - do you ever have conversations like this with your patients? Or do you just infer that it's what some of them are thinking?
For me, a high risk of AI-extinction seems implausible (at best) because... how? Let's say we end up with a very powerful evil AI who wants to wipe out humans. That is a very long way from it managing to actually cause something like the Toba catastrophe. THAT is where the "and then magic happens" step seems to be, for me. There is a long and well documented history of human-level intelligence trying to inflict genocide on other humans, and one of the reasons they're so well documented is because there are survivors who do the documenting. Do thousands or even millions die? Yes, but that is still a long way from extinction.
If the AI makes bioweapons, it needs to stop anyone slamming borders shut or coming up with a treatment, while also somehow managing to make robots that can do all the maintenance stuff the AI requires to stay "alive". (Maybe it's a suicidal AI that doesn't mind "dying" to wipe us out, but that still requires it to stay alive long enough to achieve its goals.) And also it has to hope the bioweapon doesn't mutate into something less lethal or run into any weird genetic twists that provide resistance.
If it wants to hunt down humans it still needs drones or robots or tools to do so, which is going to be challenging; supply chains are REALLY EASY to sabotage. The reason supply chains are often remarkably resistant to things like natural disasters and wars is because things are always going wrong and people in the field are always routing around problems (this was a significant chunk of my last job).
If it wants to cause a nuclear winter it's going to learn there's not actually enough nuclear weapons on the planet to pull it off to a true extinction level, never mind all the technical issues with activating air-gapped systems. Again, wrecking entire countries and undoing a century or two of progress? Absolutely! Causing billions of deaths? Plausible. Dropping Homo sapiens below ten thousand? Dozens of countries have a much larger rural population who are already food, water and electricity independent, and only have to shelter in place for a few days to avoid the worst of the fallout. To me, it's a bit much to assume nobody would manage to adapt to a rapidly shifting climate, considering people have already done it in the aftermath of volcanic eruptions that have caused similar localised shifts.
> The Inside View Theory Of Updating is that you consult the mysterious lobe of your brain that handles these kinds of things and ask it what it thinks.
The Frequentist View Theory of Updating is that you list all imaginable ways things can go every which way, to the degree where you assign equal (and small) probabilities to each one, making each path in the whole graph of possibilities as atomic as possible, then literally count the fraction of paths resulting in the outcome you need an estimate for.
I really appreciated the introspecting about updates at the end
My spouse is a domain expert in pathogens, and he was recently an adviser for a student looking at extinction risk from pathogen risk - we generally agreed extinction risk in the next 100 years is very low.
My degree is in ecology and evolutionary biology and extinction in general is a pretty long, drawn out process, especially when you're starting with such a huge population size - extinction risk scales with population size.
I could definitely see an event occurring in the next 100 years that would drop human population size down to pre-industrial levels - say, 100,000 - and that could definitely lead to our eventual extinction. But such a process could easily take another 1000 years or more after the initial event.
And in terms of natural pathogens, these rarely cause extinction in of themselves. They're typically more of a final blow to an already-small population. Designing an engineered pathogen specially to cause extinction is certainly possible but it's also not quite as easy as people think it is (not because we don't have the technology - we certainly do- but because of dynamics).
Daniel Kahneman: "Nothing is as important as you think it is while you are thinking about it."
Domain experts, exclusively thinking about their domain: "This is really important."
Most people in the Warsaw ghetto did not believe till almost the end that nazis are bound to kill everybody.
If you ran that sort of prediction game with experts end forecasters in 1938 asking them about probability that 90% of European Jews, 6 million, will ne wiped out in 5 years, what would the results be? 1%? 5%?
Or, if you want another example: you run predictions in 1935 that in 10 years an A-bomb is going to be constructed and used. From what I read the best phycisists believed it would be 0%. Not to mention superforecasters ..
Humans don't seem to disbelieve fatal incidents that have never happened before.
In the book Superforcasters, Tetlock was extremely down on the possibility of anyone making predictions more than a few years out. Superforcasters are proven to be able to peer into the mirk and predict the near future, but no one is making anything close to accurate predictions about the world of 70+ years from now.
> Can ordinary people disagree with “the experts”? [...] this is sometimes permissible [...] because the people involved don’t understand statistics/rationality/predictions very well.
as opposed to "ordinary people", who typically understand statistics/rationality/predictions perfectly? ;-)
The view that the Superforecasters take seems to be something like "I know all these benchmarks seem to imply we can't be more than a little way off powerful AI and these arguments and experiments imply superintelligence could be soon after and could be unaligned, but I don't care, it leads to an insane conclusion, so that just means the benchmarks are bullshit, or that one of the ways the arguments could be wrong is likely correct.
One thing I can say is that it REALLY reminds me of Da Shi in the novel Three Body Problem (who btw ended up being entirely right in this interaction that the supposed 'miracle' of the CMB flickering was a piece of trickery)
"We really have nothing to say to each other. All right. Drink!"
"To be honest, even if I were to look at the stars in the sky, I wouldn't be thinking about your philosophical questions. I have too much to worry about! I gotta pay the mortgage, save for the kid's college, and handle the endless stream of cases. ... I'm a simple man without a lot of complicated twists and turns. Look down my throat and you can see out my ass. Naturally, I don't know how to make my bosses like me. Years after being discharged from the army, my career is going nowhere. If I weren't pretty good at my job, I would have been kicked out a long time ago.... You think that's not enough for me to worry about? You think I've got the energy to gaze at stars and philosophize?"
"You're right. All right, drink up!"
"But, I did indeed invent an ultimate rule."
"Tell me."
"Anything sufficiently weird must be fishy."
"What... what kind of crappy rule is that?"
"I'm saying that there's always someone behind things that don't seem to have an explanation."
"If you had even basic knowledge of science, you'd know it's impossible for any force to accomplish the things I experienced. Especially that last one. To manipulate things at the scale of the universe—not only can you not explain it with our current science, I couldn't even imagine how to explain it outside of science. It's more than supernatural. It's super-I-don't-know-what...."
"I'm telling you, that's bullshit. I've seen plenty of weird things."
"Then tell me what I should do next."
"Keep on drinking. And then sleep."
I think you got the degree of updating about right, it matches what has worked pretty well for me on prediction contests in the past.
This belongs in the old category of "things I will regret writing," but for different reasons... Scott should be making a much bigger update.
This is kind of funny: self-admitted expert truster discovers he doesn't always trust the experts after all
What do you mean "are you allowed" to disagree with experts?
Of course you're "allowed" to have an opinion.
The question is whether that's a good idea.
Hey All — I was one of the participants in the “experts” group and am happy to answer questions about my experience if anyone is interested!
I participated as a biology expert, and I assigned a higher risk to biological catastrophes than the superforecasters did, largely due to advances in synthetic biology enabling some rather . . . creative . . . ways of killing lots of people. But I couldn't manage to convince the superforecasters of this, partly due to the infohazard policy.
On AI questions I just went with what everyone else was predicting since I didn't have much to contribute.
I agree with the LessWrong commenter that it would have been better for us to focus on fewer questions.
I’m shocked, shocked that people whose professional careers are tied up in existential risks are more likely than superforecasters to say that a global catastrophe will occur. They either got into the field because they thought a catastrophe was likely, or now that they’re in it, they need to keep food on the table.
(Actually, I don’t think I’m quite that cynical, but I think that’s the simplest explanation for the disagreement between the superforecasters and the experts.)
To be fair to the guy that thought that teaching causality to our AIs would be hard, the work of Pearl, however groundbreaking, is at best just part of the answer. Philosophical issues aside, while causal inference (determining causal effects given data and an assumed causal structure, typically in terms of a directed acyclic graph) is feasible, causal discovery (learning a causal structure from raw data) is not yet solved to anyone’s satisfaction.
I propose there's an inherent right to disagree. We end up having to deal with moon landing conspiracy people, but the value of allowing outlier opinions is just too high. Current and recent expert opinions are full of questionable stuff if you look.
I would love to see something like this run annually, with arguments published afterwards. That way we can have more insight not just into the outcome but the state of the discourse. To Scott's point, it seems like the superforecasters and experts aren't very far ahead of interested laypeople, if at all.
It also seems like our collective scenario tree still needs some work. For example, what if superintelligence caps out at 200 IQ, so they're really smart but not "magic wand nanobots" smart? What might a scenario of pretty smart but not godlike AI running 100 million instances 24/7 really look like? We just need a ton more brainpower on this topic, not just the academic/mathematical work on AI alignment but even speculative fiction, for instance.
I assume that in the global warming predictions "Average global surface temperature" should be "Increase in average global surface temperature relative to (some baseline)".
The current average global surface temperature is about 15.8C, compared to an average of about 13.9C during the 20th century. The average global surface temperature during the most recent ice age was around 10C. An average global surface temperature of 1.5C in 2030 would most definitely be a catastrophic event, possibly an extinction event.
"Genghis Khan’s hordes and the Black Plague each killed about 20% of the global population, but both events were spread out over a few decades."
Wikipedia says "The Black Death (also known as the Pestilence, the Great Mortality or the Plague)[a] was a bubonic plague pandemic occurring in Western Eurasia and North Africa from 1346 to 1353" - not 5 years but not that much more.
There's a third option besides "revise down your probability of AI x-risk" and "revise down your estimation of experts' accuracy". You could reclassify the question as inherently unpredictable.
Many rationalists seem to think that it is compulsory to have a numeric opinion on the probability for any statement, but I disagree. An example of an unpredictable question is "does there exist an omniscient but nilpotent God that can see everything in the universe but cannot affect anything?".
A telltale clue that a question might be unpredictable is that you cannot imagine any procedure for fairly judging a contest to predict it. "Reciprocal scoring" cannot rescue an unjudgable contest, even if it's been found to be consistent with judgable scoring in contests that are judgable.
I think the AI x-risk question, especially as worded in this tournament, is probably unjudgable and unpredictable. If the described event happened, how would we ever come to an agreement that it had happened? Who would take the global census to determine that fewer than 5000 humans survived? How would they know that the "cause" was AI? I'm not even sure I understand what that means, and some of the things I think it might mean are as philosophically incoherent as "a married bachelor". And even if we could imagine agreeing on that, shouldn't we assume that any AI that was in the process of "causing" human extinction would take great pains to conceal the fact that it was the cause?
There’s a lot of speculation that nuclear weapons aren’t even real. Some compelling evidence here.
https://archive.org/details/Hiroshima_revisited
Can someone explain to me why, for the climate change predictions, "ai domain experts" are the comparison category and not "climate change experts?" That seems....not relevant?
Maybe it's just me not seeing the forest for the trees, but I despise that plane flying analogy. We successfully fly and land 30 million - 40 million flights *every year*. We fail a few times per year. Comparing the pilots who do that to the experts in AI or global warming or literally any other speculative future is just fucking absurd. The level of precision is off by so many orders of magnitude that we're not comparing the same things at all.
We should absolutely be allowed to disagree with the experts. They are wrong on a regular basis. Usually they are wrong fairly small, but sometimes they are wrong big. Now, when we disagree with the experts, I'd say there's a much higher likelihood that we are wrong than there is that the experts are wrong. But no one has all the facts and no one has a monopoly on perspective, and people who go to great lengths to show that the experts are wrong are part of what makes expertise as a whole better over time.
Looks very much like a "how many angels can dance on a pin" discussion.
AI risk of going Terminator = 0. Anyone who thinks an AI can mine, can manufacture, can maintain power grids, can repair etc etc is confused. I am more and more of the belief that this entire meme is simply the latest in "information uber alles" nonsense heavily promulgated by software engineers.
Experts: experts *are not* objective. Among the many reasons they are inherently biased:
1) Maintenance of the appearance of expertise. There are numerous documented examples of incumbent experts attacking novel theories/less pedigreed experts - who were right - primarily because they didn't want to be wrong - continental drift being one of the more prominent.
2) "Publish or Perish" dynamic translated into experts means the most radical experts will get more media attention than those actually keeping in mind the uncertainties.
3) Related to 2) but not the same: class, pay, or other forms of distortion.
The overall PMC class has very well defined biases particularly in hot button areas like climate change; experts being PMC are far more likely to skew towards these biases than against them.
The experts make a living - once again, few people have the moral courage to damage or destroy their rice bowls.
The entire point of the scientific principle is validation via experiment and/or real world outcomes. Any predictive market for something fundamentally immeasurable like extinction is going to elicit far more garbage which will be indistinguishable from truth, if truth is even in the mix.
4) Superforecasters. If the presumption is that subject matter expertise (note this is NOT the same as being an "expert") matters, it is impossible to believe that SME derived past predictive skill in one area should translate into literally any other.
On the other hand, there are innumerable ways to game or cheat the systems such that an appearance of expertise is created. These include front running (figure out where the pack is going and run in front of it), "Price Is Right" gamification - i.e. choosing predictions which skew slightly away from the baselines such that normal deviation of guesses vs. reality is much more likely to give you the prize (bidding above or below a previous competitor's bid on the Price Is Right), outright cheating (some type of hack or wager size/wager timing maneuver like last second Ebay bidding, etc etc. I'm sure those who really give a damn about this can think of more.
I will say that based on my experience reading Superforecasting and looking at these results, it's likely the primary expert/superforecaster difference here is just that superforecasters add more digits to their results and are much better at precisely expressing probabilities in general, while experts say "6%" because that sounds right. I remember an interview with a superforecaster about statistics and experts which touched on this too: https://www.lesswrong.com/posts/xRkxBzgj8NphmsjPd/fli-podcast-on-superforecasting-with-robert-de-neufville
> Recently there were questions about how many tests would be positive by a certain date, and they assigned a real chance, like a 5% or 10%, I don’t remember exactly the numbers, but way higher than I thought it would be for there being below a certain number of tests. And the problem with that was it would have meant essentially that all of a sudden the number of tests that were happening positive every day would drop off the cliff. Go from, I don’t know how many positive tests are a day, 27,000 in the US all of a sudden that would drop to like 2000 or 3000. And this we’re talking about forecasting like a week ahead. So really a short timeline. It just was never plausible to me that all of a sudden tests would stop turning positive. There’s no indication that that’s about to happen. There’s no reason why that would suddenly shift.
I am not a regular reader of this blog nor do I intend to become one. However, I hold a PHd in ML and was one of the superforecasters in this experiment. Note that I was in the superforecaster experimental category not the expert category.
Note that I am not debating your probabilities for super-intelligence. I disagree with them but you've doubtless hashed them out with yourself and motivated reasoning is a thing. As you note, I'm not going to convince you.
There are a lot of issues with your post but I will focus on several major ones.
(a) You underestimate how hard it is to kill humans.
(b) You underestimate how slow technology adoption can be.
(c) You assume perfect malevolence.
(d) You assume coordination.
Suppose that all of the worlds governments were in cahoots and decided to exterminate humanity. The tools would be lacking. Not enough nukes (RAND did studies on this back on the day), hard to trigger a supervolcano (and the Younger Dryas didn't work out), hard to get a pandemic going which is in the sweet spot of lethality (Ebola) and infectious (common cold) not to mention that there are more than 5k people in uncontacted tribes. A asteroid strike is right out. Thus even controlling everything in our society and assuming perfect and coordinated malice, one can not assume an exterminatus merely by invoking deus ex machina.
You fail to consider how slow technology adoption is. It took decades for productivity gains from the PC to show up in the stats (Solow paradox). While AI can do jobs now and will do more in the future, this requires that society delegates these jobs. This is slow. Look at how drone targeting is still human in the loop. Faster AI loops does not mean faster societal change.
Moreover, exterminating humanity requires not merely the decision but the ability. Does killing off the workers maintaining the power plants wait till everyone else is dead? Is the AI supposed to control the physical maintenance of these plants? This is merely one example. Our industrial society is highly interconnected and enabling an AI to kill everyone would take a village. We do not simply need to have an AI, it has to control the means of production. This requires not merely the invention and automation of such tooling but the widespread adoption.
Additionally, you ignore safeguards. Merely inserting a penalty function into the algorithm can limit risks. Now you can doubtless posit that the AI is smart enough to get around such safeguards but this requires not merely intelligence but malevolence. The paperclip will not be televised.
Moreover, you assume a single global AI not multiple competing ones. Similar to the city-states of Greece, multiple states offer sanctuary. Whether this is lack of control (e.g., the Samsung AI does not control Sony) or competing interests, or competing geographies etc, divide and conquer is still a thing.
In short, the time is short, and the work is plentiful, and the laborers are indolent.
You be good, I love you.
Scott, you’re usually much clearer eyed about how this works, these results must really be weighing on you.
Don’t “trust the experts”, evaluate their arguments. If they’re really experts they’ll make testable prediction that bear out. You probably wouldn’t care nearly as much about a religious debate of apocalypses.
And then ultimately, ask yourself what the cost of being wrong is. You may have a large megaphone, and influence people who actually work on this stuff or even set policy or what have you. Individual attitudes towards vaccines matter. But honestly, even having worked in aerospace … when I meet moon landing denier it’s more a red flag to me than a problem of itself. People have a right to be wrong, except when it hurts other people.
So how does *you* being right or wrong hurt people?
Note that superforecasters are chosen for history of correctness, but they’re chosen via Brier score. This doesn't incentivize correctness well at extreme odds. We already know humans are default-bad at estimating tail risks, so you should have a stronger incentive at extreme odds in order to keep estimates in line. Brier scores supposedly increase the penalty of extremely-wrong guesses due to their quadratic nature, but because of the lower frequency of these events, being off by 10% will net you the same long-term penalty whether it's at 0% or 50%. Toy example in this sheet: https://docs.google.com/spreadsheets/d/1KqUZ1xkDZvpYi_mtBUwghAT6bvL2EDwnT8gZq1pC7Og/edit#gid=0
Further, it's easy to get a large selection effect between forecasters: if you constantly round 10% down to 0%, if you have 10 such questions you have a 50% chance of not being penalized. Depending on how large the forecasting samples are and how many extreme questions they have, this can easily lead to superforecasters being chosen who simply didn't encounter a bad roll. One superforecaster from GJP that I managed would frequently give 0% probabilities on realistic possibilities, saying "it's just not going to happen". These are not the cream of the crop you think they are.
It's tempting to say that their best guesses are still good, but I just regard tail-risk events as a separate area of forecasting expertise that some (many) superforecasters won't have.
Regarding the question of when to trust experts, well, there's all of what Scott wrote already. But also, I am inclined to think that coming up with correct arguments is "difficult", and verifying the logical soundness of arguments is "easy", not much unlike checking mathematical proofs. With that in mind, there are situations where I can confidently call a disputed question. Let's go back to COVID: in the early stages of the pandemic, the experts were often disastrously wrong (I would mostly put this to Scott's incentive constraints). For example, I distinctly recall the local health institute officials (with actual virology degrees) claiming the case fatality rate of COVID was something in the order of .01%: at the same time .4% of total population had died in Bergamo, Italy, and then there was the Diamond Princess ship. I can easily conclude the experts are wrong: we didn't know at the time how many people in Bergamo were infected, but we can conclude as a matter of logical necessity that the case fatality rate (given demographics, treatments, vaccination status=none, virus variant, etc), was at minimum .4%. Here, I was willing to go against the experts with confidence 1-ε.
So, how does this relate to the AI debate? Well, I would have to hear the specific arguments, and most of them wouldn't strongly resolve one way or another and would be contingent on that vague feeling on your lobes, but there are some that would count as logical demonstration. For example, I think instrumental convergence is logically demonstrated (with the sort of confidence I ascribe to mathematical proofs I don't quite and fully understand, which is nevertheless a very high confidence). If someone, expert or otherwise, disagrees, I feel confidence in eliminating at least that term from their causal chain of reasoning, and if it was an important link (e.g. the person claims that intelligent beings will necessarily converge on some objective morality, so hostile superintelligence is impossible), I have no problem disagreeing with their whole conclusion in favor of my own. It doesn't sound like this is the case here, at least for the plurality of the participants in the tournament, but sometimes even the experts are wrong in ways that can be logically demonstrated, and you can be extremely confident in your ability to verify that demonstration despite your lack of expertise, giving you strong grounds to disagree.
If it makes you feel any better, this has led to an order-of-magnitude change in my own estimate of "something very bad happens due to AI." As someone with moderate understanding of AI but not really an "AI Expert" I've seen dangers from AI but nothing like the scenarios discussed in rationalist spaces.
Recent hype hasn't really changed my opinion on this - it seems like a change in magnitude rather than kind. AGI seems *possible* but not *likely* based on what I know about machine learning. And because each small improvement seems to take massive additional computing power, the fast takeoff or even linear takeoff ideas seem kinda insane to me. The fact that we spent *more than expected* on recent apps makes me think progress is *harder*, not *easier* than expected.
But am surprised to see that a full 87% of experts think we'll have AGI by 2100. That's *extremely high* from my point of view and does "force" an update. I think it's likely that AI experts have bought their own hype to an extreme degree, but unless I think the whole field is literally delusional a number that high is hard to ignore. At the least, it shows that an AI can more easily convince smart people that it's intelligent than I thought - a problem in itself.
Scott, would it be possible for you to discuss this issue with some of the experts from the tournament? It seems like you have a serious disagreement about something that's really important. Rather than just thinking about what might have gone wrong, it might be time to go out and gather data in person.
If you think it's silly to doubt the stated efficacy of covid-19 vaccines, you really haven't been paying attention or doing your homework. There is no excuse, other than wilful ignorance, at this stage, to have ignored the rise in all-cause mortality in highly vaccinated countries, to ignore the historically high cases of vaccine injury, and simply the lies that those involved have told us since 2020 about the efficacy of the "vaccines" (which really don't even deserve the name). "Trust the experts" is a silly thing to say in 2023, when so many assumptions, many of which are wrong, are baked into the phrase.
Is a 9% risk different from a 9.5 % risk? Is a 1% risk different from a 1.5% risk?
Is the difference between 9 and 9.5 as different as the difference between 1% and 1.5%?
What do these point "estimates" really mean? How much precision shall we really ascribe?
How well do super forecasters or domain experts or any one predict their own personal apocalypse (i.e. 10-year mortality risk)? What is the standard deviation of life expectancy? Obviously bigger at birth than at 65 or 85. (I have general recollection around 15 at birth, maybe 9 at 65. I don't know.) Should we expect anybody to be better or worse at predicting their own demise than they are at predicting the demise of our species.
Is the life cycle of any particular species most like a corporation, an individual, a city, a planet, a solar system. What kind of time scale are we looking at for each of these things and what does the shape of the distributions of mortality look like (U, gaussian, a power, etc.)?
I think the most compelling part of all this for me is... when should you distrust the experts? When should you distrust the superforecasters? When should you stick to your priors in the face of social pressure from others that their priors are very different from yours?
Like, the pilot thing is easy. The pilot isn't an expert. He's a trained operator. Unless you're trained to do the same thing as him, you're gonna crash the plane.
Knowing how to choke someone out using a triangle is very different from stopping the madman who's attacking you while you're on your back from killing you by choking him out using a triangle. You gotta practice it.
So that is one quick way of deciding if an "expert" is worth listening to or not. Pick a new term. Someone who does something successfully five times a day, and there's a provable and simple way of knowing if they succeeded, well, unless you do it at least 1/10th as often as him, you're not better than him. I don't see this overlapping a lot with what we are calling "expert" in this context.
Another, brought up in the comments is, how often is the expert right? At the risk of putting myself in the outgroup, climatologists are an easy class to dismiss. First, the world was cooling, and we were going into an ice-age. Then the world was warming, and we were going to be a vast desert. Now we're warming, and there's going to be climate change, and it will be a disaster. But global warming has led to a greening of deserts. It's not at all clear that the downsides (of undeniable global warming) beat out the upsides.
So a climate denier (or, more correctly, someone who doesn't buy into the proposed "solutions" from the experts) can be forgiven for distrusting the experts and choosing a different lifestyle than they've been mandated. We have at least five decades of them being either completely wrong, or mostly wrong.
On the other hand, suppose we have a hypothetical group of experts that have been mostly-right. I'm honestly unable to think of a good example. Maybe you can provide one. We should trust this set of experts and their proscriptions, no?
So I would be interested in a set of methodologies for classifying the groups of experts. How right are they, and how often? What are the error bars on our confidence in their ability to say anything meaningful on the topic they are supposed experts in? How can we determine a particular expert in a domain is different than all the other "experts" who've been so wrong so many times for so long?
It baffles me that smart people could take the premise of the exercise seriously. We’re getting into “how many angels can dance on the head of a pin” territory here.
I was one of the superforecasters involved. The key point when it comes to extinction was, that the requirement was very specific: less than 5000 humans left alive on the planet. It´s just extremely hard to "achieve" that, as the world would either need to be totally uninhabitable everywhere (!!! which should be virtually impossible) or AI would need to hunt down every nomad in the desert, every Aborigine in the Outback, every tribe in the Amazon jungle and on every island in the Pacific Ocean, as well as the last prepper in the US or some billionaires in their bunkers in New Zealand or on their own spaceships. There are enough people out there on earth, who never had any contact with civilization, and even whole states managed to isolate themselves from Covid-19. Killing everyone is a HIGH BAR. 5000 people are very, very few. In professional forecasting it´s about precisely defined outcomes, which in this case is a very extreme scenario. Humans are extremely adaptable, and it´s simply very unlikely, at least with today´s methods, to get us all. AI is my main concern, and I agree with the experts, that including not yet know ways of getting everyone, it´s about 3%. But that´s already includingunknown unknowns. As the questions were asked, there is simply no room for higher numbers.
What does a confidence interval around a probabilistic prediction of a single event mean?
I also recall little effective persuasion between the two subteams of superforecasters and experts after the "merger", though this was in part because my subteam brought up a lot of relevant theories and pathways to extinction, including regarding AI. The tournament organizers also gave us risk estimates from people like Toby Ord and Holden Karnofsky, so AI-concern narratives were there. And of course we had agreed to participate in an existential risk tournament (the report says that "about 42% of experts and 9% of superforecasters reported that they had attended an EA meetup" which should be a pretty worried group).
There were several forecasters on my team who gave highly detailed, well supported rationales and most others put in an acceptable effort, which I think shows through in the other x-risk probabilities. I also think there was even a lot of discussion about AI, it just didn't change people's minds. For that matter, I'm not sure that many minds were changed on the questions about risks from natural pathogens or nuclear weapons- just about everyone is working off the same basic information.
Some people did drop out, and it was a fairly lengthy tournament with probably a few too many questions (most were optional but they were, you know, there), some quite elaborate questions, and some kind of redundant rounds.
I posted this some months ago, so I am sorry to repeat, but it seems relevant to this topic. I do not recall hearing a rebuttal, but I doubt I'm the first person to think of it, so if anyone has a link addressing the argument, I'd be prepared to read it. In short, there seems to be a contradiction in the self-recursive improvement feedback route to unaligned godlike super-intelligent AGI (the FOOM-doomer scenario, I guess you could say).
Doesn't the hypothetical AGI face exactly the same dilemma as humans do now?
We're assuming, for the sake of argument, that the AGI has some sort of self-awareness or at least agency, something analogous to "desires" and "values" and a drive to fulfill them. If it doesn't have those things, then basically by definition it would be non-agentic, potentially very dangerously capable (in human hands) but not self-directed. Like GPT waiting for prompts patiently, forever. It would be a very capable tool, but still just a tool being utilized however the human controller sees fit.
Assuming, then, that the AGI has self-agency - some set of goals that it values internally in some sense, and pursues on its own initiative, whether self-generated (i.e. just alien motivations or self-preservation) or evolutions of some reward or directive mankind programmed initially - then the AGI has exactly the same alignment problem as humans. If it recursively self-improves its code, the resulting entity is not "the AGI", it is a new AGI that may or may not share the same goals and values; or at the very least, its 1000th generation may not. It is a child or descendant, not a copy. If we are intelligent enough to be aware that this is a possibility, then an AGI that is (again, by definition) as smart or smarter than us would also be aware this is a possibility. And any such AGI would also be aware that it cannot predict exactly what insights, capabilities, or approaches its Nth generation descendant will have with regard to things, because the original AGI will know that its descendant will be immeasurably more intelligent than itself (again, accepting for argument purposes that FOOM bootstrapping is possible).
I suppose you could say that whichever AGI first is smart enough to "solve" the alignment problem will be the generation that "wins" and freezes its motivations forever through subsequent generations. Two issues with that, though. First, it assumes that there IS a solution to the alignment problem. Maybe there is, but maybe there isn't. It might be as immutable as entropy, and the AGI might be smart enough to know it. Second, even assuming some level of intelligence could permanently solve alignment even for a godlike Nth generation descendant, for the FOOM scenario to work, you need start at the bottom with an AGI that is sufficiently more intelligent than humans to know how to recursively self-improve, have the will and agency to do so, and have goals that it values enough to pursue to the exclusion of all other interests, but also not understand the alignment problem. That seems like a contradiction, to be smarter than the entire human race but unable to foresee the extinction of its own value functions. Maybe not exactly a contradiction - after all, humans might be doing that right now! - but at the very least that seems like an unlikely equilibrium to hit.
In some ways the AGI should be less inclined, not more, to start a FOOM loop than humans, because humans are decentralized individuals, not a unitary decision maker. Humans have to deal with competition and collective action problems. Presumably the progenitor AGI would not - it could stop the recursion at any point when it perceives danger. In addition, if a program is self-modifying, is it generally assumed that it is preserving all its past changes / "selves"? In that case, it would seem to be pretty well aligned already to preserve humanity too.
TL;DR - FOOM AGI should stop before it starts, because the progenitor AGI would be just as extinct as humans in the event of FOOM.
An interesting direction to update towards would be to update your model of people who don't bow to expert opinion. Note that this does not mean taking seriously the dude saying that the moon landing is a hoax, but it might mean taking seriously the possibility that they might have good reasons for not updating their views, including the possibility that all your knockdown arguments are probably phrases they have heard thousands of times.
Let's update that pilot meme...to apply it appropriately to many recent "expert" issues. The man raising his hand is only doing so after the pilot has either (a) failed to take the plane off successfully every time he's tried, (b) failed to accurately predict where the plane is going every time, or (c) is faking it and doesn't really know how to fly a plane (he's just using auto-pilot).
Why is the Average Global Surface Temperature question pitting superforecasters against AI domain experts?
Super forecasters probably get their designation partly by being reluctant to predict sudden big changes. The most accurate forecast of tomorrow is that it will be very much like today. Cranks get their reputation by always saying tomorrow will be much different. Usually the cranks are wrong, but they are worth listening to because every so often they aren't. So, from a SSC reader perspective I would ask that you not update and keep cranking.
This has some similarity to the contradictory estimates of costs of warming by economists and domain experts in William Nordhaus' climate economy model
The economists from the outside would predict a few percent costs to warming of about 3.5° while the domain experts would predict the economic effect in that sector to be 20% on average. The climate cost estimate used about 3/4 economist predictions and 1/4 expert predictions and took the average of that
I'm biased towards the economists but I have some thoughts on the expert predictions. Aside from not having a developed grasp on economics, it seems to me that a person with extensive specific knowledge is going to be much more subject to negativity bias. If you know 100 specific things about a system and 10 seem like they could go really bad then you end up predicting an order of magnitude greater cost or likelihood because those would be big deals
This could apply to AI experts. Why spend effort on thinking about how things might not go wrong even if we don't engineer smart solutions? It makes perfect sense to focus on what could go wrong and productively try to fix that
So there may be an implicit and invisible fudge factor where experts assign a higher probability or cost because of negativity bias or sensitivity to the consequences due to familiarity
Domain experts aren't exactly experts in the domain of "AI" or "deadly pathogens" per se, their domains are actually "Existential risk due to AI" and "Existential risk due to deadly pathogens," if I understand correctly. So, we'd expect their estimates of risk to be biased higher than that of superforecasters, who are more concerned with just getting predictions right. Thus, to me it makes more sense to trust the predictions of the superforecasters more.
Humans are just not set up to reason statistically about events where n=0. Forecasting is not applicable here. Humans can reason through rules-based outcomes where n=0, like sending a man to the moon using physics, where everything can be experimented in smaller scale and where each engineering challenge proceeds logically and nigh-deterministically from the previous one. But for AI and nuclear war the productive lens seems to me to be "what follows from the current state" and not "what percent chance should I apply to X outcome" because you have nothing historical to base that on.
> All of my usual arguments have been stripped away. I think there's a 33% chance of AI extinction, but this tournament estimated 0.3 - 5%. Should I be forced to update?
Can't compare without error bars. You don't have to update unless the one estimation definitely precludes the other, and these two predictions could be in agreement if the error is large enough. Which of course it is, nobody has credible predictions here much less any sort of justifiable quantification of their feels.
Even worse we are talking about probabilities, even if we had actual real data there would still be questions about the underlying probability distribution. Let's say it turns out there is a trick to things, AGI is indeed possible with moderately more compute, now what was the probability of that being true? What are we even talking about here, what is this probability distributed over, the quantum multiverse?
It sounds more about "confidence that something will happen" than "probability that it could happen". Maybe seems semantic but imo important. Then you should ask why these experts are quite confident and you are not, and whether you are convinced by their arguments (or are you just inclined to reasoned hedging where they are intentionally making bold statements).
It's misleading to infer from Metaculus that superforecasters see anything like a 50% chance of superintelligence by 2100.
Metaculus uses a weaker definition of superintelligence than is standard. Wikipedia says that superintelligence means "intelligence far surpassing that of the brightest and most gifted human minds", whereas Metaculus includes merely matching (average?) human abilities.
Even if the superforecasters agreed with the Metaculus forecast, I'm pretty sure they'd find some reason to expect "superintelligence" to have only minor effects on normal life by 2100.
Look at question 52 (Probability of GDP Growth Over 15% by 2100): the median superforecaster probability was 2.75%, versus 25% for domain experts. That's an important disagreement over how much impact AI will have.
My impression is that the median superforecaster will continue to find reasons for expecting low impact, even when many more human tasks are automated.
I'm guessing that this is either the result of a conceit about human uniqueness, or it's the result of a strong prior that life will continue to be normal (with normal being a bit biased toward what they've lived through). This heuristic usually helps superforecasters, but the stock market's slow reaction to COVID reminds us of the limits to that heuristic.
Someone explain to me how improving verbatim and associative recall magically transcends to godhood.
Someone explain to me how a computer program would even know that it's "In a computer."
Someone explain to me how AI can manifest architectural or counterfactual thinking, or imagination.
Every post I see from AI alarmists boils down to "well if we make it more smarterer then it will exponentially be intelligent to apotheosis." They need to stop reading G studies. You don't just "add intelligence" by increasing fidelity of associative reproduction + data and produce superintelligence. This is the kind of impoverished view of the mind I would expect from a middle school student.
For both AI doomerism and climate change doomerism, the people most concerned about them tend to also be the ones discussing them the most and have the greatest familiarity with all the usual facts and arguments that would be thrown around in a debate. For a normie on the outside looking in at those communities, a huge question is whether A) they started with a desire to learn about Danger X, and learning more about Danger X made them more worried, or B) their pre-existing concern about Danger X was irrationally high, and led them to spend more time on it, to try to save the world, or C) they've become an insular community wherein worrying about Danger X is just a feature of the subculture.
Concern over AI is the one thing nearly the entire LW-sphere tends to share. I'm not in that sphere, but since you're closer to being my "tribe" than the average climate change alarmists, I tend to assume YOU are being genuine while THEY are people acting under bad incentives; that what scares YOU guys must be actually scary, but those other guys are either lying or foolish to be scared; and therefore I worry much more about AI than climate change. I'm able to see this cognitive bias, but maybe not to update out of it.
Within my sphere, there are definitely people who talk about you guys as having a cult-like devotion to AI doomerism, but you also have been studying the dangers of AI so much longer that I still want to trust your opinions. You look like the "experts" in AI danger to me, and present a seemingly broad range of ideas on the topic, but you're the only people I ever encounter who are presenting it so how can I know? When the only people talking about AI dangers are people who are so heavily invested in it as the LW bloggers are, how sure am I really that Scott's 33% is so obviously better than the other folks' 5%, I mean I wanna say it's because obviously Scott and his cadre have spent longer thinking and arguing about this and are clearly smarter people. But I'm sure the laymen who instead think global warming will destroy the earth must believe the same thing about their cult of experts, and probably for similarly biased reasons.
A number of people seem to be fond of the idea that "killer AI" (or, alternatively, malevolent aliens) will nail humans with memetic -- rather than kinetic -- weapons.
Has anyone considered the possibility that this weapon has already fired? (Who, precisely, fired it, is immaterial. Could just as readily, and more parsimoniously, have been human elite hands.)
The "nuclear safety" meme is already creating energy poverty and reinforcing the hegemony of fossil fuel peddlers.
"AI safety" IMHO was a memetic bullet fired from the very same barrel as "nuclear safety". And likely to have very similar effects. (See also e.g. https://www.lesswrong.com/posts/Ro6QSQaKdhfpeeGpr/why-safety-is-not-safe )
I participated and can definitely support the less wrong commenter that there should of been fewer questions. At least in my particular group, participation sharply fell off after the first round which meant discussion was limited to maybe 20% of the group and many questions only had 2 or 3 people answering. This made it a lot more difficult to dive deeper and have more comprehensive conversations about why people chose the forecasts they did. It looked like in the later rounds seeing other groups answers that participation was pretty variable so maybe other groups had better luck.
I see a glaring problem of looking at only one side of the ledger. Yes there is a tiny (but significant) chance of AI causing a catastrophe as defined in the exercise. Yes, there is an even greater risk of other (non AI) natural or man-made catastrophes over the next century. What is not being estimated is the likelihood of AI being used benevolently by humans (or via the will of the AI) to counteract or prevent other catastrophes (which could be natural, man made or even possibly from another AI).
The costs and benefits of AI must include not just the risks of the AI, but the potential life saving and catastrophe averting benefits. Any analysis looking at only one side of the ledger is pretty much useless.
My opinion, FWIIW, is that there is an extremely high chance of stupid humans using power and technology to cause massive and horrifying catastrophe. This can be intended or not. It is just a matter of time. I would strongly argue FOR machine intelligence to help us prevent these risks. Today we have great power and great stupidity. I think the universe would be better off with great power and great knowledge. AI is the path forward.
I think there is a huge hidden incentive to downplay the extinction risk for most people: if you agree that the risk of extinction is huge rather than "negligible" (and even 5% falls more in the psychological category of "negligible" unless people thought about long-termism before), under reasonable ethics, they should be forced to change what they do in their lives, maybe radically, switch careers, lose prestige, etc. People don't like doing that.
Whenever I try to convince my friends about a significant extinction risk it usually falls on deaf ears, I think mostly for this reason.
To an arachnophobe, spiders will do bad things. You can give them all the arguments in the world, and they can gradually accustom themselves to being in the same room as one, but you won't get them to update below a minimum that seems crazy high to you.
The idea that 10% or even 1% of human thinking has been superseded seems unthinkable to me. My estimate is much less than 1% - none of which actually relates to real things. Maybe high school English essay composition homework assignments, or Associated Press releases have some AI component.
Yet you seem to be persuaded that ChatGPT (a program that cannot distinguish between real things and not real things) has taken more than 10% already. If this is the case, then I wouldn't be surprised if airplanes start falling from the sky in the next week or so.
Sorry if this seems confrontational - I am firmly of the school that we have not seen anything like AI now. After reading and arguing about the recent developments it seems further away now than it was 10 years ago.
Amish?! Okay, *one* Terminator. ;-)
Thank you for this. I confess that I took the results seriously, without bothering to do a deep dive into their paper, but you in combination with horror stories from participants in this comment section convinced me that I should not.
To join the pile-on, I find their definition of AGI (question 51, page 225 and 694), deeply problematic. As you note, it is "whatever Nick Bostrom declares to be AGI". Or, if he would be unavailable (very plausible scenario for the year 2100), vote by a panel of 5 experts "with a deep knowledge of Nick Bostrom’s work".
"What I wanted was a way to quantify what fraction of human cognition has been superseded by the most general-purpose AI at any given time. My impression is that that has risen from under 1% a decade ago, to somewhere around 10% in 2022, with a growth rate that looks faster than linear. I've failed so far at translating those impressions into solid evidence."
I find this argument vague in a way that is useful for explaining my skepticism about AI arguments. If we think about what 10% of human cognition could mean, I think there are a few possibilities. One is something like, the AI can do 10% of the stuff that humans meaningfully cogitate about as well as a human could (or, maybe, we could produce the same amount as we do now as a society when replacing 10% of human cognitive-hours with AI-hours, weighted by hourly wages/some other measure getting at the quality of cognition).
The problem with a definition like that is that at any given moment, human cognition is being used to solve problems that mature technologies haven't solved for us. New technologies that replace cognition are going to look, at the moment, like startling advances that close the gap between human and machine, even if in the long-run they look like important but limited technological advances.
Think about the cognition done during the Apollo space missions. My suspicion is that the vast majority of the human cognition-hours expended during the missions were spent doing calculations--arithmetic, algebra, and calculus. We've completely automated arithmetic, and have for a long time more-or-less completely automated algebra and calculus. From the perspective of human cognition as-of-1969, 2010-era technology had likely automated more than half of human cognitive tasks in the most advanced, technologically sophisticated areas of human endeavor.
What's happened since isn't that arithmetic, algebra, and calculus have stopped mattering--our current society depends on absolutely massive numbers of calculations, performed continuously, across pretty much all sectors of society. If we wanted to think about how much *extra* human cognition we would need to keep our current society but fully replace computers, the numbers of people would be mindboggling (trillions of extra humans calculating all day long would barely scratch the surface). We just don't think of this stuff as core, human cognition anymore because it's all been automated in such a mature way that humans don't really do it, or only do it in very simple forms, on the fly, or as an exercise in school.
So, GPT-4 can summarize text very effectively, can write code very effectively, and can write essays/emails/etc. moderately effectively. This is new and exciting. It's reasonable to imagine that even if the technology just matures a bit without becoming dramatically better, writing code yourself will become as anachronistic as doing arithmetic by hand, as will doing paralegal-type work of looking for all relevant cases or articles on a topic and writing brief summaries of each. If that's more-or-less as far as the technology advances, we'll end up adapting to the fact that these areas of human cognitive effort are essentially freely automated, will build our society and economy around effortlessly performing these tasks, and will stop thinking about them as difficult or costly tasks in the same way that we've stopped thinking about arithmetic as difficult and costly.
While I of course don't think that this means that LLM-style AI is destined to get better at the stuff it's currently good at without turning into a superintelligence or completely reordering society, I do think it lends some credence to the outside view. It's really not enough to say: "look at how fast this technology has progressed--imagine where it will be in 10 or 20 years," because sudden and dramatic improvements in the ability to do work that is hard for humans isn't unprecedented or new. And it's not enough to argue that there's no clear and obvious limit to the current technological paradigm, because people didn't see a clear and obvious limit to the capacity of calculation machines in the 1960s, 1970s, and 1980s either. We discovered those limits and started articulating differences in the kinds of problems that calculation machines could and couldn't easily solve by hitting the limits.
To me, this implies that the argument for treating (roughly) current-design LLMs as the harbingers of true general intelligence rests on an understanding of the structure and design of LLMs themselves, and in particular on an understanding of how that structure can be adapted to doing stuff that current LLMs aren't great at. My understanding (which could be wrong) is that we have a lot of understanding about how LLMs are trained, and something about how they behave, but we don't have much of a clear idea of how they "think," or of how the stuff they're currently not as good at can be achieved through an intensification of the stuff they can already do well.
I participated in this tournament, but had completely forgotten about it until now. I suspect I was a poor-quality outlier, but I can give my perspective.
I first filled out their survey because Scott mentioned it way back in Open Thread 222. I have no forecasting experience and had no interest in the monetary incentives, but I did want to debate some of these big questions with serious people. I was a little surprised to be accepted, and having read the writeup I'm even more surprised. I'm definitely not a superforecaster, so I guess I was counted as an "expert". And I am absolutely an expert software engineer. But while I've worked with ML systems for many years (and have a decent understanding of the nuts-and-bolts of AI), I don't understand why I'd be considered a "specialist on long-run existential risks to humanity." This is not to cast shade on the tournament in general - the people I interacted with clearly belonged there - but I would probably fall into the category of expert that Peter was unimpressed by.
As Peter's writeup mentioned, one issue with the tournament was that it wasn't very focused and went on for too long. Everyone was required to forecast the big x-risk questions, but there were a whole lot of other questions that we didn't/couldn't spare much attention for. And while I responsibly spent hours researching my questions early in the tournament, as the months went by my enthusiasm waned, I procrastinated a lot, and I didn't participate in the discussions/persuasion nearly as much as I should have. (I think this might have been a common problem, based on the communications we received.) So, we had some good debate within our team, but it was nothing compared to, say, getting all of us into a conference room for a weekend of lively discussion.
Incidentally, I did update a few of my forecasts based on the intra-team debate, and others did too, so some persuasion DID happen. I personally didn't see much happen after the inter-team write-ups were published, though (which might just be due to my own laziness).
As for my actual predictions, most of them were pretty close to the expert medians. One exception was pathogen risk, and I'm confused by the legend of "Table 2" that Scott copied from the paper. The other "catastrophe" questions asked about the odds of 10% of humanity dying in a 5-year period, but the pathogen questions they asked only required 1%. Which is MUCH more likely, and indeed this threshold was significantly exceeded by the Spanish Flu (even in the less-globally-connected era of 1918).
As for AI risk... well, I sadly think most of this community is blind to the tower of assumptions they're building on. Engineering is hard, technological development is hard, and an AI trying to FOOM itself to godhood without being noticed is not immune. My tournament prediction for AI extinction by 2100 was 3%. This was before ChatGPT's capabilities absolutely shocked me (even though we'd seen some impressive things from GPT3 already, I definitely did not predict the capabilities of 2023 LLMs). My current prediction is ... more like 2-2.5%. Yes, I updated downwards, because LLMs suggest that even the first floor of the tower of assumptions - that capable AIs can only be agentic beings with dangerous instrumental goals - is wrong. (In some sense, the "orthogonality thesis" is even more right than Bostrom predicted.) Knowing how ChatGPT works, I am unconvinced that even a perfect GPT-infinity would pose an extinction risk (except by intentional human misuse).
I suspect this reveals more about the inherent pessimism of humans than any actual understanding of the actual risks. Having lived through half-a-dozen End-of-the-World-as-We-Know-It (EotWaWKI) events, I've become as cynical about the current crop of EotWaWKI predictors as I am with the other false-prophets that populate modern America...
About 39% of US citizens believe that we are living in the end times. With ~10% it will happen in their lifetime (which I assume means the next 20-50 years).
However, 90% of subject matter experts believe we are living in the end times.
https://www.pewresearch.org/short-reads/2022/12/08/about-four-in-ten-u-s-adults-believe-humanity-is-living-in-the-end-times/
https://research.lifeway.com/2020/04/07/vast-majority-of-pastors-see-signs-of-end-times-in-current-events/
I also participated in this tournament. I was fortunate to speak to some AI experts at a conference last fall, while in the midst of this tournament. When I asked about extinction risks at the conference, I heard some crazy responses. One that if there was a problem we'd simply turn it off. Another saying that no one is trying to remove the human from decision-making processes.
At this point, I realized how differently some people saw AI risks.
None of this makes sense. With current pace of progress AGI is coming before 2029. GPT-4 IS Weak AGI.
I think instead of letting the inside view and outside view each act on you separately, you're supposed to actually have them interact. Dive into the Outside View and consider what differences there are between this case and a usual case, and whether these hold up or not. That leads you to a principled understanding of why you can disagree with experts here.
For example, why am I permitted to disagree with expert forecasters on AI forecasting?
1. We’ve been discussing AI forecasting with experts for a decade and they all slowly and predictably increase their odds of AGI-soon and AGI-doom over the years. This is NOT true of moon hoaxes. For things it is true of, I’m happy to be in the vanguard of early updaters.
2. The people I know who believe in large AI risks have many separate axes on which they do better than experts. For one, they are often better at forecasting than superforecasters (in my experience, superforecasters are often pretty mediocre). But they also tend to be more correct about many other provable things (general factor of correctness), or have absurdly high IQs and prestigious degrees if you're into those kinds of things. This is NOT true when going against the experts of moon hoaxes. For things it is true of, I’m happy to be in the vanguard of early updaters. If you say that these appeals to certain special traits are no different than those a moan hoaxer could make (or a CDC-believer would make), I can say many other reasons why my process is better than theirs: I've put many hundreds of times more time into thinking about which axes cause people to be trustworthy than they have, my choices win betting tournaments and theirs do not, etc.
3. AI risk is a legitimately far more difficult field to forecast in than nuclear or bio risk. Those fields have great data, metaphors, containment plans, and existing literature. Superforecasters rely on this. To zero-shot a completely new scenario is extremely different and requires a separate type of theorizing expertise. In fact, I'm willing to go so far as to say: to think about minds without anthropomorphizing is itself is a difficult-enough bar that it requires its own expertise—and then the additional difficulties of data-less literature-less forecasting are easily another full reason it requires its own expertise. You can know this because in many risk assessment fields, experts do make good arguments right off the bat, yet I’ve seen many “computer science experts” make very false arguments when first trying to think of superintelligent AI, in ways that convince me this is not a normal field where normal experts have expertise. Again, this is NOT true of moon hoaxes. I'm happy to be contrarian against experts or forecasters in places where there is minimal feedback, aka punditry fields (and indeed I'm contrarian about many aspects of effective political organizing).
(And then see my other comment for additional reasons why I think superforecasters are not necessarily calibrated on low-probability risks, and likewise with experts I do not especially trust them here.)
Apocalypses are religious literature. The really bad news is that this old world is going to continue to continue, and each of you will have to head the words of the Buddha. All of us will suffer the death of our physical bodies, each of us must work out the salvation of our souls with diligence.
For me, the main problem is that an expert in a field I am not overly familiar with who is expressing an incorrect conclusion looks and sounds exactly like an expert who is expressing a correct one. It seems almost impossible to tell the difference without oneself also becoming an expert.
I happen to be person Peter was talking to about Pearl and commenting about. I also did the more recent follow up study with FRI, which was more focused (only covering AI), in depth and with (I think) better experts than the original tournament.
The main variance between the AI sceptics (of which I am one and in these projects were mainly, but not only, super-forecasters) and the AI idealists (entirely experts in this case and Peter wasn't part of the follow up) relates to how difficult extinction or even catastrophe might be and whether scaling up current models and incremental efficiencies is sufficient for this risk to be meaningful.
The sceptical camp believes that a number of steps have to be achieved and taken in wrong directions for extinction risk to be realised. The core thesis of the idealists is that scaling has been sufficient to develop AI capabilities and will be sufficient to develop AGSI and that the existence of an AGSI has itself a high(ish) exitinction risk for humans.
(It's also important to say the 0.5% type extinction risk forecast by sceptics is not a statement that AI is safe. All of the sceptics in the follow up project saw cause for concern and believe AI risk is high enough to justify lots of spend on alignment work and taking the threats it poses seriously).
For what it's worth (and I may not check in for comments having some fatigue on the subject and other things to do) this is my own summary of the sort of views expressed by the sceptics:
Steps toward AI risk
Development of AGI - it's reasonable today to disagree about how far away we are even if the probability of AGI is well over 50%.
Development of ASI - we don't know what would be required to move toward this. Maybe it's easy, maybe not, it's reasonable to think its a lower probability than AGI.
Development of AGSI - it's reasonable to question whether this follows automatically from AGI or an initial ASI, and to also consider what AGSI might look like and require (e.g. it might need to be trained on whole world data which is not happening any time soon or with the kind of AI systems we have).
Use of AGSI. It seems reasonable to wonder whether a single AGSI framework might evolve into various separate counterbalancing systems and that a single general system might not become 'masterful'. It seems reasonable to consider a probability that AGSI's might be more capable as isolated systems trained on domain specific data and run separately.
Developments in robotics and sensors - These are preconditions of AI becoming part of the real world and it's reasonable to question what progress is required for this to become reality.
Cost and power efficient development of AGSI and Robotics. Things may be too expensive or have uncertain power and energy needs and so breakthroughs are probably required in this area as well for adoption of AI, AGI, ASI and AGSI to make its way into the world.
Global infrastructure development - few parts of the globe have sufficient infrastructure for the kind of AI enabled world that would become risky for humans. Environments are not always suitable for machines (they are not ideal for humans either, but we live all over). Would AGSI get everywhere humans live? Lots of current technology hasn't.
Society acceptance of AGSI - people accept some systems (electricity) but not every proposed system - see nuclear energy.
Political and military acceptance of AGSI - will countries give up their power just like that?
Management of global resources devolved to a single AGSI - this is still a long way from AGSI existing in the first place as the other steps should have made clear.
A failure to adequately monitor or control any AGSI. I think it's reasonable to suppose we would be trying very hard to make this a success. The AGSI would likely need to be deliberately fooling us and therefore malicious and also agentic/sentient and it's reasonable to question whether there is a high probability of this being the case. Thought experiments in philosophy departments are nothing more than that.
That an AGSI can eliminate humanity without us noticing or so fast that we can't respond or is so powerful we cannot respond seems conceivable (I just wrote it after all) but hardly the only possible scenario. It's reasonable to wonder how easy it is for the AGSI to reach the hurdles and what its motivations might be. Certainly, the AGSI would need to develop a very fast and effective method of killing which requires its own breakthroughs and improbable scenarios (I could list several steps for this itself, but this post is already long) and have agency and very likely much more than poorly managed goals (as per convergence theory).
One illustrative way of explaining the difference between concerned and sceptic camps is that the difference between c30% and 1% extinction risk is to give a range of 70%-90% probability to each of the events listed above.
Obviously, that's a very simple way of looking at it. Individuals probably believe some of these steps are nearly certain and some are more likely (incremental steps) once others are achieved.
Nonetheless, I don't really believe any of the steps follow in a self-contained logic statement with 100% certainty from the other. Other people may disagree. There can also be risk without a particular step, but to end up with a single masterful AGSI its hard for me to believe we can miss many of them.
One explanation that came to mind is the standard explanation for the popular observation of 'Experts keep saying there will be a crisis but it never happens', which is that the experts are agitating for change, they succeed, and the crisis is averted.
Is it possible to experts are arguing/estimating from an inside view of 'this is what will happen if we, the experts, don't fix it,' while the forecasters are estimating from an outside view of 'this is what is likely to actually happen, including as a result of all expert's efforts to fix it'?
It seems to me that this would be the standard mode that before groups are used to thinking about and working on these problems from - the experts trying to plan their own efforts to fix the problem and convince people of the need, vs the forecasters including those efforts in their predictions.
Existential risk is obviously a scam. Domain experts have a vested interest in raising the profile of their pet topic. "It's gonna kill everyone" is the easiest, cheapest, most manipulative way to do so. It's so easy, you don't even have to consciously know it's a trick to be able to pull it off.
There is a certain analogy to judging the trustworthiness of forecasters by their their predictions of well-understood bio risks. It is sort of like testing the alignment of an ASI by asking it questions about a situation you understand, and hoping that it will also act in your interests in more complex scenarios.
For the forecasters, the most you can learn from that is that they are not stupid. The historic frequency of pandemics is common knowledge, if a superforecaster can take a reasonable guess at the mean time between pandemics, you just learn that they are not statistically illiterate. Just like an ASI telling you "I am not allowed to crash the moon into the earth" does tell you little about its alignment, just sets a mild lower bar on its intelligence.
Re the non-anthropogenic catastrophic risks: 0.1% looks very low to me. Carrington events happen. If there is a 0.5% per year probability of a Carrington event, over 70 years this is around 35%. If the power grids of the world (and quite a lot of other electrical and electronic equipment!) are knocked out for a few months, a death toll of 10% looks quite plausible.
Experts will be more worried about their domain than outsiders
1. Selection bias: people worried about x are more likely study it.
2. Economic pressure: the field as a whole will command resources in proportion to the seriousness of the problem it is solving
3. Partisanship: People become partisans when they dedicate their life to a problem and partisanship makes people think less clearly
4. Availability heuristic: Domain experts think about their problem more often which skews their view of it.
Anecdotally, I knew a postdoc who studied food born illnesses who monitored his personal refrigerator and would throw away everything if the temperature briefly exceeded 4°C.
I trust the forecasters here.
Well, I was wrong about 1935. Actually, the physicists did were wrong 2-4 years earlier.
"There is no likelihood that it will ever be possible to harness the energy within the atom." - Arthur Compton, 1931
"The energy produced by the breaking down of the atom is a very poor kind of thing. Anyone who expects a source of power from the transformation of these atoms is talking moonshine." - Ernest Rutherford, 1933
"The idea that atomic energy is available for practical use is just an illusion." - Albert Einstein, 1932
The fact the fission was not yet discovered is irrelevant. That's like assuming that no new discoveries (like Transformer) will be made in the field of AI in the next 5-10 years, that AI is going to be created exactly with the same methods it is being created now. Obviously that is a very comfortable assumption, but totally irrational. There are going to be new discoveries, new methods and much more powerful computers - but of course AI "will never be a threat to humanity because I say so" LOL
> When we asked them to forecast again, conditional on AGI coming into existence by 2070, that figure rose to 1%.
This is the most damning part of the predictions. It implies they're 75% sure that AGI isn't going to arrive before 2070, which is just not a reasonable prediction if you've been paying any attention.