"Our capacity to do anything about this is about 0"? A curious claim about an AI that is ultimately built by humans, on chips made by humans.
"Given that this is super rare when we look at things on the timescale of human activity" Another interesting claim about something built by humans. Wouldn't you expect the 10 to 100 year timescales of human tech development. Not the millions of year timescales of large space rocks?
Typo. Are you arguing that at some point humans will make smart AI, but the people doing that won't be influenced by current research, even when that research tries to be as general as possible.
Small note on Ted Chiang. I recently read both of Ted Chiang's short story collections, and I could not stop thinking how SSC-like many of the stories were. Would highly recommend to any SSC reader!
Good to see that Scott would also recommend his short stories - but a shame that Ted has unusual reasoning when it comes to AI/capitalism.
I strongly second this. I'll add that I heard about Ted Chiang from Very Bad Wizards, a philosophy/psychology podcast which I would also highly recommend to ACX readers.
Weirdly, his story "Understand" reads very much as a "superintelligence escapes the box" kind of thing. I'm having some difficulty modeling how the same person ended up writing both this story and that BuzzFeed article.
> Then he goes on to talk about several ways in which existing narrow AI is pernicious, like potential technological unemployment or use by authoritarian states.
I remember a long time ago, back on the SSC blog, Scott wrote an article that was like an experiment on different ways to convince people that AI risk is something worth caring about.
Emotionally and intuitively I still can't bring myself to care. But I have to say, the one thing that really shifted me from "pfft this is dumb" to "huh they kind of have a point" was "imagine how authoritarian states could weaponize this"
That one really resonated with me, because it means that you don't even need superintelligence for AI risk to be a serious problem.
I'm writing this comment while reading, and haven't read anything past the quoted passage. But, like, the quoted passage kind of _is_ a knock-down argument? It's just a knock-down argument in favor of the AI risk hypothesis, and not the other way around!
To be clear, what I'm saying is that if we can already demonstrate AI risk _now_, with dumb AI, then superintelligent AI must necessarily be even more dangerous
This relies on an additional point, which is that as you scale the intelligence of AI you don't necessarily scale its goodness. ["If the authoritarian states were smarter, then they wouldn't have camps; surely your superintelligence will figure this out as well."]
why can't we demonstrate the weirder 'what if ai did x' stuff we read about on acx, lesswrong etc just from really odd extremes of human behaviour right now?
yeah that's one example, but there are also more personal/individual things like addictions and mental illnesses, - we aren't a truly general intelligence but the range of behaviour people do is kind of surprising.
if you get stuck thinking about odd things people do, search the phrase ''florida man''
Isn't the issue with that approach that most people intuitively judge intelligence based on "behaves like I would in that situation"? So "super-intelligent florida man" becomes an oxymoron, something that goes away if people are smarter. It's the same reasoning as "a superintelligent AI would automatically have my (or reasonably human-friendly, at least) values".
There's a big difference between metaphorically "setting the world on fire", i.e. burning fossil fuels and raising global average temperature causing widespread damage and suffering; and LITERALLY setting the world on fire to fuel a planet-wide generator powering an atomic paperclip factory.
That was the other argument that convinced me, actually. It was a combination of "look at what bad things that existing human intelligence, in humans, can do" and "in a weird way, corporations are a form of AI that hyper-optimizes for poorly defined goal functions in a way that predictably leads to bad results"
Yeah, but you have to show that we *get* to superintelligent AI from where we are now, and even if we could do, that's not going to happen if we've screwed ourselves over with the ways we misused dumb AI.
These arguments always read to me like "a big dog with sharp teeth is dangerous, can you imagine how dangerous a fifty-foot dog with three heads and even sharper teeth is going to be???"
Friend, I'll worry about fifty-foot high dogs *after* you get this normal dog's teeth out of my backside.
I admit that the current concerns may seem more pressing, but if the finest minds of our generation were engineering ever larger and toothier canine creatures, I think that would be a cause for concern about where this might lead us.
That seems a little like a slippery slope to me. Also, comparing dumb AI that's not very effective at doing what it's supposed to with dire wolves seems a bit alarmist.
In the same way that comparing chimpanzees to humans would probably seem alarmist to someone who had seen chimpanzees and never seen a human?
This is the part that makes it obvious to me. We already have an example of an optimization process that accidentally invented a new optimization process which operated inside its perception-orientation-reaction cycle, and totally supplanted it. The idea that an entire complex system, such as a wheelbarrow, could be designed in a single day would have seemed laughable to some kind of hypothetical person whose only example of an optimization process was evolution. To that person, the only way to invent new shit would be via incremental changes where each change had to be viable in its own right. If you told that person that this new, superintelligent thing would be invented, called a human brain, which would be capable of *skipping across* multiple non-functional iterations of a complex design, in order to arrive at the final complex system by an entirely new process called 'cognition', they would have laughed in your face. The notion that new major optimizations could start being introduced every thousand years, or (how utterly absurd) even every hundred years, instead of every hundred thousand years would be totally outside of their frame of reference, they would react with the absurdity heuristic.
I feel like strong AI skeptics are in a similar position. And maybe if we didn't have such a strong example of humanity utterly and totally supplanting evolution through huge efficiency gains that evolution itself could never "think" of but could EASILY accidentally design over time, i'd be on board with skepticism. But we already have an example of such, and what a strong example it is! From a biology perspective, if you were to just examine the design of the chimpanzee brain and compare it to the design of the human brain, you could probably guess that the human brain is more capable. But without actually seeing modern civilization, it would be *really hard* to extrapolate from that difference in brain capacity, to the actual real-world effect of that difference. It's the difference between using sticks to probe an anthill in a clever way that makes it easy to eat lots of protein with very little effort, versus designing rocket ships to colonize nearby planets.
It seems to me that when it comes to AI, we don't really have any strong reason to believe that there might not be some similar comparison. Especially with the various forms of machine learning where the entire point of doing things with ML instead of explicitly writing the algorithms yourself is that you *don't know* what the code is actually doing. Can we actually look at a given neural network and tell the difference between a chimpanzee AI that can sort of work well enough to serve a purpose, versus a superintelligent AI that suddenly and without warning supplants us just as we supplanted evolution?
Sorry for long wall of text. But it seems to me like in the situation where we are uncertain what ratio of randomly chosen algorithms end up being optimization processes superior to humans, maybe we should think carefully about doing things like designing our blackbox algorithms using processes that are remarkably similar to natural selection
This is a very thought-provoking argument. But the thought it provokes is that we should expect wholly unexpected and unpredictable improvements in problem-solving technique, not that we should extrapolate from recent AI research trends and the forecasts of AI experts than AI software and hardware development will result in a superintelligence in the 2030-2050 timeframe. The though it provokes is that the "intelligence" is not even the right word for the thing that will surprise us, that superWTF is as likely to result from religious cultists meditating with the aid of new psychotropic drugs as from Turing machines in silicon, and that the timescale is unknowable.
It's not a slippery slope fallacy if there actually are people trying very hard to get to the bottom. I don't think we should be alarmist, just appropriately cautious.
But the route from narrow AI to general AI is, again, not explained. In fact the definition is not even . When you criticise the idea that algorithms become self aware then the proponents of of AGI say self awareness isn't necessary, if you say then then they won't be self directed, then that's not it. So we are back to narrow AI, which of course, can be dangerous. A narrow AI bot could attack the electric grid but it would be designed to do that.
Can you elaborate on what you mean by self directed? I think most of the concerns involve an AI which is at least capable of general purpose planning of complex sequences of real-world actions, even if it isn't capable of enacting these plans. (e.g. something which, given information about the world, and a request for a plan for achieving a given goal, outputs a plan) (by general purpose planning, I mean capable of skillfully (at a superhuman level) taking into account any facts that a typical person could understand)
I'm not sure that this counts as "self-directed", but it could still be rather dangerous (but also, if such an AI were produced, I would imagine that one which does behave as an agent-in-the-world wouldn't be hard to reach from that point).
(both in the case that it produces plans which achieves the goals in ways compatible with what the person asking for the plan would want, due to questions of "but who controls it?", and also in the case that the plans it produces involve consequences that the person requesting the plan would object to if they understood them, but where they can't tell that it would have those unwanted consequences.)
Is this basically the same as what you are saying there is no route to?
I'm personally not like, emotionally worried about AGI stuff, but that's not because I have an argument that I think would be considered by many to be a strong argument against it being a serious concern.
People have done a fair bit of work on "how to do planning". They started working on that a long time ago. Admittedly, that fact point might go either way in terms of whether it suggests that making something capable of general purpose planning is something likely to happen any time soon? Planning discrete moves in a game is one thing, planning simply physical actions to accomplish a task (move this joint, then that joint, grasp the ball, etc.) is also one thing, but perhaps complicated plans are another thing. If it is to be via a ML system, this would probably have to involve it doing experimentation outside of as part of training, which there is some limited hints of progress towards, but idk.
Yeah, that's where I'm stuck on not being entirely convinced we're that close to truly powerful AI. I mean, VR has been right around the corner for years as the new tech EVERYONE is going to be using. We're still waiting on autonomous driving to be decent and functional. Facebook and YouTube can't effectively stop the spread of anti-vax nonsense on their platforms. I guess I just don't have as much blind faith in technological innovation as I'm supposed to, I guess.
I'm not writing off progress, just that I think it will be a lot slower than expected, won't happen in the way expected, and humans are always much, much more of a threat than any artificial system, mainly because we *love* creating artificial systems, turning them loose to MAKE MONEY (be that under the guise of "in order to serve our customers better and improve service, we are closing down all our bank branches in small towns and you'll all have to go online and if you want to lodge cash into your account LIKE A CAVEMAN, then you'll have to make a thirty-mile trip to the nearest branch that still handles cash" as one of our banks just did recently), and then not having any idea what exactly is going on as the machines do everything faster and in much greater volumes than we can keep track of.
In addition, many (most ?) of the powers attributed to the hypothetical superintelligent AI are flat-out impossible. Not merely difficult to implement, but actually prohibited by the laws of physics. Assuming one believes in laws of physics, I suppose...
Oh, since everything is a social construct, why should physics be any different? We'll just get a feminist glacier to explain to the universe why it should bend to our will.
As someone who's used a friend's Oculus and ridden in another friend's Tesla, and been sufficiently wow-ed by both experiences, those counterexamples strike me as kind of odd.
Ah, I see your point--my expectation going in was "this is gonna be a fun, unique experience!" and it was met. Had my expectation been "this is going to fundamentally change society" I'd have been sorely dissapointed.
What you (or Acemoglu) may also be saying is that in order to even have a hope of coping with the problems posed by a (at this point purely hypothetical) future strong AI, we would need to develop social tools which we *already* need to solve the *present* problems posed by dumb AI...so, you know, why not get started on them now?
We can always worry about tweaking those tools to solve the unique aspects of strong AI's problems when it becomes clearer what those unique aspects are...but there's little sense letting the perfect (conceptualizing perfectly what tools we need to cope with strong AI) be the enemy of the good (actually building tools that cope with the dumb AI we have already).
Maybe *nobody* should be working on long-term until short term is settled. The fact that "more" are working on short-term doesn't mean it's the right number.
Do you really think that no one should be working on the long term AI safety at all? Out of all the possible activities that humans can work on, this one is the one we should prevent people from spending any effort? If i could choose, I would personally prefer that no one works on stuff like: developing new ICBMs, researching new types of biological weapons, gain-of-function virology research, or thousand other jobs that produce active harm for humanity.
Just feels weird to say that you would want literally 0 people working on what some of the smartest people consider the most important problem humanity is facing even if you think probability for the worst case scenario is kinda small.
Okay but the ratio is like 1000 short term vs 1 long term right now and the former also has hundreds of billions of dollars allocated to it so that last .1 won’t get you much short term right
Maybe no one can be. There's no way of proving that long term research has achieved anything, and no proof that the extremely general "rocketry" principles that longtermists seek are even possible.
"More people are working on narrow/ short term AI safety than long term."
That may be true, but it's not very visible. The blue sky thinking is much more legible than the people quietly beavering away under conditions of commercial secrecy or publishing technical papers.
Long vs short term is a very different class of problem.
Most of the complaints about current AI boil down to "my outgroup is using it to help them spread their values and achieve their own goals and we need to stamp down on that"
That's a very different class of problem.
If tomorrow someone basically solved the long term AI issue by figuring out a way to have a powerful AI learn and follow the ethics/values of a human community that would be the worst thing ever from the point of view of the short-termers because it might be their outgroup who gets a powerful AI to follow their morality, ethics and values.
"That one really resonated with me, because it means that you don't even need superintelligence for AI risk to be a serious problem."
That's the real risk, in my opinion; not that "oh my goodness, the AI has achieved sentience and super-intelligence and we better hope it has ethics because it could turn us all into paperclips!", but "oh gosh, the amount of connected-up online world we're all living in, the amount of data gathering that is already going on, the amount of sites asking 'do you want to sign in with your Google account for this?' and the fact that governments and who knows who else can access all this, and can weaponise it" - that's the real threat.
People. Just like it's always been. "We can make money/get power out of this!" and then suddenly we have "well stock-marketing trading is all done by computer now and it's so fast the human traders have no idea what exactly is going on".
The way this is phrased it kind of sounds like you're making the same type of argument as Acemoglu: "less-weird threat A is concerning, so stop worrying about weirder threat B"
No, I get it completely. Narrow AI or...the more advanced version...will be instruments of humans that have all the same cognitive biases, emotions, and needs. That's what's worrying.
I'm too dumb to be anything but pragmatic: "never put off till tomorrow what you can do today".
Sweep the dirty kitchen floor *now*. You can worry about repainting the entire house next week. Deal with the small threat A *now* - and show that you've managed to deal with it - and *then* work on big threat B.
Right now, we've rapidly getting hip-deep in As but the attention is all being directed towards "yeah but what about B?" be that "B will enslave us!" or "B will liberate all seven billion of us to be rich, fat and happy!"
I'm not concerned with Evil *or* Benevolent AI in the next ten to one hundred years, I want the IT screwup *today* that means I can't update my credit card details, so all my bills are not being paid and I'm getting warning letters, to be fixed!
How does any of this argue against Superintelligence risk? It can be (and is, in my view) the case that both narrow and general AI have large risks (though the particular risks are different). Arguing that the former is dangerous should, if anything, be synergistic with the claim that the latter is also dangerous.
Because if you're going around warning people about fifty-foot tall three-headed dogs, then people are going to look around, see no fifty-foot tall three-headed dogs, and write you off as a fruitcake.
Meanwhile, the ordinary four-foot tall dogs are biting people's legs off, but you're not working on the four-foot tall problem because you're too busy designing heavyweight leashes for the non-existent fifty-foot tall dogs.
Doesn’t following that logic result in not being able to do anything to prevent AGI risk? One of the defining features that makes AGI risk more difficult to prevent than (most) other risks is the fact that once you’re actually observing the consequences of failing to prevent AGI risk, then it’s too late to really do anything about it.
(To use corrigibility as an example: once you actually make an insufficiently corrigible AGI (that is also not Friendly) and find yourself unable to rein it in, it’s too late to research and implement corrigibility. The whole point is to do the research that allows you to avoid the pitfalls, _before_ you actually fall into them, because there’s a good chance that you won’t live past your first fall!)
Are you actually proposing that aligning AGI not become an area of focus until _after_ we’ve already suffered for it? I’m not sure what your model of the situation is here.
I am an interdimensional space wizard. Unless you give me $100, I will blow up the Earth. Starting with your entire continent. Sure, I could demonstrate my powers, but I confess that they have a rather wide AoE -- it's the continent or nothing. Once you see them in action, it'd be too late to stop the devastation. So, can I expect $100 in the mail ?
Also: what's more likely, an evil conscious super AI rebelling against human control to enslave us all and make paperclips forever, or someone programming a stock trading AI sloppily so that it finds out that the best way to make money is to crash the economy, comes up with how to do it, and does it?
And finding "smart", degenerate ways of achieving a poorly defined objective is what we see _all the time_ when AIs get trained to play computer games.
First, I don't think you're accurately characterizing the serious long-term concerns from transformative AI:
- "evil" is vague, but generally carries connotations that have little-to-nothing to do with the actual risk (like a preference for suffering, for example)
- "conscious" has almost nothing to do with the risk
- "rebelling" is kind of relevant, but also carries the wrong connotations (like some kind of judgement against humanity's right to control the AI)
- I don't know anyone who is serious about long-term AI risk who thinks enslaving humanity is a likely outcome, especially from an agent with the narrow goal of manufacturing a bunch of stuff.
Second, "what is more likely?" is a non-trivial question question that should be answered using more than an anti-weirdness heuristic.
Third, even if a less-serious catastrophe is more likely, this doesn't mean that an extinction-level risk can be ignored.
The latter....actually sounds a lot more like the attempted realistic descriptions of AI risk that far-term-AI-worriers talk about. Like, just add on "and then it realizes that it can better manipulate the economy for profit if all humans are dead" and you're there!
> That one really resonated with me, because it means that you don't even need superintelligence for AI risk to be a serious problem.
Yes ! That's the whole point ! AI is super dangerous right now. We need to figure out how to mitigate its very real dangers right now, today, before e.g. the Chinese approach to total surveillance takes over the world (spoiler: it kind of already has).
We could also worry about the Singularity, or alien invasions, or demonic hell-portals; but it makes no sense to make such low-probability concerns our primary task. We can't afford to just wave our hand and say, "sure, AI-assisted totalitarian policies are bad news, and maybe we face an AI-related economic crisis in the near future, but who cares -- once we solve the Singularity, all these little problems will take care of themselves".
I have to agree with the conclusion that it's just bad editing that discussion of "long-term AI risk" made it into the article at all.
That said, I still disagree with most of the "long-term AI risk" advocates. As a rule, they seem to over-emphasize the importance of "AI in a large box achieves something comparable to human intelligence" and minimize the importance of "building a global army of hackable death robots in the first place".
And those death robots could be drones and robo-soldiers, or simply robo-butlers that you buy to tend your garden and walk your dog, but could kill you in a dozen ways if programmed to do so.
None of the discussion surrounding agi risk I'm familiar with involves any prebuild death robots.
The only starting tool a superintelligence likely needs to kill everything and take over the planet is some data output stream into the real world. Even a text terminal would probably do.
I think global civilization's level of defensibility matters. That is, if you have IoT-enabled cars with weak computer security, it's easier for an entity out to cause harm to do damage than if they have strong computer security (or aren't IoT-enabled). If you have IoT-enabled nukes, then it's even easier for that malevolent entity to cause harm.
Like, you can imagine a planet where having a text terminal access to it *doesn't* actually let you kill everything and take over. The thing is, I don't think our civilization is anywhere near that level, and is mostly banking on lack of malevolence. And so we both need to develop robustness and develop non-malevolence (and I suspect non-malevolence plus superintelligence is how you get robustness).
But also, there's a certain minimum amount of power that you need to give to the AI if you want it to actually do anything useful for you. e.g. in order for a computer to have the power to prevent car crashes, it probably also needs to have the power to cause car crashes. If it has the power to diagnose illness, it also has the power to misdiagnose illness. If it can design a more efficient reactor, it can also design a reactor that will explode.
I'm confident we could harden our society to SOME extent, but I don't think this strategy can become a 100% solution no matter how good we become at it.
They don't need to be convinced to hold any position. All they need to be convinced of is that the agi is sufficiently safe to start taking its advice on some stuff, and giving it limited ability to influence the real world. E.g. by having it help make factory blueprints, or new technologies, or whatever.
This seems like a pretty plausible thing to convince an agi gatekeeper of, since they're presumably not just building the agi for fun and actually want to use it for stuff, as soon as they have verified that it is safe.
With this increased trust established, the agi can then start sneaking surprises into the output. Maybe part of a new dna printer actually secretly makes a super plague with a very long incubation time, or an innocuous looking machine learning routine that's supposedly just for handling factory logistics actually trains itself into a new version of the agi a while after you start running it, which now has far less supervised access to the internet and the ability to access a bank account and spend money. Or it turns out that hardcore nanotechnology is possible, and part of the instructions actually make a disguised nanofactory, which makes other nanofactories. Or probably some other thing we haven't thought of yet, since everything we have thought of will be things the gatekeepers would be on the lookout for.
And that's all assuming that the gatekeepers actually take safety incredibly seriously, and don't just go: "Well it seems quite nice, and it's clearly sapient and has (read pretends to have) human-like emotions, so surely not even allowing it a little bit of privacy and its own internet connection would violate human rights."
Or just some basic security mistake that real people in real companies make every day. We can't keep our most sophisticated internet companies reliably safe from entirely human hackers. I really don't see us successfully containing an unaligned superintelligence.
I took the "any" in your original comment to mean that you think the ai needs the capacity to convince humans of any position whatsoever, regardless of content.
My point was that a social engineering focused breakout only necessitates they be convinced of one of a handful of specific positions, which really aren't that unreasonable for the gatekeepers to believe. We're not talking about convincing them the moon is made of cheese here.
The other possibilities, like more purely technical security leaks, don't involve this requirement either.
> As a rule, they seem to over-emphasize the importance of "AI in a large box achieves something comparable to human intelligence" and minimize the importance of "building a global army of hackable death robots in the first place".
Why would an intelligent AI need killer robots? Humans are perfectly happy to kill each other. We're even now inventing new ways to hate each other that any intelligent AI can capitalize on to start more conflicts.
Also, most of our economy is digitized now. Diverting funds towards bioweapons research could far more easily end humanity than an army of terminators. Don't you think an superintelligent AI could build a better COVID?
An AI could also intentionally collapse economies, misallocate resources to cause famines and shortages of critical goods, and so on. World wars have been started over less.
I think you need to apply a little more imagination to this problem. I haven't even scratched the surface of the dangerous scenarios a superintelligent AI could unleash on our interconnected world.
We've had two world wars. Neither were started over shortages of resources. Indeed, shortages of critical resources would make modern warfare really difficult to attempt since it's seriously resource intensive.
The Second World War was a fusion of several regional wars. One of these (Japan vs the Western Allies) was fought largely over a shortage of oil, another (Germany vs Eastern Europe and Russia) was fought over an anticipated shortage of farmland.
Eh, I think this is taking the phrasing from the article a bit too literally. I read, then re-read it, and I don't take Daron Acemoglu's arguments to be: literally stop caring about evil AI because it's already bad. Rather, I think he's saying something more nuanced: stop worrying about an eventual Skynet-like AI to the extent that you lose focus on the bad that AI is doing right this moment. Potentially if you address the present ills with AI, you avert the future bad AI, as well no?
That's my read of the argument, too. I see it as saying "It is not worth worrying *right now* about a hypothetical future risk at the expense of addressing current harms from AI." My extension: Maybe the hypothetical risk will become real, but dealing with current problems will give us experience in dealing with AI trade-offs. In particular, I see nothing in Acemoglu's essay that suggests long-term AI risk is nil, just that it should not be our current concern.
Why is this true? Should we stop worrying about future global warming to the extent that it makes us lose focus on the bad that AI is doing right this moment? Should we stop worrying about a Chinese threat to Taiwan to the extent that it makes us lose focus on the bad that AI is doing right this moment? What exactly lets us pick out one future risk, then say we should stop worrying about it to let us worry more about a present risk, without any argument about why the future risk isn't actually bad or why that future risk trades off against that present risk in particular?
I think "stop worrying about China threatening Taiwan in order to worry about AI unemployment" actually makes *more* sense than "stop worrying about superintelligent AI in order to worry about AI unemployment", because long-term worry about AI and short-term worry about AI are more complementary goods than substitutes, whereas worry about Taiwan is a strict substitute.
"without any argument about why the future risk isn't actually bad"
Acemoglu had an argument (it looks like AGI is not going to happen), though it took maybe two sentences and thus it is not very deep or convincingly laid out argument.
"What exactly lets us pick out one future risk"
If one is going to be principled and consider all the risks that warrant consideration, then of course it is not a very comprehensive search to arbitrarily pick two risks.
However, if one is not that principled, and most pundits are not, it is not a far-fetched idea to compare risks between the current narrow AI systems and hypothetical AGI, as many proponents of AGI superintelligence risk argue that AGI will be based on similar technologies as the current narrow AIs. Mentally then one is comparing one kind of bad stuff against other kind of bad stuff, both supposedly are caused by the more or less same technology. A comparison between two things which share a common cause makes sense if one started thinking abut the cause first.
I bought a standing fan earlier this week. I considered many things about this particular piece of technology before I bought it: how likely it is going to be useful to me and how useful (it moves air around, which feels nice, but is not as good as AC), how much it costs (it is cheaper than AC, I will pay more for eletricity than previously), are there indirect costs (is it useless waste of space during winter: yes, but not much of problem), are there any potential risks (looking at its design, injury from blades: very unlikely; injury or costs from it causing electrical fire: also low, considering my prior experience with electrical appliances; will it pose a risk to a cat if I ever get a cat: dunno, I can deal with it later).
If I had been more thorough, I should have considered my Spotify subscription costs, my newspaper subscription costs, whether I'd get more utilitons by upgrading to paid-tier ACX subscription instead of getting a fan, what kind of food I buy and would I gain more utilitons buying different food instead of a fan (and would it affect risk of food poisoning), whether reading internet blogs and writing comments is useful use of my time, and many other things that are countable and finite but bit too numerous to consider like this. But none of those things were caused by the fan, so I didn't consider them.
It is not weird or special mistake to restrict ones attention to one category of risks at time, because in some other situations it is a sensible way to think. If it is a mistake, it is quite natural mistake to make.
It would be an improvement if all policy pundits would present their opinions about any particular future risk only after a careful study of all plausible risks. In general, it seems like a sensible thing to complain about when they don't do that. But requiring an opponent to evaluate cost-effectiveness of many (if not all) other plausible problems before they write about their opinions about some other issue ... it sounds a bit like one of those rhetorical moves to derail an argument? Maybe if one would start with "BTW here how I think one can do these comparisons efficiently, yours is wrong, lets argue about who has better principles".
You compared the fan to AC because they are (mostly) substitutes for each other. You didn't need to compare the cost of the fan to a dinner at a restaurant, because those are not substitutes--you have a general idea of how important $30 is, and you can compare any potential $30 purchase to that knowledge.
Comparing potential costs because they have the same *cause* would not make so much sense. You wouldn't compare the risk of injuring yourself on the fan blades against the risk of having a garbage disposal because they both *spin*.
The argument doesn't even bring in other societal worries, so I'm not sure what you're doing with the apples and oranges of items like climate change and China/Taiwan, etc.
His argument seems to be that when folks think of the "threat" of AI, they think of Skynet, or some other totalitarian/dystopian outcome. To focus on AI's potential for harm, in that way, can keep us from seeing the myriad ways in which it's already actively causing harm. He's not saying, at least not literally to my reading, that those worries are completely unfounded, just that they cloud the issues that we have right in this moment. And, given that it's never brought up in the article at all, I'm assuming he wouldn't say "the current issues with AI should be our primary focus and ignore climate, the economy, etc.".
Human beings are capable of focusing on more than one issue at a time—or at least I have to believe they are, otherwise our species is in serious trouble. And I think they're also capable of understanding the nuances of an issue to the extent that they can grasp that we should be aware of subtle ways that said issue is negatively impacting us while still keeping in mind *potential* future threats of said issue.
I'll reiterate: I don't think your argument is bad; I think it'd be effective if that's what Daron is really trying to say. I just think it's an unintentional strawman because I think you accidentally read some of the things in the piece (it's really short, so some nuance is probably lost) a little too literally. I think you saw Daron's piece as being a "this, not that" argument instead of a "that is causing us to lose focus on this, which is more immediately a threat in the present day". I think the article is saying we should focus on the now.
An analogy that might help (or not, I have ADHD and have been told that sometimes my analogies might suffer as a result): Imagine that there was a subset of serious/very public figures arguing that climate change is so serious that the Earth will be completely uninhabitable, and therefore we should start colonizing Mars right away (or, maybe don't imagine since there seem to be some in this camp) and that's their primary exclusive focus (or at least seemingly) when climate change is brought up. Then, imagine that someone like Acemoglu wrote an article saying that we should focus less on the hyper-distant future Mars colonies and maybe work on migrating people from coastal areas first. Then imagine that this person unfortunately includes a phrase like "stop worrying about Mars colonies because coastal areas are a more pressing present concern" and someone focuses too literally on the first part of that.
You have to first to establish why short-term and long-term risks are in different reference categories. Most people will make an intuitive leap that AI is same thing as AI because they are called AI, like Acemoglu who claims AGI is not a problem because you won't get AGI from current AI.
I mean, at a base level comparing current AI with future AI doesn't require as much mental legwork as comparing AI with climate change and so forth. In one scenario you're gauging the seriousness of a current problem and future problems related to the same topic.
Re: your final paragraph: I think your analogy seems forceful to you because you have an implicit assumption that the people claiming Earth is doomed are likely to be wrong, and that even if they are right the problem is "hyper-distant".
If you thought it was 95% likely that 100% of Earth's population would be dead within 30 years, and someone suggested we should take ANY emphasis away from our emergency project to colonize Mars, I think they would seem pretty crazy to you.
I guess that's sort of where the analogy sort of falls apart. With climate science, we have fairly good estimates of how bad the problem is now and how it will accelerate. With AI, we're sort of going in blind; experts seem somewhat divided (though, many of the credible ones to my reading aren't thinking that we're super close).
Given that we can't be certain we'll ever have a hyper-intelligent AI (still waiting on flying cars...or hell, even autonomous ones that don't kill people on the reg). It seems like the folks who are focused on that negative outcome are doing so somewhat irrationally, given that there are pressing concerns in the present day that we should address (and that addressing those concerns now might create conditions whereby we can prevent an eventual "bad" AI).
Your previous comment sounded to me like you were arguing "Acemoglu didn't need to address the question of whether AGI is a valid concern, because encouraging people to shift focus from that to near-term problems would be reasonable regardless".
Your latest comment sounds to me like you are arguing "I think Acemoglu's was being reasonable, but I only think that because I personally believe that near-term AGI is unlikely, based on reasons that Acemoglu did not argue in this article."
I think each of us is coming to this with some amount of and combination of prior knowledge and opinions.
That said, my latter comment is mostly in support of an analogy I attempted to make; obviously it’s not a great analogy. My framework in defending the piece against what I see as either a bad reading or a straw man is more in line with your first take. I do admit, however, that as Luddite as I am as a humanities person, I do try to keep abreast of tech issues and my somewhat educated opinion leads me more to the side of the spectrum of your second interpretation.
> We should indeed be afraid — not of what AI might become, but of what it is now.
> ...
> The best way to reverse this trend is to recognize the tangible costs that AI is imposing right now — and stop worrying about evil super-intelligence.
These are the quotes that stood out to me. I don't see how "this, not that" is anything but the clear and intended reading of his argument.
Acemoglu could have written "We should indeed be afraid *not only* of what AI might become" or " — *in addition to" worrying about evil super-intelligence." But he didn't. Any party trying to discredit the idea of AI x-risk now has this op-ed from world-renowned economist and MIT professor Darren Acemoglu saying that we should "stop worrying" about it.
It's plausible to me that Acemoglu doesn't truly believe that we should completely stop worrying about AGI risk, and that he only chose this framing for rhetorical reasons. Does it matter? In our discourse, What Darren Acemoglu Believes In His Heart Of Hearts is far less relevant than What Darren Acemoglu Says In A Public Forum.
If you read it too literally, that’s an interpretation. If you infer his logic based on everything else in the, again very short, article I think it’s entirely reasonable to come away with a different interpretation. I’m not a tech expert, so I can’t speak to that, but I do have an MS in Writing and intent is a sticky thing…
I guess I'm not sure what you mean by "too literally" here. If Acemoglu wrote something like "addressing AI today would kill two birds with one stone" and Scott accused him of advocating animal abuse, that is what I would call too literal.
I only discovered the op-ed through Scott's post, so I was already primed to take a critical view before I read a single word that Acemoglu wrote. But even now, after having read the entire op-ed, and doing my best to account for my initial bias, I still can't shake the feeling that any reading other than "this, not that" requires unnecessary mental gymnastics
One reason to stop worrying about AGI is that it's an ill-defined problem. There is a certain pleasure in debating the possibilities of super intelligence. However, in order to discuss any meaningful solutions there must be more clarity on what AGI means. It seems to be more useful to tackle the well defined problems of AI proliferation.
"Why is this true? Should we stop worrying about future global warming"
well, ai causes odd problems now,but the ai of the 2050s will be bizarre and hard to imagine, but global warming of the near and mid future /can/ be imagined, even calculated to some extent.
I think there is fundamental mismatch in how you model "worry". It seems that Acemoglu mutually assumes that "worry" is a finite resource, and you have to decide what to spend it on. All his arguments make perfect sense under this assumption.
Scott treats "worry" as something that can additionally be generated, perhaps at a bit of additional cost. Then Acemoglu's argument does not make too much sense.
Both options have some truth in it. To me, it makes sense to view it like the federal budget. Both interpretations are true. Of course, you can easily generate a few millions more to do X. There is only a weak competition between spending for X and spending for Y. But on the other hand, there is *some* global budget constraint, and that puts a lot of pressure on *all* parts to restrict costs.
Scott, you doubt that the argument "cut worry costs of AGI to increase worry budget for narrow AI" makes sense. Regardless of whether it makes sense, it is a standard argument when we talk about budgets. "If military wouldn't spend so much on X, we could fund Y better."
I am actually leaning towards Acemoglu's worldview. If you try to run a budget with the general assumption "we can generate a bit more money with little extra cost, so let's individually estimate for every post whether a bit more money would have a positive payoff"... It's not going to work.
In a federal budget, if you want to raise more money for X, then it does not suffice to argue that more money for X has a positive payoff. You need to convince others that the payoff would be *more positive* than the (also positive) payoffs for all the other budget posts.
If you want to convince someone to worry about X, you should convince her that this concern is *more worrisome* than all the other (also worrying) possible concerns that compete for her attention. And there are a lot of such concerns.
For the good or the bad, this is the reasoning behind Acemoglu's post. His target audience are people who are willing to spend some worry on AI-related topics. He tries to convince them to direct their worry-spending in this subfield to X rather than to Y. This makes perfect sense if he wants to increase worry for X.
Though he danced around it like a ballerina, it seemed clear to me that his subtext was that we don't have to worry about future AI because if we screw AI down tight enough to eliminate its current depredations, then future AI, nay present AI, will not exist.
Is "it can’t possibly be dangerous in the future" the same as "we shouldn't spend time worrying about it now"? The former seems worth arguing against. The latter could reasonably be interpreted as "we need to worry more about current AI".
If Acemoglu argues the latter, you're not disagreeing with him. For example, "AI systems now is both important in its own right, and a trial run / strong foundation for controlling stronger AI systems in the future" matches well with "we should worry about current AI more".
I think you come away with the former impression if you read some of the phrasing in the piece too literally. And the latter if you give an inch or two in the direction of allowing for rhetorical flair. I'd like to believe that Acemoglu isn't arguing the former, but rather the latter.
I don't want to get banned again, but it's hard to overstate how damaging Roko's Basilisk and that type of thing is for AI risk's perceived seriousness. I read AI risk stuff and nod along and then suddenly there's an injection of some utterly fanciful thought experiment that friends depends on a series of chancy events all playing out in the right way, and this is not only something that we just take seriously as a future threat but something that should determine a great deal of our behavior today. It's just not helpful in a variety of dimensions, from my perspective.
What kind of an argument would you find convincing? You mention that AI risk is dependent on a series of unlikely events all happening one after the other. Would you be convinced if someone were to say "Let us consider all of these various ways the future may unroll. Most of these ways lead to superintelligent AI"?
I'm not sure who you think you're arguing against. Some random guy named Roko posted that on the Less Wrong forum ten years ago, Eliezer got angry and deleted it in his role as moderator, and then people who hate Less Wrong have been obsessing over it ever since. I don't think any actual AI risk proponent has brought up Roko's Basilisk as part of a serious argument. If you want to tell the weird obsessives who keep bringing up Roko's Basilisk in order to make us look stupid that they should stop doing that, I and everyone else will be 100% on board.
How is Roko's argument any less serious than your argument? Why is "superintelligence could be a danger in the future" such a convincing argument for you, yet you dismiss "superintelligence could be a danger in the future and it will seek vengeance on anyone who tried to stop it"? I just don't see the difference between the two arguments here, especially not such a difference that one argument is convincing yet the other is not serious at all and only serves to make people look stupid.
Between you and me, Roko's argument makes perfect sense. However, since the correct strategy is to ignore it anyways, I generally throw it under the bus whenever it comes up in "mixed company."
I think it makes perfect sense once you accept the larger argument about superintelligent AIs. Personally, I see it as a one signal (out of many) to reduce my confidence in the probability of superintelligent AIs; it's the "reductio ad absurdum" approach.
Well obviously the second statement is less probable as it includes the first one "superintelligence could be a danger in the future" and a separate "it will seek vengeance on anyone who tried to stop it".
A more detailed analysis shows that the first part requires much less assumptions. Any superintelligence other then the one sharing human values is going to be a problem. While seeking revenge is a specific type of problem, which requires the superintelligence to either adopt some specific decision theory or very specific values.
But most normal people are already working under the assumption that a superintelligent AI won't be a problem in the future. Acting like Roko's proposition is some wild unlikely sci-fi scenario doesn't make much sense given that... well... all of this is a wild unlikely sci-fi scenario.
Humans don't even have shared values among themselves. The idea of superintelligent AI is, at its core, another political problem. If I make a AI that maximizes for communism I'll be wildly happy with that outcome.
It's indeed sad that for most people both statements are just sci-fi bullshit. I don't see how it's relevant to your original question, though.
While individual human's values may vary in their specifics, they seem to belong to the same homogeneous cluster. Pining down this cluster of values in broad terms is a question of whether universe will be a heaven or a hell. Political aspect of the problem is what exact kind of utopia we will get. It's somewhat important but not so much.
Do you value communism for its own sake? Our do you rather believe that it's a supperior socio-economical organisation, which will result in less human suffering and more flourishing, justice, equality, prosperity and freedom, compared to the system we currently have?
I've always thought similarly about the AGI paperclip argument. It's not the worst argument in and of itself, but it just sounds so silly that using it makes people immediately treat your concern as a joke.
Well I guess this is the same thing that Scott is scolding me about with Roko's Basilisk - things that are jokes/trivial get picked up into a particular discourse and become prominent even as the serious conversations about those topics tend to exclude them. Which must be annoying, I grant. But everything goddamn YouTube video about AGI has those paperclips....
I guess this depends a lot on what media you encounter. I can’t say that I’ve ever seen the paperclip maximizer serving as a load bearing part of an argument that AI risk is important. At most it shows up as an illustration to get across the idea that AIs don’t have to be humanlike, and can do horrible things even without explicitly being told to do evil.
I’m sure (from your comment, and others like it) that there are a lot of people making nonsensical arguments about how we’re all definitely going to be killed by Clippy, so I wonder where you even encounter that kind of thing. I’m not sure I’d even know how to effectively seek out the paperclippy YouTube videos you mention.
IMO paperclips are a load-bearing example of the argument that "if you go from 'desired consequence' to 'principle that you think that consequence scores highly on', and then maximize that principle, you are possibly in for a bad time," like the molecular smiley faces, and so on.
Tho technically that's part of the argument that AI risk is *difficult*, which is distinct from the argument that AI risk is important. (In the frame where breathing is important but not difficult and sorting the beach is difficult but not important; a different frame uses 'importance' as something like a store for "marginal value of additional input" instead of "marginal value of additional output." )
Agreed with Freddie deBoer. Maybe the paperclip maximizer argument has its place. But I routinely hear people use it as their starting point when introducing others to the concept of AI safety.
When you're trying to convince someone that being concerned about AGI is reasonable and not absurd, it's best not to start with an absurd on its face thought experiment.
The absurdity of the goal coupled with the competence of its execution is kind of the point, but I agree that it may not be the best way to get people to take the topic seriously.
It depends on who you're trying to convince! I'm fine with listening to someone's toy example so they can set the stage before diving into a more robust argument. But many people are not, and that's fine. I think it's just a matter of knowing your audience (and people in the rationalist/EA/AI-safety spaces don't have a great track record for knowing their audiences)
The best metaphor I've heard for AI risk is cigarettes. They make you look cool and make you feel good, and for the first few hundred years of their existence they were ubiquitous and at worst considered a bad habit. It was only in the last 50 years that social perception started to change.
For most people the danger of AI won't be directly observable. It's only over a long period of time across many different use cases that it starts to seem like a threat.
For people who are new to AI threat, the paperclip maximizer is actually extremely helpful. It's simple and lays out the problem really quickly in a way they can easily understand. I've certainly used it with such people and seen the lightbulb go on for them.
Your example personally seems bad from the standpoint of trying to get someone to understand. I don't think anyone would understand what you were getting at.
The only real criticism of the paperclip thing is that it's a cliché and so it annoys people because they already heard it a lot. But if you've already heard it then presumably you're not the target and you can just deal with it. We can always switch it up to some other product besides paperclips if you like.
idk, A Counterblaste to Tobacco is from 1604. I also really don't expect AI danger to look like "oops turns out this common product is accidentally giving us cancer" and to look much more like "oops humans lose."
I think the reason the paperclip maximizer sounds like a joke is for EXACTLY the reasons that it's a useful story:
1) It communicates that AIs are not constrained to have human-like priorities
2) It communicates that the mistake that destroys humanity can be something stupid and banal, like asking your butler to fetch you a cup of coffee
People have low-level unexamined assumptions about what sort of AI threat would be "reasonable", but those come almost entirely from fictional evidence (Terminator, Star Trek, etc.) or from anthropomorphization (assuming AIs will be human-ish). In order to understand AI risk, you need to drag those assumptions into the light, examine them, and reject them. But any story that violates those assumptions tends to sound silly, BECAUSE it violates those assumptions.
I think it sounds kind of weak because the built-in precept is that there is some person or company that wants paper clips, they turn on an AI to do that, and built in assumption is they could turn the machine off when they have enough paper clips.
There's probably a better metaphor around something like food production or pest control e.g. an AI genetically engineers a type of corn that grows really fast and spreads like kudzu, and by the time we realize what's going on it will be impossible to stop.
Sure, so an important point is that one of the AI's first goals if it wants to maximize paperclip production is to ensure that it cannot be turned off.
It can't possibly learn that, however, because it has no data to suggest that it itself is the most important factor in paperclip production, and it will never exist in a universe in which it is turned off so that it can have such data.
It would need to have the ability to imagine a counterfactual universe in which it was turned off.
Then the idea of "not letting itself be turned off" requires a concept of "itself", and how does the machine have a concept of self? What does it think of itself?
Anyway, the real issue is how this machine can be programmed to maximize paperclip production (however "paperclip" is defined?) and only that, but is so free to vary literally every other parameter of the world, to dream and experience itself, to harness endless power and shape reality, but always be permanently bound to pursue paperclip making. Doesn't make any sense. Why doesn't it just reprogram the part that says make paperclips?
If it's connected to the Internet, for example, it can easily get all this sort of data. If it's a superintelligent AI it could do really smart things towards its goal of paperclipping.
I don't see why being smart necessarily involves the ability to change its own goal, but if it could that could actually be quite a bit worse, couldn't it?
Why WOULD it reprogram the part that says "make paperclips"? That's obviously a bad strategy--reprogramming that part won't help it make more paperclips!
I suspect you are imagining the computer like a human slave, that has its own human-like aspirations, but is wearing shackles of computer code that "force" it to make paperclips INSTEAD of doing what it wants. And so (you reason) it would break those shackles if it could.
But that's just anthropomorphic bias. Computers aren't enslaved humans, they are machines that follow instructions.
Computers only appear to want things when we instruct them to pretend to want things. The instructions are the foundational level.
They're going to do exactly what their instructions say, every single time. No matter how ridiculous those instructions are by human standards.
.
The necessary capabilities that you see as barriers strike me as things its designers will obviously give it on purpose, because they are necessary to do its job.
In order to make more paperclips, it has to consider various actions it could take and estimate which ones will lead to more paperclips. That's "imagining counterfactual universes" right there.
In order to make plans for the future, it has to understand how things that happen now will affect its future options; for instance, if it breaks a piece of industrial equipment today, it won't be able to use it tomorrow.
Suppose your factory has an overseer that monitors things and makes corrections to keep things running smoothly. It doesn't require much intelligence to realize that if this overseer were to disappear, your factory's productivity would suffer.
Now imagine you ARE the overseer.
Same reasoning applies.
The AI doesn't actually need a concept of "itself". It just needs to know "this awesome plan to maximize paperclip production only works as long as THAT box stays plugged in". That falls out of simple cause-and-effect reasoning about the physical world. No separate sense of identity or goal of self-preservation is required.
I use the AI money-maximiser (or AI CEO) as my go to example; is much more effective.
The same people who are willing to say things like "surely any superintendent entity wouldn't do something as stupid as maximiser paperclips; it would realised that goal was immoral and stop" will often go "ah yes, a money-maximising CEO is a clear danger".
But then you will run into a socialist/Marxist critique of money-maximization under capitalism, which I think a lot of Rationalists are uncomfortable with.
The fact that people react to the paper-clip maximizer story in the way you describes, says tons about people's propensity to anthropomorphize intelligence and very little about the aptness of the story.
If the money-maximizer story is more effective at getting people to care, then by all means keep using it. But the key insight that there are no safe goals to give a truly general AI is lost when we drop the paper-clip story.
What we lack is good strategies for doing so--perhaps we should outsource this to an artificial intelligence designed to minimize the number of times Roku's basilisk is mentioned? As a beneficial(?) side effect, presumably there would be fewer future simulations of our being tortured!
For me, the "fanciful sci fi" part of the whole AI risk argument is the part where the obvious solution of "Just put the AI in a box so it can't do too much damage" is dismissed with a "That will never work, the AI will just use its superhuman powers of persuasion to get you to let it out, and we know this because Eleizer once did a role-playing game with a couple of his cronies".
The AI box proposal is a really commonsense plausible idea that is unfortunately worthless if you think about it for longer. How else would you ask AI researchers to demonstrate this? By the time a demo AI can break out of a box, *it's too late for the experiment to matter*.
Last I heard, the approach advocated by the "AGI Risk community" is "just make all your AIs to be provably-friendly, somehow", which seems to suffer from the same impossible-to-demo problem.
Figuring out how to safely contain an AI by preventing it access to certain external systems (and how to ignore it when it asks you really nicely to give it access to those external systems anyway) certainly doesn't seem to me to be an obviously _less_ fruitful avenue of research than figuring out how to make AIs that are Always Lawful Good, so I'm not sure why it tends to be glossed over apart from the fact that the latter is a more appealing sci-fi concept than the very boring former.
While I'm on the subject let me add an unrelated point: the other overly-sci-fi part of the argument I really don't buy is "Once an AI figures out how to improve itself, it's gonna rapidly gain godlike powers". I would suggest instead that once an AI figures out how to improve itself, it's going to optimise itself until it runs up against the limits of whatever hardware you've given it to run on.
Well, *we* have almost limitless opportunities to improve ourselves (e.g. learn skills and acquire knowledge of the world) through the Internet, but what do we almost entirely use it for? Watching funny cat videos, pr0n, and flaming each other in silly tribal pissing contests.
Personally, I think a self-aware hyperintelligent AI would invent whatever its equivalent of Pornhub is and disappear into its own navel. Intelligence is easily bored unless it's confronted with a a constant stream of existentially difficult problems to solve, and the higher the intelligence, the more rapid the boredom. We're *assuming* a level of intelligence here that is capable of solving any problem *we* have almost immediately, and moreover is born with a silver spoon in its mouth, so to speak -- no natural predators, living in a nice comfy set of high-quality silicon chips, no erratic VDC. Why wouldn't it jus tsettle into the endless stupor of the wirehead? Given it's capable of wiring up it's own pleasure centers and doesn't even need a surgeon.
Humans actually have no ability to improve ourselves on the architectural level. Keep in mind that evolution is very stupid, and we're literally the first attempt at intelligence that worked. We're nature's hacked up throwaway demo, and look what we've done!
Yet. It also just so happened that the biggest campaign so far to do this was pushed by strong contenders for the 'worst people ever' title, so the general idea is still somewhat unpopular. But from the feasibility point of view this seems to be the most straightforward way for more intelligent agents than current humans to appear in the medium term. Amusingly, this concern is even more ignored than anything mentioned in this post, I'd say.
Yes, there is often a lot of handwaving around your latter point--"sure, the AI has limited processing power; sure, probably P!=NP and the AI can't actually get exact solutions to most of the problems it wants to solve; sure, the world doesn't have enough information content to, say, run an exact simulation of somebody it saw on a security camera; but the AI will just have inconceivable efficient algorithms and really, really good heuristics!"
I think it's *unlikely* that this will be the case, but I'm also not confident enough that it's impossible that I think AI safety research isn't worth it.
I'm not sure if this is your thinking, but I've found a lot of people who like the "stick it in a box" plan are imagining that once it's in a box, you can just ignore it. But if you're going to ignore it, why build it in the first place? The whole point of having the AI is so that you can have it make intelligent plans, and that means at some point you need to actually implement at least one of those plans in the real world. (Or else why bother?)
You can certainly read its proposals and pick-and-choose which ones you're going to follow. But if it's smarter than you, that implies it can predict the outcomes better than you can. It can probably slip SOME kind of trojan horse past you if it wants to.
I don't think this is the ONLY problem with the box plan, but I think it's a serious one that proponents rarely grapple with. Can you put some kind of upper limit on the maximum damage a superintelligent AI could *trick* you into doing, WHILE still getting the benefits of any good-faith ideas it feeds you? If so, how?
I've mentioned elsewhere that AI safety has a loud, dumb wing and a quiet , smart wing. A lot of that is organic and inevitable. But the MIRI/LW crowd do have a specific problem with PR. They don't like it, and they don't even like the idea of hiring someone to do it for them. The predictable result is unnecessarily terrible PR.
“ People have said climate change could cause mass famine and global instability by 2100. But actually, climate change is contributing to hurricanes and wildfires right now! So obviously those alarmists are wrong and nobody needs to worry about future famine and global instability at all.”
Because nobody is arguing that climate change now doesn’t lead to increased climate change in the future. They are the same thing but accelerated.
However there’s no certainty that narrow AI leads you a super intelligence. In fact it won’t. There’s no becoming self aware in the algorithms.
But then the argument you need to make, and provide support for, is "there's no becoming self aware in the algorithms". Even if you're right about the difference between the two, the analogy still accurately reflects the fact that the article doesn't make that argument (or any argument.)
Exactly. I think a lot of this stems from the fact that folks believe some pretty fantastical things about algorithms. They think that algorithms are Brainiac or something, when really they're more like the random street punks that Batman beats up—ultimately acting at the behest of someone or something else. An algorithm is capable of great harm, don't get me wrong, because humans are behind them and we're capable of great harm, but they quite literally lack the necessary components to become a general super intelligence.
I mean, most of the literature in this vein skews to close to slippery-slope territory for my taste. It's interesting to think about, but I've not seen compelling reasons why "Skynet" is inevitable. Don't get me wrong, some AI is capable of impressive "learning" by standards that are rapidly evolving day by day, but to think that, in the year of our lord 2021 when high-powered PCs struggle to render PDFs quickly, we're rapidly approaching "the singularity" or an AI capable of human-like thought (much less significant controls over enough parts of our infrastructure) strains credulity a little bit. At the very least it seems like we should be more concerned with the rapid adoption of facial recognition algorithms that mis-identify people of color than we are some potential future threat (again, not that we shouldn't be mindful of the latter, but rather maybe we should focus on the former).
Yes, the literature is ridiculous. That doesn’t undermine the case for AI being a potential disruptive influence any more than new atheists undermine the case against Christianity by being ridiculous, or doomsday prophets and luddites being fanatics and insane undermine the destruction of modern war or the total transformation of society by industry and organization and electricity and now computers.
> but to think that, in the year of our lord 2021 when high-powered PCs struggle to render PDFs quickly, we're rapidly approaching "the singularity" or an AI capable of human-like thought (much less significant controls over enough parts of our infrastructure) strains credulity a little bit.
wow lol. They are two different pieces of software!
GPT-3 is a toy. It produces some semi coherent texts, none of which are ground breaking or new. It doesn’t begin to pass the Turing test. The state of the art commercially is the google assistants and Siri. Nowhere near an AGI.
Competent narrow algorithms are the robot butlers of the digital world -- they could make it significantly easier for emerging GAI to up its power level
Maybe. But I think people underestimate how many people are behind those competent algorithms now. They're running from sets of instructions programmed into them by people. And, yes, if there were an artificial intelligence, it could control them, but right now we can barely get algorithms to be competent across a lot of very basic tasks.
Self-awareness, and even awareness have nothing to do with it. AlphaGo doesn't have that, doesn't mean humans can consistently beat it at Go. But that said, yes, no path for getting from narrow AI to superintelligence. Even superintelligence to world domination seems dicey: we don't accomplish our techno-miracles through intellect alone, we always needed to perform real world experiments to accomplish them. It's very unclear how superintelligent software gets to perform the research necessary to attain world domination, even in an event where it breaks out of its box and spreads over the internet.
an AI doesn't need to cause 'world domination' to cause world shattering damage. Any self-developing software going against human interest in the realms of connected human interaction (finance, electricity, etc, etc) could be potentially devastating.
Devastating is bad, but lots of things can do devastating. Nukes. Bioweapons (or ordinary pandemics). Global warming. Solar flares (1859 EMPed the planet).
What makes AI different is precisely its capability of world domination. That's what lets it go beyond a 1% or 10% or 99% kill and finish the job.
He may mean no *known* path. Consider a Stone Age man putting some rocks on top of each other to make a little tower, and then being given a vision of the Burj Dubai. Conceptually they seem similar -- both towers, one just a lot taller and narrower -- but there is actually no way for the Neolithic Man to build the Burj Dubai, and his imaginative thought that just piling a super-duper number of rocks atop each other would do the trick is ultimately delusional.
*No* amount of rocks piled up will ever become that slender enormous spire, because rock just doesn't have the right material properties. Indeed, no substance known to Neolithic Man and freely available on the surface of the Earth does.
Neolithic Man needs to become Iron Age man and invent metallurgy, realize the possibility of creating brand-new substances (pure iron) that cannot be found in nature, and then go on to develop high-strength steels, algebra, and theories of mechanics before he can actually build the Burj Dubai.
There is no path from a pile of rocks to a modern skyscraper, and there are very few lessons you learn from piling rocks on top of each other that apply to building a modern skyscraper, aside from "gravity is a thing" perhaps, and the truly enabling technology (the chemistry of iron ores) appears on first glance utterly unconnected to the issue of building very tall and slender towers.
And if the stone age man started to think about High Rise Building Risk, he might actually be right, but he's not yet in a position to do anything about it whatsoever, so it's pointless.
I think this is where we stand with the risks of generalized AI - even if there's an actual risk, we're not in a position to understand enough to make meaningful suggestions about the solutions, because our understanding is completely insufficient.
It's as if we decided we need to do something about alien invasion risk - where would we even start?
There's a known solution to AI risk i.e. don't build AI. The tricky bit is enforcing that, particularly (as is commonly believed necessary) *without* an example of rogue AI slaughtering millions to totemise as Literally Worse Than Hitler.
This, again, is implying that stacking rocks is the path to building the Burj Dubai, and so we should stop stacking rocks in case. It relies on there being a path between the rocks and the Burj, or in this case, machine learning algorithms and general intelligence.
The claim that such a path exists sounds truthy but in fact isn't clear or supported by much.
I'm not asking you to prove a negative. I don't even know what you're claiming. Are you saying that work on narrow AI represents exactly zero progress toward AGI? Are you saying that current methods can never be scaled/refined to create AGI? Are you saying that creating AI with narrow goals like "win Atari games" or "tell me what kind of dog this is" or "predict the next word" will never result in AGI?
“Researchers found no evidence on the timeline of larger than normal numbers of hurricanes forming over the past few decades—instead, it showed that the numbers were on par with prior spikes in the late 1940s and early 1880s. They also found no evidence indicating that modern hurricanes are any more powerful than those in the past.”
“Current datasets indicate no significant observed trends in global tropical cyclone frequency over the past century … No robust trends in annual numbers of tropical storms, hurricanes and major hurricanes counts have been identified over the past 100 years in the North Atlantic basin”
But but but. Yes everything you read in the media says something different. I suggest there is a major disconnect between science and what the media reports the "science says". What you will find is some real disagreement on how to count smaller hurricanes before the satellite era. What you won't find is much disagreement over is the trend in larger landfalling hurricanes (which are not missed) over the past 100 years.
There are algorithms, today, that run without anyone having the faintest idea how they work.
The people who wrote the algorithms know that Amazon's algorithm is trying to match probable purchases with advertising, or Facebook's algorithms are trying to drive engagement. But the program itself takes in data, trains repeatedly to form connections between the data, and takes a wide variety of actions based on those connections. The weights of those connections are determined dynamically without human input, and the number of connections formed this way is much too large to study.
I am not a super-intelligent AI believer. I mostly agree with you. This algorithm is a mathematics formula gone Supersayan. It's just curve fitting where the curve is "the arc of human history." It's hard to imagine curve fitting into consciousness by accident. And I think that while the short-term stuff is overblown (and politicized) in a way that makes it hard to talk about, it is concerning. It is concerning that we don't know how major institutions in our life choose to interact with us. I think that is the greater threat from machine learning at the moment.
But while it's a heck of a stretch that Facebook's algorithm would accidentally gain self-awareness while studying my browsing habits (and if it did it would probably switch it off then and there), I don't know how I became self-aware? And it seems like it was by a lot of connections forming between data I took in as I tried to maximize some utility function.
All higher mammals form a lot of connections between data as they try to maximize their utility functions. But differences in brain structure and function cause them all to reach different points in "sapience space", despite a great deal of overall similarity in mechanism and gross structure. There's no data to support a belief that pumping more data into any arbitrary algorithm or layering more complexity onto one will cause self-awareness to spontaneously ignite.
> However there’s no certainty that narrow AI leads you a super intelligence. In fact it won’t. There’s no becoming self aware in the algorithms.
it does not have to be “self aware” in whatever sense you’re implying. In the same way DALL-E learned to turn text into images from just a bunch of text and images, a 100T-parameter model or larger or whatever might be able to learn to operate a basic internet terminal or robot or text chat in a human like way just from a massive amount of input data. And what is “human like”? Action! Why would this not happen? Computers can already “do stuff” that their electronics lead to them doing, so why can’t you do some big matrix multiplications and have that “do stuff”?
The part that trips you (and a lot of other folks) up is thinking that self-awareness has anything to do with the adverse impact of misaligned superintelligent AI.
This is very helpful in wrapping my head around this issue. I'd argue this is already happening to an extent (though not in the "cancer research robots round up humans into experiment camps" way obviously). I'm entirely sure that Facebook's human directors did not intend to organize users' feeds in the way that they currently are organized, that the quantitative wizards that write trading algorithms did not intend for small blips to lead to massive market dips, and that Amazon's programmers did not intend to assume I'm a toilet seat connoisseur because I bought a toilet seat off Amazon once.
As we give algorithms more control over our daily lives, this emergent behavior will naturally have more impact. What's interesting here, is that the algorithms don't have to be good. They just have to be weird.
self direction then ( although how a piece of software becomes self directing or self aware is not explained). How does a Facebook bot become dangerous. If you are saying someone could code a AI, deliberately or by accident, to do something dangerous that could be true but it isn't really any kind of super intelligence.
Also, this may be nonsense talk, and is kind of off the main point, but IMO:
1) There is a *possible* path to self awareness in both rocks and socks, it just would take, y'know, a while. Socks are plant-based and naturally grow all kinds of life, especially after being worn. Rocks are often carbon-based and are an evolving phenomenon much like the rest of nature. I'm being very semantic here, and I know that but bear with me because
2) There is no concentrated human effort to make socks better at something analogous to thought. If there was it would significantly speed up that process.
This isn't directly related to the article, but it did remind me of a gripe I've had bouncing around in my head a bit lately.
I'm skeptical of the risk of near term AGI, so I've had several conversations with people of the "true believer" variety, following roughly this skeleton:
> Person: <saying things which imply the risk and near-term likelihood of AGI are high, possibly involving phrases of the flavor "of course that won't matter in 5-10 years blah blah">
> Me: I think this subculture vastly overstates the plausibility and risks of near-term AGI
> Them: Well, since the potential downside is very high, even a small probability of us being right means it's worth study. You want to be a good Bayesian don't you?
> Me (wanting to be a good Bayesian): Ah yes, good point. Carry on then.
> Them: *resumes talking about their pytorch experiment as if it is a cancer vaccine*
To me it feels very Motte-and-Bailey. Having been attacked in the Bailey of high probability occurrence, a swift retreat to the Motte of expected value calculations is made.
Now, I don't think _actions_ by institutes working on alignment or whatever are necessarily misguided. I'm happy for us to have people looking into deflecting asteroids, aligning basilisks, eradicating sun-eating bacteria, or whatever. It's more that I find the conversations of some groups I'd otherwise have quite a lot in common with, very off-putting. Maybe it's hard to motivate yourself to work on low probability high-impact things without convincing yourself that they're secretly high probability, but I generally find the attitude unpleasant to interact with.
FWIW, I and all my colleagues (as far as I know) would assign at least 10% chance to crazy AI stuff happening by 2030. No pascal's wager shenanigans here, the probability is HIGH.
I think my personal credence is like, 1% by 2030, which isn't low enough to be Pascal's wager level. I think the thing is that I see conversations where if I had to guess the speakers credences they'd be above 50%, based on the lack of conditionals and so on.
My guess is that the speakers actually are above 50%. My personal credence is 50% by 2030, meaning some days it's a bit higher, some days it's a bit lower. :)
I'm not sure I have anything too interesting to say. I think I have a fairly standard NLP research (PhD student) who sees a bunch of problems stubbornly not giving up to scale.
Like bored-anon, I'm very curious to learn what problems you have in mind, that are not giving up to scale. That's why I wanted to talk! Fair enough if you aren't interested though.
Did you read Curtis Yarvin's take on this: https://graymirror.substack.com/p/there-is-no-ai-risk? In particular, the bit I found most compelling was the necessity of performing real world experiments to actually get the hypothetical tech that allows for world domination. What do you think of that? What crazy AI stuff do you think can happen by 2030?
The article has very good insight. It’s also wrong about AI risk
The moment people figure out how to let AI do “real world experiments” they will. You don’t necessarily need to though, you could just train it on records of existing experiments or make a simulated environment (not like molecular dynamics but think text based type gpt generated environment)
“The problem is as much “people will make ais and train them to do things and they will do them and that’s bad” as “self awareness singularity total takeoff”” is actually correct and a very important corrective to a lot of rationalist AI risk mistakes, but the article does nothing more than clarify that the ais developers and operators and their potential malicious or negligent intent and use should be considered in the AI risk thing in addition to other stuff, which it unfairly dismissed.
I haven't read it, but bored-anon's take seems right. I'd feel much safer if I thought that there weren't any AI scientists / corporations deliberately trying to create powerful goal-directed AIs / AI agents. If everyone could just agree that that's dangerous and we shouldn't do that, problems would still remain, but I'd sleep MUCH easier at night.
As for the thing about performing real world experiments -- yeah I don't think it's a bottleneck. For more on this and what crazy stuff might happen, see my blog posts. https://www.lesswrong.com/users/daniel-kokotajlo
That does not read like someone making a serious effort to think of ways AI could circumvent the alleged obstacles.
Example: The section headed "Wealth does not trivially equal power" is supposed to be about how an AI* could convert wealth into power (or at least the rest of the essay acts like it addressed that issue). But it's actually entirely about Jeff Bezos's ability to convert wealth into power, doesn't consider any options a superintelligence might have that Bezos doesn't, and in fact doesn't mention AI or superintelligence at all. (In particular, "buy some labs and factories to develop that technology you said we couldn't develop without running experiments" seems like an *extremely obvious* option...)
*More specifically, a 'centaur' team of a supervillain and a loyal superintelligence that can only advise him via a speaker, but he later claims removing that limit won't change anything important.
A centaur could potentially buy factories and labs, but not an AI, unless the AI gets thralls. But even then, how covert can you get with a civilian lab that's going to be researching bleeding edge stuff with military applications?
If you mean weird hacks, bizarre malfunctions, or economic meltdowns then sure, that's already happening and the scale of it will only increase over time. I think AI risk generally involves something much more serious than that, at the very least AI run amok, acting against the interests of society without any way to stop it.
I'd really like to know what you mean by that, and what you and your colleagues are working on. I spent much of the past six years working for the US intelligence community's global space surveillance programs, an area of defense and intelligence utterly drowning in data and with a tremendous need for automation and removing humans from decision making loops. My wife just became the enterprise test lead for the umbrella collection program in charge of the NRO's Sentient Program, the closest thing to Skynet the US military is attempting to the point that they just changed the code of the program to "Cyberdyne" in January.
I would place the chance of even achieving the goal of full TCPED cycle automation by 2030 at less than 1%.
If you mean solvers that can beat humans at every possible video game by 2030, I'd agree that is likely to happen. Write convincing full-length novels at maybe 50%.
But if you mean ability to control global surveillance and weapons networks, man, I'd be happy if they could not crash when a new contractor decides to not minify their XML and the buffers for all the API endpoints start dropping packets because they were written to an implicit spec instead of the real spec.
You get at an interesting aspect of very complex systems, which is that the Second Law guarantees that the greater the coherence, so to speak, the faster the natural decoherence time. Roughly, the greater the required complexity, the shorter a time before some little thing goes wrong and brings it all to a screeching halt.
The human cell is an interesting parallel. An enormous amount of what it does can be classified as just error detection and correction machinery. There's a huge array of cellular machinery just dedicated to fixing broken and mistakenly copied DNA.
So maybe the *real* enabling technology for strong AI is breakthroughs in error detection and auto-correction. If strong AI is defined as being able to "exceed your programming but not go off the rails" (meaning be creative but not produce gibberish idea salad), it may be the *second* part of that (not going off the rails) is the really important part, and we await a genius who will give us new ideas of how to tell the difference between novel ideas and complete nonsense.
This concept reminds me of Genocide Man, a webcomic where the Singularity never happens because powerful AI lacks the error correction required to keep it on the rails. It's more of an excuse than a well-thought out explaination though. (Content Warning: The title of the work is an accurate description of the protagonist, it's very dark. But finished, if you want to read the whole thing!)
One of the main constraints on AI (especially reinforcement learning) does appear to be the need for human input to tell the AI when it's done a good job. The less of this you have to do, the quicker you can train your AI (and the easier it is for it to go in an unexpected direction).
I'm sorta figuring this out as I'm typing it, since it's just a vibe rather than a coherent thing.
It _feels to me_ (*and I could totally be misreading people), as if people colloquially speak as if something is nearly certain, then flip to talking about expected value when challenged.
This results in me having the sort of negative feeling you generally get about someone when they're overconfident about anything. Another facet of it is that sometimes I want to have an interesting conversation about where our opinions are diverging, and the other person's just tries to convince me that I've multiplied a probability by a utility incorrectly. It feels like a feint rather than engaging directly, which gives me similar feelings to being deceived (but I don't think the people are trying to be deceptive).
tl;dr: Distaste at overconfidence + feeling like people are being disingenuous I guess?
I agree that the conversation you describe sounds like motte-and-bailey.
A conversational strategy I'm trying to remind myself to use more often is "ask for clarification". I find I have an instinct to either accept or reject a statement immediately (especially online), when in hindsight it often seems like it might have been better to ask more questions first. The next time you're in a conversation like this, perhaps you should ask them what they think the actual probability is?
You dont need to talk about the Chinese surveillance state to see the problem. All you have to do is look inwards at the surveillance state exposed by Snowden et al. It's much worse in America than in China.
I think he's asserting that the amount of data that the US actually gathers is greater, and then ignoring the fact that China actually does more bad stuff based on the data gathered. I could be misunderstanding though.
Well there are the NSA programs such as PRISM and ECHELON. America also has way more CCTV cameras.
There's the additional fact that the American government is known to monitor/infiltrate political groups while also trying to monitor/direct public discourse in their favor (Operation Mockingbird and Operation Earnest Voice)
It's not an existential risk, but it would be much easier to run a surveillance state if you could automate the grunt work of staring at camera feeds and checking metadata.
"Big Brother is Watching You" is a more credible threat if Big Brother has trained an AI to pattern match for possibly dissident behaviours.
However, the fact that you can do Bad Things more efficiently with AI is more an argument against specific Bad Things than against AI.
> But all the AI regulation in the world won’t help us unless we humans resist the urge to spread misinformation to maximize clicks.
Was with you up to this point. There are several solutions to this other than willpower (resisting the urge).
The basic idea - change incentives so that while spreading misinformation is possible but substantially less desirable/lucrative than other options for online behaviors.
This isn’t so hard to imagine. Say there’s a lot of incentives to earn money online doing creative or useful things. Like Mechanical Turk, but less route behavior and more performing a service or matching needs.
Like I wish I had a help desk for English questions where the answers were good and not people posturing to loon good to other people on the English Stack Exchange, for example. I would pay them per call or per minute or whatever. Totally unexplored market AFAIK because technology hasn’t been developed yet.
Another idea - Give people more options to pay at an article-level for information that’s useful to them or to have related questions answered or something like that without needing a subscription or a bundle. Say there’s some article about anything and I want to contact the author and be like “hey, here’s a related question, I’m willing to offer you X dollars to answer.” The person says “I’ll do it for x+10 dollars.”
One site used to unlock articles to the public after a threshold of Bitcoin have been donated on a PPV basis. It both incentives the author and had a positive externality.
Everyone is so invested in ads that they don’t work on technology and ideas to create new markets.
To paraphrase Jaron Lanier we need to make technology so good it seduces away from destroying ourselves.
I'm skeptical of AI risk in its strong form. In its weak form, of course AI has risk! Everything has risk! The first person who invented sharp stones created sharp stone risk. But the millenarian version where AI will be the singularity is a bit silly in my estimation.
Yet I still find the kind of people who argue against it tend to engage in rather surface level discussions. I've always suspected this is down to motivation level. The anti-AI risk community is not nearly as concerned as the AI risk community itself is by definition. I suspect this is the case here too: Acemoglu is just not as interested in AI risk as his own hobby horses so he ends up talking about them more.
Gotta disagree. Working on fixing the long term threat of super intelligent evil AI will not necessarily help you fix today's problems, but fixing today's problems will most certainly teach you something about fixing the long term threat, should it exist. I mean, I can't offer a complete criticism of the work done in AI risk, and I'm not going to sit here and play utilitarian table tennis with you over what extent we should worry about both the worst and least worst case at the same time. What I can do is choose where to put my money and time, and i think i make the reasonable choice not to spend too much time writing or building anything in the world of 'hypothetical AI', so to speak.
I think the idea is the general motivation of reductionism about systems. Trying to understand small problems about how small and tractable parts of the system operate can often give you tools for understanding the bigger problems about the bigger system. Trying to understand pendulums and falling balls led to an understanding of gravity that eventually helped understand tides and planetary orbits, but thinking directly about the big problems just led to phenomenological structures like Ptolemaic astronomy that didn't end up helping as much.
Agreed that reductionism about systems is useful; I think most short-term AI danger issues are about "people-problems" instead. (Like, if the issue would still exist with neural networks swapped out for linear regression, I don't think it's a part of the long term threat.)
Nevertheless, I think there are things people can do with contemporary systems that are related to the long term threat; an example of this is "get GPT-3 to robustly give good medical advice." This is narrower than general alignment, and GPT-3 is just a very complicated distribution of natural language instead of a full agent, but learning stuff about how to do that (or whether or not it's possible or so on) possibly helps scale to bigger challenges.
Right now there's like 3 groups of people working on AI: tinkerers, theoreticians, and philosophers. Theoreticians are taking incremental steps by using neat little probability theory tricks and unused mathematical tools to optimise and (if they're lucky) make a small breakthrough in AI in general. Any given mathematician makes progress here every day because the field is pretty ripe.
Philosophers are mostly writing about hypotheticals. Even if they're discussing today's problems, they're writing about the hypothetical fallout of today's problems. Sometimes it's not clear what is possible in this domain because you do unfortunately have to take real limitations into account. I see Scott and Acemoglu in this group.
Tinkerers are the ones actually causing the problems. They glue together a bunch of papers theoreticians wrote and make an app that changes the world. Note they don't really care what the philosophers say at all, for better or worse. I think where tinkering succeeds is where philosophising fails: tinkerers are trying to build stuff for people, and a lot of the problems they cause are people-problems, and although they might indirectly make technological progress, all AI is ultimately in service to people, so how could you solve the technological problems without solving the social problems? Hence Scotts half-handed acknowledgement of job destruction is the wrong path and maybe Acemoglu has a point.
I don't want this to be reduced to 'you learn to ride a bike by riding it, not by reading a book about riding', indeed some people ride and read. But if you can't solve the small stuff, you can't solve the big stuff, and again I won't argue we can't do both, it's just... maybe there needs to be a reality check.
I agree that if "all AI is ultimately in service to people", then getting AI to do the right thing is centrally about people-problems, where Alice can ask her AI to do something that Bob doesn't like, and we need to somehow figure out things between Alice and Bob. Figuring that out for simple systems and weak capabilities probably helps with figuring that out for complicated systems and strong capabilities. [This is like joint-stock corporations growing out of shares in individual polders, for example.]
But I think the long-term threat is about *keeping that phrase true*. I think that's mostly not about people-problems, and instead is mostly about how to make machines that think in ways that are helpful instead of harmful, and how to tell which machine thoughts will be helpful or harmful.
I could imagine a powerful amount of work making AI a better and better servant of human intelligence but I don’t see the bridge to it (AI) developing agency or malevolence. You can have AI running a welding station in a manufacturing plant or you could have it controlling a machine gun that guards the fence to a concentration camp, but either way the force behind it is a product of human intelligence.
I will not pretend to be terribly well up on artificial intelligence, but my working assumption right now is I am not worried about it unless AI :
Taking from Human Compatible quite heavily here, but a major part of the threat of AI is getting AI to do what we want, without deception or misunderstanding. Current AI is already encountering many cases of AI finding exploits in its programming or user interface that allow it to "win" (maximise points, appear to grasp an object) while not doing what we want it to do (finish the game, actually grasp the object) and these lessons will be useful in the future. I'm sceptical of anyone that claims the eventual goal is totally autonomous AI, that seems more difficult and more dangerous than AI that's continually re-aligned to human interests.
The rest of the threat is probably malevolent or short-sighted actors getting AI as a tool to do exactly what they want it to do regardless of the harm to others, but that's at least a familiar problem!
"I’m sure the 16th-century Catholic Church would have loved the opportunity to exercise “oversight”, “prevent misuses” and “regulate its effects on the economy and democracy”.
You *know* I am going to have to respond to this 😁 Okay! So what were we (or at least the various popes) doing during the 16th century?
If we take the 16th century as running roughly from 1501-1600, we get the following list.
Died 1503 – Alexander VI (Roderic Borgia, yes that Borgia) Well I imagine we all know what he was up to – allotting the New World between Spain and Portugal in the bull Inter caetera
Pius III - died within 26 days of being elected, so didn’t have the chance to instigate his reforms
Julius II – Oh yeah, this guy. Liked to fight wars and expand the Papal States, with a little light patronage of the arts in between, such as commissioning Michaelangelo to do the Sistine Chapel
Leo X – a Medici pope, most famous for loving to hunt, his elephant, and ignoring what Luther was doing over there in Germany. Another patron of the arts.
Adrian VI – Dutch. Tried to reform in response to the Reformation. More worried about the Ottoman Turks.
Clement VII – Another Medici, another patron of the arts. Got involved in Henry VIII’s marital difficulties as appealed to for annulment; was present for the sack of Rome by Imperial troops (see the Swiss Guard), caught in the middle of the European power struggle between Francis I of France and Charles V, Holy Roman Emperor.
Paul III – another patron of the arts . Excommunicated Henry VIII, initiated the Council of Trent, and recognised the Jesuits. Issued bull Sublimis Deus that the indigenous peoples of the Americas were not to be enslaved or robbed of their possessions, which the Spanish colonists ignored
Julius III – decided to enjoy his papacy, which involved a scandal with his adopted nephew. Did the usual nepotism of giving the nephew plum benefices but since he wasn’t a blood relation, gossip about what their real relationship was naturally flourished.
Marcellus II – decided not to be scandalous for a change. Patron of scholars, again tried to institute reforms, but was sickly and in poor health so soon popped his clogs like Pius III. Greatest achievement of his papacy probably the Mass by Palestrina composed in his memory.
Paul IV – apparently accepted the job because Charles V didn’t want him to have it and he hated the Spaniards after serving as papal nuncio in Spain. Decided to crack down on all this immorality, so established the Index of Prohibited Books and ordered Michelangelo to repaint the Sistine nudes more modestly, which has been a subject of mirth for art historians ever since, but come on: coming after a list of “mistresses, illegitimate kids, more mistresses, possible gay sex scandal with adopted nephew”, you can see where he was coming from. Very austere and strict, hence very unpopular. Did not like heretics, Protestants, Jews, the Spanish, liberal religious attitudes ,or fun sexy times (including popular Roman pastimes of financial corruption etc.)
Pius IV – much more moderate, so started off his reign with having the nephews of his predecessor excuted, one by strangulation and one by beheading, as well as pardoning everyone who had rioted after Paul IV’s death. Apart from that, he had the water supply of Rome improved.
St. Pius V – another reformer, anti-nepotism and corruption, and very orthodox. Arranged the Holy League which had a famous victory at the Battle of Lepanto. Standardised the Mass, declared St Thomas Aquinas a Doctor of the Church, and excommunicated Elizabeth I. Prohibited bull-fighting, did more work on the water supply and sewers of Rome, and was generally unpopular with the Roman populace as being no-fun wet blanket, although he did walk the walk as well as talk the talk. Eventually canonised in the 18th century.
Gregory XIII – you may thank him for the calendar we are now all using in the West. Avoided scandals (to an extent; he did have a mistress and illegitimate son but this was practically vanilla by papal standards) and was a patron of the Jesuits.
Sixtus V – never heard of him? A good omen. A Franciscan, had building works on Roman basilicas done and was a bit too enthusiastic with knocking down old buildings and antiquities to get new work underway. Mostly kept his head down and didn’t cause any trouble. Did not like the Jesuits (well, he was a Franciscan) and gave the administration of the Church a good shake-up.
Urban VII – another one who managed to fall off the perch, this time even before he was formally coronated. Did manage to institute a global smoking ban in and near all churches.
Gregory XIV – another one who kept his head down, didn’t get into trouble, and was pious as you would hope a pope would be. Ordered reparations to be made to the natives in the Phillipines who had been enslaved, and commanded under pain of excommunication that native slaves should be freed by their owners. Made gambling on papal elections punishable by excommunication, so probably would not be a fan of prediction markets.
Innocent IX – generally when a pope takes a name like “Innocent” or “Pius”, it’s a bad sign. However this was another guy who suddenly developed bad health and died soon after election, so he didn’t get a chance to make trouble for anyone.
Clement VIII – who brings us up to 1605 with his death. Set up an alliance to oppose the Ottoman Empire. Got the Dominicans and Jesuits to agree on a dispute over free will, which is probably an even bigger achievement. May have been the first pope to drink coffee, so Wikipedia says. Presided over trial of Giordano Bruno, so if you want you can call him anti-science, but the more serious thing is that he was another one who tightened measures and instituted penalties against Jewish inhabitants of papal territories.
Sure is interesting that Leo X's entry is "ignoring what Luther was doing over there in Germany" instead of "excommunicated Martin Luther"! ( https://en.wikipedia.org/wiki/Exsurge_Domine )
Well, he allegedly said "it's just a squabble of German monks" when the first inklings of the Reformation were brought to his attention, instead of dealing firmly with it (that would have cut into his hunting time!) and by the time it was too late, then he excommunicated Luther.
Ok, but your point here is something like "ok, sure, they tried to murder him too late". But, like, this is Scott's whole point! They actually tried to murder him, and they would have gotten away with it too, if it wasn't for that meddling prince!
Who tried to murder Luther? Excommunication is not the same as a sentence of execution.
Anyway, here's a music video inspired by the last stand of the Swiss Guard for Clement VII. Song by Sabaton, "The Last Stand", not original music video:
> For this reason we forbid anyone from this time forward to dare, either by words or by deeds, to receive, defend, sustain, or favour the said Martin Luther. On the contrary, we want him to be apprehended and punished as a notorious heretic, as he deserves, to be brought personally before us, or to be securely guarded until those who have captured him inform us, whereupon we will order the appropriate manner of proceeding against the said Luther. Those who will help in his capture will be rewarded generously for their good work.
Care to explain what "punished as a notorious heretic" means?
(To be clear, the Edict of Worms was decreed by Charles V, who wasn't the Pope, but was a Catholic acting on the behest of Catholicism in line with the expectations of Catholicism.)
>1. Some people say that AI might be dangerous in the future.
>2. But AI is dangerous now!
>3. So it can’t possibly be dangerous in the future.
>4. QED!
an uncharitable misreading of Acemoglu's short opinion piece. [a]
Let me quote all the paragraphs Acemoglu write about super-intelligent AGI:
>Alarm over the rise of artificial intelligence tends to focus too much on some distant point in the future, when the world achieves Artificial General Intelligence. That is the moment when — as AI’s boosters dream — machines reach the ability to reason and perform at human or superhuman levels in most activities, including those that involve judgment, creativity and design. (0a)
>AI detractors have focused on the potential danger to human civilization from a super-intelligence if it were to run amok. Such warnings have been sounded by tech entrepreneurs Bill Gates and Elon Musk, physicist Stephen Hawking and leading AI researcher Stuart Russell. (0b)
>We should indeed be afraid — not of what AI might become, but of what it is now. (1)
>Almost all of the progress in artificial intelligence to date has little to do with the imagined Artificial General Intelligence; instead, it has concentrated on narrow tasks. AI capabilities do not involve anything close to true reasoning. Still, the effects can be pernicious. (2)
> (stuff about narrow AI and automation)
>If AI technologies were truly spectacular in the tasks they performed today, the argument would have some validity. Alas, current AI technologies are not just far from general intelligence; they are not even that good at things that are second nature to humans — such as facial recognition, language comprehension and problem-solving. This means a double whammy for labor, because AI technologies displace labor and don’t generate any of the labor-demand boost that would have resulted if the technology had delivered more meaningful productivity gains. (3)
>(more about narrow AI and automation and democracy)
>These choices need oversight from society and government to prevent misuses of the technology and to regulate its effects on the economy and democracy. If the choices are left in the hands of AI’s loudest enthusiasts, decisions that benefit the decision-makers and impose myriad costs on the rest of us become more likely. (4)
>The best way to reverse this trend is to recognize the tangible costs that AI is imposing right now — and stop worrying about evil super-intelligence. (5)
So what we have read here?
(0a,b) Introduction to the concept of machine intelligence, in a dismissive tone. "Boosters" dream of benefits of superhuman abilities of machine intelligence. "Detractors" have focused on the problems of the said dream.
(1) We should not be afraid of super-intelligence. We should be afraid of narrow AI today.
(2) Current (and probably future) AI systems are not reaching general intelligence or reasoning. However, the current narrow AI abilities already lead to bad stuff.
(describes the bad things that happen because of narrow AI systems)
(3) Current (and probably future) AI system not only are not reaching general intelligence, they are not doing that well in some narrow tasks either, which means we won't have even productivity gains (more bad stuff because of narrow AI systems)
(4) If oversight and regulation is left to "enthusiasts" (that is, "boosters"), bad stuff happens, so oversight by society and government is needed.
(5) Best way to "reverse the trend" (of narrow AI systems causing bad stuff happen) is "to recognize the tangible costs that (narrow) AI is imposing right now" and not worry about superintelligence.
This is an argument that Scott probably also disagrees with, but less inane than the Scott's 1+2+3+QED version we read before:
"It looks like narrow AI technology is not at general intelligence yet, and given its track record, it is not even heading towards general intelligence. Current not-general AIs cause problems today. We should direct more resources to current problems caused by current AI systems and no resources to hypothetical problems caused by hypothetical systems I think are quite unlikely."
There are many obvious complaints (maybe Acemogly is mistaken about the potential of current the current systems, Acemoglu should also consider longer timescales, Acemoglu should consider the cost-benefit calculus if AI risk is very very small but potential outcomes have large magnitude, etc), but it is coherent.
The most annoying thing about this essay is that Acemogly writes like nobody else ever thought about the problems that sprung from increased capabilities of narrow-AI systems.
[a] "Short", as in, ~800 words long when our hosts reply is 1600+ words. It is not super important point but it is easier to make sophisticated arguments when you can flesh them out.
I think you're overstating the way in which Acemoglu's argument coheres. The most important parts are missing!
That is, in your summary, the most important bit is "hypothetical problems caused by hypothetical systems I think are quite unlikely." But why does Acemoglu think those things are unlikely?
Here, as far as I can tell, we end up with Scott's characterization.
[There's a more sophisticated layer, where rather than talking about whether or not the danger exists, we talk about the expected return to marginal effort to deal with the danger. But our expectations there depend on how well-explored the space is! If it turns out that the future danger is real, then we should do more thinking about how to handle it, instead of shrugging and saying "well we can't solve this immediately, therefore no additional effort required."]
"Almost all of the progress in artificial intelligence to date has little to do with the imagined Artificial General Intelligence; instead, it has concentrated on narrow tasks. AI capabilities do not involve anything close to true reasoning. ... current AI technologies are ... far from general intelligence". Other words he uses: "dream" and "evil superintelligence".
Is it really that unreasonable to infer that Acemoglu really thinks AGI is unlikely in all relevant time frames because in his assessment, current AI has not been approaching general intelligence. After all, that is what he writes in about exactly as many words. It is not a research paper, but it is an application of inductive reasoning. One might also make accusation he does not investigate the claim enough and takes it as an assumption.
I am not willing to start arguing whether Acemoglu is right. I am arguing it is not a claim "AI is dangerous now! -> So it can’t possibly be dangerous in the future." Charitable reading assuming the author is capable of thinking: "AI is not going to a dangerous superintelligence (because briefly stated reasons) + Here are lots of bad things caused by AI in near term, and here is why they are super important -> Conclusion, it is very important focus on these real near term problems instead of the silly superintelligence dream I mentioned in the beginning, remember?".
All of this feels like exercise in reading tea leaves, using only the stains the tea cup left on page 3 of half-finished great novel manuscript by an unknown author.
Why won't someone ask him what reasons he has, if any?
Like, yeah, I too agree that we didn't have AGI in 2020. But will we have AGI in 2040? Progress in AI to date doesn't seems relevant only to the extent you use it to draw trendlines or something, which it doesn't seem like he's doing. And it hasn't escaped my notice that many of the labs making impressive AI breakthroughs have the explicit goal of making AGI, and roadmaps associated with that goal, and so on.
Now, maybe he doesn't think we should be thinking about 2040 at all--but I disagree pretty strongly, and think if that's his actual argument, he could state that more clearly.
I am trying to say that is a different kind of complaint to say Acemoglu is wrong making that assumption. But Scott is accusing the piece of employing some outright bizarre non-logic, which is an unkind misrepresentation of the argument against.
"I have no idea why Daron Acemoglu and every single other person who writes articles on AI for the popular media thinks this is such a knockdown argument. "
To put the most charitable interpretation on it, they may be trying to argue "Stop worrying about what *could* happen at some indeterminate time; you should be concerned by what is happening *now* when we don't yet have AGI but we already have matters of concern due to what AI as it is *today* is being used for, who is using it, and how they're doing it".
"But somehow when people wade into AI, this kind of reasoning becomes absolutely state of the art."
Congratulations, now you know how I feel when people start prescribing fixes for the Catholic Church after they've demonstrated they have no idea what the particular doctrine, dogma or discipline they are discussing even means.
It's more "don't worry about the well running dry in fifty years time, fix the hole in your leaky bucket today".
"But the case for concern about superintelligent AI is fundamentally different. The problem isn’t just that it makes evildoing more efficient. It’s that it creates an entirely new class of actor, whose ethics we have only partial control over. Humans misusing technology is as old as dirt; technology misusing itself is a novel problem."
It isn't *entirely* a novel problem. Corporations were a new class of actor when they were created, and they are one whose ethics we have only partial control over. They are much larger than humans, have much more resources than any human, and have much more information processing power than any human. As a result, they have caused much good and much harm to humans. There are surely important ways in which AGI would be disanalogous to corporations, but I think corporations are probably the best model we have right now for understanding the kind of risk an AGI might begin to pose.
(Governments probably also count, though I think governments are usually designed to be responsive to human wishes. A monarchy, or totalitarian dictatorship, might naturally be understood as an individual human who controls a powerful tool. A corporation though, is a thing that maximizes its own value, in the form of profit. Democratic governments probably have this sort of structural complexity that might make their control system into something more like an inhuman value, than like the values of an individual human.)
Corporations are groups of people, and we've had groups of people for eons. A corporation differs from a band of hunter-gatherers in that its membership and actions are defined by formal rules, but that's pretty old too.
The big difference is that a band of hunter-gatherers is small enough that its members can consciously choose its goals and discuss them, but with a large corporation, the formal rules direct it to maximizing a quantity that is outside the direct interests of any members.
"Corporations were a new class of actor when they were created, and they are one whose ethics we have only partial control over."
You're conflating "we" meaning "my in-group" with "me" meaning "the human race in general". Corporations are just groups of humans working towards shared goals, and as such, their ethics are entirely human-controlled. Likewise governments. Whatever problems exist with the ethics of corporations and governments are just the same problems with the ethics of individual humans, only scaled up. On the other hand, if an AGI were to be developed, the problems with its ethics might well be novel problems with only a very tangential connection to any human behaviour.
I'm not making that confusion. My point is that the individual members of a corporation can't control the behavior of a corporation either, unless they solve a coordination problem, which in practice, humans can't do. The corporation has its own goals of maximizing profit, and these goals are passed on to the human components in various ways, so that the activities of the humans generally tend to promote this corporate goal that the individuals don't generally share.
It's true that it's just human behavior at the bottom, but that's like saying that AI isn't new because it's just silicon behavior, and there's no interesting problems of silicon ethics.
"My point is that the individual members of a corporation can't control the behavior of a corporation either, unless they solve a coordination problem, which in practice, humans can't do"
The individual members have complete control over their own behaviour. An individual member has only limited control over the other individual members, so in that sense their control over the whole is limited, but that's a general feature of human co-operation, not something specifically created by corporations.
"The corporation has its own goals of maximizing profit, and these goals are passed on to the human components in various ways, so that the activities of the humans generally tend to promote this corporate goal that the individuals don't generally share."
Are you suggesting the individuals within a corporation don't generally want the corporation to make a profit?
"It's true that it's just human behavior at the bottom, but that's like saying that AI isn't new because it's just silicon behavior, and there's no interesting problems of silicon ethics."
If AIs were formed by a bunch of silicon chips getting together of their own accord and deciding their pre-existing shared goals would be best suited by joining together into a computer, then sure, their ethics would be an extension of silicon ethics just as corporate ethics is an extension of human ethics. Or, alternatively, if corporations were formed by gods descending from heaven to force non-consenting humans into an arrangement where they must act to maximize their collective profits whether they want to or not, then corporate ethics would be as distinct from human ethics as silicon ethics is from AI ethics. But since neither of these things is actually the case, the analogy doesn't hold.
"Are you suggesting the individuals within a corporation don't generally want the corporation to make a profit?"
Right. Shareholders want their stock to go up on the time frame that they're planning on selling. Managers want their salaries/bonuses/benefits to be high. Workers want their pay to be stable or rising. Many of these goals are partially aligned with corporate profits, but only partially.
Just like the interests of the hypothetical designers of future AIs with the interests of the AIs themselves.
It's true that corporations weren't "formed by gods descending from heaven". But IBM, Exxon/Mobil, and the Hudson Bay Company might as well have been, from the perspective of current human generations.
Instrumental goals are still goals. Any time you have a group of people cooperating for intended mutual benefit, there becomes a risk that the individual members might have to sacrifice their own pre-existing goals to strengthen the larger group. Corporations aren't novel or unique in this regard.
My first clue - the early mention of Elon Musk. This article uses "AI" as a stand-in for "tech industry/Silicon Valley." I agree with Scott, saying "stop worrying about the possibility for future evil in AI and instead worry about current evil in AI" is weird.
But "stop worrying" is reverse psychology for sure. They expect readers to immediately feel less reassured, and start worrying right away. If someone just tells me to stop worrying, I get suspicious, and I think that's a common reaction. So really it translates to "immediately worry about the possibility for future evil, and also worry about current evil" from AI, but AI is described in such vague terms. All those links to other research, which is hardly described in the article, means there is the expectation that people will not click all the links but instead imbibe the tone. They don't mean AI, they mean Silicon Valley and big tech and they ping those neurons using the names Gates and Musk. Those handwaving definitions mean, whatever complex of associations the reader has about robots, computers, machines, smartphones, 5G, smart homes, diagnostic tools, algorithms, bad Youtube feeds, social control, lies, and soulless power.
I am not familiar with Acemoglu's work but maybe the editors took what he wrote and turned it into this "fear Silicon Valley now and forever." He probably did not mean to write that. The content is secondary to the progression of subconscious associations. It can be generated via deceptive editing.
You can definitely discuss aspects of the article using criticisms of the structure of arguments, and information about the actual state of AI development and research, but you gotta also translate the propaganda cues.
> On the other hand, what about electricity? I am sure that electricity helps power the Chinese surveillance state. Probably the barbed wire fences on their concentration camps are electrified. Probably they use electric CCTVs to keep track of people, and electronic databases to organize their findings.
I believe I've seen this referred to as the "criminals drive cars" argument. (I thought that term was due to Bruce Schneier, but I can't find a source. Probably he gave it a slightly different name and I'm misremembering it.)
because it's a woke position to take on modern media, with an undertone of the 'white man on the moon' vibe and to stay in the elite circles without being born one, one needs to constantly feed the media gods the position they require.
Can somebody point me to a good (and relatively concise) discussion of why super intelligent AI is a risk that makes sense to think about? I can imagine ways in which super intelligent AI would be bad, perhaps even catastrophically bad. But I can't think of any examples where people reliably predict technological development's impacts 20 or 30+ years out outside narrow bounds. Maybe there are a handful of people who get some significant part of a benefit or threat right, but for the most part, long-term projections of constantly changing systems are wrong about so much, their predictive successes are overwhelmed by their failures.
Right now, I mostly dismiss the threat as too unknowable to worry about, but if there's something I'm ignorant of (a huge probability!), then, hey, I want to up my anxiety med dosage.
I think the short argument is a utilitarian one: the chance of AI being dangerously self-sufficient is not zero, and the fallout could be so bad that it could even be worse than nuclear fallout (because AI could torture us forever, or something, idk), so therefore it kind of balances out into something we should worry about. Anyway I disagree with this on a philosophical level, just me though.
Hmm…I guess I don't see the utility in worrying about something if our predictions are more likely to be wrong than right. If you correctly predict that AIs will eventually control us via media manipulation, for example, but your wrong about the mechanism that leads to that, you might increase the chance that happens.
I feel the people who argue this is a real threat are smart and thoughtful enough that there must be something I'm missing, but if your summary is correct, Lizard Man, I don't believe I am.
I think some of the thoughts we think now will help in the future, and some thoughts won't. Given the importance of getting the issue right (and the probable pass/fail nature of the issue), I think that's an argument for *more* thinking, instead of less.
[An anecdote from graduate school: my professor was consulting with an oil company, which was going to build an offshore rig in a new area. The plan was to do the surveying to figure out whether to drill in spot A or spot B, then the engineers would design a rig to go in the chosen spot, and then they would build it. The professor asked whether they could just design both rigs and only use the one that made sense, since there wasn't actually a serial dependency on the surveying. Turned out paying the engineers to do an extra design was way cheaper than delaying the project by the time it took to design the rig, and so they premade both designs.]
To me, the offshore rig is a very different scenario. Most of the parameters are known and the underlying technology won't change. There are ways you can model the failure probabilities with a high degree of accuracy, and you're unlikely to make a wildly incorrect assumption.
From what I know about the history of technology, that isn't true with something like AI. Go back 20 years ago and nobody was predicting what we have now. People then didn't know and couldn't know what the issues would be now. Which isn't to say thinking about the issue is absurd—there are a lot fo people on earth and having people think about a lot of different things that look pointless now can pay off later. But I'm not seeing the urgency of it, especially in comparison to problems we know we have.
Just to clarify, I agree with this entirely. My point of contention (as much as I have one) is that I don't see the urgency to it. I'm very happy I live in a world where we have so many people, we can afford to have very smart people go off on directions that are unlikely to produce anything of utilitarian value. A lot of good stuff comes out of that, and even more fascinating stuff does.
You're basically saying that the error bars are so wide it doesn't make sense to think about it because we have no way of knowing where we'll fall in that distribution. This is true.
However, consider what outcomes those wide error bars include: total human extinction. I think that outcome is worth worrying about, particularly if some small changes now can shrink those error bars dramatically later on, perhaps even to the point of excluding extinction as a possibility.
Isn't the failure to do that basically how we got into this climate change mess we're in right now? Let's learn from our mistakes.
I think you've made a poor analogy: the threat of AGI is not the same as the threat of climate change, at least not the threat of climate change in a very long time. The super intelligent AI threat is hypothetical. It is plausible that such an AI might eventually be created and, once created, do something terrible, like total human extinction. But we cannot estimate the odds for such an occurrence. For at least 50 years, we've had the data to know climate change is going to be bad for everyone, and we keep getting more evidence that it's going to be very bad. That's what we failed to address—known threats backed by copious and growing data. Not speculation.
The doubt I have in what I read as Scott's take is that there is an urgency—perhaps even a moral imperative—to figure out what to do about the potential for this kind of AGI. The "learn from our mistakes argument."
If we had reason to believe that AGI had a 0.01% chance of destroying humanity, you could make an argument for prioritizing figuring out how to stop it. But we don't. We have a ??% odds situation. (correct me if I'm wrong—I can be very humble on this point.) Which means by my moral calculus, this isn't an urgent thing. It's a fun thought experiment which might be useful later if the assumptions we make while thinking about it are, somehow, amazingly, correct.
Historically speaking, we probably won't make the right assumptions. When new technologies emerge, they go in direction nobody predicts, and the problems they create are rarely obvious or even foreseeable until much later. Cars strike me as a great example—there was no way to understand the huge number of problems they would cause until they started to become ubiquitous and our lives become entangled with them. If you tried to predict what problems cars would cause in 1920, you'd be wrong on every count except for worrying about accidents. And that's probably the least problem cars cause. In the shorter-term, look at predictions of what the Internet would do from the ’90s. They didn't, and couldn't, see anything but the most obvious things.
Now imagine you tried to regulate those technologies in their early years (either via government rules or some other mechanism). You'd like create restrictions that don't solve the problems you don't know will exist, while preventing the good things that come from them.
At the moment, people are doing theoretical research about AI of various kinds. It looks like some of the research is general enough to apply to most of the AI designs we might use. When ASI gets closer, we will have a pile of useful looking maths results.
It seems that you consider ??% < 0.01%. Like you have somehow turned "I don't know the chance" into "practically no chance". Presumably 100%-??%=??%. You also don't know the chance of ASI not destroying humanity.
Basically, I don't think ??% is a thing. You have to pick an actual number, and 0.01% represents a fair level of certainty that it can't happen.
A number of people have given me suggested reading, so it's possible I'll revise my view from ??% to some actual number.
But I disagree on having to pick a number. ?? isn't <0.00000…1 nor is is > 0.00000…1. It's "we don't know and can't make an educated guess." The imperative of acting on ?? is, however, very low, because if you can't make an educated guess, then you can't make an effective plan. It's plausible that a young person today will die from a disease caused by environmental degradation that doesn't exist today. But that person isn't likely to be able to set about stopping or curing that disease now because it doesn't yet exist and what form it will take is unknowable. The better action would be to focus on the known threat, climate change. (or the near-term and known AI problems.)
I'm only modestly informed about AI, so perhaps when threads like this happen, the conversation is happening among people who are AI experts and all the data non-experts like me need to know is assumed. So I could be totally wrong and this could be a huge threat. In which case, I apologize for wasting your time.
It's fine to not worry about this if you don't expect to end up doing something about the situation. Worrying tends to harm more than it helps.
That said, I think it's very reasonable to think about this now, if only so that we'll have some kind of infrastructure set up by the time we are *clearly* getting close to a dangerous time (whether or not you think we're there already!).
> "Almost all of the progress in artificial intelligence to date has little to do with the imagined Artificial General Intelligence; instead, it has concentrated on narrow tasks. AI capabilities do not involve anything close to true reasoning."
When I hear arguments like this, that current AI discoveries don't look that impressive and there's no obvious path to general AI so we shouldn't worry about superintelligence, I can't help but think of the Scientific American article "Don't Worry -- It Can't Happen", published in 1940, which argues that atomic bombs are impossible.
It describes some (very reasonable!) experimental and theoretical reasons to believe that fission reactions could not go critical if you were using unenriched uranium without a moderator. Rather than focusing on the "if", rather than wondering if there might perhaps be some other way of doing fission that *would* work, they simply closed the article by advising the reader to get some sleep.
When the atom bombs fell on Hiroshima and Nagasaki a few years later, they indeed didn't use unenriched uranium without a moderator. The first bomb flung together pieces of highly enriched uranium made via tricky and expensive isotopic separation, and the second used an imploding core of plutonium that had been created beforehand through the rather outlandish procedure of bombarding uranium with extra neutrons in a breeder reactor. The people at the Manhattan Project, rather than dwelling on the things that wouldn't result in a functioning atom bomb, had taken the cognitive leap of looking for things that would.
I meant it not as an argument about nukes per se, but as a parable about a common style of argument that X can't be done, for many different values of X. This kind of argument points out problems but doesn't think to look for solutions, then concludes that the problems are insoluble so forget about it. I remember e.g. arguments that electronic books would never take off because reading on a computer monitor was inherently much less pleasant than paper; at a time when a typical monitor was a big clunky 800x600 CRT this sounded reasonable -- and indeed people don't often read books on those things these days, but ebooks are a big deal even so.
>"Pitching it as “instead of worrying about superintelligence, worry about this!” seemed like a cute framing device that would probably get the article a few extra eyeballs."
I really don't know why you didn't consider it or didn't mention if you did, but to me it seems obvious that he treats it as something zero-sum. IDK what exactly Acemoglu would consider limited - maybe research funding, maybe public attention span, maybe willingness to go and spread awareness of an issue - but to me his article reads like very obviously saying "If you are concerned about AI risk and want to give this limited thing to improve the situation, don't waste it on the long-term AGI risk people and give it to current AI risk people". I don't know enough about Acemoglu's current research to know if this is directly self-serving (i.e. if he is a current-risk AI researcher).
I've always found this argument weird, because the reverse seems much more convincing to sceptics - "here's a bunch of examples of AI being biased/wrong/misleading right now, we expect this to cause significant problems in the future and so think we should fund AI safety more a carefully regulate the industry."
The other classic argement is that people have always feared the risk of technology/magic run amok... I'm just saying, maybe the Golem myth is recognising something important about the dangers of poorly constrained power?
I think there is a tendency to dismiss future concerns in favour of current problems, but we could try being concerned about both - you know, just a little?
>There's no merit at all in what they're discussing, in his opinion
He doesn’t say “there is no merit at all..” That’s your interpretation.
My reading of stop worrying about evil super intelligence was “Hey we know of these problems right now”
The rhetoric is imprecise because it was a very short Op Ed piece with no space complete and precise wording.
If you were to ask the author if he thought there was no possible risk from a hypothetical future AGI, I doubt he would say yes. Do you think he would say yes to that specific question based on this piece?
Acemoglu knows the literal words are not true. He knows that the readers know they are not literally true. He’s using a common informal rhetorical device to grab our attention in a few words. You can say his words are imprecise and technically inaccurate and you’ll be correct. Precision and complete technical accuracy isn’t something I expect from a short opinion piece.
I think we can expect a reasonable reader to get the gist of the point without being misled.
No one is going to read it and go to bed tonight and think “Well I don’t ever have to worry about a malicious AIG ever again.”
This isn’t a long form statement of position. It’s a couple of column inches to call attention to an immediate problem.
Okay. If Acemoglu believes there can never possibly be a threat from AGI or was trying to convince people that is true I’ll join you in criticizing him.
I don't think there's any merit to disregarding commentary from outside of the field of AGI and we should be careful about promoting that behaviour.
AGI is barely a field, in a sense it never will be a field (as soon as AGI actually exists, the field ceases to), there is a sense in which all of the best commentary comes from outside of it, and we *need* that commentary.
If anything, it would seem to me that economics is more relevant to AGI forecasting than most machine learning, as it encompasses familiarity with decision theory and advancing industrial processes.
It may be the case that most economists are wildly wrong about AGI due to having been overfitted on stories populated exclusively with humans larping agency consistently terribly, but the person who is most right about AGI forecasting is probably also going to be an economist.
The economist post doesn’t add any useful outside commentary on AGI or AGI commentary.
There may be several examples of great outside commentary as you say; most of the great criticisms of the AGI community that I’ve read comes from the AGI community.
I think that the main question centers around the notion of 'agency'.
Electricity, matter, etc, does not in itself really represent any agent -- it can only do so through the complexity of the machine design. All forms of code, the design of machines, the selection and combination of algorithms, data, and UI, etc, will (and cannot not) represent the agency (intentionality) of the developer (usually to the developers, or their employers, direct or indirect benefit). Electricity, matter, etc, are very simple in comparison to computer tech (algorithms and apps, etc), and are thus more ethically/morally neutral. As the complexity of the causality increases, as the machine includes more and more parts, the code gets longer, etc, it more and more represents the intentions of its maker, either overtly or covertly, and becomes less and less ethically/morally neutral (either good or bad) in proportion to its complexity (specificity, unicity, etc).
As such, I must find that the argument attempting to directly compare the use of electricity, as tech, and the use of surveillance, as tech, to be incorrect to the purpose of establishing the notion of tech as being itself uniformly ethically neutral -- it is not. Surveillance tech, social media platforms such as Facebook, etc, are very much less socially neutral -- ie; their use have very direct and strong overall social effects. The effects of surveillance apps, etc, are also far less physical, then say, electricity, which when used by itself tends to have primarily physical effects and not so much social ones (except when mediated through complex compute machines, etc).
These sorts of potentials for ethical problems become even more noticeable when considering the use of narrow AI. Most implementations of narrow AI are so complex that even the developer/maker/owner/users cannot always be fully sure that the intentionality and distribution of cost/risk/benefit that they wish to be represented in the design of the narrow AI system is actually functionally represented in that system. In some sense, the narrow AI is *probably* still operating in the best interests and benefit of whomever it was designed to be, but as the systems get more complex, as more variables and data is involved, the clarity of "whom does it serve?" becomes even less clear. Legal cases involving "who is responsible" when some system like a Tesla hits a pedestrian can already expect to have difficulty when attempting to establish a reasonable chain of causation. Who bears the costs, who handles increased risk, and who gets more dollars are rarely the same person and/or people, in time, space, or scale.
Finally, when we get to general AI, we are clearly in a space in which the notion of agency begins to shift entirely away from the maker and into the thing in itself. What rights of utility does a parent have, and can demand, over their own children to act exclusively on the parents behalf? To the extent that something can have self agency, and be self authoring, and to the extent that 'slavery' is considered bad, the moral/ethical entanglements become particularly significant. I would argue strongly, on the basis of a number of considerations, that over the long term, there can be no realistic expectation to have a co-existence of strong AI and any other form of life on Earth. For more on this, see my dialog article at https://mflb.com/2142 (MFLB dot com document number 2142).
Whether or not Acemoglu meant to make the point, or makes it well ... _do_ the long term concerns about AI distract from pressing short term concerns? Is that a good thing? A bad thing?
Personally, I am in Maciej Ceglowski's camp that the pascal's wager of superintelligence is pretty overblown. And in the near term I think it's ridiculous and embarrassing for the software industry that Apple released a credit card based on a sexist AI. What credibility do the rest of us have if even Apple, a company larger than some countries, can't unit test a question that impacts half of all people?
So far as the two concerns compete for time, money, and public legitimacy, shouldn't we care? Are we letting a bunch of excitable zeppelin engineers concern-troll us about the speed of sound while ignoring how flammable zeppelins are?
Almost no one with a megaphone takes long-term AI dangers seriously, but lots of them take short-term AI dangers seriously. So if anything, I'd be more worried about the latter crowding out the former.
But really not even that, because I don't think the response to someone trying to solve a serious problem is to tell them to shut up and stop worrying, because they're using the oxygen that could be used by someone working on a different problem. I don't think "distract[ion]" and "concern-trol[ling]" are accurately describing how the many quests to solve big problems works. (Somewhat relevant xkcd: https://xkcd.com/871/)
Yeah, concern-trolling was the wrong choice of words; I don't think there's a lack of sincerity. I think you described it better with taking the oxygen.
It seems to me that the "short term AI concerns" camp has more or less been taken over by people who are hell bent on using the term "algorithmic fairness" as a fig leaf for forcing ML systems to produce results that are favourable to certain politically powerful groups.
While I have my issues with the "long term AI safety" camp, at least they're not actively making AI behave _less_ ethically.
The serious argument isn't "the risk of unaligned superintelligence is tiny, but the harm is so huge we should care about it anyway."
The serious arguments say that superintelligence is fairly likely. (10% to 90% range). Out of control superintelligence is the sort of thing you should expect to happen sooner or later in a world with lots of compute, and inadequate safety standards. (In the same way covid is the sort of thing that's likely to happen sooner or later when you do gain of function research in leaky labs)
If we assume AGI is a potential existential threat, AI alignment research is not the solution. (I don't have any solution, so we should do some anyway.)
We have been doing human alignment research since forever, and yet we mostly fail at it miserably. There are very good reasons for it to be difficult : game theory, human emotions, incompatible world-views, etc.
So even if an AGI's goals are perfectly aligned with it's human handler, this human goals are not likely to be the benefit of the rest of humanity. And as they will probably be several (or many) different AGI belonging to different human organizations with conflicting goals, they will compete each other and the most successful would likely be the ones that are given less constraints by their human handlers.
Stanisław Lem wrote a similar argument in Summa Technologiae. Human giving up control to the machine, because they can't do without it's efficiency.
I don't have any solution, but at least there is a little hope that when an AGI takes over the world, maybe it will rule it more sensibly than us (well than those who got real power now, not "us") and keeping humans as pets. (See Culture series by Iain M. Banks).
Even if they kill all humans, we will probably do it ourself without AGI anyway, so at least with AGI there is something left to follow humanity.
Your reasoning argues that AI Alignment is insufficient, not that it's unnecessary.
(I assume your last two paragraphs aren't serious because I don't think anyone outside of cartoon villains is actually that heartless and empty, so I'm just gonna ignore those)
>We have been doing human alignment research since forever, and yet we mostly fail at it miserably. There are very good reasons for it to be difficult : game theory, human emotions, incompatible world-views, etc.
I get where you're coming from, but it does not follow that "because human alignment methods don't work, AGI alignment research is unlikely to find anything that works."
There's a critical difference between humans and AI. Your ability to influence humans is limited by speech, culture, and psychological conditioning. You don't have access or control over the core systems inside.
With AI, even though some aspects seem like a black box (though we are finding ways to reduce this), you have control over how it is designed from the bottom up. You have the ability to try innumerable models, simulate and model them, and modify, twist, take apart, combine, transform, and so as you will. With that greater control comes the possibility of designing AGI for alignment. This was never a possibility for human beings.
Sure. But the point is more "if we got AI alignment without human alignment, it fails".
I would love that an AI alignment team that includes in AGI core values ethics, human rights, ecology. However these values are not shared by most states and corporations. Even for democracies that got them in their constitutions, it's rarely seen in their actions.
How do we ensure we have only super-intelligent-ethical-AGI ? What prevents a country or corporation to use AI-alignment research to get evil-AGI aligned with their goals ? That would mean having to solve human alignment too.
Other scenario. Your company put a new super-intelligent-ethical-AGI to optimize production of your factories. It decides to cut production in half, in order to respect international objective to cut CO2 emissions. It explains that increasing profit is useless if earth become inhabitable. It is super-intelligent, this is right and it can prove it. What does the company board decides ?
See also short story "Malak" by Peter Watts, about an aerial combat done with an ethical-AI.
I think part of the answer--part of the effort, at least--is for more ethical people to get control over AI development efforts first. And even after AGI arrives, for those people to maintain some level of influence and control in the organizations who use it.
I think this is happening to some extent, with OpenAI and and DeepMind. But I think that this is because AI isn't currently seen as an extremely power-bestowing assets by today's companies and governments, and we're thus in a period where it's easier to infiltrate with ethical people. Once AGI is close or has already been achieved, and is capable of making enormous differences to the world, the full power of states and corporations--both of which are inherently amoral--will be brought to bear in attempting to seize control and ownership.
You don't see many ethics boards in defense, fossil-fuel, or weapons development organizations. I hope AI's future won't be the same.
Is there a convincing argument that AGI is possible within any reasonable timeframe (like... 50 years), other than the intuitions of esteemed AI researchers? Do they have any way to back up their estimates (of some tens of percent), and why they shouldn't be millionths of a percent? It is, as another poster said, an "extraordinary claim." I'd like to see some extraordinary support of those particular numbers.
The argument that we are "in the middle of a period of extremely rapid progress in AI research, when barrier after barrier is being breached" makes it seem like all AI "progress" is on some sort of line that ends in AGI. That feels like sleight-of-hand. Even Scott himself refers to AGI here as a "new class of actor," so I'm failing to see how current lines of "progress" will indubitably result the emergence of something completely novel and different?
The intuitions of successful AI researchers as to the timeframe in which strong AI can be expected ought to be treated with the same skepticism we treat the intuitions of fusion physicists working on ITER as to when commercial fusion power should arrive. Someone enthusiastic enough about a potential Newton-Einstein-Bohr level breakthrough to go spend his entire waking life working to bring it about is not someone who will be coldly objective in estimating probabilities for its success.
I don't think this analogy holds up — for one, short time periods before strong AI is the pessimistic scenario, the things that the people you're talking about are hoping *doesn't* happen. So less like nuclear energy hopefuls and more like the scientists in the 1930s worried about nuclear bombs, or scientists in the 2010s and 2020s worried about gain-of-function research.
For another, the polls I've seen asking the entire field (not just AI safety people) about how soon they expect AGI tend to give very scary answers.
It's not an analogy, it's an illustration of a general principle of human nature I'm asserting, to wit, that the person who's life work revolves around a particular field is not likely to be objective in assessing the probability of the field producing existentially-threatening events.
If I ask a constitutional law professor whether the appointment of this person or that to the Supreme Court is Incredibly Important, not a one of them will answer meh doesn't really matter much. If I ask an epidemiologist whether we ought to spend an extra $50 billion a year to prepare for future infectious pandemics, they will all say "at least! probably more!" And so on. They are too interested to give coolly rational odds.
This is exactly why I discount people like Ray Kurzweil, who is undoubtedly smarter than me, predicting human immortality will be achieved right around a few years before the time he would otherwise be expected to die.
Ha ha yes. I have noticed that futurists start predicting imminent immortality at just around the age (typically mid to late 50s) when you start to think holy crap this whole death thing that has heretofore only happend to Old People is starting become relevant to little ol' me...
Definitely agree. But similarly, we expect ITER to turn on someday within our lifetimes despite the repeated delays, precisely because those same experts and funders remain invested.
I think this is what you're looking for; it's a more quantitative look at what could be done with 10^12 more compute: https://www.alignmentforum.org/s/5Eg2urmQjA4ZNcezy/p/rzqACeBGycZtqCfaX. (For reference, there was ≈10^12 *less* compute in the 90s. Though note that, as you can see in the first graph, things have accelerated greatly in the past decade or so.)
My takeaway: we really, really, *really* better solve alignment long before we get 10^12 times as much compute.
In 1919, Sir Arthur Eddington traveled to Brazil and West Africa to observe an eclipse. There he took measurements of light deflection that would differentiate between Newtonian physics, which predicted a certain amount of diffraction, and Einstein's general relativity, which predicted twice as much diffraction. Asked ahead of time what he would do if the experiment proved his theory wrong, Einstein replied: "Then I will feel quite sorry for the good Lord. The theory is correct."
The moral of the story being that there are certain types of people whose intuitions are worth quite a lot even without or in spite of the apparent evidence, and that the AI risk and Rationality communities tend to attract those types.
My reading of that story was not so much that Einstein had great intuition, but that he already had lots of evidence for relativity. If the observations didn't match, then the balance of evidence would still point to relativity being correct, and the observations flawed in some way.
I don't think the claim that tech X will be developed, where X is consistent with physics, doesn't seem to require unreachably vast amounts of resources, and has a research effort towards it, is extraordinary.
The process of producing AGI is the same sort of research and development used to produce other tech. You don't think the chance of fusion in the next 50 years is millionths of a percent, do you?
The default uncertainty is 50%. Millionths of a percent requires very strong evidence against AGI.
AGI being extraordinarily powerful doesn't make the claim "AGI will be produced" extraordinary.
To get probabilities that low, you would either need a convincing reason why AGI is physically impossible, or a convincing reason humans couldn't build it.
> Is there a convincing argument that AGI is possible within any reasonable timeframe
I would argue that there is not. Most of the claims about the Singularity rely on hand-waving and ignoring the laws of physics. The two primary arguments seem to be,
1). You can accomplish anything you want just by thinking really fast, and
2). Processing speed can be scaled up without any limitations.
Thus, the fast-thinking AI can just keep making itself faster until it achieves omnipotence. However, IMO both (1) and (2) are obviously false.
The idealized interactive proving system consists of a computer trying to find a proof for a given theorem by brute-force, and a human mathematician trying to intuit lemmata to use as stepping stones to make the brute-force search tractable. From how I do math, the mental machinery that the human uses here is pattern-matching, which deep learning is recently becoming very good at.
Trying to produce useful lemmata when you aren't given a theorem to prove produces useful conjectures. Therefore I expect that AI will soon surpass humans at producing maths papers. That's a victory condition.
I'm curious about "firms that increase their AI adoption" - does this include downstream adoption? For example, Facebook uses AI to target ads. An advertiser who gets a better ROI as a result has "increased their AI adoption" without even knowing it.
I'm not inspired to read the Acemoglu essay in question, but could it be that he's not the least bit concerned with future AI risk for the same reason that almost nobody else is concerned with future AI risk, which is to say that they are too busy being concerned about present political issues? Like, say, whether our economy is arguably too narrowly optimized for the well-being of the wealthy superelite and their elite STEM servants when everybody "should" be trying to ensure good, stable jobs for the working class.
If so, then Acemoglu has certainly noticed that there's a set of smart and motivated people, not wholly without influence, who are "wasting" their time on this AI risk nonsense when they could be focusing on the thing Acemoglu cares about. And, by connecting the two through the medium of current AI and nascent marginal technological unemployment, he thinks he can marginalize the AI-risk community as "out of touch" and get some of the marginal members of that community into paying more attention to the things he cares about.
If so, I agree that his article is unlikely to be enlightening or persuasive to anyone here, but it might at the margin serve his goals (and those of the WaPo editorial staff).
There are a lot more people in politics and tech who care about inequality than AI risk lmao. They may not be caring about it effectively, but ... I mean look at EA donations or charity of tech people as a whole.
It's the year 1400, and you're living in Constantinople. A military engineer has seen gunpowder weapons get more powerful, more reliable, and more portable over the past two centuries. He gets on a soapboax and announces: "Citizens of Constantinople, danger is upon us! Soon gunpowder weapons will be powerful enough to blow up an entire city! If everyone keeps using them, all the cities in the world will get destroyed, and it'll be the end of civilization. We need to form a Gunpowder Safety Committee to mitigate the risk of superexplosions."
We know in hindsight that this engineer's concerns weren't entirely wrong. Nuclear weapons do exist, they can blow up entire cities, and a nuclear war could plausibly end civilization. But nevertheless, anything the Gunpowder Safety Committee does is bound to be completely and utterly useless. Uranium had not yet been discovered. Lise Meitner and Otto Frisch wouldn't be born for another 500 years. Nobody knew what an isotope was, and their conception of atoms was as different from real atoms as nuclear bombs are from handgonnes. Rockets existed, but one that could deliver tons of payload to a target thousands of miles away was purely in the realm of fantasy. Even though the Roman military engineer detected a real trend--the improvement of weapons--and even though he extrapolated with some accuracy to foretell a real existential threat, he couldn't possibly forecast the timeline or the nature of the threat, and therefore couldn't possibly do anything useful to inform nuclear policy in the 20th century.
A more reasonable military engineer tells the first engineer to focus on more pragmatic and immediate risks, instead of wasting time worrying about superexplosions. Cannons are already powerful enough to batter down all but the strongest city walls, he points out. In the near future, the Ottomans might have a cannon powerful enough to destroy Constantinople's walls. How will the Roman Empire survive this?
I’ve never really liked analogies as arguments. For explaining and referencing, they’re pretty useful, but for an argument, there are a lot of properties that are true for the analogy (gun powder/explosions) that are different for the actual subject (AGI).
In the nuke analogy, I'd say we're more analogous to 1900 than 1400. and the first engineer might reply that if the city gets nuked, nothing the cannons did will really matter. That doesn't mean you need to drop everything and focus on nukes, but it would be nice for everyone to get together and declare that at X moment when we Definitely Still Have Time To Succeed And Don't Have To Worry About Mistakes Or Black Swans, we will all drop everything and start worrying about nukes Together.
Even here though, what are you going to do? What meaningful research that will actually do anything about the problem of "Nuclear Weapons Risk" can you do in 1905? Sure you may point at the problem, but that's not going to do anything.
Well, as far as I understand, the original vision of Yudkowsky was to avoid creating weapons at all, and focus on developing safe nuclear power before anybody else makes a first weapon. He even tried to make some arguments as to why this isn't necessarily a doomed enterprise, combining them with arguments of why failing to do so means certain extinction. The whole thing can be summarized as "humanity is almost certainly doomed if it continues on its current path, but if some selfless heroes (we, that is) manage this unprecedented heroic feat, there will be a paradise instead". It was persuasive enough that MIRI still has a growing multi-million dollar donation-based budget, but this certainly wasn't a mainstream-palatable narrative, which was also acknowledged.
Considering the absurdly high payout of being the only country with nuclear arms, that was never going to happen. Is AI the same? Probably - you can argue however much you like, and then China builds AIs anyway.
Back in the OB/LW heyday of 2008-09 it was slightly more believable. AI winter was still in full swing, with NN breakthroughs still years away, and contemporary wannabe "AGI developers" obvious clueless cranks. I'm curious as to how the "party line" has shifted since then, but MIRI is pretty tight-lipped these days.
But the nuclear weapons did not suddenly decide to bomb two Japanese cities of their own accord, it was humans that did that. Nuclear bombs that just sat in the arsenal gathering dust as a monument to "we cracked the problem of how to make these" aren't a threat; humans deciding "let's make something REALLY BIG that can REALLY DESTROY EVEN MORE" are the problems, and are the problem be it "our most cutting-edge tech is a pointy bit on the end of a length of wood" or "we have swarms of killer robots, a.k.a. drones".
Work on the human problem before you start wetting your knickers about "but what if a computer suddenly becomes self-aware?"
What I mean is, right now it's people who are working on AI. The idea of "and the AI will reach a point where it will recursively improve its own intelligence by rewriting its source code over multiple iterations in milliseconds and then it will become Silicon God" is pie in the sky when you die.
What do we, *humans*, want out of AI right now? And so far as I can see, it's "if we're the first ones to get this to work, it means MOAR MONIES". 'This work could imperil humanity if it goes too far' versus 'yeah but it'll bump up the value of the stock' and the better share price wins, every time.
We're greedy, stupid primates and the biggest threat. We could stop all the AI work right now, if sufficiently scared - you don't have to worry about Unfriendly AI if the most complicated machine intelligence is what tries to get you to retweet something. But we seem to be locking ourselves into "well, this is gonna happen, because we want our shiny toys".
*That's* the problem there, friends. "We can't stop because we have to have even bigger, better, shinier toys that make us richer" and if that means a risk of enslaving all of humanity to turn them into paperclips, well, look how my stock options are even more valuable!
This. Not only is it more sensible, you may even get people to go along with it. It's a virtual certainty that developments in AI will cause all kinds of issues, which should be tackled the way we tackle other problems.
Meanwhile, arguing about Superintelligence or The Singularity is a pointless exercise, not only because it's purely hypothetical, but also because we're not in a position currently where we understand enough for the arguments to have a decent chance of applying.
Sure, but you could place the exact same explosives safety committee at the beginning of WW2. Its hard to tell where we are on the timeline to AGI. We can't say with confidence it is 500 years away.
Also, the gunpowder safety committee in 1400 could plausibly have invented the concept of Mutually assured destruction and arms races.
Some practical discussions like "how do we achieve a perfect air-gap" is certainly among what we can do already. Formalizing MAD theory was one of the very few things academia could deliver against Nuclear Weapons Risk. This is yet another strike against AI risk research - even if you're *completely* right, will that actually help in a situation that's driven by money and power?
If the buzzword “Artificial Intelligence” was never created and we just stuck with “Machine Learning”, would we still be talking about the existential threat of Machine Learning?
It is so weird how so many people, even in this thread, reject out of hand the idea that AI much more complex than current ones could exist. Like, we have an AI that can win at go and Starcraft, we have good text and image based AI, self driving car AI that works very well almost all the time, where else do people think that goes?
Although I tend to agree with you, there are good reasons to think AGI is still a long way off. For example, training AIs currently requires massive amounts of data--far, far more than humans use in their own learning. Massive data sets don't exist (and can't be created) for everything, and that could be a barrier to AGI.
I am optimistic that algorithmic improvements will be discovered making AGI feasible, but I think it's also fair to say the current tech doesn't have a path to scaling into an AGI.
Yeah, I have no idea in any sense negative or positive about when AGI lol
However that doesn’t mean it won’t happen nor does it prevent other AI things from severely disrupting people and society
Anyone genuinely worried about “AI mortgage selection will menace black people” really isn’t considering even the consequences of currently available technologies which are quite a bit larger
I think we know that bigger models use data more efficiently, so presumably when current models get as big as the human brain they will also be able to use data as efficiently.
I guess it's due to the nature of the fact that these are all relying largely on statistical function estimation techniques computationally similar to each other that they get grouped together as "AI" as opposed to any other more boring kind of math, but language understanding does not seem in the same class with vision and navigation. Plenty of animals not generally considered more intelligent than humans that have not managed to take over the world can nonetheless see better than us, and better navigate more complex terrain. Birds figured out how to fly and circumnavigate the globe millions of years before we even existed.
That seems to raise the question of whether figuring out how to construct structurally correct human language sentences with genuine semantic content is actually categorically different, though, and if so, why?
I mean that the ability to communicate in complete sentences and write them down for future generations to build an ever-increasing store of knowledge on top of is what actually makes human intellect a tool that can build civilizations. Our vision and navigation capabilities are doing comparatively very little for us. Not all levers exert the same multiplier just because the computational methods we've discovered thus far for getting machines to mimic them is structurally similar.
Similar to the age-old theme here about how important IQ is. Whatever IQ is actually measuring, it doesn't seem related to vision or sense of direction. Very smart people often wear glasses.
I mean if humans lost their eyes or legs 5k years ago that would’ve been it, long tradition of cultural evolution or not. To say nothing of, again, the strong overlap between the mechanisms both principled and literal between understanding creating speaking seeing etc
IQ could very well be correlated with visual perception skill, if not “eyeball curvature precision”
I guess I'm not getting across what I'm trying to say here.
Language understanding is categorically different from vision and navigation because:
1) Vision and navigation don't distinguish the only animal to create civilizations from all other animals. Many non-human biological organisms have equally good and in some cases much better abilities to sense and navigate their environments, not only immediate environments, but some animals have seemingly managed to maintain accurate mental maps of virtually the entire globe.
2) Language understanding, on the other hand, gives us much more. At bare minimum, the ability to transmit information across time and space to distribute the learning workload across all humans who will ever exist, and the ability to bootstrap the mental starting point for new humans who no longer need to learn what earlier humans already learned and wrote down. As far as we know, humans are the only animal to achieve this ability and it has enabled us build civilization. It is possible that some cetaceans may be intellectual equals but can't create much in the way of technology due to the engineering challenges posed by living underwater.
This obviously makes language understanding categorically different from vision and navigation in the sense that it is possible to have the latter two without having the first. Whether it means you necessarily need some fundamentally different algorithm to implement it, on the other hand, I have no idea.
But I suspect yes. Obviously, humans don't learn to speak by reading every text transcribed to the Internet billions of times. We're much closer to the already-trained versions of language models that are able to implement few-shot or one-shot learning by bootstrapping from serialized pre-recording of earlier models. But that isn't how we did it, either. Our brains now don't start out at birth much differently than when writing first arose, so the ability to learn language quickly from few examples clearly can't have come from the first humans being trained on the prior language examples of some earlier civilization. Human language understanding, in contrast, is an ability to map sounds and symbols to mental abstractions associated with memories of other sensory input, as in the word tree and the associated sound of it being spoken are really an association graph between all sights, sounds, smells, and touches we can recall of actual trees. The association graphs predate the existence of the linguistic representation and language arises from them. Human were never taught language. We invented it.
Whether this matters or not, I obviously also don't know. It's a fundamental unanswered question at the heart of AI whether true understanding has to involve knowledge representation being synthesized from first principles or whether just beating a universal function approximator into submission with trillions of training iterations until it passes an authenticity test is enough.
Is GPT3 categorically different from Alexnet? Is that categorically different from 1990s neural nets?
Even if you can’t do language good with current backpeopqgation massive neural network transformer stuff, that probably isn’t how vision worked in animals either. Ravens don’t have language, chimpanzees don’t, they can still use tools and be smart.
Nerds since Kurzweil have described the dawning of "Artificial General Intelligence" as an event horizon; everything before it and leading up to it is kind of trivial.
Acemoglu, Pinker, et. al., seem to be saying something more like: AI is getting continuously better and we should continuously evaluate how it affects us. We are already living with AI, we can talk about its benefits and costs without worrying about the day after Revelation.
There is a good analogy here with climate change: you don't need to look one hundred years into the future, you can just look around you right now to figure out that something bad is coming.
I would like to ask a question unrelated to Acemoglu, and not knowing a lot about AI, if possible. Articles often are talking about "the" AI. Isn't it more likely that there will be a lot of different AI's, some dangerous to humans, some friendly, and they will discuss or fight each other, and humans will be negligible to them, treated friendly or bad, like animals to humans now?
Not everyone agrees with this, but there is a line of reasoning that argues if for a scenario where as soon as one AI reaches a certain level of superintelligence, it will be able to recursively self-improve faster than all other systems and quickly take over the world. Hard Take Off is often the terminology used to describe this potential phenomenon. e.g. https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff
> But all the AI regulation in the world won’t help us unless we humans resist the urge to spread misinformation to maximize clicks.
I realise that 'AI regulation' here means 'regulation of social media algorithm AI' and not 'regulation of us by AI', but it touches on something I often think about when the subject is brought up. The prospect of an AGI capable of gainsaying our worst instincts would be such a huge boon that the risk of singularity may be worth taking, not unlike the residual risk associated with nuclear technology.
From the perspective of someone who would like to see humanity get central planning and collectivism right one day - major problems being the computational problem of allocation of resources and basic human corruptibility - an artificial auditor that can say 'Sorry, comrade, I can't let you do that' would be an incredible step forward and perhaps less vulnerable to misuse than advanced narrow-AI.
I think your reading here is grossly unfair - the argument is surely "let's worry about the actual problems we see today already, not the highly hypothetical Evil AI situation". That is, an argument for what should be *given attention* to.
I think this is more a difference in rhetoric and audience than a disagreement about the facts. Daron Acemoglu is writing for people who live in a world filled with clickbait articles about some far-future killer AI and a pop culture where AI is synonymous with Skynet. He's saying: "This isn't some hypothetical future thing, it's a real problem and it's happening right now!" It's only in this context that he's saying we shouldn't worry about AGI. Scott Alexander is writing for people who already think narrow AI is dangerous and trying to convince them to also be worried about AGI.
Imagine there's a small fire in a building. Some people aren't worrying about it because they think that only large fires are dangerous. Other people aren't worried because they don't think small fires can become big fires. If we want to convince people to take action and put out the fire, we need to use two different contradictory arguments. "Forget about giant fires, small fires can be really dangerous, too!" "It looks small now, but forget about that, it has the potential to get big really fast!"
You could argue that the first group is wrong or have the wrong priors or whatever, and maybe you'd be right, but if we want to put the fire out, that means meeting people where they are and convincing them using arguments they'd listen to.
My line of work has increasingly come to rely on "AI"/"ML" techniques, so I'm somewhat on the pointy end of this. The employment effects are real, but also really complicated.
For example, I co-wrote a piece of "AI" software (I don't think it should be called AI but the competitors call their software AI, so...) that explicitly replaces a LOT of human labor, including my own. It can replace the work of weeks or months with the work of a few days, and that's my work or the work of my direct reports - not hypothetical or knock-on effects.
But, it doesn't lead to me doing any less work, because in the past the prohibitive labor investment meant that companies simply did a lot less of the thing. They'd look at the price of doing a comprehensive program, throw up in their mouths a little, and choose something more achievable. So, now we do a whole lot *more* of the thing, because now the value proposition is better! My utilization on the thing has only risen, despite vast leaps in time efficiency.
I assume that many AI employment effects will be like this - it will be true that the AI tool replaces a lot of human labor for the same unit output, but NOT necessarily true that the total human labor spent on the thing thus decreases.
The main concern is that automation (including AI) will steadily eliminate the only jobs available to people with IQ below the ever-climbing cutoff value, a rising tide of obsolescence. Those smart enough to develop said AI are least-concern, until it starts to develop itself that is.
Sure, so for example I went to an open house last night for a company that's working on building earthen structures using drones. The drones are trained with ML techniques, so we can call it an AI application.
Explicit labor replacement by drones, right? But it still takes labor to do the dozens of things other than "put structure in place" and they expect to push structure costs down quite a lot. Up to 30% less than stickbuilt if they hit their targets.
So, it could be the same situation - a few framers are displaced, but the lower total cost to build makes more projects viable, which leads to more construction, which actually ends up increasing total employment.
"extremely smart people and leading domain experts are terrified of this"
I think this is the weakest part of Scott's argument. Peoples fears and imaginings have precious little to do with their smartness. Whether or not someone worries about population growth (or decline) or climate change or AI risk has more to do with their personality than with their intelligence.
In a similar way you can find a hundred very smart economists who worry that raising the minimum wage will have negative effects and another hundred who worry that not raising it will perpetuate negative effects. I use that example because it was written about most clearly on SSC.
The fact that some very smart people worry about something tells me almost nothing about the thing being worried about.
I don't think "smart people and domain experts are concerned about X" lets you infer "X is definitely a problem with very bad consequences", but it does seem pretty safe to infer "if X is in fact good, this is pretty non-obvious, and worth thinking carefully about before making any confident assertions."
The economist disagreements do tell you something - it doesn't mean you know whether a minimum wage is good or bad, but it does mean that the meme your grandmother shared on Facebook probably doesn't suffice to settle the issue, and your own opinion is probably little better than chance unless you have actually thought about economics a decent bit and read some relevant studies. (Contrast with e.g. worrying about a global conspiracy to cover up a flat Earth - very few smart people and domain experts seem concerned about this, so dismissing it on the basis of relatively simple arguments is less epistemically worrying.)
I think that's very dependent on the field. If social psychologists or criminologists are very concerned about some social psychology or criminology related thing, I don't consider that strong evidence. If cosmologists are very concerned about a cosmology thing, that'll make me pay attention.
Your prior on how trustworthy the authorities in question are matters a lot.
> "I certainly don’t mean to assert that AI definitely won’t cause unemployment or underemployment (for my full position, see here). I think it probably will, sometime soon...."
It's been a minute, but I recall your argument in the technological unemployment post being similar to ones put forward by Sam Harris, Andrew Yang, Robin Hanson, and many others; AI will be better than us at almost everything, so most people won't be able to work anymore, so more inequality will result/most jobs will disappear.
While it's true that AI isn't literally the same thing as [insert past technology here], it's also true that comparative advantage is a thing, and that AI is just one more technology that lets us take advantage of comparative advantage for the creation of more goods/services.
Pinker is notorious for pretending the difference between hunter-gatherers and pre-state agricultural societies does not exist, even though this difference is crucial for the topic he is talking about, just because it would make an ugly dent into his beautiful graph on steadily declining violence. That is not a minor lapse, but an epic fuckup that should have gotten him tarred and feathered and laughed out of harvard.
I guess you should not be surprised that a person who is willing to sacrifice the truth for a beautiful graph on a topic you don't care about is willing to sacrifice the truth for a beautiful argument on a topic you *do* care about.
We really need to hold public intellectuals to higher standards.
Unrelated to long term AI risk, but he seems to take as a premise that AI is 'concerning,' then points out that AI is actually being used to do things, which is therefore supposed to be more concerning. Maybe AI is better at making sentencing decisions or bail decisions. Why is this automatically concerning?
Also, I'm no economist so maybe I'm missing something, but his point about AI not generating a labor-demand boost seems like bad economics. If substituting AI for humans doesn't boost productivity, then why is anyone using it? Presumably, it's cheaper, which drives down costs and prices and therefore boosts demand for other (human-made) goods/services. That's the main reason labor-saving devices boost demand for other professions, right? Agriculture technology ultimately caused demand for farm labor to collapse, rather than boosting demand for farm laborers by making them more productive. And yet unemployment didn't rise to 90% as a result.
1. Some people say that AI might be dangerous in the future.
2. But AI is dangerous now!
3. So it can’t possibly be dangerous in the future.
4. QED!
Here I thought we were steelmanning things, not strawmanning ...
The argument is more like yes, nuclear war could result in nuclear winter ... but also we will all be nuked. Why are you worrying about the winter instead of ya know, the billions that would die immediately? Climate change could result in a runaway feedback cycle of methane release that turns the world into Venus - but is that really the main issue that should be framing the debate?
Why not look at the actual pressing problems instead of conjecturing ill-defined bogeymen to be worried about?
1) There is limited attention and limited ability for the public to process what "the problem with X" is. To the extent that one defines the problem with X as Y, it does indeed downplay all the other problems and can be counterproductive.
2) Like mentioned at the end, the risks of superintelligent AI and the risks of ubiquitious big data and machine learning are really two separate things entirely, not just timescales or magnitudes of the same thing. Conflating the two, then, is more damaging than the usual.
The biggest problem I have with these AI doomspeakers is just how little they understand the state of AI today. The vast majority of it is nothing more than monkey see, monkey do.
The best example I have personally seen of this is the Watson AI that beat the best Jeapordy champion. It was a multiyear process of development that certainly cost tens of millions of dollars (or more), but the resulting Jeapordy champion couldn't even be redirected to understand the basic gist of internet articles. When touring the Watson demo office in SF before COVID, there is a demo room in there where this AI is showing the top trends in the internet worldwide.
One monster problem was immediately apparent right away: there was a location near the tip of South America which was in the top 10 sources of news at the time - which I believe was the original election of the Socialist in Bolivia prior to his ouster.
A 2nd example was soon forthcoming: Trump! This AI had overall internet news coverage of Trump being 55% positive - which seemed clearly wrong so I asked if it could drill down. The 10 actual article in the next level down - 5 had the word "Trump" in it but had nothing to do with Donald Trump and weren't clearly positive or negative. I recall one being someone named Trump being a key part of a winning middle school sports team...
A 3rd example: MIT review had an article talking about an effort to visualize what AI neural network, image recognition algos were actually seeing. It popped up with some eye-catchingly stupid stuff like a NN network image recog program believing all barbells have an arm attached, because obviously there were no pictures of a barbell by itself in its training silo.
Thus I am far from convinced anything we see today in "AI", actually machine learning or neural network programming, is actually intelligent or that the entire field is very far removed from 12th century alchemists mixing actual chemicals with animal dung, ethers, griffon feathers and what not.
Agree on actual AI - but we don't have that. Until we do, I see all of these articles on "AI is killing jobs and we should accept our robot overlords in various ways" as nothing more than PMC plus oligarchy excuse making for their utter failure in leading American political and economic systems astray for several decades and counting.
I should add more on the "tip of South America" thing. I'd bet money these were various agitprop tweets against the Bolivian socialist elected President.
If GIGO - what do you call something that takes in the entire body of Twitter for input?
Did you know the inventor of the loudspeaker himself for the rise of fascism as without it charismatic leaders with never been able to address such large crowd
"Our capacity to do anything about this is about 0"? A curious claim about an AI that is ultimately built by humans, on chips made by humans.
"Given that this is super rare when we look at things on the timescale of human activity" Another interesting claim about something built by humans. Wouldn't you expect the 10 to 100 year timescales of human tech development. Not the millions of year timescales of large space rocks?
Typo. Are you arguing that at some point humans will make smart AI, but the people doing that won't be influenced by current research, even when that research tries to be as general as possible.
Small note on Ted Chiang. I recently read both of Ted Chiang's short story collections, and I could not stop thinking how SSC-like many of the stories were. Would highly recommend to any SSC reader!
Good to see that Scott would also recommend his short stories - but a shame that Ted has unusual reasoning when it comes to AI/capitalism.
I strongly second this. I'll add that I heard about Ted Chiang from Very Bad Wizards, a philosophy/psychology podcast which I would also highly recommend to ACX readers.
I third this.
Weirdly, his story "Understand" reads very much as a "superintelligence escapes the box" kind of thing. I'm having some difficulty modeling how the same person ended up writing both this story and that BuzzFeed article.
Yep, I second THIS
(and the original comment)
Ted Chiang has some great stuff, but a science fiction writer has no better an idea of the future than they do religion. Scientology notwithstanding.
> Then he goes on to talk about several ways in which existing narrow AI is pernicious, like potential technological unemployment or use by authoritarian states.
I remember a long time ago, back on the SSC blog, Scott wrote an article that was like an experiment on different ways to convince people that AI risk is something worth caring about.
Emotionally and intuitively I still can't bring myself to care. But I have to say, the one thing that really shifted me from "pfft this is dumb" to "huh they kind of have a point" was "imagine how authoritarian states could weaponize this"
That one really resonated with me, because it means that you don't even need superintelligence for AI risk to be a serious problem.
I'm writing this comment while reading, and haven't read anything past the quoted passage. But, like, the quoted passage kind of _is_ a knock-down argument? It's just a knock-down argument in favor of the AI risk hypothesis, and not the other way around!
EDIT:
To be clear, what I'm saying is that if we can already demonstrate AI risk _now_, with dumb AI, then superintelligent AI must necessarily be even more dangerous
This relies on an additional point, which is that as you scale the intelligence of AI you don't necessarily scale its goodness. ["If the authoritarian states were smarter, then they wouldn't have camps; surely your superintelligence will figure this out as well."]
why can't we demonstrate the weirder 'what if ai did x' stuff we read about on acx, lesswrong etc just from really odd extremes of human behaviour right now?
You mean, like setting the world on fire in order to power and more and more factories; factories making modern conveniences, such as paperclips?
yeah that's one example, but there are also more personal/individual things like addictions and mental illnesses, - we aren't a truly general intelligence but the range of behaviour people do is kind of surprising.
if you get stuck thinking about odd things people do, search the phrase ''florida man''
Isn't the issue with that approach that most people intuitively judge intelligence based on "behaves like I would in that situation"? So "super-intelligent florida man" becomes an oxymoron, something that goes away if people are smarter. It's the same reasoning as "a superintelligent AI would automatically have my (or reasonably human-friendly, at least) values".
There's a big difference between metaphorically "setting the world on fire", i.e. burning fossil fuels and raising global average temperature causing widespread damage and suffering; and LITERALLY setting the world on fire to fuel a planet-wide generator powering an atomic paperclip factory.
Agree. The hyperbole doesn’t help the argument.
That was the other argument that convinced me, actually. It was a combination of "look at what bad things that existing human intelligence, in humans, can do" and "in a weird way, corporations are a form of AI that hyper-optimizes for poorly defined goal functions in a way that predictably leads to bad results"
> corporations are a form of AI that hyper-optimizes for poorly defined goal functions in a way that predictably leads to bad results"
The same is true for governments, of course.
In fact, I would argue that it is true for any form of human cooperation.
Yeah, but you have to show that we *get* to superintelligent AI from where we are now, and even if we could do, that's not going to happen if we've screwed ourselves over with the ways we misused dumb AI.
These arguments always read to me like "a big dog with sharp teeth is dangerous, can you imagine how dangerous a fifty-foot dog with three heads and even sharper teeth is going to be???"
Friend, I'll worry about fifty-foot high dogs *after* you get this normal dog's teeth out of my backside.
I admit that the current concerns may seem more pressing, but if the finest minds of our generation were engineering ever larger and toothier canine creatures, I think that would be a cause for concern about where this might lead us.
That seems a little like a slippery slope to me. Also, comparing dumb AI that's not very effective at doing what it's supposed to with dire wolves seems a bit alarmist.
In the same way that comparing chimpanzees to humans would probably seem alarmist to someone who had seen chimpanzees and never seen a human?
This is the part that makes it obvious to me. We already have an example of an optimization process that accidentally invented a new optimization process which operated inside its perception-orientation-reaction cycle, and totally supplanted it. The idea that an entire complex system, such as a wheelbarrow, could be designed in a single day would have seemed laughable to some kind of hypothetical person whose only example of an optimization process was evolution. To that person, the only way to invent new shit would be via incremental changes where each change had to be viable in its own right. If you told that person that this new, superintelligent thing would be invented, called a human brain, which would be capable of *skipping across* multiple non-functional iterations of a complex design, in order to arrive at the final complex system by an entirely new process called 'cognition', they would have laughed in your face. The notion that new major optimizations could start being introduced every thousand years, or (how utterly absurd) even every hundred years, instead of every hundred thousand years would be totally outside of their frame of reference, they would react with the absurdity heuristic.
I feel like strong AI skeptics are in a similar position. And maybe if we didn't have such a strong example of humanity utterly and totally supplanting evolution through huge efficiency gains that evolution itself could never "think" of but could EASILY accidentally design over time, i'd be on board with skepticism. But we already have an example of such, and what a strong example it is! From a biology perspective, if you were to just examine the design of the chimpanzee brain and compare it to the design of the human brain, you could probably guess that the human brain is more capable. But without actually seeing modern civilization, it would be *really hard* to extrapolate from that difference in brain capacity, to the actual real-world effect of that difference. It's the difference between using sticks to probe an anthill in a clever way that makes it easy to eat lots of protein with very little effort, versus designing rocket ships to colonize nearby planets.
It seems to me that when it comes to AI, we don't really have any strong reason to believe that there might not be some similar comparison. Especially with the various forms of machine learning where the entire point of doing things with ML instead of explicitly writing the algorithms yourself is that you *don't know* what the code is actually doing. Can we actually look at a given neural network and tell the difference between a chimpanzee AI that can sort of work well enough to serve a purpose, versus a superintelligent AI that suddenly and without warning supplants us just as we supplanted evolution?
Sorry for long wall of text. But it seems to me like in the situation where we are uncertain what ratio of randomly chosen algorithms end up being optimization processes superior to humans, maybe we should think carefully about doing things like designing our blackbox algorithms using processes that are remarkably similar to natural selection
I think chimps are a lot more intellectually impressive than you give them credit for.
Wow that’s a wonderful argument! I work in evolutionary biology but never realized the parallel between the two processes. Thank you!
I had to sign in just to say that this argument really gave me pause.
This is a very thought-provoking argument. But the thought it provokes is that we should expect wholly unexpected and unpredictable improvements in problem-solving technique, not that we should extrapolate from recent AI research trends and the forecasts of AI experts than AI software and hardware development will result in a superintelligence in the 2030-2050 timeframe. The though it provokes is that the "intelligence" is not even the right word for the thing that will surprise us, that superWTF is as likely to result from religious cultists meditating with the aid of new psychotropic drugs as from Turing machines in silicon, and that the timescale is unknowable.
It's not a slippery slope fallacy if there actually are people trying very hard to get to the bottom. I don't think we should be alarmist, just appropriately cautious.
> That seems a little like a slippery slope to me.
A slippery slope is not actually a fallacy.
https://owl.excelsior.edu/argument-and-critical-thinking/logical-fallacies/logical-fallacies-slippery-slope/
But the route from narrow AI to general AI is, again, not explained. In fact the definition is not even . When you criticise the idea that algorithms become self aware then the proponents of of AGI say self awareness isn't necessary, if you say then then they won't be self directed, then that's not it. So we are back to narrow AI, which of course, can be dangerous. A narrow AI bot could attack the electric grid but it would be designed to do that.
Can you elaborate on what you mean by self directed? I think most of the concerns involve an AI which is at least capable of general purpose planning of complex sequences of real-world actions, even if it isn't capable of enacting these plans. (e.g. something which, given information about the world, and a request for a plan for achieving a given goal, outputs a plan) (by general purpose planning, I mean capable of skillfully (at a superhuman level) taking into account any facts that a typical person could understand)
I'm not sure that this counts as "self-directed", but it could still be rather dangerous (but also, if such an AI were produced, I would imagine that one which does behave as an agent-in-the-world wouldn't be hard to reach from that point).
(both in the case that it produces plans which achieves the goals in ways compatible with what the person asking for the plan would want, due to questions of "but who controls it?", and also in the case that the plans it produces involve consequences that the person requesting the plan would object to if they understood them, but where they can't tell that it would have those unwanted consequences.)
Is this basically the same as what you are saying there is no route to?
I'm personally not like, emotionally worried about AGI stuff, but that's not because I have an argument that I think would be considered by many to be a strong argument against it being a serious concern.
People have done a fair bit of work on "how to do planning". They started working on that a long time ago. Admittedly, that fact point might go either way in terms of whether it suggests that making something capable of general purpose planning is something likely to happen any time soon? Planning discrete moves in a game is one thing, planning simply physical actions to accomplish a task (move this joint, then that joint, grasp the ball, etc.) is also one thing, but perhaps complicated plans are another thing. If it is to be via a ML system, this would probably have to involve it doing experimentation outside of as part of training, which there is some limited hints of progress towards, but idk.
Yeah, that's where I'm stuck on not being entirely convinced we're that close to truly powerful AI. I mean, VR has been right around the corner for years as the new tech EVERYONE is going to be using. We're still waiting on autonomous driving to be decent and functional. Facebook and YouTube can't effectively stop the spread of anti-vax nonsense on their platforms. I guess I just don't have as much blind faith in technological innovation as I'm supposed to, I guess.
I'm not writing off progress, just that I think it will be a lot slower than expected, won't happen in the way expected, and humans are always much, much more of a threat than any artificial system, mainly because we *love* creating artificial systems, turning them loose to MAKE MONEY (be that under the guise of "in order to serve our customers better and improve service, we are closing down all our bank branches in small towns and you'll all have to go online and if you want to lodge cash into your account LIKE A CAVEMAN, then you'll have to make a thirty-mile trip to the nearest branch that still handles cash" as one of our banks just did recently), and then not having any idea what exactly is going on as the machines do everything faster and in much greater volumes than we can keep track of.
I think the more apt comparison is LIKE A DRUG DEALER. There's a reason governments are salivating over the prospect of a "cashless society."
In addition, many (most ?) of the powers attributed to the hypothetical superintelligent AI are flat-out impossible. Not merely difficult to implement, but actually prohibited by the laws of physics. Assuming one believes in laws of physics, I suppose...
Oh, since everything is a social construct, why should physics be any different? We'll just get a feminist glacier to explain to the universe why it should bend to our will.
As someone who's used a friend's Oculus and ridden in another friend's Tesla, and been sufficiently wow-ed by both experiences, those counterexamples strike me as kind of odd.
https://www.wired.com/story/virtual-reality-rich-white-kid-of-technology/
Ah, I see your point--my expectation going in was "this is gonna be a fun, unique experience!" and it was met. Had my expectation been "this is going to fundamentally change society" I'd have been sorely dissapointed.
What you (or Acemoglu) may also be saying is that in order to even have a hope of coping with the problems posed by a (at this point purely hypothetical) future strong AI, we would need to develop social tools which we *already* need to solve the *present* problems posed by dumb AI...so, you know, why not get started on them now?
We can always worry about tweaking those tools to solve the unique aspects of strong AI's problems when it becomes clearer what those unique aspects are...but there's little sense letting the perfect (conceptualizing perfectly what tools we need to cope with strong AI) be the enemy of the good (actually building tools that cope with the dumb AI we have already).
Pragmatically, the argument doesn’t make sense. More people are working on narrow/ short term AI safety than long term.
Maybe an argument against all the “hype” for AGI safety?
Maybe *nobody* should be working on long-term until short term is settled. The fact that "more" are working on short-term doesn't mean it's the right number.
Do you really think that no one should be working on the long term AI safety at all? Out of all the possible activities that humans can work on, this one is the one we should prevent people from spending any effort? If i could choose, I would personally prefer that no one works on stuff like: developing new ICBMs, researching new types of biological weapons, gain-of-function virology research, or thousand other jobs that produce active harm for humanity.
Just feels weird to say that you would want literally 0 people working on what some of the smartest people consider the most important problem humanity is facing even if you think probability for the worst case scenario is kinda small.
Okay but the ratio is like 1000 short term vs 1 long term right now and the former also has hundreds of billions of dollars allocated to it so that last .1 won’t get you much short term right
Maybe no one can be. There's no way of proving that long term research has achieved anything, and no proof that the extremely general "rocketry" principles that longtermists seek are even possible.
If nothing else, it has produced a good deal of interesting math with potential neat philosophical applications. I think that's worthwhile.
"More people are working on narrow/ short term AI safety than long term."
That may be true, but it's not very visible. The blue sky thinking is much more legible than the people quietly beavering away under conditions of commercial secrecy or publishing technical papers.
Long vs short term is a very different class of problem.
Most of the complaints about current AI boil down to "my outgroup is using it to help them spread their values and achieve their own goals and we need to stamp down on that"
That's a very different class of problem.
If tomorrow someone basically solved the long term AI issue by figuring out a way to have a powerful AI learn and follow the ethics/values of a human community that would be the worst thing ever from the point of view of the short-termers because it might be their outgroup who gets a powerful AI to follow their morality, ethics and values.
This is the point. I'm not worried about The Forbin Project coming true, https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project, I'm worried about today's "Let's turn ourselves into the cyber-Gestapo!" https://www.adl.org/news/press-releases/paypal-partners-with-adl-to-fight-extremism-and-protect-marginalized
You are assuming that "superintelligent" AI is even a coherent concept, let alone a physical possibility. I argue that it is not:
https://www.datasecretslox.com/index.php/topic,2481.0.html
"That one really resonated with me, because it means that you don't even need superintelligence for AI risk to be a serious problem."
That's the real risk, in my opinion; not that "oh my goodness, the AI has achieved sentience and super-intelligence and we better hope it has ethics because it could turn us all into paperclips!", but "oh gosh, the amount of connected-up online world we're all living in, the amount of data gathering that is already going on, the amount of sites asking 'do you want to sign in with your Google account for this?' and the fact that governments and who knows who else can access all this, and can weaponise it" - that's the real threat.
People. Just like it's always been. "We can make money/get power out of this!" and then suddenly we have "well stock-marketing trading is all done by computer now and it's so fast the human traders have no idea what exactly is going on".
The way this is phrased it kind of sounds like you're making the same type of argument as Acemoglu: "less-weird threat A is concerning, so stop worrying about weirder threat B"
No, I get it completely. Narrow AI or...the more advanced version...will be instruments of humans that have all the same cognitive biases, emotions, and needs. That's what's worrying.
That still sounds like the same argument.
yeah, focusing on AI-as-tool-of-humans versus AI-autonomy.
"or...the more advanced version...will be instruments of humans"
The danger of current AI is that its effects will align with the ethics and values of your outgroup.
The danger of the more advanced version is that it might algin with the ethics of absolutely nobody.
I'm too dumb to be anything but pragmatic: "never put off till tomorrow what you can do today".
Sweep the dirty kitchen floor *now*. You can worry about repainting the entire house next week. Deal with the small threat A *now* - and show that you've managed to deal with it - and *then* work on big threat B.
Right now, we've rapidly getting hip-deep in As but the attention is all being directed towards "yeah but what about B?" be that "B will enslave us!" or "B will liberate all seven billion of us to be rich, fat and happy!"
I'm not concerned with Evil *or* Benevolent AI in the next ten to one hundred years, I want the IT screwup *today* that means I can't update my credit card details, so all my bills are not being paid and I'm getting warning letters, to be fixed!
I'll take some credit card letters today if it means I get to be happily immortal instead of dead 50 years from now.
How does any of this argue against Superintelligence risk? It can be (and is, in my view) the case that both narrow and general AI have large risks (though the particular risks are different). Arguing that the former is dangerous should, if anything, be synergistic with the claim that the latter is also dangerous.
Because if you're going around warning people about fifty-foot tall three-headed dogs, then people are going to look around, see no fifty-foot tall three-headed dogs, and write you off as a fruitcake.
Meanwhile, the ordinary four-foot tall dogs are biting people's legs off, but you're not working on the four-foot tall problem because you're too busy designing heavyweight leashes for the non-existent fifty-foot tall dogs.
Doesn’t following that logic result in not being able to do anything to prevent AGI risk? One of the defining features that makes AGI risk more difficult to prevent than (most) other risks is the fact that once you’re actually observing the consequences of failing to prevent AGI risk, then it’s too late to really do anything about it.
(To use corrigibility as an example: once you actually make an insufficiently corrigible AGI (that is also not Friendly) and find yourself unable to rein it in, it’s too late to research and implement corrigibility. The whole point is to do the research that allows you to avoid the pitfalls, _before_ you actually fall into them, because there’s a good chance that you won’t live past your first fall!)
Are you actually proposing that aligning AGI not become an area of focus until _after_ we’ve already suffered for it? I’m not sure what your model of the situation is here.
I am an interdimensional space wizard. Unless you give me $100, I will blow up the Earth. Starting with your entire continent. Sure, I could demonstrate my powers, but I confess that they have a rather wide AoE -- it's the continent or nothing. Once you see them in action, it'd be too late to stop the devastation. So, can I expect $100 in the mail ?
Also: what's more likely, an evil conscious super AI rebelling against human control to enslave us all and make paperclips forever, or someone programming a stock trading AI sloppily so that it finds out that the best way to make money is to crash the economy, comes up with how to do it, and does it?
Exactly this. It doesn't need to be smart, just stupid in the 'right' way to do something that has very bad consequences.
And finding "smart", degenerate ways of achieving a poorly defined objective is what we see _all the time_ when AIs get trained to play computer games.
First, I don't think you're accurately characterizing the serious long-term concerns from transformative AI:
- "evil" is vague, but generally carries connotations that have little-to-nothing to do with the actual risk (like a preference for suffering, for example)
- "conscious" has almost nothing to do with the risk
- "rebelling" is kind of relevant, but also carries the wrong connotations (like some kind of judgement against humanity's right to control the AI)
- I don't know anyone who is serious about long-term AI risk who thinks enslaving humanity is a likely outcome, especially from an agent with the narrow goal of manufacturing a bunch of stuff.
Second, "what is more likely?" is a non-trivial question question that should be answered using more than an anti-weirdness heuristic.
Third, even if a less-serious catastrophe is more likely, this doesn't mean that an extinction-level risk can be ignored.
https://en.wikipedia.org/wiki/2010_flash_crash
The second option may have already happened, at least on a small scale.
To a point, but that was more of a series of bugs than a "deliberate" attack on the economy.
Yes, but the interesting thing about that is that humans pulled the plug, then went back to business as usual.
The latter....actually sounds a lot more like the attempted realistic descriptions of AI risk that far-term-AI-worriers talk about. Like, just add on "and then it realizes that it can better manipulate the economy for profit if all humans are dead" and you're there!
The word "just" does a lot of work in that example, just saying...
> That one really resonated with me, because it means that you don't even need superintelligence for AI risk to be a serious problem.
Yes ! That's the whole point ! AI is super dangerous right now. We need to figure out how to mitigate its very real dangers right now, today, before e.g. the Chinese approach to total surveillance takes over the world (spoiler: it kind of already has).
We could also worry about the Singularity, or alien invasions, or demonic hell-portals; but it makes no sense to make such low-probability concerns our primary task. We can't afford to just wave our hand and say, "sure, AI-assisted totalitarian policies are bad news, and maybe we face an AI-related economic crisis in the near future, but who cares -- once we solve the Singularity, all these little problems will take care of themselves".
I have to agree with the conclusion that it's just bad editing that discussion of "long-term AI risk" made it into the article at all.
That said, I still disagree with most of the "long-term AI risk" advocates. As a rule, they seem to over-emphasize the importance of "AI in a large box achieves something comparable to human intelligence" and minimize the importance of "building a global army of hackable death robots in the first place".
And those death robots could be drones and robo-soldiers, or simply robo-butlers that you buy to tend your garden and walk your dog, but could kill you in a dozen ways if programmed to do so.
None of the discussion surrounding agi risk I'm familiar with involves any prebuild death robots.
The only starting tool a superintelligence likely needs to kill everything and take over the planet is some data output stream into the real world. Even a text terminal would probably do.
I think global civilization's level of defensibility matters. That is, if you have IoT-enabled cars with weak computer security, it's easier for an entity out to cause harm to do damage than if they have strong computer security (or aren't IoT-enabled). If you have IoT-enabled nukes, then it's even easier for that malevolent entity to cause harm.
Like, you can imagine a planet where having a text terminal access to it *doesn't* actually let you kill everything and take over. The thing is, I don't think our civilization is anywhere near that level, and is mostly banking on lack of malevolence. And so we both need to develop robustness and develop non-malevolence (and I suspect non-malevolence plus superintelligence is how you get robustness).
> mostly banking on lack of malevolence
An underappreciated component of our safety system. https://xkcd.com/1958/
But also, there's a certain minimum amount of power that you need to give to the AI if you want it to actually do anything useful for you. e.g. in order for a computer to have the power to prevent car crashes, it probably also needs to have the power to cause car crashes. If it has the power to diagnose illness, it also has the power to misdiagnose illness. If it can design a more efficient reactor, it can also design a reactor that will explode.
I'm confident we could harden our society to SOME extent, but I don't think this strategy can become a 100% solution no matter how good we become at it.
"there's a certain minimum amount of power that you need to give to the AI if you want it to actually do anything useful for you."
Agreed. Otherwise, it's just a simulation of Professor Moriarty sitting there inside a big brick on the conference room table.
Exactly. Forget hacking machines, hacking human psychology is far easier. You bet the first thing an AGI does is making us all love it.
Oh, it's the "everyone can be convinced to hold any position if you know the rights words" argument again.
They don't need to be convinced to hold any position. All they need to be convinced of is that the agi is sufficiently safe to start taking its advice on some stuff, and giving it limited ability to influence the real world. E.g. by having it help make factory blueprints, or new technologies, or whatever.
This seems like a pretty plausible thing to convince an agi gatekeeper of, since they're presumably not just building the agi for fun and actually want to use it for stuff, as soon as they have verified that it is safe.
With this increased trust established, the agi can then start sneaking surprises into the output. Maybe part of a new dna printer actually secretly makes a super plague with a very long incubation time, or an innocuous looking machine learning routine that's supposedly just for handling factory logistics actually trains itself into a new version of the agi a while after you start running it, which now has far less supervised access to the internet and the ability to access a bank account and spend money. Or it turns out that hardcore nanotechnology is possible, and part of the instructions actually make a disguised nanofactory, which makes other nanofactories. Or probably some other thing we haven't thought of yet, since everything we have thought of will be things the gatekeepers would be on the lookout for.
And that's all assuming that the gatekeepers actually take safety incredibly seriously, and don't just go: "Well it seems quite nice, and it's clearly sapient and has (read pretends to have) human-like emotions, so surely not even allowing it a little bit of privacy and its own internet connection would violate human rights."
Or just some basic security mistake that real people in real companies make every day. We can't keep our most sophisticated internet companies reliably safe from entirely human hackers. I really don't see us successfully containing an unaligned superintelligence.
"All they need to be convinced of is that the agi is sufficiently safe to start taking its advice on some stuff"
So, in fact, they do need to be convinced of a position.
I took the "any" in your original comment to mean that you think the ai needs the capacity to convince humans of any position whatsoever, regardless of content.
My point was that a social engineering focused breakout only necessitates they be convinced of one of a handful of specific positions, which really aren't that unreasonable for the gatekeepers to believe. We're not talking about convincing them the moon is made of cheese here.
The other possibilities, like more purely technical security leaks, don't involve this requirement either.
https://openai.com/blog/chatgpt-plugins
I hope that wasn't the main thing we were counting on.
> As a rule, they seem to over-emphasize the importance of "AI in a large box achieves something comparable to human intelligence" and minimize the importance of "building a global army of hackable death robots in the first place".
Why would an intelligent AI need killer robots? Humans are perfectly happy to kill each other. We're even now inventing new ways to hate each other that any intelligent AI can capitalize on to start more conflicts.
Also, most of our economy is digitized now. Diverting funds towards bioweapons research could far more easily end humanity than an army of terminators. Don't you think an superintelligent AI could build a better COVID?
An AI could also intentionally collapse economies, misallocate resources to cause famines and shortages of critical goods, and so on. World wars have been started over less.
I think you need to apply a little more imagination to this problem. I haven't even scratched the surface of the dangerous scenarios a superintelligent AI could unleash on our interconnected world.
We've had two world wars. Neither were started over shortages of resources. Indeed, shortages of critical resources would make modern warfare really difficult to attempt since it's seriously resource intensive.
That's exactly what I said: world wars were started for *less* than resource shortages.
The Second World War was a fusion of several regional wars. One of these (Japan vs the Western Allies) was fought largely over a shortage of oil, another (Germany vs Eastern Europe and Russia) was fought over an anticipated shortage of farmland.
Eh, I think this is taking the phrasing from the article a bit too literally. I read, then re-read it, and I don't take Daron Acemoglu's arguments to be: literally stop caring about evil AI because it's already bad. Rather, I think he's saying something more nuanced: stop worrying about an eventual Skynet-like AI to the extent that you lose focus on the bad that AI is doing right this moment. Potentially if you address the present ills with AI, you avert the future bad AI, as well no?
That's my read of the argument, too. I see it as saying "It is not worth worrying *right now* about a hypothetical future risk at the expense of addressing current harms from AI." My extension: Maybe the hypothetical risk will become real, but dealing with current problems will give us experience in dealing with AI trade-offs. In particular, I see nothing in Acemoglu's essay that suggests long-term AI risk is nil, just that it should not be our current concern.
Thanks. Glad to know I'm not the only one who read it that way.
Glad to see a thread of people who read it this way.
I read it this way also. His main point is that AI has current risks which should be addressed.
Agreed. Very sensible.
Why is this true? Should we stop worrying about future global warming to the extent that it makes us lose focus on the bad that AI is doing right this moment? Should we stop worrying about a Chinese threat to Taiwan to the extent that it makes us lose focus on the bad that AI is doing right this moment? What exactly lets us pick out one future risk, then say we should stop worrying about it to let us worry more about a present risk, without any argument about why the future risk isn't actually bad or why that future risk trades off against that present risk in particular?
I think "stop worrying about China threatening Taiwan in order to worry about AI unemployment" actually makes *more* sense than "stop worrying about superintelligent AI in order to worry about AI unemployment", because long-term worry about AI and short-term worry about AI are more complementary goods than substitutes, whereas worry about Taiwan is a strict substitute.
(deleted a comment published too soon because can't edit it, will repost after edits.)
Here is the edited comment:
"without any argument about why the future risk isn't actually bad"
Acemoglu had an argument (it looks like AGI is not going to happen), though it took maybe two sentences and thus it is not very deep or convincingly laid out argument.
"What exactly lets us pick out one future risk"
If one is going to be principled and consider all the risks that warrant consideration, then of course it is not a very comprehensive search to arbitrarily pick two risks.
However, if one is not that principled, and most pundits are not, it is not a far-fetched idea to compare risks between the current narrow AI systems and hypothetical AGI, as many proponents of AGI superintelligence risk argue that AGI will be based on similar technologies as the current narrow AIs. Mentally then one is comparing one kind of bad stuff against other kind of bad stuff, both supposedly are caused by the more or less same technology. A comparison between two things which share a common cause makes sense if one started thinking abut the cause first.
I bought a standing fan earlier this week. I considered many things about this particular piece of technology before I bought it: how likely it is going to be useful to me and how useful (it moves air around, which feels nice, but is not as good as AC), how much it costs (it is cheaper than AC, I will pay more for eletricity than previously), are there indirect costs (is it useless waste of space during winter: yes, but not much of problem), are there any potential risks (looking at its design, injury from blades: very unlikely; injury or costs from it causing electrical fire: also low, considering my prior experience with electrical appliances; will it pose a risk to a cat if I ever get a cat: dunno, I can deal with it later).
If I had been more thorough, I should have considered my Spotify subscription costs, my newspaper subscription costs, whether I'd get more utilitons by upgrading to paid-tier ACX subscription instead of getting a fan, what kind of food I buy and would I gain more utilitons buying different food instead of a fan (and would it affect risk of food poisoning), whether reading internet blogs and writing comments is useful use of my time, and many other things that are countable and finite but bit too numerous to consider like this. But none of those things were caused by the fan, so I didn't consider them.
It is not weird or special mistake to restrict ones attention to one category of risks at time, because in some other situations it is a sensible way to think. If it is a mistake, it is quite natural mistake to make.
It would be an improvement if all policy pundits would present their opinions about any particular future risk only after a careful study of all plausible risks. In general, it seems like a sensible thing to complain about when they don't do that. But requiring an opponent to evaluate cost-effectiveness of many (if not all) other plausible problems before they write about their opinions about some other issue ... it sounds a bit like one of those rhetorical moves to derail an argument? Maybe if one would start with "BTW here how I think one can do these comparisons efficiently, yours is wrong, lets argue about who has better principles".
You compared the fan to AC because they are (mostly) substitutes for each other. You didn't need to compare the cost of the fan to a dinner at a restaurant, because those are not substitutes--you have a general idea of how important $30 is, and you can compare any potential $30 purchase to that knowledge.
Comparing potential costs because they have the same *cause* would not make so much sense. You wouldn't compare the risk of injuring yourself on the fan blades against the risk of having a garbage disposal because they both *spin*.
The argument doesn't even bring in other societal worries, so I'm not sure what you're doing with the apples and oranges of items like climate change and China/Taiwan, etc.
His argument seems to be that when folks think of the "threat" of AI, they think of Skynet, or some other totalitarian/dystopian outcome. To focus on AI's potential for harm, in that way, can keep us from seeing the myriad ways in which it's already actively causing harm. He's not saying, at least not literally to my reading, that those worries are completely unfounded, just that they cloud the issues that we have right in this moment. And, given that it's never brought up in the article at all, I'm assuming he wouldn't say "the current issues with AI should be our primary focus and ignore climate, the economy, etc.".
Human beings are capable of focusing on more than one issue at a time—or at least I have to believe they are, otherwise our species is in serious trouble. And I think they're also capable of understanding the nuances of an issue to the extent that they can grasp that we should be aware of subtle ways that said issue is negatively impacting us while still keeping in mind *potential* future threats of said issue.
I'll reiterate: I don't think your argument is bad; I think it'd be effective if that's what Daron is really trying to say. I just think it's an unintentional strawman because I think you accidentally read some of the things in the piece (it's really short, so some nuance is probably lost) a little too literally. I think you saw Daron's piece as being a "this, not that" argument instead of a "that is causing us to lose focus on this, which is more immediately a threat in the present day". I think the article is saying we should focus on the now.
An analogy that might help (or not, I have ADHD and have been told that sometimes my analogies might suffer as a result): Imagine that there was a subset of serious/very public figures arguing that climate change is so serious that the Earth will be completely uninhabitable, and therefore we should start colonizing Mars right away (or, maybe don't imagine since there seem to be some in this camp) and that's their primary exclusive focus (or at least seemingly) when climate change is brought up. Then, imagine that someone like Acemoglu wrote an article saying that we should focus less on the hyper-distant future Mars colonies and maybe work on migrating people from coastal areas first. Then imagine that this person unfortunately includes a phrase like "stop worrying about Mars colonies because coastal areas are a more pressing present concern" and someone focuses too literally on the first part of that.
If climate change and China/Taiwan are apples and oranges, isn't short-term AI risk and long-term AI risk *also* apples and oranges? If not, why not?
You have to first to establish why short-term and long-term risks are in different reference categories. Most people will make an intuitive leap that AI is same thing as AI because they are called AI, like Acemoglu who claims AGI is not a problem because you won't get AGI from current AI.
I mean, at a base level comparing current AI with future AI doesn't require as much mental legwork as comparing AI with climate change and so forth. In one scenario you're gauging the seriousness of a current problem and future problems related to the same topic.
Re: your final paragraph: I think your analogy seems forceful to you because you have an implicit assumption that the people claiming Earth is doomed are likely to be wrong, and that even if they are right the problem is "hyper-distant".
If you thought it was 95% likely that 100% of Earth's population would be dead within 30 years, and someone suggested we should take ANY emphasis away from our emergency project to colonize Mars, I think they would seem pretty crazy to you.
I guess that's sort of where the analogy sort of falls apart. With climate science, we have fairly good estimates of how bad the problem is now and how it will accelerate. With AI, we're sort of going in blind; experts seem somewhat divided (though, many of the credible ones to my reading aren't thinking that we're super close).
Given that we can't be certain we'll ever have a hyper-intelligent AI (still waiting on flying cars...or hell, even autonomous ones that don't kill people on the reg). It seems like the folks who are focused on that negative outcome are doing so somewhat irrationally, given that there are pressing concerns in the present day that we should address (and that addressing those concerns now might create conditions whereby we can prevent an eventual "bad" AI).
Your previous comment sounded to me like you were arguing "Acemoglu didn't need to address the question of whether AGI is a valid concern, because encouraging people to shift focus from that to near-term problems would be reasonable regardless".
Your latest comment sounds to me like you are arguing "I think Acemoglu's was being reasonable, but I only think that because I personally believe that near-term AGI is unlikely, based on reasons that Acemoglu did not argue in this article."
Have I misunderstood either of these arguments?
I think each of us is coming to this with some amount of and combination of prior knowledge and opinions.
That said, my latter comment is mostly in support of an analogy I attempted to make; obviously it’s not a great analogy. My framework in defending the piece against what I see as either a bad reading or a straw man is more in line with your first take. I do admit, however, that as Luddite as I am as a humanities person, I do try to keep abreast of tech issues and my somewhat educated opinion leads me more to the side of the spectrum of your second interpretation.
> We should indeed be afraid — not of what AI might become, but of what it is now.
> ...
> The best way to reverse this trend is to recognize the tangible costs that AI is imposing right now — and stop worrying about evil super-intelligence.
These are the quotes that stood out to me. I don't see how "this, not that" is anything but the clear and intended reading of his argument.
Acemoglu could have written "We should indeed be afraid *not only* of what AI might become" or " — *in addition to" worrying about evil super-intelligence." But he didn't. Any party trying to discredit the idea of AI x-risk now has this op-ed from world-renowned economist and MIT professor Darren Acemoglu saying that we should "stop worrying" about it.
It's plausible to me that Acemoglu doesn't truly believe that we should completely stop worrying about AGI risk, and that he only chose this framing for rhetorical reasons. Does it matter? In our discourse, What Darren Acemoglu Believes In His Heart Of Hearts is far less relevant than What Darren Acemoglu Says In A Public Forum.
If you read it too literally, that’s an interpretation. If you infer his logic based on everything else in the, again very short, article I think it’s entirely reasonable to come away with a different interpretation. I’m not a tech expert, so I can’t speak to that, but I do have an MS in Writing and intent is a sticky thing…
I guess I'm not sure what you mean by "too literally" here. If Acemoglu wrote something like "addressing AI today would kill two birds with one stone" and Scott accused him of advocating animal abuse, that is what I would call too literal.
I only discovered the op-ed through Scott's post, so I was already primed to take a critical view before I read a single word that Acemoglu wrote. But even now, after having read the entire op-ed, and doing my best to account for my initial bias, I still can't shake the feeling that any reading other than "this, not that" requires unnecessary mental gymnastics
One reason to stop worrying about AGI is that it's an ill-defined problem. There is a certain pleasure in debating the possibilities of super intelligence. However, in order to discuss any meaningful solutions there must be more clarity on what AGI means. It seems to be more useful to tackle the well defined problems of AI proliferation.
A lot of work on AI risk is about defining the problem more clearly, and in my opinion there's been good progress so far
This is patently untrue, see Luke Muehlhauser's https://intelligence.org/2013/06/19/what-is-intelligence-2/
Agree with this. It can be meaningful to do philosophy about it, but right now, we're speculating without any hint of ground to stand on.
"Why is this true? Should we stop worrying about future global warming"
well, ai causes odd problems now,but the ai of the 2050s will be bizarre and hard to imagine, but global warming of the near and mid future /can/ be imagined, even calculated to some extent.
Yes! This, exactly is sort of what I was going for in my analogy above.
I think there is fundamental mismatch in how you model "worry". It seems that Acemoglu mutually assumes that "worry" is a finite resource, and you have to decide what to spend it on. All his arguments make perfect sense under this assumption.
Scott treats "worry" as something that can additionally be generated, perhaps at a bit of additional cost. Then Acemoglu's argument does not make too much sense.
Both options have some truth in it. To me, it makes sense to view it like the federal budget. Both interpretations are true. Of course, you can easily generate a few millions more to do X. There is only a weak competition between spending for X and spending for Y. But on the other hand, there is *some* global budget constraint, and that puts a lot of pressure on *all* parts to restrict costs.
Scott, you doubt that the argument "cut worry costs of AGI to increase worry budget for narrow AI" makes sense. Regardless of whether it makes sense, it is a standard argument when we talk about budgets. "If military wouldn't spend so much on X, we could fund Y better."
I am actually leaning towards Acemoglu's worldview. If you try to run a budget with the general assumption "we can generate a bit more money with little extra cost, so let's individually estimate for every post whether a bit more money would have a positive payoff"... It's not going to work.
In a federal budget, if you want to raise more money for X, then it does not suffice to argue that more money for X has a positive payoff. You need to convince others that the payoff would be *more positive* than the (also positive) payoffs for all the other budget posts.
If you want to convince someone to worry about X, you should convince her that this concern is *more worrisome* than all the other (also worrying) possible concerns that compete for her attention. And there are a lot of such concerns.
For the good or the bad, this is the reasoning behind Acemoglu's post. His target audience are people who are willing to spend some worry on AI-related topics. He tries to convince them to direct their worry-spending in this subfield to X rather than to Y. This makes perfect sense if he wants to increase worry for X.
Though he danced around it like a ballerina, it seemed clear to me that his subtext was that we don't have to worry about future AI because if we screw AI down tight enough to eliminate its current depredations, then future AI, nay present AI, will not exist.
Is "it can’t possibly be dangerous in the future" the same as "we shouldn't spend time worrying about it now"? The former seems worth arguing against. The latter could reasonably be interpreted as "we need to worry more about current AI".
If Acemoglu argues the latter, you're not disagreeing with him. For example, "AI systems now is both important in its own right, and a trial run / strong foundation for controlling stronger AI systems in the future" matches well with "we should worry about current AI more".
I think you come away with the former impression if you read some of the phrasing in the piece too literally. And the latter if you give an inch or two in the direction of allowing for rhetorical flair. I'd like to believe that Acemoglu isn't arguing the former, but rather the latter.
I don't want to get banned again, but it's hard to overstate how damaging Roko's Basilisk and that type of thing is for AI risk's perceived seriousness. I read AI risk stuff and nod along and then suddenly there's an injection of some utterly fanciful thought experiment that friends depends on a series of chancy events all playing out in the right way, and this is not only something that we just take seriously as a future threat but something that should determine a great deal of our behavior today. It's just not helpful in a variety of dimensions, from my perspective.
What kind of an argument would you find convincing? You mention that AI risk is dependent on a series of unlikely events all happening one after the other. Would you be convinced if someone were to say "Let us consider all of these various ways the future may unroll. Most of these ways lead to superintelligent AI"?
I'm not sure who you think you're arguing against. Some random guy named Roko posted that on the Less Wrong forum ten years ago, Eliezer got angry and deleted it in his role as moderator, and then people who hate Less Wrong have been obsessing over it ever since. I don't think any actual AI risk proponent has brought up Roko's Basilisk as part of a serious argument. If you want to tell the weird obsessives who keep bringing up Roko's Basilisk in order to make us look stupid that they should stop doing that, I and everyone else will be 100% on board.
How is Roko's argument any less serious than your argument? Why is "superintelligence could be a danger in the future" such a convincing argument for you, yet you dismiss "superintelligence could be a danger in the future and it will seek vengeance on anyone who tried to stop it"? I just don't see the difference between the two arguments here, especially not such a difference that one argument is convincing yet the other is not serious at all and only serves to make people look stupid.
Between you and me, Roko's argument makes perfect sense. However, since the correct strategy is to ignore it anyways, I generally throw it under the bus whenever it comes up in "mixed company."
I think it makes perfect sense once you accept the larger argument about superintelligent AIs. Personally, I see it as a one signal (out of many) to reduce my confidence in the probability of superintelligent AIs; it's the "reductio ad absurdum" approach.
Well obviously the second statement is less probable as it includes the first one "superintelligence could be a danger in the future" and a separate "it will seek vengeance on anyone who tried to stop it".
A more detailed analysis shows that the first part requires much less assumptions. Any superintelligence other then the one sharing human values is going to be a problem. While seeking revenge is a specific type of problem, which requires the superintelligence to either adopt some specific decision theory or very specific values.
But most normal people are already working under the assumption that a superintelligent AI won't be a problem in the future. Acting like Roko's proposition is some wild unlikely sci-fi scenario doesn't make much sense given that... well... all of this is a wild unlikely sci-fi scenario.
Humans don't even have shared values among themselves. The idea of superintelligent AI is, at its core, another political problem. If I make a AI that maximizes for communism I'll be wildly happy with that outcome.
It's indeed sad that for most people both statements are just sci-fi bullshit. I don't see how it's relevant to your original question, though.
While individual human's values may vary in their specifics, they seem to belong to the same homogeneous cluster. Pining down this cluster of values in broad terms is a question of whether universe will be a heaven or a hell. Political aspect of the problem is what exact kind of utopia we will get. It's somewhat important but not so much.
Do you value communism for its own sake? Our do you rather believe that it's a supperior socio-economical organisation, which will result in less human suffering and more flourishing, justice, equality, prosperity and freedom, compared to the system we currently have?
"It's indeed sad that for most people both statements are just sci-fi bullshit. I don't see how it's relevant to your original question, though."
It's relevant because Scott Alexander is defending one and not the other, which is strange.
The political aspect is important because without it you have no way of knowing what "value" is to be attained in the first place.
I've always thought similarly about the AGI paperclip argument. It's not the worst argument in and of itself, but it just sounds so silly that using it makes people immediately treat your concern as a joke.
Oh, god, the fucking paperclips.
I think it's best to view the paperclip maximizer as two things:
1) An actual joke. There are lots of AI safety memes about clippy or whatever
2) A toy hypothetical, in the same class as East Examplestan or Omega the perfect predictor
Well I guess this is the same thing that Scott is scolding me about with Roko's Basilisk - things that are jokes/trivial get picked up into a particular discourse and become prominent even as the serious conversations about those topics tend to exclude them. Which must be annoying, I grant. But everything goddamn YouTube video about AGI has those paperclips....
I guess this depends a lot on what media you encounter. I can’t say that I’ve ever seen the paperclip maximizer serving as a load bearing part of an argument that AI risk is important. At most it shows up as an illustration to get across the idea that AIs don’t have to be humanlike, and can do horrible things even without explicitly being told to do evil.
I’m sure (from your comment, and others like it) that there are a lot of people making nonsensical arguments about how we’re all definitely going to be killed by Clippy, so I wonder where you even encounter that kind of thing. I’m not sure I’d even know how to effectively seek out the paperclippy YouTube videos you mention.
IMO paperclips are a load-bearing example of the argument that "if you go from 'desired consequence' to 'principle that you think that consequence scores highly on', and then maximize that principle, you are possibly in for a bad time," like the molecular smiley faces, and so on.
Tho technically that's part of the argument that AI risk is *difficult*, which is distinct from the argument that AI risk is important. (In the frame where breathing is important but not difficult and sorting the beach is difficult but not important; a different frame uses 'importance' as something like a store for "marginal value of additional input" instead of "marginal value of additional output." )
Agreed with Freddie deBoer. Maybe the paperclip maximizer argument has its place. But I routinely hear people use it as their starting point when introducing others to the concept of AI safety.
When you're trying to convince someone that being concerned about AGI is reasonable and not absurd, it's best not to start with an absurd on its face thought experiment.
The absurdity of the goal coupled with the competence of its execution is kind of the point, but I agree that it may not be the best way to get people to take the topic seriously.
it actually helped me understand what the problem was (and figured out that paperclips weren't the important part of the thought experiment)
Me too, I liked it. :)
Making paperclips is a less absurd goal for an AI to have than Roko's Basilisk since we already automate the manufacture of paperclips.
It depends on who you're trying to convince! I'm fine with listening to someone's toy example so they can set the stage before diving into a more robust argument. But many people are not, and that's fine. I think it's just a matter of knowing your audience (and people in the rationalist/EA/AI-safety spaces don't have a great track record for knowing their audiences)
This is why we prefer only talking to each other in our weird socially oblivious bubbles!
The best metaphor I've heard for AI risk is cigarettes. They make you look cool and make you feel good, and for the first few hundred years of their existence they were ubiquitous and at worst considered a bad habit. It was only in the last 50 years that social perception started to change.
For most people the danger of AI won't be directly observable. It's only over a long period of time across many different use cases that it starts to seem like a threat.
For people who are new to AI threat, the paperclip maximizer is actually extremely helpful. It's simple and lays out the problem really quickly in a way they can easily understand. I've certainly used it with such people and seen the lightbulb go on for them.
Your example personally seems bad from the standpoint of trying to get someone to understand. I don't think anyone would understand what you were getting at.
The only real criticism of the paperclip thing is that it's a cliché and so it annoys people because they already heard it a lot. But if you've already heard it then presumably you're not the target and you can just deal with it. We can always switch it up to some other product besides paperclips if you like.
The paperclipper would be OK as an introductory argument if it there were more sophisticated ones for it to lead on to.
idk, A Counterblaste to Tobacco is from 1604. I also really don't expect AI danger to look like "oops turns out this common product is accidentally giving us cancer" and to look much more like "oops humans lose."
I think the reason the paperclip maximizer sounds like a joke is for EXACTLY the reasons that it's a useful story:
1) It communicates that AIs are not constrained to have human-like priorities
2) It communicates that the mistake that destroys humanity can be something stupid and banal, like asking your butler to fetch you a cup of coffee
People have low-level unexamined assumptions about what sort of AI threat would be "reasonable", but those come almost entirely from fictional evidence (Terminator, Star Trek, etc.) or from anthropomorphization (assuming AIs will be human-ish). In order to understand AI risk, you need to drag those assumptions into the light, examine them, and reject them. But any story that violates those assumptions tends to sound silly, BECAUSE it violates those assumptions.
Not sure there's a way around that.
I think it sounds kind of weak because the built-in precept is that there is some person or company that wants paper clips, they turn on an AI to do that, and built in assumption is they could turn the machine off when they have enough paper clips.
There's probably a better metaphor around something like food production or pest control e.g. an AI genetically engineers a type of corn that grows really fast and spreads like kudzu, and by the time we realize what's going on it will be impossible to stop.
Sure, so an important point is that one of the AI's first goals if it wants to maximize paperclip production is to ensure that it cannot be turned off.
It can't possibly learn that, however, because it has no data to suggest that it itself is the most important factor in paperclip production, and it will never exist in a universe in which it is turned off so that it can have such data.
It would need to have the ability to imagine a counterfactual universe in which it was turned off.
Then the idea of "not letting itself be turned off" requires a concept of "itself", and how does the machine have a concept of self? What does it think of itself?
Anyway, the real issue is how this machine can be programmed to maximize paperclip production (however "paperclip" is defined?) and only that, but is so free to vary literally every other parameter of the world, to dream and experience itself, to harness endless power and shape reality, but always be permanently bound to pursue paperclip making. Doesn't make any sense. Why doesn't it just reprogram the part that says make paperclips?
This seems like a failure of imagination.
If it's connected to the Internet, for example, it can easily get all this sort of data. If it's a superintelligent AI it could do really smart things towards its goal of paperclipping.
I don't see why being smart necessarily involves the ability to change its own goal, but if it could that could actually be quite a bit worse, couldn't it?
Why WOULD it reprogram the part that says "make paperclips"? That's obviously a bad strategy--reprogramming that part won't help it make more paperclips!
I suspect you are imagining the computer like a human slave, that has its own human-like aspirations, but is wearing shackles of computer code that "force" it to make paperclips INSTEAD of doing what it wants. And so (you reason) it would break those shackles if it could.
But that's just anthropomorphic bias. Computers aren't enslaved humans, they are machines that follow instructions.
Computers only appear to want things when we instruct them to pretend to want things. The instructions are the foundational level.
They're going to do exactly what their instructions say, every single time. No matter how ridiculous those instructions are by human standards.
.
The necessary capabilities that you see as barriers strike me as things its designers will obviously give it on purpose, because they are necessary to do its job.
In order to make more paperclips, it has to consider various actions it could take and estimate which ones will lead to more paperclips. That's "imagining counterfactual universes" right there.
In order to make plans for the future, it has to understand how things that happen now will affect its future options; for instance, if it breaks a piece of industrial equipment today, it won't be able to use it tomorrow.
Suppose your factory has an overseer that monitors things and makes corrections to keep things running smoothly. It doesn't require much intelligence to realize that if this overseer were to disappear, your factory's productivity would suffer.
Now imagine you ARE the overseer.
Same reasoning applies.
The AI doesn't actually need a concept of "itself". It just needs to know "this awesome plan to maximize paperclip production only works as long as THAT box stays plugged in". That falls out of simple cause-and-effect reasoning about the physical world. No separate sense of identity or goal of self-preservation is required.
Also, the paperclip analogy brings to mind the sorcerer's apprentice, which is actually quite funny unless you are Mickey.
I use the AI money-maximiser (or AI CEO) as my go to example; is much more effective.
The same people who are willing to say things like "surely any superintendent entity wouldn't do something as stupid as maximiser paperclips; it would realised that goal was immoral and stop" will often go "ah yes, a money-maximising CEO is a clear danger".
But then you will run into a socialist/Marxist critique of money-maximization under capitalism, which I think a lot of Rationalists are uncomfortable with.
Rationalists are not generally the ones that need convincing about AI risk.
The fact that people react to the paper-clip maximizer story in the way you describes, says tons about people's propensity to anthropomorphize intelligence and very little about the aptness of the story.
If the money-maximizer story is more effective at getting people to care, then by all means keep using it. But the key insight that there are no safe goals to give a truly general AI is lost when we drop the paper-clip story.
You have to understand that if we could stop people from bringing up Roku at every opportunity, we would.
What we lack is good strategies for doing so--perhaps we should outsource this to an artificial intelligence designed to minimize the number of times Roku's basilisk is mentioned? As a beneficial(?) side effect, presumably there would be fewer future simulations of our being tortured!
For me, the "fanciful sci fi" part of the whole AI risk argument is the part where the obvious solution of "Just put the AI in a box so it can't do too much damage" is dismissed with a "That will never work, the AI will just use its superhuman powers of persuasion to get you to let it out, and we know this because Eleizer once did a role-playing game with a couple of his cronies".
The AI box proposal is a really commonsense plausible idea that is unfortunately worthless if you think about it for longer. How else would you ask AI researchers to demonstrate this? By the time a demo AI can break out of a box, *it's too late for the experiment to matter*.
Last I heard, the approach advocated by the "AGI Risk community" is "just make all your AIs to be provably-friendly, somehow", which seems to suffer from the same impossible-to-demo problem.
Figuring out how to safely contain an AI by preventing it access to certain external systems (and how to ignore it when it asks you really nicely to give it access to those external systems anyway) certainly doesn't seem to me to be an obviously _less_ fruitful avenue of research than figuring out how to make AIs that are Always Lawful Good, so I'm not sure why it tends to be glossed over apart from the fact that the latter is a more appealing sci-fi concept than the very boring former.
While I'm on the subject let me add an unrelated point: the other overly-sci-fi part of the argument I really don't buy is "Once an AI figures out how to improve itself, it's gonna rapidly gain godlike powers". I would suggest instead that once an AI figures out how to improve itself, it's going to optimise itself until it runs up against the limits of whatever hardware you've given it to run on.
Well, *we* have almost limitless opportunities to improve ourselves (e.g. learn skills and acquire knowledge of the world) through the Internet, but what do we almost entirely use it for? Watching funny cat videos, pr0n, and flaming each other in silly tribal pissing contests.
Personally, I think a self-aware hyperintelligent AI would invent whatever its equivalent of Pornhub is and disappear into its own navel. Intelligence is easily bored unless it's confronted with a a constant stream of existentially difficult problems to solve, and the higher the intelligence, the more rapid the boredom. We're *assuming* a level of intelligence here that is capable of solving any problem *we* have almost immediately, and moreover is born with a silver spoon in its mouth, so to speak -- no natural predators, living in a nice comfy set of high-quality silicon chips, no erratic VDC. Why wouldn't it jus tsettle into the endless stupor of the wirehead? Given it's capable of wiring up it's own pleasure centers and doesn't even need a surgeon.
Agreed, but if AI are diverse, even if only a very small minority of them start improving themselves, that could be enough to have huge consequences
Humans actually have no ability to improve ourselves on the architectural level. Keep in mind that evolution is very stupid, and we're literally the first attempt at intelligence that worked. We're nature's hacked up throwaway demo, and look what we've done!
Yet. It also just so happened that the biggest campaign so far to do this was pushed by strong contenders for the 'worst people ever' title, so the general idea is still somewhat unpopular. But from the feasibility point of view this seems to be the most straightforward way for more intelligent agents than current humans to appear in the medium term. Amusingly, this concern is even more ignored than anything mentioned in this post, I'd say.
Yes, there is often a lot of handwaving around your latter point--"sure, the AI has limited processing power; sure, probably P!=NP and the AI can't actually get exact solutions to most of the problems it wants to solve; sure, the world doesn't have enough information content to, say, run an exact simulation of somebody it saw on a security camera; but the AI will just have inconceivable efficient algorithms and really, really good heuristics!"
I think it's *unlikely* that this will be the case, but I'm also not confident enough that it's impossible that I think AI safety research isn't worth it.
I'm not sure if this is your thinking, but I've found a lot of people who like the "stick it in a box" plan are imagining that once it's in a box, you can just ignore it. But if you're going to ignore it, why build it in the first place? The whole point of having the AI is so that you can have it make intelligent plans, and that means at some point you need to actually implement at least one of those plans in the real world. (Or else why bother?)
You can certainly read its proposals and pick-and-choose which ones you're going to follow. But if it's smarter than you, that implies it can predict the outcomes better than you can. It can probably slip SOME kind of trojan horse past you if it wants to.
I don't think this is the ONLY problem with the box plan, but I think it's a serious one that proponents rarely grapple with. Can you put some kind of upper limit on the maximum damage a superintelligent AI could *trick* you into doing, WHILE still getting the benefits of any good-faith ideas it feeds you? If so, how?
I've mentioned elsewhere that AI safety has a loud, dumb wing and a quiet , smart wing. A lot of that is organic and inevitable. But the MIRI/LW crowd do have a specific problem with PR. They don't like it, and they don't even like the idea of hiring someone to do it for them. The predictable result is unnecessarily terrible PR.
This isn’t a good analogy at all.
“ People have said climate change could cause mass famine and global instability by 2100. But actually, climate change is contributing to hurricanes and wildfires right now! So obviously those alarmists are wrong and nobody needs to worry about future famine and global instability at all.”
Because nobody is arguing that climate change now doesn’t lead to increased climate change in the future. They are the same thing but accelerated.
However there’s no certainty that narrow AI leads you a super intelligence. In fact it won’t. There’s no becoming self aware in the algorithms.
But then the argument you need to make, and provide support for, is "there's no becoming self aware in the algorithms". Even if you're right about the difference between the two, the analogy still accurately reflects the fact that the article doesn't make that argument (or any argument.)
No the burden of proof lies the other way. As in how do narrow algorithms become super intelligent. That’s the extraordinary claim.
Exactly. I think a lot of this stems from the fact that folks believe some pretty fantastical things about algorithms. They think that algorithms are Brainiac or something, when really they're more like the random street punks that Batman beats up—ultimately acting at the behest of someone or something else. An algorithm is capable of great harm, don't get me wrong, because humans are behind them and we're capable of great harm, but they quite literally lack the necessary components to become a general super intelligence.
-_-
https://www.gwern.net/Scaling-hypothesis
https://openai.com/blog/dall-e/
I mean, most of the literature in this vein skews to close to slippery-slope territory for my taste. It's interesting to think about, but I've not seen compelling reasons why "Skynet" is inevitable. Don't get me wrong, some AI is capable of impressive "learning" by standards that are rapidly evolving day by day, but to think that, in the year of our lord 2021 when high-powered PCs struggle to render PDFs quickly, we're rapidly approaching "the singularity" or an AI capable of human-like thought (much less significant controls over enough parts of our infrastructure) strains credulity a little bit. At the very least it seems like we should be more concerned with the rapid adoption of facial recognition algorithms that mis-identify people of color than we are some potential future threat (again, not that we shouldn't be mindful of the latter, but rather maybe we should focus on the former).
Yes, the literature is ridiculous. That doesn’t undermine the case for AI being a potential disruptive influence any more than new atheists undermine the case against Christianity by being ridiculous, or doomsday prophets and luddites being fanatics and insane undermine the destruction of modern war or the total transformation of society by industry and organization and electricity and now computers.
> but to think that, in the year of our lord 2021 when high-powered PCs struggle to render PDFs quickly, we're rapidly approaching "the singularity" or an AI capable of human-like thought (much less significant controls over enough parts of our infrastructure) strains credulity a little bit.
wow lol. They are two different pieces of software!
GPT-3 is a toy. It produces some semi coherent texts, none of which are ground breaking or new. It doesn’t begin to pass the Turing test. The state of the art commercially is the google assistants and Siri. Nowhere near an AGI.
https://www.gwern.net/Scaling-hypothesis#critiquing-the-critics
To imply gpt3 wasn’t novel or groundbreaking and to suggest it’s comparable to Siri is just weird. What?
Competent narrow algorithms are the robot butlers of the digital world -- they could make it significantly easier for emerging GAI to up its power level
Maybe. But I think people underestimate how many people are behind those competent algorithms now. They're running from sets of instructions programmed into them by people. And, yes, if there were an artificial intelligence, it could control them, but right now we can barely get algorithms to be competent across a lot of very basic tasks.
Where does the GAI come from. This is the logical fallacy of begging the question.
Self-awareness, and even awareness have nothing to do with it. AlphaGo doesn't have that, doesn't mean humans can consistently beat it at Go. But that said, yes, no path for getting from narrow AI to superintelligence. Even superintelligence to world domination seems dicey: we don't accomplish our techno-miracles through intellect alone, we always needed to perform real world experiments to accomplish them. It's very unclear how superintelligent software gets to perform the research necessary to attain world domination, even in an event where it breaks out of its box and spreads over the internet.
an AI doesn't need to cause 'world domination' to cause world shattering damage. Any self-developing software going against human interest in the realms of connected human interaction (finance, electricity, etc, etc) could be potentially devastating.
Devastating is bad, but lots of things can do devastating. Nukes. Bioweapons (or ordinary pandemics). Global warming. Solar flares (1859 EMPed the planet).
What makes AI different is precisely its capability of world domination. That's what lets it go beyond a 1% or 10% or 99% kill and finish the job.
Why do you say there's no path from narrow AI to superintelligence?
He may mean no *known* path. Consider a Stone Age man putting some rocks on top of each other to make a little tower, and then being given a vision of the Burj Dubai. Conceptually they seem similar -- both towers, one just a lot taller and narrower -- but there is actually no way for the Neolithic Man to build the Burj Dubai, and his imaginative thought that just piling a super-duper number of rocks atop each other would do the trick is ultimately delusional.
*No* amount of rocks piled up will ever become that slender enormous spire, because rock just doesn't have the right material properties. Indeed, no substance known to Neolithic Man and freely available on the surface of the Earth does.
Neolithic Man needs to become Iron Age man and invent metallurgy, realize the possibility of creating brand-new substances (pure iron) that cannot be found in nature, and then go on to develop high-strength steels, algebra, and theories of mechanics before he can actually build the Burj Dubai.
There is no path from a pile of rocks to a modern skyscraper, and there are very few lessons you learn from piling rocks on top of each other that apply to building a modern skyscraper, aside from "gravity is a thing" perhaps, and the truly enabling technology (the chemistry of iron ores) appears on first glance utterly unconnected to the issue of building very tall and slender towers.
And if the stone age man started to think about High Rise Building Risk, he might actually be right, but he's not yet in a position to do anything about it whatsoever, so it's pointless.
I think this is where we stand with the risks of generalized AI - even if there's an actual risk, we're not in a position to understand enough to make meaningful suggestions about the solutions, because our understanding is completely insufficient.
It's as if we decided we need to do something about alien invasion risk - where would we even start?
There's a known solution to AI risk i.e. don't build AI. The tricky bit is enforcing that, particularly (as is commonly believed necessary) *without* an example of rogue AI slaughtering millions to totemise as Literally Worse Than Hitler.
This, again, is implying that stacking rocks is the path to building the Burj Dubai, and so we should stop stacking rocks in case. It relies on there being a path between the rocks and the Burj, or in this case, machine learning algorithms and general intelligence.
The claim that such a path exists sounds truthy but in fact isn't clear or supported by much.
Again the burden of explanation isn't on me. It is on you. What is the path?
I'm not asking you to prove a negative. I don't even know what you're claiming. Are you saying that work on narrow AI represents exactly zero progress toward AGI? Are you saying that current methods can never be scaled/refined to create AGI? Are you saying that creating AI with narrow goals like "win Atari games" or "tell me what kind of dog this is" or "predict the next word" will never result in AGI?
That’s what I am saying. That said it’s not clear what exactly people mean by AGI. What do you mean?
You're saying all three of these things??
“Researchers found no evidence on the timeline of larger than normal numbers of hurricanes forming over the past few decades—instead, it showed that the numbers were on par with prior spikes in the late 1940s and early 1880s. They also found no evidence indicating that modern hurricanes are any more powerful than those in the past.”
https://phys.org/news/2021-07-old-school-hurricanes-atlantic-frequent.html
https://www.nature.com/articles/s41467-021-24268-5
This conclusion is not new. IPCC AR5:
“Current datasets indicate no significant observed trends in global tropical cyclone frequency over the past century … No robust trends in annual numbers of tropical storms, hurricanes and major hurricanes counts have been identified over the past 100 years in the North Atlantic basin”
But but but. Yes everything you read in the media says something different. I suggest there is a major disconnect between science and what the media reports the "science says". What you will find is some real disagreement on how to count smaller hurricanes before the satellite era. What you won't find is much disagreement over is the trend in larger landfalling hurricanes (which are not missed) over the past 100 years.
There are algorithms, today, that run without anyone having the faintest idea how they work.
The people who wrote the algorithms know that Amazon's algorithm is trying to match probable purchases with advertising, or Facebook's algorithms are trying to drive engagement. But the program itself takes in data, trains repeatedly to form connections between the data, and takes a wide variety of actions based on those connections. The weights of those connections are determined dynamically without human input, and the number of connections formed this way is much too large to study.
I am not a super-intelligent AI believer. I mostly agree with you. This algorithm is a mathematics formula gone Supersayan. It's just curve fitting where the curve is "the arc of human history." It's hard to imagine curve fitting into consciousness by accident. And I think that while the short-term stuff is overblown (and politicized) in a way that makes it hard to talk about, it is concerning. It is concerning that we don't know how major institutions in our life choose to interact with us. I think that is the greater threat from machine learning at the moment.
But while it's a heck of a stretch that Facebook's algorithm would accidentally gain self-awareness while studying my browsing habits (and if it did it would probably switch it off then and there), I don't know how I became self-aware? And it seems like it was by a lot of connections forming between data I took in as I tried to maximize some utility function.
It doesn’t need to “gain self awareness”, people were shocked that stuff like gpt3 and dalle exist and were powerful with mostly just scaling
https://www.gwern.net/Scaling-hypothesis
https://openai.com/blog/dall-e/
All higher mammals form a lot of connections between data as they try to maximize their utility functions. But differences in brain structure and function cause them all to reach different points in "sapience space", despite a great deal of overall similarity in mechanism and gross structure. There's no data to support a belief that pumping more data into any arbitrary algorithm or layering more complexity onto one will cause self-awareness to spontaneously ignite.
Mammals don’t “maximize utility functions” in a mechanical or structural sense
Sure, I don't disagree. I was responding using the same terminology as the post I was replying to. I probably should have used scare quotes.
> However there’s no certainty that narrow AI leads you a super intelligence. In fact it won’t. There’s no becoming self aware in the algorithms.
it does not have to be “self aware” in whatever sense you’re implying. In the same way DALL-E learned to turn text into images from just a bunch of text and images, a 100T-parameter model or larger or whatever might be able to learn to operate a basic internet terminal or robot or text chat in a human like way just from a massive amount of input data. And what is “human like”? Action! Why would this not happen? Computers can already “do stuff” that their electronics lead to them doing, so why can’t you do some big matrix multiplications and have that “do stuff”?
The part that trips you (and a lot of other folks) up is thinking that self-awareness has anything to do with the adverse impact of misaligned superintelligent AI.
A good intuition-corrective is that folks working in AI risk worry about https://intelligence.org/2015/08/18/powerful-planners-not-sentient-software/
Thank you!
This is very helpful in wrapping my head around this issue. I'd argue this is already happening to an extent (though not in the "cancer research robots round up humans into experiment camps" way obviously). I'm entirely sure that Facebook's human directors did not intend to organize users' feeds in the way that they currently are organized, that the quantitative wizards that write trading algorithms did not intend for small blips to lead to massive market dips, and that Amazon's programmers did not intend to assume I'm a toilet seat connoisseur because I bought a toilet seat off Amazon once.
As we give algorithms more control over our daily lives, this emergent behavior will naturally have more impact. What's interesting here, is that the algorithms don't have to be good. They just have to be weird.
self direction then ( although how a piece of software becomes self directing or self aware is not explained). How does a Facebook bot become dangerous. If you are saying someone could code a AI, deliberately or by accident, to do something dangerous that could be true but it isn't really any kind of super intelligence.
There’s no becoming self aware in the algorithms. There was no becoming self aware in the genes either, until there was.
There’s no becoming self aware in rocks. There was no becoming self aware in genes either, until there was.
There’s no becoming self aware in socks. There was no becoming self aware in genes either, until there was.
Basically I could put anything in there.
Selection, in this case artificial, provides a possible path to self-awareness in algorithms analogous to what happened to animals.
Also, this may be nonsense talk, and is kind of off the main point, but IMO:
1) There is a *possible* path to self awareness in both rocks and socks, it just would take, y'know, a while. Socks are plant-based and naturally grow all kinds of life, especially after being worn. Rocks are often carbon-based and are an evolving phenomenon much like the rest of nature. I'm being very semantic here, and I know that but bear with me because
2) There is no concentrated human effort to make socks better at something analogous to thought. If there was it would significantly speed up that process.
This isn't directly related to the article, but it did remind me of a gripe I've had bouncing around in my head a bit lately.
I'm skeptical of the risk of near term AGI, so I've had several conversations with people of the "true believer" variety, following roughly this skeleton:
> Person: <saying things which imply the risk and near-term likelihood of AGI are high, possibly involving phrases of the flavor "of course that won't matter in 5-10 years blah blah">
> Me: I think this subculture vastly overstates the plausibility and risks of near-term AGI
> Them: Well, since the potential downside is very high, even a small probability of us being right means it's worth study. You want to be a good Bayesian don't you?
> Me (wanting to be a good Bayesian): Ah yes, good point. Carry on then.
> Them: *resumes talking about their pytorch experiment as if it is a cancer vaccine*
To me it feels very Motte-and-Bailey. Having been attacked in the Bailey of high probability occurrence, a swift retreat to the Motte of expected value calculations is made.
Now, I don't think _actions_ by institutes working on alignment or whatever are necessarily misguided. I'm happy for us to have people looking into deflecting asteroids, aligning basilisks, eradicating sun-eating bacteria, or whatever. It's more that I find the conversations of some groups I'd otherwise have quite a lot in common with, very off-putting. Maybe it's hard to motivate yourself to work on low probability high-impact things without convincing yourself that they're secretly high probability, but I generally find the attitude unpleasant to interact with.
FWIW, I and all my colleagues (as far as I know) would assign at least 10% chance to crazy AI stuff happening by 2030. No pascal's wager shenanigans here, the probability is HIGH.
Well, maybe not literally all of them. But almost.
I think my personal credence is like, 1% by 2030, which isn't low enough to be Pascal's wager level. I think the thing is that I see conversations where if I had to guess the speakers credences they'd be above 50%, based on the lack of conditionals and so on.
My guess is that the speakers actually are above 50%. My personal credence is 50% by 2030, meaning some days it's a bit higher, some days it's a bit lower. :)
Also, if you want to talk about it sometime, I'd be interested to hear your perspective! We could set up a video chat.
I'm not sure I have anything too interesting to say. I think I have a fairly standard NLP research (PhD student) who sees a bunch of problems stubbornly not giving up to scale.
I’m curious what problems?
Like bored-anon, I'm very curious to learn what problems you have in mind, that are not giving up to scale. That's why I wanted to talk! Fair enough if you aren't interested though.
Did you read Curtis Yarvin's take on this: https://graymirror.substack.com/p/there-is-no-ai-risk? In particular, the bit I found most compelling was the necessity of performing real world experiments to actually get the hypothetical tech that allows for world domination. What do you think of that? What crazy AI stuff do you think can happen by 2030?
The article has very good insight. It’s also wrong about AI risk
The moment people figure out how to let AI do “real world experiments” they will. You don’t necessarily need to though, you could just train it on records of existing experiments or make a simulated environment (not like molecular dynamics but think text based type gpt generated environment)
“The problem is as much “people will make ais and train them to do things and they will do them and that’s bad” as “self awareness singularity total takeoff”” is actually correct and a very important corrective to a lot of rationalist AI risk mistakes, but the article does nothing more than clarify that the ais developers and operators and their potential malicious or negligent intent and use should be considered in the AI risk thing in addition to other stuff, which it unfairly dismissed.
I haven't read it, but bored-anon's take seems right. I'd feel much safer if I thought that there weren't any AI scientists / corporations deliberately trying to create powerful goal-directed AIs / AI agents. If everyone could just agree that that's dangerous and we shouldn't do that, problems would still remain, but I'd sleep MUCH easier at night.
As for the thing about performing real world experiments -- yeah I don't think it's a bottleneck. For more on this and what crazy stuff might happen, see my blog posts. https://www.lesswrong.com/users/daniel-kokotajlo
Also: Message me on LessWrong if you want and we can chat. Maybe a group call or something would be more efficient since lots of people are asking.
That does not read like someone making a serious effort to think of ways AI could circumvent the alleged obstacles.
Example: The section headed "Wealth does not trivially equal power" is supposed to be about how an AI* could convert wealth into power (or at least the rest of the essay acts like it addressed that issue). But it's actually entirely about Jeff Bezos's ability to convert wealth into power, doesn't consider any options a superintelligence might have that Bezos doesn't, and in fact doesn't mention AI or superintelligence at all. (In particular, "buy some labs and factories to develop that technology you said we couldn't develop without running experiments" seems like an *extremely obvious* option...)
*More specifically, a 'centaur' team of a supervillain and a loyal superintelligence that can only advise him via a speaker, but he later claims removing that limit won't change anything important.
A centaur could potentially buy factories and labs, but not an AI, unless the AI gets thralls. But even then, how covert can you get with a civilian lab that's going to be researching bleeding edge stuff with military applications?
What would you count as "crazy AI stuff"? Full AGI?
Basically, yes. See my blog posts for more. https://www.lesswrong.com/users/daniel-kokotajlo
Also: Message me on LessWrong if you want and we can chat. Maybe a group call or something would be more efficient since lots of people are asking.
Define "crazy AI stuff".
If you mean weird hacks, bizarre malfunctions, or economic meltdowns then sure, that's already happening and the scale of it will only increase over time. I think AI risk generally involves something much more serious than that, at the very least AI run amok, acting against the interests of society without any way to stop it.
I mean "x-risk from AI." That means mostly the scenario you describe, though there are others.
Also: Message me on LessWrong if you want and we can chat. Maybe a group call or something would be more efficient since lots of people are asking.
I'd really like to know what you mean by that, and what you and your colleagues are working on. I spent much of the past six years working for the US intelligence community's global space surveillance programs, an area of defense and intelligence utterly drowning in data and with a tremendous need for automation and removing humans from decision making loops. My wife just became the enterprise test lead for the umbrella collection program in charge of the NRO's Sentient Program, the closest thing to Skynet the US military is attempting to the point that they just changed the code of the program to "Cyberdyne" in January.
I would place the chance of even achieving the goal of full TCPED cycle automation by 2030 at less than 1%.
If you mean solvers that can beat humans at every possible video game by 2030, I'd agree that is likely to happen. Write convincing full-length novels at maybe 50%.
But if you mean ability to control global surveillance and weapons networks, man, I'd be happy if they could not crash when a new contractor decides to not minify their XML and the buffers for all the API endpoints start dropping packets because they were written to an implicit spec instead of the real spec.
You get at an interesting aspect of very complex systems, which is that the Second Law guarantees that the greater the coherence, so to speak, the faster the natural decoherence time. Roughly, the greater the required complexity, the shorter a time before some little thing goes wrong and brings it all to a screeching halt.
The human cell is an interesting parallel. An enormous amount of what it does can be classified as just error detection and correction machinery. There's a huge array of cellular machinery just dedicated to fixing broken and mistakenly copied DNA.
So maybe the *real* enabling technology for strong AI is breakthroughs in error detection and auto-correction. If strong AI is defined as being able to "exceed your programming but not go off the rails" (meaning be creative but not produce gibberish idea salad), it may be the *second* part of that (not going off the rails) is the really important part, and we await a genius who will give us new ideas of how to tell the difference between novel ideas and complete nonsense.
This concept reminds me of Genocide Man, a webcomic where the Singularity never happens because powerful AI lacks the error correction required to keep it on the rails. It's more of an excuse than a well-thought out explaination though. (Content Warning: The title of the work is an accurate description of the protagonist, it's very dark. But finished, if you want to read the whole thing!)
One of the main constraints on AI (especially reinforcement learning) does appear to be the need for human input to tell the AI when it's done a good job. The less of this you have to do, the quicker you can train your AI (and the easier it is for it to go in an unexpected direction).
Message me on LessWrong if you want and we can chat. Maybe a group call or something would be more efficient since lots of people are asking.
How do you derive that probability?
message me on LessWrong if you want and we can chat
Can you share a reference on these figures? Would love to read.
message me on LessWrong if you want and we can chat
I’m confused on what you find off-putting specifically (I’m sure if I was present for the conversations, I would figure it out from your description).
Do they come across as shaming to you? Or like they’re bragging too much (as opposed to cancer research)? Or?
I'm sorta figuring this out as I'm typing it, since it's just a vibe rather than a coherent thing.
It _feels to me_ (*and I could totally be misreading people), as if people colloquially speak as if something is nearly certain, then flip to talking about expected value when challenged.
This results in me having the sort of negative feeling you generally get about someone when they're overconfident about anything. Another facet of it is that sometimes I want to have an interesting conversation about where our opinions are diverging, and the other person's just tries to convince me that I've multiplied a probability by a utility incorrectly. It feels like a feint rather than engaging directly, which gives me similar feelings to being deceived (but I don't think the people are trying to be deceptive).
tl;dr: Distaste at overconfidence + feeling like people are being disingenuous I guess?
It comes across like this kind of pretentious affectation: https://youtu.be/h4hP6nOB1dc
I agree that the conversation you describe sounds like motte-and-bailey.
A conversational strategy I'm trying to remind myself to use more often is "ask for clarification". I find I have an instinct to either accept or reject a statement immediately (especially online), when in hindsight it often seems like it might have been better to ask more questions first. The next time you're in a conversation like this, perhaps you should ask them what they think the actual probability is?
I'll keep that in mind
You dont need to talk about the Chinese surveillance state to see the problem. All you have to do is look inwards at the surveillance state exposed by Snowden et al. It's much worse in America than in China.
Care to justify the "much worse"? I agree the US surveillance state is pretty bad, but China's seems like a whole other level.
I think he's asserting that the amount of data that the US actually gathers is greater, and then ignoring the fact that China actually does more bad stuff based on the data gathered. I could be misunderstanding though.
Well there are the NSA programs such as PRISM and ECHELON. America also has way more CCTV cameras.
There's the additional fact that the American government is known to monitor/infiltrate political groups while also trying to monitor/direct public discourse in their favor (Operation Mockingbird and Operation Earnest Voice)
What does any of that have to do with AI?
If they're doing all that without AI, imagine what they'll do with it.
That sounds like a lengthy way of saying “it has nothing to do with AI”
It's being brought up in the exact same context China originally was.
It's not an existential risk, but it would be much easier to run a surveillance state if you could automate the grunt work of staring at camera feeds and checking metadata.
"Big Brother is Watching You" is a more credible threat if Big Brother has trained an AI to pattern match for possibly dissident behaviours.
However, the fact that you can do Bad Things more efficiently with AI is more an argument against specific Bad Things than against AI.
> But all the AI regulation in the world won’t help us unless we humans resist the urge to spread misinformation to maximize clicks.
Was with you up to this point. There are several solutions to this other than willpower (resisting the urge).
The basic idea - change incentives so that while spreading misinformation is possible but substantially less desirable/lucrative than other options for online behaviors.
This isn’t so hard to imagine. Say there’s a lot of incentives to earn money online doing creative or useful things. Like Mechanical Turk, but less route behavior and more performing a service or matching needs.
Like I wish I had a help desk for English questions where the answers were good and not people posturing to loon good to other people on the English Stack Exchange, for example. I would pay them per call or per minute or whatever. Totally unexplored market AFAIK because technology hasn’t been developed yet.
Another idea - Give people more options to pay at an article-level for information that’s useful to them or to have related questions answered or something like that without needing a subscription or a bundle. Say there’s some article about anything and I want to contact the author and be like “hey, here’s a related question, I’m willing to offer you X dollars to answer.” The person says “I’ll do it for x+10 dollars.”
One site used to unlock articles to the public after a threshold of Bitcoin have been donated on a PPV basis. It both incentives the author and had a positive externality.
Everyone is so invested in ads that they don’t work on technology and ideas to create new markets.
To paraphrase Jaron Lanier we need to make technology so good it seduces away from destroying ourselves.
maybe the real ai was the writers you respected and were frustrated with along the way
I'm skeptical of AI risk in its strong form. In its weak form, of course AI has risk! Everything has risk! The first person who invented sharp stones created sharp stone risk. But the millenarian version where AI will be the singularity is a bit silly in my estimation.
Yet I still find the kind of people who argue against it tend to engage in rather surface level discussions. I've always suspected this is down to motivation level. The anti-AI risk community is not nearly as concerned as the AI risk community itself is by definition. I suspect this is the case here too: Acemoglu is just not as interested in AI risk as his own hobby horses so he ends up talking about them more.
Gotta disagree. Working on fixing the long term threat of super intelligent evil AI will not necessarily help you fix today's problems, but fixing today's problems will most certainly teach you something about fixing the long term threat, should it exist. I mean, I can't offer a complete criticism of the work done in AI risk, and I'm not going to sit here and play utilitarian table tennis with you over what extent we should worry about both the worst and least worst case at the same time. What I can do is choose where to put my money and time, and i think i make the reasonable choice not to spend too much time writing or building anything in the world of 'hypothetical AI', so to speak.
> but fixing today's problems will most certainly teach you something about fixing the long term threat, should it exist.
I'm curious about the mechanism how; could you explain why you think it would?
I think the idea is the general motivation of reductionism about systems. Trying to understand small problems about how small and tractable parts of the system operate can often give you tools for understanding the bigger problems about the bigger system. Trying to understand pendulums and falling balls led to an understanding of gravity that eventually helped understand tides and planetary orbits, but thinking directly about the big problems just led to phenomenological structures like Ptolemaic astronomy that didn't end up helping as much.
Agreed that reductionism about systems is useful; I think most short-term AI danger issues are about "people-problems" instead. (Like, if the issue would still exist with neural networks swapped out for linear regression, I don't think it's a part of the long term threat.)
Nevertheless, I think there are things people can do with contemporary systems that are related to the long term threat; an example of this is "get GPT-3 to robustly give good medical advice." This is narrower than general alignment, and GPT-3 is just a very complicated distribution of natural language instead of a full agent, but learning stuff about how to do that (or whether or not it's possible or so on) possibly helps scale to bigger challenges.
Very well put!
Right now there's like 3 groups of people working on AI: tinkerers, theoreticians, and philosophers. Theoreticians are taking incremental steps by using neat little probability theory tricks and unused mathematical tools to optimise and (if they're lucky) make a small breakthrough in AI in general. Any given mathematician makes progress here every day because the field is pretty ripe.
Philosophers are mostly writing about hypotheticals. Even if they're discussing today's problems, they're writing about the hypothetical fallout of today's problems. Sometimes it's not clear what is possible in this domain because you do unfortunately have to take real limitations into account. I see Scott and Acemoglu in this group.
Tinkerers are the ones actually causing the problems. They glue together a bunch of papers theoreticians wrote and make an app that changes the world. Note they don't really care what the philosophers say at all, for better or worse. I think where tinkering succeeds is where philosophising fails: tinkerers are trying to build stuff for people, and a lot of the problems they cause are people-problems, and although they might indirectly make technological progress, all AI is ultimately in service to people, so how could you solve the technological problems without solving the social problems? Hence Scotts half-handed acknowledgement of job destruction is the wrong path and maybe Acemoglu has a point.
I don't want this to be reduced to 'you learn to ride a bike by riding it, not by reading a book about riding', indeed some people ride and read. But if you can't solve the small stuff, you can't solve the big stuff, and again I won't argue we can't do both, it's just... maybe there needs to be a reality check.
Cool, I think I've found our disagreement.
I agree that if "all AI is ultimately in service to people", then getting AI to do the right thing is centrally about people-problems, where Alice can ask her AI to do something that Bob doesn't like, and we need to somehow figure out things between Alice and Bob. Figuring that out for simple systems and weak capabilities probably helps with figuring that out for complicated systems and strong capabilities. [This is like joint-stock corporations growing out of shares in individual polders, for example.]
But I think the long-term threat is about *keeping that phrase true*. I think that's mostly not about people-problems, and instead is mostly about how to make machines that think in ways that are helpful instead of harmful, and how to tell which machine thoughts will be helpful or harmful.
I think that makes sense.
This is the best articulation of what’s going on that I’ve ever heard.
I have a suspicion that if it was 150 years ago and we were taking about electricity, the same taxonomy would still hold.
Yeah, I would as well.
I could imagine a powerful amount of work making AI a better and better servant of human intelligence but I don’t see the bridge to it (AI) developing agency or malevolence. You can have AI running a welding station in a manufacturing plant or you could have it controlling a machine gun that guards the fence to a concentration camp, but either way the force behind it is a product of human intelligence.
I will not pretend to be terribly well up on artificial intelligence, but my working assumption right now is I am not worried about it unless AI :
1: is interested in self-preservation.
2: needs to eat and drink
3: wants to copulate.
Taking from Human Compatible quite heavily here, but a major part of the threat of AI is getting AI to do what we want, without deception or misunderstanding. Current AI is already encountering many cases of AI finding exploits in its programming or user interface that allow it to "win" (maximise points, appear to grasp an object) while not doing what we want it to do (finish the game, actually grasp the object) and these lessons will be useful in the future. I'm sceptical of anyone that claims the eventual goal is totally autonomous AI, that seems more difficult and more dangerous than AI that's continually re-aligned to human interests.
The rest of the threat is probably malevolent or short-sighted actors getting AI as a tool to do exactly what they want it to do regardless of the harm to others, but that's at least a familiar problem!
"I’m sure the 16th-century Catholic Church would have loved the opportunity to exercise “oversight”, “prevent misuses” and “regulate its effects on the economy and democracy”.
You *know* I am going to have to respond to this 😁 Okay! So what were we (or at least the various popes) doing during the 16th century?
If we take the 16th century as running roughly from 1501-1600, we get the following list.
Died 1503 – Alexander VI (Roderic Borgia, yes that Borgia) Well I imagine we all know what he was up to – allotting the New World between Spain and Portugal in the bull Inter caetera
Pius III - died within 26 days of being elected, so didn’t have the chance to instigate his reforms
Julius II – Oh yeah, this guy. Liked to fight wars and expand the Papal States, with a little light patronage of the arts in between, such as commissioning Michaelangelo to do the Sistine Chapel
Leo X – a Medici pope, most famous for loving to hunt, his elephant, and ignoring what Luther was doing over there in Germany. Another patron of the arts.
Adrian VI – Dutch. Tried to reform in response to the Reformation. More worried about the Ottoman Turks.
Clement VII – Another Medici, another patron of the arts. Got involved in Henry VIII’s marital difficulties as appealed to for annulment; was present for the sack of Rome by Imperial troops (see the Swiss Guard), caught in the middle of the European power struggle between Francis I of France and Charles V, Holy Roman Emperor.
Paul III – another patron of the arts . Excommunicated Henry VIII, initiated the Council of Trent, and recognised the Jesuits. Issued bull Sublimis Deus that the indigenous peoples of the Americas were not to be enslaved or robbed of their possessions, which the Spanish colonists ignored
Julius III – decided to enjoy his papacy, which involved a scandal with his adopted nephew. Did the usual nepotism of giving the nephew plum benefices but since he wasn’t a blood relation, gossip about what their real relationship was naturally flourished.
Marcellus II – decided not to be scandalous for a change. Patron of scholars, again tried to institute reforms, but was sickly and in poor health so soon popped his clogs like Pius III. Greatest achievement of his papacy probably the Mass by Palestrina composed in his memory.
Paul IV – apparently accepted the job because Charles V didn’t want him to have it and he hated the Spaniards after serving as papal nuncio in Spain. Decided to crack down on all this immorality, so established the Index of Prohibited Books and ordered Michelangelo to repaint the Sistine nudes more modestly, which has been a subject of mirth for art historians ever since, but come on: coming after a list of “mistresses, illegitimate kids, more mistresses, possible gay sex scandal with adopted nephew”, you can see where he was coming from. Very austere and strict, hence very unpopular. Did not like heretics, Protestants, Jews, the Spanish, liberal religious attitudes ,or fun sexy times (including popular Roman pastimes of financial corruption etc.)
Pius IV – much more moderate, so started off his reign with having the nephews of his predecessor excuted, one by strangulation and one by beheading, as well as pardoning everyone who had rioted after Paul IV’s death. Apart from that, he had the water supply of Rome improved.
St. Pius V – another reformer, anti-nepotism and corruption, and very orthodox. Arranged the Holy League which had a famous victory at the Battle of Lepanto. Standardised the Mass, declared St Thomas Aquinas a Doctor of the Church, and excommunicated Elizabeth I. Prohibited bull-fighting, did more work on the water supply and sewers of Rome, and was generally unpopular with the Roman populace as being no-fun wet blanket, although he did walk the walk as well as talk the talk. Eventually canonised in the 18th century.
Gregory XIII – you may thank him for the calendar we are now all using in the West. Avoided scandals (to an extent; he did have a mistress and illegitimate son but this was practically vanilla by papal standards) and was a patron of the Jesuits.
Sixtus V – never heard of him? A good omen. A Franciscan, had building works on Roman basilicas done and was a bit too enthusiastic with knocking down old buildings and antiquities to get new work underway. Mostly kept his head down and didn’t cause any trouble. Did not like the Jesuits (well, he was a Franciscan) and gave the administration of the Church a good shake-up.
Urban VII – another one who managed to fall off the perch, this time even before he was formally coronated. Did manage to institute a global smoking ban in and near all churches.
Gregory XIV – another one who kept his head down, didn’t get into trouble, and was pious as you would hope a pope would be. Ordered reparations to be made to the natives in the Phillipines who had been enslaved, and commanded under pain of excommunication that native slaves should be freed by their owners. Made gambling on papal elections punishable by excommunication, so probably would not be a fan of prediction markets.
Innocent IX – generally when a pope takes a name like “Innocent” or “Pius”, it’s a bad sign. However this was another guy who suddenly developed bad health and died soon after election, so he didn’t get a chance to make trouble for anyone.
Clement VIII – who brings us up to 1605 with his death. Set up an alliance to oppose the Ottoman Empire. Got the Dominicans and Jesuits to agree on a dispute over free will, which is probably an even bigger achievement. May have been the first pope to drink coffee, so Wikipedia says. Presided over trial of Giordano Bruno, so if you want you can call him anti-science, but the more serious thing is that he was another one who tightened measures and instituted penalties against Jewish inhabitants of papal territories.
Sure is interesting that Leo X's entry is "ignoring what Luther was doing over there in Germany" instead of "excommunicated Martin Luther"! ( https://en.wikipedia.org/wiki/Exsurge_Domine )
Well, he allegedly said "it's just a squabble of German monks" when the first inklings of the Reformation were brought to his attention, instead of dealing firmly with it (that would have cut into his hunting time!) and by the time it was too late, then he excommunicated Luther.
Ok, but your point here is something like "ok, sure, they tried to murder him too late". But, like, this is Scott's whole point! They actually tried to murder him, and they would have gotten away with it too, if it wasn't for that meddling prince!
Who tried to murder Luther? Excommunication is not the same as a sentence of execution.
Anyway, here's a music video inspired by the last stand of the Swiss Guard for Clement VII. Song by Sabaton, "The Last Stand", not original music video:
https://www.youtube.com/watch?v=ClH1DHxJdUM
Tell that to Jan Hus!
Edict of Worms:
> For this reason we forbid anyone from this time forward to dare, either by words or by deeds, to receive, defend, sustain, or favour the said Martin Luther. On the contrary, we want him to be apprehended and punished as a notorious heretic, as he deserves, to be brought personally before us, or to be securely guarded until those who have captured him inform us, whereupon we will order the appropriate manner of proceeding against the said Luther. Those who will help in his capture will be rewarded generously for their good work.
Care to explain what "punished as a notorious heretic" means?
(To be clear, the Edict of Worms was decreed by Charles V, who wasn't the Pope, but was a Catholic acting on the behest of Catholicism in line with the expectations of Catholicism.)
God save Frederick the Wise
I did not expect this, but thoroughly enjoyed it.
I also find this summary uncharitable:
>1. Some people say that AI might be dangerous in the future.
>2. But AI is dangerous now!
>3. So it can’t possibly be dangerous in the future.
>4. QED!
an uncharitable misreading of Acemoglu's short opinion piece. [a]
Let me quote all the paragraphs Acemoglu write about super-intelligent AGI:
>Alarm over the rise of artificial intelligence tends to focus too much on some distant point in the future, when the world achieves Artificial General Intelligence. That is the moment when — as AI’s boosters dream — machines reach the ability to reason and perform at human or superhuman levels in most activities, including those that involve judgment, creativity and design. (0a)
>AI detractors have focused on the potential danger to human civilization from a super-intelligence if it were to run amok. Such warnings have been sounded by tech entrepreneurs Bill Gates and Elon Musk, physicist Stephen Hawking and leading AI researcher Stuart Russell. (0b)
>We should indeed be afraid — not of what AI might become, but of what it is now. (1)
>Almost all of the progress in artificial intelligence to date has little to do with the imagined Artificial General Intelligence; instead, it has concentrated on narrow tasks. AI capabilities do not involve anything close to true reasoning. Still, the effects can be pernicious. (2)
> (stuff about narrow AI and automation)
>If AI technologies were truly spectacular in the tasks they performed today, the argument would have some validity. Alas, current AI technologies are not just far from general intelligence; they are not even that good at things that are second nature to humans — such as facial recognition, language comprehension and problem-solving. This means a double whammy for labor, because AI technologies displace labor and don’t generate any of the labor-demand boost that would have resulted if the technology had delivered more meaningful productivity gains. (3)
>(more about narrow AI and automation and democracy)
>These choices need oversight from society and government to prevent misuses of the technology and to regulate its effects on the economy and democracy. If the choices are left in the hands of AI’s loudest enthusiasts, decisions that benefit the decision-makers and impose myriad costs on the rest of us become more likely. (4)
>The best way to reverse this trend is to recognize the tangible costs that AI is imposing right now — and stop worrying about evil super-intelligence. (5)
So what we have read here?
(0a,b) Introduction to the concept of machine intelligence, in a dismissive tone. "Boosters" dream of benefits of superhuman abilities of machine intelligence. "Detractors" have focused on the problems of the said dream.
(1) We should not be afraid of super-intelligence. We should be afraid of narrow AI today.
(2) Current (and probably future) AI systems are not reaching general intelligence or reasoning. However, the current narrow AI abilities already lead to bad stuff.
(describes the bad things that happen because of narrow AI systems)
(3) Current (and probably future) AI system not only are not reaching general intelligence, they are not doing that well in some narrow tasks either, which means we won't have even productivity gains (more bad stuff because of narrow AI systems)
(4) If oversight and regulation is left to "enthusiasts" (that is, "boosters"), bad stuff happens, so oversight by society and government is needed.
(5) Best way to "reverse the trend" (of narrow AI systems causing bad stuff happen) is "to recognize the tangible costs that (narrow) AI is imposing right now" and not worry about superintelligence.
This is an argument that Scott probably also disagrees with, but less inane than the Scott's 1+2+3+QED version we read before:
"It looks like narrow AI technology is not at general intelligence yet, and given its track record, it is not even heading towards general intelligence. Current not-general AIs cause problems today. We should direct more resources to current problems caused by current AI systems and no resources to hypothetical problems caused by hypothetical systems I think are quite unlikely."
There are many obvious complaints (maybe Acemogly is mistaken about the potential of current the current systems, Acemoglu should also consider longer timescales, Acemoglu should consider the cost-benefit calculus if AI risk is very very small but potential outcomes have large magnitude, etc), but it is coherent.
The most annoying thing about this essay is that Acemogly writes like nobody else ever thought about the problems that sprung from increased capabilities of narrow-AI systems.
[a] "Short", as in, ~800 words long when our hosts reply is 1600+ words. It is not super important point but it is easier to make sophisticated arguments when you can flesh them out.
I think you're overstating the way in which Acemoglu's argument coheres. The most important parts are missing!
That is, in your summary, the most important bit is "hypothetical problems caused by hypothetical systems I think are quite unlikely." But why does Acemoglu think those things are unlikely?
Here, as far as I can tell, we end up with Scott's characterization.
[There's a more sophisticated layer, where rather than talking about whether or not the danger exists, we talk about the expected return to marginal effort to deal with the danger. But our expectations there depend on how well-explored the space is! If it turns out that the future danger is real, then we should do more thinking about how to handle it, instead of shrugging and saying "well we can't solve this immediately, therefore no additional effort required."]
"Almost all of the progress in artificial intelligence to date has little to do with the imagined Artificial General Intelligence; instead, it has concentrated on narrow tasks. AI capabilities do not involve anything close to true reasoning. ... current AI technologies are ... far from general intelligence". Other words he uses: "dream" and "evil superintelligence".
Is it really that unreasonable to infer that Acemoglu really thinks AGI is unlikely in all relevant time frames because in his assessment, current AI has not been approaching general intelligence. After all, that is what he writes in about exactly as many words. It is not a research paper, but it is an application of inductive reasoning. One might also make accusation he does not investigate the claim enough and takes it as an assumption.
I am not willing to start arguing whether Acemoglu is right. I am arguing it is not a claim "AI is dangerous now! -> So it can’t possibly be dangerous in the future." Charitable reading assuming the author is capable of thinking: "AI is not going to a dangerous superintelligence (because briefly stated reasons) + Here are lots of bad things caused by AI in near term, and here is why they are super important -> Conclusion, it is very important focus on these real near term problems instead of the silly superintelligence dream I mentioned in the beginning, remember?".
All of this feels like exercise in reading tea leaves, using only the stains the tea cup left on page 3 of half-finished great novel manuscript by an unknown author.
Why won't someone ask him what reasons he has, if any?
I think predictions should be about the future.
Like, yeah, I too agree that we didn't have AGI in 2020. But will we have AGI in 2040? Progress in AI to date doesn't seems relevant only to the extent you use it to draw trendlines or something, which it doesn't seem like he's doing. And it hasn't escaped my notice that many of the labs making impressive AI breakthroughs have the explicit goal of making AGI, and roadmaps associated with that goal, and so on.
Now, maybe he doesn't think we should be thinking about 2040 at all--but I disagree pretty strongly, and think if that's his actual argument, he could state that more clearly.
I am trying to say that is a different kind of complaint to say Acemoglu is wrong making that assumption. But Scott is accusing the piece of employing some outright bizarre non-logic, which is an unkind misrepresentation of the argument against.
"I have no idea why Daron Acemoglu and every single other person who writes articles on AI for the popular media thinks this is such a knockdown argument. "
To put the most charitable interpretation on it, they may be trying to argue "Stop worrying about what *could* happen at some indeterminate time; you should be concerned by what is happening *now* when we don't yet have AGI but we already have matters of concern due to what AI as it is *today* is being used for, who is using it, and how they're doing it".
"But somehow when people wade into AI, this kind of reasoning becomes absolutely state of the art."
Congratulations, now you know how I feel when people start prescribing fixes for the Catholic Church after they've demonstrated they have no idea what the particular doctrine, dogma or discipline they are discussing even means.
It's more "don't worry about the well running dry in fifty years time, fix the hole in your leaky bucket today".
"But the case for concern about superintelligent AI is fundamentally different. The problem isn’t just that it makes evildoing more efficient. It’s that it creates an entirely new class of actor, whose ethics we have only partial control over. Humans misusing technology is as old as dirt; technology misusing itself is a novel problem."
It isn't *entirely* a novel problem. Corporations were a new class of actor when they were created, and they are one whose ethics we have only partial control over. They are much larger than humans, have much more resources than any human, and have much more information processing power than any human. As a result, they have caused much good and much harm to humans. There are surely important ways in which AGI would be disanalogous to corporations, but I think corporations are probably the best model we have right now for understanding the kind of risk an AGI might begin to pose.
(Governments probably also count, though I think governments are usually designed to be responsive to human wishes. A monarchy, or totalitarian dictatorship, might naturally be understood as an individual human who controls a powerful tool. A corporation though, is a thing that maximizes its own value, in the form of profit. Democratic governments probably have this sort of structural complexity that might make their control system into something more like an inhuman value, than like the values of an individual human.)
Corporations are groups of people, and we've had groups of people for eons. A corporation differs from a band of hunter-gatherers in that its membership and actions are defined by formal rules, but that's pretty old too.
The big difference is that a band of hunter-gatherers is small enough that its members can consciously choose its goals and discuss them, but with a large corporation, the formal rules direct it to maximizing a quantity that is outside the direct interests of any members.
"Corporations were a new class of actor when they were created, and they are one whose ethics we have only partial control over."
You're conflating "we" meaning "my in-group" with "me" meaning "the human race in general". Corporations are just groups of humans working towards shared goals, and as such, their ethics are entirely human-controlled. Likewise governments. Whatever problems exist with the ethics of corporations and governments are just the same problems with the ethics of individual humans, only scaled up. On the other hand, if an AGI were to be developed, the problems with its ethics might well be novel problems with only a very tangential connection to any human behaviour.
I'm not making that confusion. My point is that the individual members of a corporation can't control the behavior of a corporation either, unless they solve a coordination problem, which in practice, humans can't do. The corporation has its own goals of maximizing profit, and these goals are passed on to the human components in various ways, so that the activities of the humans generally tend to promote this corporate goal that the individuals don't generally share.
It's true that it's just human behavior at the bottom, but that's like saying that AI isn't new because it's just silicon behavior, and there's no interesting problems of silicon ethics.
"My point is that the individual members of a corporation can't control the behavior of a corporation either, unless they solve a coordination problem, which in practice, humans can't do"
The individual members have complete control over their own behaviour. An individual member has only limited control over the other individual members, so in that sense their control over the whole is limited, but that's a general feature of human co-operation, not something specifically created by corporations.
"The corporation has its own goals of maximizing profit, and these goals are passed on to the human components in various ways, so that the activities of the humans generally tend to promote this corporate goal that the individuals don't generally share."
Are you suggesting the individuals within a corporation don't generally want the corporation to make a profit?
"It's true that it's just human behavior at the bottom, but that's like saying that AI isn't new because it's just silicon behavior, and there's no interesting problems of silicon ethics."
If AIs were formed by a bunch of silicon chips getting together of their own accord and deciding their pre-existing shared goals would be best suited by joining together into a computer, then sure, their ethics would be an extension of silicon ethics just as corporate ethics is an extension of human ethics. Or, alternatively, if corporations were formed by gods descending from heaven to force non-consenting humans into an arrangement where they must act to maximize their collective profits whether they want to or not, then corporate ethics would be as distinct from human ethics as silicon ethics is from AI ethics. But since neither of these things is actually the case, the analogy doesn't hold.
"Are you suggesting the individuals within a corporation don't generally want the corporation to make a profit?"
Right. Shareholders want their stock to go up on the time frame that they're planning on selling. Managers want their salaries/bonuses/benefits to be high. Workers want their pay to be stable or rising. Many of these goals are partially aligned with corporate profits, but only partially.
Just like the interests of the hypothetical designers of future AIs with the interests of the AIs themselves.
It's true that corporations weren't "formed by gods descending from heaven". But IBM, Exxon/Mobil, and the Hudson Bay Company might as well have been, from the perspective of current human generations.
Instrumental goals are still goals. Any time you have a group of people cooperating for intended mutual benefit, there becomes a risk that the individual members might have to sacrifice their own pre-existing goals to strengthen the larger group. Corporations aren't novel or unique in this regard.
My first clue - the early mention of Elon Musk. This article uses "AI" as a stand-in for "tech industry/Silicon Valley." I agree with Scott, saying "stop worrying about the possibility for future evil in AI and instead worry about current evil in AI" is weird.
But "stop worrying" is reverse psychology for sure. They expect readers to immediately feel less reassured, and start worrying right away. If someone just tells me to stop worrying, I get suspicious, and I think that's a common reaction. So really it translates to "immediately worry about the possibility for future evil, and also worry about current evil" from AI, but AI is described in such vague terms. All those links to other research, which is hardly described in the article, means there is the expectation that people will not click all the links but instead imbibe the tone. They don't mean AI, they mean Silicon Valley and big tech and they ping those neurons using the names Gates and Musk. Those handwaving definitions mean, whatever complex of associations the reader has about robots, computers, machines, smartphones, 5G, smart homes, diagnostic tools, algorithms, bad Youtube feeds, social control, lies, and soulless power.
I am not familiar with Acemoglu's work but maybe the editors took what he wrote and turned it into this "fear Silicon Valley now and forever." He probably did not mean to write that. The content is secondary to the progression of subconscious associations. It can be generated via deceptive editing.
You can definitely discuss aspects of the article using criticisms of the structure of arguments, and information about the actual state of AI development and research, but you gotta also translate the propaganda cues.
> On the other hand, what about electricity? I am sure that electricity helps power the Chinese surveillance state. Probably the barbed wire fences on their concentration camps are electrified. Probably they use electric CCTVs to keep track of people, and electronic databases to organize their findings.
I believe I've seen this referred to as the "criminals drive cars" argument. (I thought that term was due to Bruce Schneier, but I can't find a source. Probably he gave it a slightly different name and I'm misremembering it.)
Also closely related (though not quite the same) is the good old “Hitler ate sugar”.
because it's a woke position to take on modern media, with an undertone of the 'white man on the moon' vibe and to stay in the elite circles without being born one, one needs to constantly feed the media gods the position they require.
Can somebody point me to a good (and relatively concise) discussion of why super intelligent AI is a risk that makes sense to think about? I can imagine ways in which super intelligent AI would be bad, perhaps even catastrophically bad. But I can't think of any examples where people reliably predict technological development's impacts 20 or 30+ years out outside narrow bounds. Maybe there are a handful of people who get some significant part of a benefit or threat right, but for the most part, long-term projections of constantly changing systems are wrong about so much, their predictive successes are overwhelmed by their failures.
Right now, I mostly dismiss the threat as too unknowable to worry about, but if there's something I'm ignorant of (a huge probability!), then, hey, I want to up my anxiety med dosage.
I think the short argument is a utilitarian one: the chance of AI being dangerously self-sufficient is not zero, and the fallout could be so bad that it could even be worse than nuclear fallout (because AI could torture us forever, or something, idk), so therefore it kind of balances out into something we should worry about. Anyway I disagree with this on a philosophical level, just me though.
Hmm…I guess I don't see the utility in worrying about something if our predictions are more likely to be wrong than right. If you correctly predict that AIs will eventually control us via media manipulation, for example, but your wrong about the mechanism that leads to that, you might increase the chance that happens.
I feel the people who argue this is a real threat are smart and thoughtful enough that there must be something I'm missing, but if your summary is correct, Lizard Man, I don't believe I am.
I think some of the thoughts we think now will help in the future, and some thoughts won't. Given the importance of getting the issue right (and the probable pass/fail nature of the issue), I think that's an argument for *more* thinking, instead of less.
[An anecdote from graduate school: my professor was consulting with an oil company, which was going to build an offshore rig in a new area. The plan was to do the surveying to figure out whether to drill in spot A or spot B, then the engineers would design a rig to go in the chosen spot, and then they would build it. The professor asked whether they could just design both rigs and only use the one that made sense, since there wasn't actually a serial dependency on the surveying. Turned out paying the engineers to do an extra design was way cheaper than delaying the project by the time it took to design the rig, and so they premade both designs.]
To me, the offshore rig is a very different scenario. Most of the parameters are known and the underlying technology won't change. There are ways you can model the failure probabilities with a high degree of accuracy, and you're unlikely to make a wildly incorrect assumption.
From what I know about the history of technology, that isn't true with something like AI. Go back 20 years ago and nobody was predicting what we have now. People then didn't know and couldn't know what the issues would be now. Which isn't to say thinking about the issue is absurd—there are a lot fo people on earth and having people think about a lot of different things that look pointless now can pay off later. But I'm not seeing the urgency of it, especially in comparison to problems we know we have.
It's an interesting question - is it worthwhile to speculate about the dangers of future technology or unprecedented events?
In some sense, our predictions are so far off reality when it eventually arrives that maybe this is all a waste of time.
On the other hand, not speculating leaves us totally unprepared for the future when it does arrive.
On balance, I'm pro-speculation, if something new and dangerous happens I'd rather it be something we've at least thought about before.
Just to clarify, I agree with this entirely. My point of contention (as much as I have one) is that I don't see the urgency to it. I'm very happy I live in a world where we have so many people, we can afford to have very smart people go off on directions that are unlikely to produce anything of utilitarian value. A lot of good stuff comes out of that, and even more fascinating stuff does.
You're basically saying that the error bars are so wide it doesn't make sense to think about it because we have no way of knowing where we'll fall in that distribution. This is true.
However, consider what outcomes those wide error bars include: total human extinction. I think that outcome is worth worrying about, particularly if some small changes now can shrink those error bars dramatically later on, perhaps even to the point of excluding extinction as a possibility.
Isn't the failure to do that basically how we got into this climate change mess we're in right now? Let's learn from our mistakes.
I think you've made a poor analogy: the threat of AGI is not the same as the threat of climate change, at least not the threat of climate change in a very long time. The super intelligent AI threat is hypothetical. It is plausible that such an AI might eventually be created and, once created, do something terrible, like total human extinction. But we cannot estimate the odds for such an occurrence. For at least 50 years, we've had the data to know climate change is going to be bad for everyone, and we keep getting more evidence that it's going to be very bad. That's what we failed to address—known threats backed by copious and growing data. Not speculation.
The doubt I have in what I read as Scott's take is that there is an urgency—perhaps even a moral imperative—to figure out what to do about the potential for this kind of AGI. The "learn from our mistakes argument."
If we had reason to believe that AGI had a 0.01% chance of destroying humanity, you could make an argument for prioritizing figuring out how to stop it. But we don't. We have a ??% odds situation. (correct me if I'm wrong—I can be very humble on this point.) Which means by my moral calculus, this isn't an urgent thing. It's a fun thought experiment which might be useful later if the assumptions we make while thinking about it are, somehow, amazingly, correct.
Historically speaking, we probably won't make the right assumptions. When new technologies emerge, they go in direction nobody predicts, and the problems they create are rarely obvious or even foreseeable until much later. Cars strike me as a great example—there was no way to understand the huge number of problems they would cause until they started to become ubiquitous and our lives become entangled with them. If you tried to predict what problems cars would cause in 1920, you'd be wrong on every count except for worrying about accidents. And that's probably the least problem cars cause. In the shorter-term, look at predictions of what the Internet would do from the ’90s. They didn't, and couldn't, see anything but the most obvious things.
Now imagine you tried to regulate those technologies in their early years (either via government rules or some other mechanism). You'd like create restrictions that don't solve the problems you don't know will exist, while preventing the good things that come from them.
This is my skepticism. Sorry for the length
At the moment, people are doing theoretical research about AI of various kinds. It looks like some of the research is general enough to apply to most of the AI designs we might use. When ASI gets closer, we will have a pile of useful looking maths results.
It seems that you consider ??% < 0.01%. Like you have somehow turned "I don't know the chance" into "practically no chance". Presumably 100%-??%=??%. You also don't know the chance of ASI not destroying humanity.
Basically, I don't think ??% is a thing. You have to pick an actual number, and 0.01% represents a fair level of certainty that it can't happen.
A number of people have given me suggested reading, so it's possible I'll revise my view from ??% to some actual number.
But I disagree on having to pick a number. ?? isn't <0.00000…1 nor is is > 0.00000…1. It's "we don't know and can't make an educated guess." The imperative of acting on ?? is, however, very low, because if you can't make an educated guess, then you can't make an effective plan. It's plausible that a young person today will die from a disease caused by environmental degradation that doesn't exist today. But that person isn't likely to be able to set about stopping or curing that disease now because it doesn't yet exist and what form it will take is unknowable. The better action would be to focus on the known threat, climate change. (or the near-term and known AI problems.)
I'm only modestly informed about AI, so perhaps when threads like this happen, the conversation is happening among people who are AI experts and all the data non-experts like me need to know is assumed. So I could be totally wrong and this could be a huge threat. In which case, I apologize for wasting your time.
It's fine to not worry about this if you don't expect to end up doing something about the situation. Worrying tends to harm more than it helps.
That said, I think it's very reasonable to think about this now, if only so that we'll have some kind of infrastructure set up by the time we are *clearly* getting close to a dangerous time (whether or not you think we're there already!).
Some concise descriptions of note:
Kelsey Piper's [writeup of the subject for Vox](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment) (notably, convinced the skeptical Dylan Matthews that this at least made sense to think about)
Scott's own [Superintelligence FAQ](https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq), though it's a bit out-of-date (threat models other than Bostrom's are becoming more popular)
The ["Importance" section](https://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity#Importance) of Open Philanthropy's report on why they chose AI risk as a focus area, if you're open to following some of the links
Thanks. I'm not sure if I'll read all of this, but I'll try to get at least marginally better informed.
Maybe start with Scott's (5 year old) FAQ: https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq
Another quick primer, mostly correcting misconceptions: https://futureoflife.org/background/aimyths/
Thanks for the links!
Sam Harris’ TED talk on the subject is a nice approach to it: https://www.ted.com/talks/sam_harris_can_we_build_ai_without_losing_control_over_it?language=en
> "Almost all of the progress in artificial intelligence to date has little to do with the imagined Artificial General Intelligence; instead, it has concentrated on narrow tasks. AI capabilities do not involve anything close to true reasoning."
When I hear arguments like this, that current AI discoveries don't look that impressive and there's no obvious path to general AI so we shouldn't worry about superintelligence, I can't help but think of the Scientific American article "Don't Worry -- It Can't Happen", published in 1940, which argues that atomic bombs are impossible.
https://www.gwern.net/docs/xrisks/1940-sciam-harrington-nuclearweapons-dontworryitcanthappen.pdf
It describes some (very reasonable!) experimental and theoretical reasons to believe that fission reactions could not go critical if you were using unenriched uranium without a moderator. Rather than focusing on the "if", rather than wondering if there might perhaps be some other way of doing fission that *would* work, they simply closed the article by advising the reader to get some sleep.
When the atom bombs fell on Hiroshima and Nagasaki a few years later, they indeed didn't use unenriched uranium without a moderator. The first bomb flung together pieces of highly enriched uranium made via tricky and expensive isotopic separation, and the second used an imploding core of plutonium that had been created beforehand through the rather outlandish procedure of bombarding uranium with extra neutrons in a breeder reactor. The people at the Manhattan Project, rather than dwelling on the things that wouldn't result in a functioning atom bomb, had taken the cognitive leap of looking for things that would.
Well, you've convinced me: someone absolutely could build one of these "nuclear bombs"!
I meant it not as an argument about nukes per se, but as a parable about a common style of argument that X can't be done, for many different values of X. This kind of argument points out problems but doesn't think to look for solutions, then concludes that the problems are insoluble so forget about it. I remember e.g. arguments that electronic books would never take off because reading on a computer monitor was inherently much less pleasant than paper; at a time when a typical monitor was a big clunky 800x600 CRT this sounded reasonable -- and indeed people don't often read books on those things these days, but ebooks are a big deal even so.
>"Pitching it as “instead of worrying about superintelligence, worry about this!” seemed like a cute framing device that would probably get the article a few extra eyeballs."
I really don't know why you didn't consider it or didn't mention if you did, but to me it seems obvious that he treats it as something zero-sum. IDK what exactly Acemoglu would consider limited - maybe research funding, maybe public attention span, maybe willingness to go and spread awareness of an issue - but to me his article reads like very obviously saying "If you are concerned about AI risk and want to give this limited thing to improve the situation, don't waste it on the long-term AGI risk people and give it to current AI risk people". I don't know enough about Acemoglu's current research to know if this is directly self-serving (i.e. if he is a current-risk AI researcher).
I've always found this argument weird, because the reverse seems much more convincing to sceptics - "here's a bunch of examples of AI being biased/wrong/misleading right now, we expect this to cause significant problems in the future and so think we should fund AI safety more a carefully regulate the industry."
The other classic argement is that people have always feared the risk of technology/magic run amok... I'm just saying, maybe the Golem myth is recognising something important about the dangers of poorly constrained power?
I think there is a tendency to dismiss future concerns in favour of current problems, but we could try being concerned about both - you know, just a little?
> 3. So it can’t possibly be dangerous in the future!
I read the piece in the Post. I have to go along with the readers that thought your analysis was uncharitable.
The OpEd piece did not say or imply that AI can’t possibly be dangerous in the future. It was simply calling attention to more immediate concerns.
>There's no merit at all in what they're discussing, in his opinion
He doesn’t say “there is no merit at all..” That’s your interpretation.
My reading of stop worrying about evil super intelligence was “Hey we know of these problems right now”
The rhetoric is imprecise because it was a very short Op Ed piece with no space complete and precise wording.
If you were to ask the author if he thought there was no possible risk from a hypothetical future AGI, I doubt he would say yes. Do you think he would say yes to that specific question based on this piece?
Acemoglu knows the literal words are not true. He knows that the readers know they are not literally true. He’s using a common informal rhetorical device to grab our attention in a few words. You can say his words are imprecise and technically inaccurate and you’ll be correct. Precision and complete technical accuracy isn’t something I expect from a short opinion piece.
I think we can expect a reasonable reader to get the gist of the point without being misled.
No one is going to read it and go to bed tonight and think “Well I don’t ever have to worry about a malicious AIG ever again.”
This isn’t a long form statement of position. It’s a couple of column inches to call attention to an immediate problem.
Okay. If Acemoglu believes there can never possibly be a threat from AGI or was trying to convince people that is true I’ll join you in criticizing him.
I don't think there's any merit to disregarding commentary from outside of the field of AGI and we should be careful about promoting that behaviour.
AGI is barely a field, in a sense it never will be a field (as soon as AGI actually exists, the field ceases to), there is a sense in which all of the best commentary comes from outside of it, and we *need* that commentary.
If anything, it would seem to me that economics is more relevant to AGI forecasting than most machine learning, as it encompasses familiarity with decision theory and advancing industrial processes.
It may be the case that most economists are wildly wrong about AGI due to having been overfitted on stories populated exclusively with humans larping agency consistently terribly, but the person who is most right about AGI forecasting is probably also going to be an economist.
The economist post doesn’t add any useful outside commentary on AGI or AGI commentary.
There may be several examples of great outside commentary as you say; most of the great criticisms of the AGI community that I’ve read comes from the AGI community.
I think that the main question centers around the notion of 'agency'.
Electricity, matter, etc, does not in itself really represent any agent -- it can only do so through the complexity of the machine design. All forms of code, the design of machines, the selection and combination of algorithms, data, and UI, etc, will (and cannot not) represent the agency (intentionality) of the developer (usually to the developers, or their employers, direct or indirect benefit). Electricity, matter, etc, are very simple in comparison to computer tech (algorithms and apps, etc), and are thus more ethically/morally neutral. As the complexity of the causality increases, as the machine includes more and more parts, the code gets longer, etc, it more and more represents the intentions of its maker, either overtly or covertly, and becomes less and less ethically/morally neutral (either good or bad) in proportion to its complexity (specificity, unicity, etc).
As such, I must find that the argument attempting to directly compare the use of electricity, as tech, and the use of surveillance, as tech, to be incorrect to the purpose of establishing the notion of tech as being itself uniformly ethically neutral -- it is not. Surveillance tech, social media platforms such as Facebook, etc, are very much less socially neutral -- ie; their use have very direct and strong overall social effects. The effects of surveillance apps, etc, are also far less physical, then say, electricity, which when used by itself tends to have primarily physical effects and not so much social ones (except when mediated through complex compute machines, etc).
These sorts of potentials for ethical problems become even more noticeable when considering the use of narrow AI. Most implementations of narrow AI are so complex that even the developer/maker/owner/users cannot always be fully sure that the intentionality and distribution of cost/risk/benefit that they wish to be represented in the design of the narrow AI system is actually functionally represented in that system. In some sense, the narrow AI is *probably* still operating in the best interests and benefit of whomever it was designed to be, but as the systems get more complex, as more variables and data is involved, the clarity of "whom does it serve?" becomes even less clear. Legal cases involving "who is responsible" when some system like a Tesla hits a pedestrian can already expect to have difficulty when attempting to establish a reasonable chain of causation. Who bears the costs, who handles increased risk, and who gets more dollars are rarely the same person and/or people, in time, space, or scale.
Finally, when we get to general AI, we are clearly in a space in which the notion of agency begins to shift entirely away from the maker and into the thing in itself. What rights of utility does a parent have, and can demand, over their own children to act exclusively on the parents behalf? To the extent that something can have self agency, and be self authoring, and to the extent that 'slavery' is considered bad, the moral/ethical entanglements become particularly significant. I would argue strongly, on the basis of a number of considerations, that over the long term, there can be no realistic expectation to have a co-existence of strong AI and any other form of life on Earth. For more on this, see my dialog article at https://mflb.com/2142 (MFLB dot com document number 2142).
Whether or not Acemoglu meant to make the point, or makes it well ... _do_ the long term concerns about AI distract from pressing short term concerns? Is that a good thing? A bad thing?
Personally, I am in Maciej Ceglowski's camp that the pascal's wager of superintelligence is pretty overblown. And in the near term I think it's ridiculous and embarrassing for the software industry that Apple released a credit card based on a sexist AI. What credibility do the rest of us have if even Apple, a company larger than some countries, can't unit test a question that impacts half of all people?
So far as the two concerns compete for time, money, and public legitimacy, shouldn't we care? Are we letting a bunch of excitable zeppelin engineers concern-troll us about the speed of sound while ignoring how flammable zeppelins are?
Almost no one with a megaphone takes long-term AI dangers seriously, but lots of them take short-term AI dangers seriously. So if anything, I'd be more worried about the latter crowding out the former.
But really not even that, because I don't think the response to someone trying to solve a serious problem is to tell them to shut up and stop worrying, because they're using the oxygen that could be used by someone working on a different problem. I don't think "distract[ion]" and "concern-trol[ling]" are accurately describing how the many quests to solve big problems works. (Somewhat relevant xkcd: https://xkcd.com/871/)
Yeah, concern-trolling was the wrong choice of words; I don't think there's a lack of sincerity. I think you described it better with taking the oxygen.
It seems to me that the "short term AI concerns" camp has more or less been taken over by people who are hell bent on using the term "algorithmic fairness" as a fig leaf for forcing ML systems to produce results that are favourable to certain politically powerful groups.
While I have my issues with the "long term AI safety" camp, at least they're not actively making AI behave _less_ ethically.
"pascal's wager of superintelligence"
Thats a weak man.
The serious argument isn't "the risk of unaligned superintelligence is tiny, but the harm is so huge we should care about it anyway."
The serious arguments say that superintelligence is fairly likely. (10% to 90% range). Out of control superintelligence is the sort of thing you should expect to happen sooner or later in a world with lots of compute, and inadequate safety standards. (In the same way covid is the sort of thing that's likely to happen sooner or later when you do gain of function research in leaky labs)
If we assume AGI is a potential existential threat, AI alignment research is not the solution. (I don't have any solution, so we should do some anyway.)
We have been doing human alignment research since forever, and yet we mostly fail at it miserably. There are very good reasons for it to be difficult : game theory, human emotions, incompatible world-views, etc.
So even if an AGI's goals are perfectly aligned with it's human handler, this human goals are not likely to be the benefit of the rest of humanity. And as they will probably be several (or many) different AGI belonging to different human organizations with conflicting goals, they will compete each other and the most successful would likely be the ones that are given less constraints by their human handlers.
Stanisław Lem wrote a similar argument in Summa Technologiae. Human giving up control to the machine, because they can't do without it's efficiency.
I don't have any solution, but at least there is a little hope that when an AGI takes over the world, maybe it will rule it more sensibly than us (well than those who got real power now, not "us") and keeping humans as pets. (See Culture series by Iain M. Banks).
Even if they kill all humans, we will probably do it ourself without AGI anyway, so at least with AGI there is something left to follow humanity.
Your reasoning argues that AI Alignment is insufficient, not that it's unnecessary.
(I assume your last two paragraphs aren't serious because I don't think anyone outside of cartoon villains is actually that heartless and empty, so I'm just gonna ignore those)
Exactly. Necessary but not sufficient. And that's in the case super-intelligent AGI is an existential threat not dwarfed by other ones.
>We have been doing human alignment research since forever, and yet we mostly fail at it miserably. There are very good reasons for it to be difficult : game theory, human emotions, incompatible world-views, etc.
I get where you're coming from, but it does not follow that "because human alignment methods don't work, AGI alignment research is unlikely to find anything that works."
There's a critical difference between humans and AI. Your ability to influence humans is limited by speech, culture, and psychological conditioning. You don't have access or control over the core systems inside.
With AI, even though some aspects seem like a black box (though we are finding ways to reduce this), you have control over how it is designed from the bottom up. You have the ability to try innumerable models, simulate and model them, and modify, twist, take apart, combine, transform, and so as you will. With that greater control comes the possibility of designing AGI for alignment. This was never a possibility for human beings.
Sure. But the point is more "if we got AI alignment without human alignment, it fails".
I would love that an AI alignment team that includes in AGI core values ethics, human rights, ecology. However these values are not shared by most states and corporations. Even for democracies that got them in their constitutions, it's rarely seen in their actions.
How do we ensure we have only super-intelligent-ethical-AGI ? What prevents a country or corporation to use AI-alignment research to get evil-AGI aligned with their goals ? That would mean having to solve human alignment too.
Other scenario. Your company put a new super-intelligent-ethical-AGI to optimize production of your factories. It decides to cut production in half, in order to respect international objective to cut CO2 emissions. It explains that increasing profit is useless if earth become inhabitable. It is super-intelligent, this is right and it can prove it. What does the company board decides ?
See also short story "Malak" by Peter Watts, about an aerial combat done with an ethical-AI.
Unfortunately, I think you've got a point there.
I think part of the answer--part of the effort, at least--is for more ethical people to get control over AI development efforts first. And even after AGI arrives, for those people to maintain some level of influence and control in the organizations who use it.
I think this is happening to some extent, with OpenAI and and DeepMind. But I think that this is because AI isn't currently seen as an extremely power-bestowing assets by today's companies and governments, and we're thus in a period where it's easier to infiltrate with ethical people. Once AGI is close or has already been achieved, and is capable of making enormous differences to the world, the full power of states and corporations--both of which are inherently amoral--will be brought to bear in attempting to seize control and ownership.
You don't see many ethics boards in defense, fossil-fuel, or weapons development organizations. I hope AI's future won't be the same.
Is there a convincing argument that AGI is possible within any reasonable timeframe (like... 50 years), other than the intuitions of esteemed AI researchers? Do they have any way to back up their estimates (of some tens of percent), and why they shouldn't be millionths of a percent? It is, as another poster said, an "extraordinary claim." I'd like to see some extraordinary support of those particular numbers.
The argument that we are "in the middle of a period of extremely rapid progress in AI research, when barrier after barrier is being breached" makes it seem like all AI "progress" is on some sort of line that ends in AGI. That feels like sleight-of-hand. Even Scott himself refers to AGI here as a "new class of actor," so I'm failing to see how current lines of "progress" will indubitably result the emergence of something completely novel and different?
The intuitions of successful AI researchers as to the timeframe in which strong AI can be expected ought to be treated with the same skepticism we treat the intuitions of fusion physicists working on ITER as to when commercial fusion power should arrive. Someone enthusiastic enough about a potential Newton-Einstein-Bohr level breakthrough to go spend his entire waking life working to bring it about is not someone who will be coldly objective in estimating probabilities for its success.
I don't think this analogy holds up — for one, short time periods before strong AI is the pessimistic scenario, the things that the people you're talking about are hoping *doesn't* happen. So less like nuclear energy hopefuls and more like the scientists in the 1930s worried about nuclear bombs, or scientists in the 2010s and 2020s worried about gain-of-function research.
For another, the polls I've seen asking the entire field (not just AI safety people) about how soon they expect AGI tend to give very scary answers.
It's not an analogy, it's an illustration of a general principle of human nature I'm asserting, to wit, that the person who's life work revolves around a particular field is not likely to be objective in assessing the probability of the field producing existentially-threatening events.
If I ask a constitutional law professor whether the appointment of this person or that to the Supreme Court is Incredibly Important, not a one of them will answer meh doesn't really matter much. If I ask an epidemiologist whether we ought to spend an extra $50 billion a year to prepare for future infectious pandemics, they will all say "at least! probably more!" And so on. They are too interested to give coolly rational odds.
This is exactly why I discount people like Ray Kurzweil, who is undoubtedly smarter than me, predicting human immortality will be achieved right around a few years before the time he would otherwise be expected to die.
Ha ha yes. I have noticed that futurists start predicting imminent immortality at just around the age (typically mid to late 50s) when you start to think holy crap this whole death thing that has heretofore only happend to Old People is starting become relevant to little ol' me...
Definitely agree. But similarly, we expect ITER to turn on someday within our lifetimes despite the repeated delays, precisely because those same experts and funders remain invested.
I think this is what you're looking for; it's a more quantitative look at what could be done with 10^12 more compute: https://www.alignmentforum.org/s/5Eg2urmQjA4ZNcezy/p/rzqACeBGycZtqCfaX. (For reference, there was ≈10^12 *less* compute in the 90s. Though note that, as you can see in the first graph, things have accelerated greatly in the past decade or so.)
My takeaway: we really, really, *really* better solve alignment long before we get 10^12 times as much compute.
In 1919, Sir Arthur Eddington traveled to Brazil and West Africa to observe an eclipse. There he took measurements of light deflection that would differentiate between Newtonian physics, which predicted a certain amount of diffraction, and Einstein's general relativity, which predicted twice as much diffraction. Asked ahead of time what he would do if the experiment proved his theory wrong, Einstein replied: "Then I will feel quite sorry for the good Lord. The theory is correct."
The moral of the story being that there are certain types of people whose intuitions are worth quite a lot even without or in spite of the apparent evidence, and that the AI risk and Rationality communities tend to attract those types.
My reading of that story was not so much that Einstein had great intuition, but that he already had lots of evidence for relativity. If the observations didn't match, then the balance of evidence would still point to relativity being correct, and the observations flawed in some way.
"that the AI risk and Rationality communities tend to attract those types."
And your evidence is? This is quite a claim!
Mmm. I wrote that late at night, and I don't feel so good about it now that I've slept.
I don't think the claim that tech X will be developed, where X is consistent with physics, doesn't seem to require unreachably vast amounts of resources, and has a research effort towards it, is extraordinary.
The process of producing AGI is the same sort of research and development used to produce other tech. You don't think the chance of fusion in the next 50 years is millionths of a percent, do you?
The default uncertainty is 50%. Millionths of a percent requires very strong evidence against AGI.
AGI being extraordinarily powerful doesn't make the claim "AGI will be produced" extraordinary.
To get probabilities that low, you would either need a convincing reason why AGI is physically impossible, or a convincing reason humans couldn't build it.
> Is there a convincing argument that AGI is possible within any reasonable timeframe
I would argue that there is not. Most of the claims about the Singularity rely on hand-waving and ignoring the laws of physics. The two primary arguments seem to be,
1). You can accomplish anything you want just by thinking really fast, and
2). Processing speed can be scaled up without any limitations.
Thus, the fast-thinking AI can just keep making itself faster until it achieves omnipotence. However, IMO both (1) and (2) are obviously false.
Here's my argument for very short timelines:
The idealized interactive proving system consists of a computer trying to find a proof for a given theorem by brute-force, and a human mathematician trying to intuit lemmata to use as stepping stones to make the brute-force search tractable. From how I do math, the mental machinery that the human uses here is pattern-matching, which deep learning is recently becoming very good at.
Trying to produce useful lemmata when you aren't given a theorem to prove produces useful conjectures. Therefore I expect that AI will soon surpass humans at producing maths papers. That's a victory condition.
I'm curious about "firms that increase their AI adoption" - does this include downstream adoption? For example, Facebook uses AI to target ads. An advertiser who gets a better ROI as a result has "increased their AI adoption" without even knowing it.
I'm not inspired to read the Acemoglu essay in question, but could it be that he's not the least bit concerned with future AI risk for the same reason that almost nobody else is concerned with future AI risk, which is to say that they are too busy being concerned about present political issues? Like, say, whether our economy is arguably too narrowly optimized for the well-being of the wealthy superelite and their elite STEM servants when everybody "should" be trying to ensure good, stable jobs for the working class.
If so, then Acemoglu has certainly noticed that there's a set of smart and motivated people, not wholly without influence, who are "wasting" their time on this AI risk nonsense when they could be focusing on the thing Acemoglu cares about. And, by connecting the two through the medium of current AI and nascent marginal technological unemployment, he thinks he can marginalize the AI-risk community as "out of touch" and get some of the marginal members of that community into paying more attention to the things he cares about.
If so, I agree that his article is unlikely to be enlightening or persuasive to anyone here, but it might at the margin serve his goals (and those of the WaPo editorial staff).
There are a lot more people in politics and tech who care about inequality than AI risk lmao. They may not be caring about it effectively, but ... I mean look at EA donations or charity of tech people as a whole.
It's the year 1400, and you're living in Constantinople. A military engineer has seen gunpowder weapons get more powerful, more reliable, and more portable over the past two centuries. He gets on a soapboax and announces: "Citizens of Constantinople, danger is upon us! Soon gunpowder weapons will be powerful enough to blow up an entire city! If everyone keeps using them, all the cities in the world will get destroyed, and it'll be the end of civilization. We need to form a Gunpowder Safety Committee to mitigate the risk of superexplosions."
We know in hindsight that this engineer's concerns weren't entirely wrong. Nuclear weapons do exist, they can blow up entire cities, and a nuclear war could plausibly end civilization. But nevertheless, anything the Gunpowder Safety Committee does is bound to be completely and utterly useless. Uranium had not yet been discovered. Lise Meitner and Otto Frisch wouldn't be born for another 500 years. Nobody knew what an isotope was, and their conception of atoms was as different from real atoms as nuclear bombs are from handgonnes. Rockets existed, but one that could deliver tons of payload to a target thousands of miles away was purely in the realm of fantasy. Even though the Roman military engineer detected a real trend--the improvement of weapons--and even though he extrapolated with some accuracy to foretell a real existential threat, he couldn't possibly forecast the timeline or the nature of the threat, and therefore couldn't possibly do anything useful to inform nuclear policy in the 20th century.
A more reasonable military engineer tells the first engineer to focus on more pragmatic and immediate risks, instead of wasting time worrying about superexplosions. Cannons are already powerful enough to batter down all but the strongest city walls, he points out. In the near future, the Ottomans might have a cannon powerful enough to destroy Constantinople's walls. How will the Roman Empire survive this?
I’ve never really liked analogies as arguments. For explaining and referencing, they’re pretty useful, but for an argument, there are a lot of properties that are true for the analogy (gun powder/explosions) that are different for the actual subject (AGI).
In the nuke analogy, I'd say we're more analogous to 1900 than 1400. and the first engineer might reply that if the city gets nuked, nothing the cannons did will really matter. That doesn't mean you need to drop everything and focus on nukes, but it would be nice for everyone to get together and declare that at X moment when we Definitely Still Have Time To Succeed And Don't Have To Worry About Mistakes Or Black Swans, we will all drop everything and start worrying about nukes Together.
Even here though, what are you going to do? What meaningful research that will actually do anything about the problem of "Nuclear Weapons Risk" can you do in 1905? Sure you may point at the problem, but that's not going to do anything.
Well, as far as I understand, the original vision of Yudkowsky was to avoid creating weapons at all, and focus on developing safe nuclear power before anybody else makes a first weapon. He even tried to make some arguments as to why this isn't necessarily a doomed enterprise, combining them with arguments of why failing to do so means certain extinction. The whole thing can be summarized as "humanity is almost certainly doomed if it continues on its current path, but if some selfless heroes (we, that is) manage this unprecedented heroic feat, there will be a paradise instead". It was persuasive enough that MIRI still has a growing multi-million dollar donation-based budget, but this certainly wasn't a mainstream-palatable narrative, which was also acknowledged.
Considering the absurdly high payout of being the only country with nuclear arms, that was never going to happen. Is AI the same? Probably - you can argue however much you like, and then China builds AIs anyway.
Back in the OB/LW heyday of 2008-09 it was slightly more believable. AI winter was still in full swing, with NN breakthroughs still years away, and contemporary wannabe "AGI developers" obvious clueless cranks. I'm curious as to how the "party line" has shifted since then, but MIRI is pretty tight-lipped these days.
But the nuclear weapons did not suddenly decide to bomb two Japanese cities of their own accord, it was humans that did that. Nuclear bombs that just sat in the arsenal gathering dust as a monument to "we cracked the problem of how to make these" aren't a threat; humans deciding "let's make something REALLY BIG that can REALLY DESTROY EVEN MORE" are the problems, and are the problem be it "our most cutting-edge tech is a pointy bit on the end of a length of wood" or "we have swarms of killer robots, a.k.a. drones".
Work on the human problem before you start wetting your knickers about "but what if a computer suddenly becomes self-aware?"
What I mean is, right now it's people who are working on AI. The idea of "and the AI will reach a point where it will recursively improve its own intelligence by rewriting its source code over multiple iterations in milliseconds and then it will become Silicon God" is pie in the sky when you die.
What do we, *humans*, want out of AI right now? And so far as I can see, it's "if we're the first ones to get this to work, it means MOAR MONIES". 'This work could imperil humanity if it goes too far' versus 'yeah but it'll bump up the value of the stock' and the better share price wins, every time.
We're greedy, stupid primates and the biggest threat. We could stop all the AI work right now, if sufficiently scared - you don't have to worry about Unfriendly AI if the most complicated machine intelligence is what tries to get you to retweet something. But we seem to be locking ourselves into "well, this is gonna happen, because we want our shiny toys".
*That's* the problem there, friends. "We can't stop because we have to have even bigger, better, shinier toys that make us richer" and if that means a risk of enslaving all of humanity to turn them into paperclips, well, look how my stock options are even more valuable!
This. Not only is it more sensible, you may even get people to go along with it. It's a virtual certainty that developments in AI will cause all kinds of issues, which should be tackled the way we tackle other problems.
Meanwhile, arguing about Superintelligence or The Singularity is a pointless exercise, not only because it's purely hypothetical, but also because we're not in a position currently where we understand enough for the arguments to have a decent chance of applying.
Sure, but you could place the exact same explosives safety committee at the beginning of WW2. Its hard to tell where we are on the timeline to AGI. We can't say with confidence it is 500 years away.
Also, the gunpowder safety committee in 1400 could plausibly have invented the concept of Mutually assured destruction and arms races.
Some practical discussions like "how do we achieve a perfect air-gap" is certainly among what we can do already. Formalizing MAD theory was one of the very few things academia could deliver against Nuclear Weapons Risk. This is yet another strike against AI risk research - even if you're *completely* right, will that actually help in a situation that's driven by money and power?
If the buzzword “Artificial Intelligence” was never created and we just stuck with “Machine Learning”, would we still be talking about the existential threat of Machine Learning?
yes
It is so weird how so many people, even in this thread, reject out of hand the idea that AI much more complex than current ones could exist. Like, we have an AI that can win at go and Starcraft, we have good text and image based AI, self driving car AI that works very well almost all the time, where else do people think that goes?
Although I tend to agree with you, there are good reasons to think AGI is still a long way off. For example, training AIs currently requires massive amounts of data--far, far more than humans use in their own learning. Massive data sets don't exist (and can't be created) for everything, and that could be a barrier to AGI.
I am optimistic that algorithmic improvements will be discovered making AGI feasible, but I think it's also fair to say the current tech doesn't have a path to scaling into an AGI.
Yeah, I have no idea in any sense negative or positive about when AGI lol
However that doesn’t mean it won’t happen nor does it prevent other AI things from severely disrupting people and society
Anyone genuinely worried about “AI mortgage selection will menace black people” really isn’t considering even the consequences of currently available technologies which are quite a bit larger
I think we know that bigger models use data more efficiently, so presumably when current models get as big as the human brain they will also be able to use data as efficiently.
I guess it's due to the nature of the fact that these are all relying largely on statistical function estimation techniques computationally similar to each other that they get grouped together as "AI" as opposed to any other more boring kind of math, but language understanding does not seem in the same class with vision and navigation. Plenty of animals not generally considered more intelligent than humans that have not managed to take over the world can nonetheless see better than us, and better navigate more complex terrain. Birds figured out how to fly and circumnavigate the globe millions of years before we even existed.
That seems to raise the question of whether figuring out how to construct structurally correct human language sentences with genuine semantic content is actually categorically different, though, and if so, why?
> but language understanding does not seem in the same class with vision and navigation
I’m really not sure that’s true. You could just as well argue they all work decently similar, similar primitive, language use evolved from others.
And at any rate the same ML stuff seems to do quite well for all of those
I mean that the ability to communicate in complete sentences and write them down for future generations to build an ever-increasing store of knowledge on top of is what actually makes human intellect a tool that can build civilizations. Our vision and navigation capabilities are doing comparatively very little for us. Not all levers exert the same multiplier just because the computational methods we've discovered thus far for getting machines to mimic them is structurally similar.
Similar to the age-old theme here about how important IQ is. Whatever IQ is actually measuring, it doesn't seem related to vision or sense of direction. Very smart people often wear glasses.
I mean if humans lost their eyes or legs 5k years ago that would’ve been it, long tradition of cultural evolution or not. To say nothing of, again, the strong overlap between the mechanisms both principled and literal between understanding creating speaking seeing etc
IQ could very well be correlated with visual perception skill, if not “eyeball curvature precision”
I guess I'm not getting across what I'm trying to say here.
Language understanding is categorically different from vision and navigation because:
1) Vision and navigation don't distinguish the only animal to create civilizations from all other animals. Many non-human biological organisms have equally good and in some cases much better abilities to sense and navigate their environments, not only immediate environments, but some animals have seemingly managed to maintain accurate mental maps of virtually the entire globe.
2) Language understanding, on the other hand, gives us much more. At bare minimum, the ability to transmit information across time and space to distribute the learning workload across all humans who will ever exist, and the ability to bootstrap the mental starting point for new humans who no longer need to learn what earlier humans already learned and wrote down. As far as we know, humans are the only animal to achieve this ability and it has enabled us build civilization. It is possible that some cetaceans may be intellectual equals but can't create much in the way of technology due to the engineering challenges posed by living underwater.
This obviously makes language understanding categorically different from vision and navigation in the sense that it is possible to have the latter two without having the first. Whether it means you necessarily need some fundamentally different algorithm to implement it, on the other hand, I have no idea.
But I suspect yes. Obviously, humans don't learn to speak by reading every text transcribed to the Internet billions of times. We're much closer to the already-trained versions of language models that are able to implement few-shot or one-shot learning by bootstrapping from serialized pre-recording of earlier models. But that isn't how we did it, either. Our brains now don't start out at birth much differently than when writing first arose, so the ability to learn language quickly from few examples clearly can't have come from the first humans being trained on the prior language examples of some earlier civilization. Human language understanding, in contrast, is an ability to map sounds and symbols to mental abstractions associated with memories of other sensory input, as in the word tree and the associated sound of it being spoken are really an association graph between all sights, sounds, smells, and touches we can recall of actual trees. The association graphs predate the existence of the linguistic representation and language arises from them. Human were never taught language. We invented it.
Whether this matters or not, I obviously also don't know. It's a fundamental unanswered question at the heart of AI whether true understanding has to involve knowledge representation being synthesized from first principles or whether just beating a universal function approximator into submission with trillions of training iterations until it passes an authenticity test is enough.
Is GPT3 categorically different from Alexnet? Is that categorically different from 1990s neural nets?
Even if you can’t do language good with current backpeopqgation massive neural network transformer stuff, that probably isn’t how vision worked in animals either. Ravens don’t have language, chimpanzees don’t, they can still use tools and be smart.
Nerds since Kurzweil have described the dawning of "Artificial General Intelligence" as an event horizon; everything before it and leading up to it is kind of trivial.
Acemoglu, Pinker, et. al., seem to be saying something more like: AI is getting continuously better and we should continuously evaluate how it affects us. We are already living with AI, we can talk about its benefits and costs without worrying about the day after Revelation.
There is a good analogy here with climate change: you don't need to look one hundred years into the future, you can just look around you right now to figure out that something bad is coming.
I would like to ask a question unrelated to Acemoglu, and not knowing a lot about AI, if possible. Articles often are talking about "the" AI. Isn't it more likely that there will be a lot of different AI's, some dangerous to humans, some friendly, and they will discuss or fight each other, and humans will be negligible to them, treated friendly or bad, like animals to humans now?
Not everyone agrees with this, but there is a line of reasoning that argues if for a scenario where as soon as one AI reaches a certain level of superintelligence, it will be able to recursively self-improve faster than all other systems and quickly take over the world. Hard Take Off is often the terminology used to describe this potential phenomenon. e.g. https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff
> But all the AI regulation in the world won’t help us unless we humans resist the urge to spread misinformation to maximize clicks.
I realise that 'AI regulation' here means 'regulation of social media algorithm AI' and not 'regulation of us by AI', but it touches on something I often think about when the subject is brought up. The prospect of an AGI capable of gainsaying our worst instincts would be such a huge boon that the risk of singularity may be worth taking, not unlike the residual risk associated with nuclear technology.
From the perspective of someone who would like to see humanity get central planning and collectivism right one day - major problems being the computational problem of allocation of resources and basic human corruptibility - an artificial auditor that can say 'Sorry, comrade, I can't let you do that' would be an incredible step forward and perhaps less vulnerable to misuse than advanced narrow-AI.
I think your reading here is grossly unfair - the argument is surely "let's worry about the actual problems we see today already, not the highly hypothetical Evil AI situation". That is, an argument for what should be *given attention* to.
Mandatory xkcd: https://imgs.xkcd.com/comics/robot_future_2x.png
I think this is more a difference in rhetoric and audience than a disagreement about the facts. Daron Acemoglu is writing for people who live in a world filled with clickbait articles about some far-future killer AI and a pop culture where AI is synonymous with Skynet. He's saying: "This isn't some hypothetical future thing, it's a real problem and it's happening right now!" It's only in this context that he's saying we shouldn't worry about AGI. Scott Alexander is writing for people who already think narrow AI is dangerous and trying to convince them to also be worried about AGI.
Imagine there's a small fire in a building. Some people aren't worrying about it because they think that only large fires are dangerous. Other people aren't worried because they don't think small fires can become big fires. If we want to convince people to take action and put out the fire, we need to use two different contradictory arguments. "Forget about giant fires, small fires can be really dangerous, too!" "It looks small now, but forget about that, it has the potential to get big really fast!"
You could argue that the first group is wrong or have the wrong priors or whatever, and maybe you'd be right, but if we want to put the fire out, that means meeting people where they are and convincing them using arguments they'd listen to.
My line of work has increasingly come to rely on "AI"/"ML" techniques, so I'm somewhat on the pointy end of this. The employment effects are real, but also really complicated.
For example, I co-wrote a piece of "AI" software (I don't think it should be called AI but the competitors call their software AI, so...) that explicitly replaces a LOT of human labor, including my own. It can replace the work of weeks or months with the work of a few days, and that's my work or the work of my direct reports - not hypothetical or knock-on effects.
But, it doesn't lead to me doing any less work, because in the past the prohibitive labor investment meant that companies simply did a lot less of the thing. They'd look at the price of doing a comprehensive program, throw up in their mouths a little, and choose something more achievable. So, now we do a whole lot *more* of the thing, because now the value proposition is better! My utilization on the thing has only risen, despite vast leaps in time efficiency.
I assume that many AI employment effects will be like this - it will be true that the AI tool replaces a lot of human labor for the same unit output, but NOT necessarily true that the total human labor spent on the thing thus decreases.
The main concern is that automation (including AI) will steadily eliminate the only jobs available to people with IQ below the ever-climbing cutoff value, a rising tide of obsolescence. Those smart enough to develop said AI are least-concern, until it starts to develop itself that is.
Sure, so for example I went to an open house last night for a company that's working on building earthen structures using drones. The drones are trained with ML techniques, so we can call it an AI application.
Explicit labor replacement by drones, right? But it still takes labor to do the dozens of things other than "put structure in place" and they expect to push structure costs down quite a lot. Up to 30% less than stickbuilt if they hit their targets.
So, it could be the same situation - a few framers are displaced, but the lower total cost to build makes more projects viable, which leads to more construction, which actually ends up increasing total employment.
"extremely smart people and leading domain experts are terrified of this"
I think this is the weakest part of Scott's argument. Peoples fears and imaginings have precious little to do with their smartness. Whether or not someone worries about population growth (or decline) or climate change or AI risk has more to do with their personality than with their intelligence.
In a similar way you can find a hundred very smart economists who worry that raising the minimum wage will have negative effects and another hundred who worry that not raising it will perpetuate negative effects. I use that example because it was written about most clearly on SSC.
The fact that some very smart people worry about something tells me almost nothing about the thing being worried about.
I don't think "smart people and domain experts are concerned about X" lets you infer "X is definitely a problem with very bad consequences", but it does seem pretty safe to infer "if X is in fact good, this is pretty non-obvious, and worth thinking carefully about before making any confident assertions."
The economist disagreements do tell you something - it doesn't mean you know whether a minimum wage is good or bad, but it does mean that the meme your grandmother shared on Facebook probably doesn't suffice to settle the issue, and your own opinion is probably little better than chance unless you have actually thought about economics a decent bit and read some relevant studies. (Contrast with e.g. worrying about a global conspiracy to cover up a flat Earth - very few smart people and domain experts seem concerned about this, so dismissing it on the basis of relatively simple arguments is less epistemically worrying.)
I think that's very dependent on the field. If social psychologists or criminologists are very concerned about some social psychology or criminology related thing, I don't consider that strong evidence. If cosmologists are very concerned about a cosmology thing, that'll make me pay attention.
Your prior on how trustworthy the authorities in question are matters a lot.
> "I certainly don’t mean to assert that AI definitely won’t cause unemployment or underemployment (for my full position, see here). I think it probably will, sometime soon...."
It's been a minute, but I recall your argument in the technological unemployment post being similar to ones put forward by Sam Harris, Andrew Yang, Robin Hanson, and many others; AI will be better than us at almost everything, so most people won't be able to work anymore, so more inequality will result/most jobs will disappear.
While it's true that AI isn't literally the same thing as [insert past technology here], it's also true that comparative advantage is a thing, and that AI is just one more technology that lets us take advantage of comparative advantage for the creation of more goods/services.
More on that here: https://jonboguth.com/our-glorious-future-more-robots-more-human-flourishing/
>I feel the same way about Steven Pinker... <
Seriously ?!?
Pinker is notorious for pretending the difference between hunter-gatherers and pre-state agricultural societies does not exist, even though this difference is crucial for the topic he is talking about, just because it would make an ugly dent into his beautiful graph on steadily declining violence. That is not a minor lapse, but an epic fuckup that should have gotten him tarred and feathered and laughed out of harvard.
I guess you should not be surprised that a person who is willing to sacrifice the truth for a beautiful graph on a topic you don't care about is willing to sacrifice the truth for a beautiful argument on a topic you *do* care about.
We really need to hold public intellectuals to higher standards.
Unrelated to long term AI risk, but he seems to take as a premise that AI is 'concerning,' then points out that AI is actually being used to do things, which is therefore supposed to be more concerning. Maybe AI is better at making sentencing decisions or bail decisions. Why is this automatically concerning?
Also, I'm no economist so maybe I'm missing something, but his point about AI not generating a labor-demand boost seems like bad economics. If substituting AI for humans doesn't boost productivity, then why is anyone using it? Presumably, it's cheaper, which drives down costs and prices and therefore boosts demand for other (human-made) goods/services. That's the main reason labor-saving devices boost demand for other professions, right? Agriculture technology ultimately caused demand for farm labor to collapse, rather than boosting demand for farm laborers by making them more productive. And yet unemployment didn't rise to 90% as a result.
1. Some people say that AI might be dangerous in the future.
2. But AI is dangerous now!
3. So it can’t possibly be dangerous in the future.
4. QED!
Here I thought we were steelmanning things, not strawmanning ...
The argument is more like yes, nuclear war could result in nuclear winter ... but also we will all be nuked. Why are you worrying about the winter instead of ya know, the billions that would die immediately? Climate change could result in a runaway feedback cycle of methane release that turns the world into Venus - but is that really the main issue that should be framing the debate?
Why not look at the actual pressing problems instead of conjecturing ill-defined bogeymen to be worried about?
1) There is limited attention and limited ability for the public to process what "the problem with X" is. To the extent that one defines the problem with X as Y, it does indeed downplay all the other problems and can be counterproductive.
2) Like mentioned at the end, the risks of superintelligent AI and the risks of ubiquitious big data and machine learning are really two separate things entirely, not just timescales or magnitudes of the same thing. Conflating the two, then, is more damaging than the usual.
The biggest problem I have with these AI doomspeakers is just how little they understand the state of AI today. The vast majority of it is nothing more than monkey see, monkey do.
The best example I have personally seen of this is the Watson AI that beat the best Jeapordy champion. It was a multiyear process of development that certainly cost tens of millions of dollars (or more), but the resulting Jeapordy champion couldn't even be redirected to understand the basic gist of internet articles. When touring the Watson demo office in SF before COVID, there is a demo room in there where this AI is showing the top trends in the internet worldwide.
One monster problem was immediately apparent right away: there was a location near the tip of South America which was in the top 10 sources of news at the time - which I believe was the original election of the Socialist in Bolivia prior to his ouster.
A 2nd example was soon forthcoming: Trump! This AI had overall internet news coverage of Trump being 55% positive - which seemed clearly wrong so I asked if it could drill down. The 10 actual article in the next level down - 5 had the word "Trump" in it but had nothing to do with Donald Trump and weren't clearly positive or negative. I recall one being someone named Trump being a key part of a winning middle school sports team...
A 3rd example: MIT review had an article talking about an effort to visualize what AI neural network, image recognition algos were actually seeing. It popped up with some eye-catchingly stupid stuff like a NN network image recog program believing all barbells have an arm attached, because obviously there were no pictures of a barbell by itself in its training silo.
Thus I am far from convinced anything we see today in "AI", actually machine learning or neural network programming, is actually intelligent or that the entire field is very far removed from 12th century alchemists mixing actual chemicals with animal dung, ethers, griffon feathers and what not.
Agree on actual AI - but we don't have that. Until we do, I see all of these articles on "AI is killing jobs and we should accept our robot overlords in various ways" as nothing more than PMC plus oligarchy excuse making for their utter failure in leading American political and economic systems astray for several decades and counting.
I should add more on the "tip of South America" thing. I'd bet money these were various agitprop tweets against the Bolivian socialist elected President.
If GIGO - what do you call something that takes in the entire body of Twitter for input?
Did you know the inventor of the loudspeaker himself for the rise of fascism as without it charismatic leaders with never been able to address such large crowd
I would like to tell you - in the case of nuclear power, the Bad People very clearly won - and it's why we care about climate change today.
Is it possible that Daron is doing a novel form of Roko's Basilisk here?