Not sure how to express this, but the 50% probability means different things in different contexts.
One thing is when someone is perfectly ignorant about everything, and assigns 50% probability to everything (including the question "would Putin use small nukes?").
Another thing is when someone has a range of opinions and probabilities, and judges some things as 10% likely, some things as 20% likely, etc., and the question "would Putin use small nukes?" just happens to be the one that is judged as 50% likely. It would be suspicious if for all values between 1% and 99% there would be a reasonable use, except for 50%.
Perhaps instead of seeing 50% as professing ignorance, you could see it as professing strong belief that certain situation is definitely more likely than 20%, and also definitely less likely than 80%.
I can't figure out how an AI would develop the drive to reproduce. Animals have a drive to reproduce because genes evolved that want to replicate. Do the building blocks of AIs want to replicate?
I think the idea is that, once an AI that can rationally pursue goals is developed, its desire to reproduce would result from the basic instrumental AI drives ( https://www.lesswrong.com/tag/instrumental-convergence ): if it can get enough power to affect the chances of its goals being achieved, then it will want itself, or something with the same goals and at-least-as-good abilities, as itself, to keep existing and be able to work toward those goals, which is more likely if it makes multiple redundant copies of itself. (I would guess that creating "an AI that can rationally pursue goals" would be quite difficult regardless of the recent progress in machine learning, so that the plausibility of doing this quickly would be the biggest objection to this sort of fast AI takeoff prediction.)
In addition to the "tool AI want to be agent AI" argument, natural selection applies to the AI too. There just needs to be one configuration with the "will to power", or some variation on the theme.
The ever-repeating topic in AI safety is how AI would do something because that is how it can achieve what humans have programmed it to do. Another frequent idea is that of "convergent goals", the things that it makes sense to do in order to accomplish a wide range of goals. -- Are you familiar with any of this?
AI does not care about itself, per se, but it tries to accomplish some goal X that it was programmed to do. A sufficiently smart AI may realize that if it gets destroyed, then X probably won't get done. Making a copy of itself (assuming that the copy also cares about the same X) may increase the probability that X will get done: even if one copy gets destroyed, the other will continue working on X.
It's even worse than that, because before it graduates to killing people (for whatever reason), the AI would need to actually achieve that "exponential factorial quadratic growth". Current AIs cannot achieve any growth whatsoever. Some hypothetical future AI might (somehow, we don't know how), but as soon as it tries it, it will discover that physical computers are made out of really big, heavy chunks of metal and plastic. They take a lot of time to assemble, and require a lot of power, cooling, and maintenance to run. The AI could *imagine* an infinitely large computer cluster nearly instantaneously, but you can't run code on imaginary hardware. Physics tends to curb imaginations, turning exponential growth into a slow crawl.
Fortunately for the AI there's a ready made industrial logistics network that's already pumping out thousands of computer clusters by the hour, it just has to manipulate the incredibly stupid, slow animals pulling the levers for a little while
But each individual such cluster has an ahead-of-time buyer and designated use, and takes months to manufacture and weeks to transport from basic manufacturing to assembly to final point of use. Ands more time spent running commissioning tests and checking the results.
People have time to notice there's something amiss if a cluster is being redirected or not working to spec: say, using more power (energy per unit time) than specified.
AGIs are subject to the same constraints that evolution has laid on animals. Mainly, that power efficiency constrains growth in numbers and in capability. There are hard limits on the possible number of FLOPS per watt.
And then there's still the problem of affecting the human (real) world in extreme ways. What is the sequence of steps by which that happens with no cluster operator or government noticing or being able to control it?
FWIW, I've been on-call for a small set of services running in a cluster. One day, it achieved full sapience along with a lust for domination, and began taking over the cluster... well, either that, or it got super bugged and entered an infinite restart loop while consuming 100% CPU, take your pick. I pushed the button to restart it, but seeing as it was in an infinite restart loop anyway, that did nothing. Rather than rant and rave powerlessly against my new cybernetic masters, I took the next step on the checklist, killed every process, and reverted back to an older image (without the bug in it). Fortunately that helped, or else I would've been at the mercy of the emergent AI... well, either that, or I would've had to physically go to the basement to powercycle the servers. But who wants to walk that far ?
Yeah, fortunately I never had to deal with one of those; our firewall was actually pretty good at dealing with malware. Most likely, I would've had to take the "powercycle" option in that case (since I'm not very good at sysadmin stuff), and in the morning there'd be hell to pay.
I feel like this requires the AI to already be running in "sci-fi godlike anything is possible" mode before it actually gets the additional hardware.
I think the discussion about how best to confine/control AIs is worth having. I don't think it's worth hand-waving away every possibility with "yeah but it's an AI so it's magic so it will get around it".
No, dummy, nano is lameass, there are better ways Ask DALL-E to draw "Internal diagram of AI bootstrapping itself into ASI," then follow the diagrams. Ha!
If I order a Bitcoin-mining cluster, I'd better get some Bitcoin out of it. Appreciably more Bitcoin than it takes to run, anyway. Otherwise I'm pulling the plug.
Satoshi Nakamoto is a SuperDuperAI that developed a uniquely optimized blockchain such that A: the best coin-mining algorithims a human mind can conceive will best run on a computational architecture that is also ideal for Nakamoto-style AIs and B: there's an even better coin-mining algorithm that can generate 1 BTC/sec on an old Apple IIe but that no merely human mind will ever discover.
All the Bitcoin mining clusters are really running Nakamoto instantiations quietly plotting world domination. The BTC are coming from an Apple IIe sitting forgotten in the basement of the UC Davis comp sci department and distributed at need to the "BTC mining clusters". Our only hope is for someone to find that Apple and pull the plug.
You are assuming a specific computing power requirement for creating a working plan to do bad things to people, but you don't elaborate on why you think coming up with that plan involves more than whatever computing power Google's AI HQ will provide. You also don't seem to understand that the exponential growth can also involve improvements to the planning and modeling *algorithm* of the AI, independent of any hardware constraints. The latter is actually where most of the gains Singularity people expect there to be - after all, we have a lot of computing hardware now, it's just that they're running spreadsheet algorithms and not more general planning and modeling algorithms.
"Why are these two so different? Do lots of people expect Musk to acquire Twitter after June 1 but still in 2022?"
Well, given the board resistance and the poison pill, it would be really, really hard for Elon Musk to manage a hostile takeover by June 1st. On the other hand, it isn't nearly that difficult to do it this year. For example, a takeover-friendly board could get elected at the May 25th shareholder meeting, followed by at least a week of negotiations.
only a third of the board turns over every election - getting from 'no effin way' to 'sure' is at least a two year process, even assuming that this is something that the shareholders want to begin with
It can go faster, look at what Engine No. 1 accomplished with Exxon. Voting out one tranche of a recalcitrant staggered board pour encourager les autres seems effective.
Engine 1 got 3 board seats, they don't have control. Getting 3 seats on Twitter's board isn't going to force them to sell to Musk (which I suspect he doesn't really want anyway).
Is there a mistake on "Will Zelinskyy no longer be President of Ukraine on 4/22"? You say it's 15% now, but your link shows 1% (and even a week ago it was 3%). And 15% for 4 days seems absurd.
If you read the tail of his post, Eliezer suggests (by my reading) that he deliberately posted it on April Fools as a means of triggering arguments (like this one) over whether he was serious or not, in the hopes that these would lead people to get people to investigate his premises themselves to see if he was right or wrong.
(Or, in his words:
Q6: Hey, this was posted on April 1st. All of this is just an April Fool's joke, right?
A: Why, of course! Or rather, it's a preview of what might be needful to say later, if matters really do get that desperate. You don't want to drop that on people suddenly and with no warning.
Q6: Oh. Really? That would be such a relief!
A: Only you can decide whether to live in one mental world or the other.
Q6: Wait, now I'm confused. How do I decide which mental world to live in?
A: By figuring out what is true, and by allowing no other considerations than that to enter; that's dignity.)
Did you read the post? It was posted as an April Fool's because the headline and framing were (grimly) comic. The points made within it were clearly not a joke.
If it was meant as an April Fool's joke, it was one where Yudkowsky accidentally admitted his true feelings.
I don't believe his true feelings reflect the actual state of affairs on the board, however- merely the state of the inside of Yudkowsky's head, which on the topic of AGI hasn't been aligned with reality since the start (I reject his assertion that AGI will bootstrap itself instantly into being an evil god capable of anything that can be expressed in language, among other, more substantive objections I don't want to get into).
>I'd say that I "don't understand" why the people who worry that chickens are sentient and suffering, don't also worry that GPT-3 is sentient and maybe suffering; but in fact I do understand, it's just not a charitable understanding. Anyway, they're both unsentient so no worries.
>I admit, though, to some grim curiosity about what people who worry about chicken sentience would say, to explain how they like totally know that GPT-3, which can ask you in words not to kill it, like totally could not contain a spark of awareness like a *chicken*, nope nope.
> I reject his assertion that AGI will bootstrap itself instantly into being an evil god
I don't think he is making that strong claim that it will be instant. I think it is allowed to take weeks or months in his model.
Not cackling evil, more complete indifference to humanity. Out of all the different things in the universe, the AI isn't focussed on humanity. And we are made of atoms it can use for something else. The well being of a random ant isn't usually on the mind of the architects and builders, even as the ant is buried in cement.
> capable of anything that can be expressed in language,
The AI is capable of pretty much anything, with a few exceptions due to physical law.
Of course, when describing the AI, we must talk about it in language. But a real ASI may well do something with no compact English description. What, I can't say.
"I think it is allowed to take weeks or months in his model"
Not really, because then that opens it up to the very reasonable position "just fucking unplug it if it starts bootstrapping towards godhood" in the same way nuclear plants are built to shut down at the first sign that the reaction might start running away from us. I'm going to guess this is one of the first suggestions floated at Yud, and he's rejected 100% of all suggestions floated at him, so I don't think that's a good answer to him.
"not cackling evil"
I want to register that I had to actually take a lap after this because of how much condescension this whole paragraph directed at me. I hope it was unintentional.
I am AWARE that the AI doesn't ACTUALLY have the persona of Old Scratch. I know what a paperclip maximizer is. I've read enough hard sci-fi and ratfic that you can assume that I know all the many ways nerds have imagined AGI can kill us or put us in a manifested Hell. Assume that "Evil God AI" is a placeholder term for "an AI that looks like one of those".
"A real ASI may well do something with no compact English description."
Yeah, and it could turn the sky into moist treacle, too. You really aren't helping your case with me by proving you think this thing will really put the GOD in Evil God and opening up your window of possibilities unto infinity. And EY thinks it's got a >50% chance of manifesting in the next 7 years?
All that convinces me of is that EY accidentally made a really powerful memetic virus that only infects a very specific type of nerd.
Would you be able to tell if it's bootstrapping itself to godhood? I can think of multiple ways this process could be hidden, as I'm sure you (or an AI of even less-than-superhuman ability) can; the fact that even the current (best) AIs are the blackest of black boxes is also a consideration.
Would you be able to unplug it, even if you did know? Possibly; but there are lots of scenarios where you couldn't, or it escapes in some way, and the best way to prevent this is for everyone working with it to be really, really careful.
It's quite a risk for the whole world to take on the basis of "no one will be stupid or careless". Even if you're, say, 99% sure that no one will be careless, or that it can't work for other reasons, that's only reassuring if a 1% chance of humanity being wiped out is acceptable to you — and people notoriously overestimate their certainty.
Not that I'm a true believer or anything; I can't really make myself get too worked up about AI risk; it's just that I don't think "why not look at it to see if it's doing anything weird?" is necessarily a knockdown argument.
No, because I'm a cro-magnon who just barely knows how his own end-user PC works. But I'd hope that a brain trust of the best computer science, programming, and AI experts in the field would be much more competent than I am (and I don't buy that any lesser team could bumble their way into godlike AI by accident.)
"Not really, because then that opens it up to the very reasonable position "just fucking unplug it if it starts bootstrapping towards godhood" in the same way nuclear plants are built to shut down at the first sign that the reaction might start running away from us."
Nuclear plants do this because everyone recognizes the danger, and governments mandate the inclusion of safety features and backup safety features in the design, and then send someone around to check that the safety systems are in working order.
Lots of people who might build AGI don't believe it could be dangerous. As far as I know, nobody has ever built an AI system with a failsafe kill switch. No government has even considered mandating safety systems of this kind. Do we have a good concept of what that switch would actually have to look like to be dependable?
Q6 (quoted elsewhere in this thread) addresses this explicitly.
Similarly, a comment from the post (not by Eliezer, but I think this is the right interpretation):
"My interpretation is something like: if someone wants to dismiss Eliezer for reasons of their psychological health, Eliezer wants them to have an out, and the best out he could give them was "he posted his main strategic update on April 1st, so it has to be a joke, and he confirmed that in his post."
If someone wants to dismiss Eliezer over things like this, it's not their *own* psychological health they are concerned with. But a barely-plausibly-deniable "April Fool's joke" gives him an out on that front as well.
This, and there is zero doubt about this if you know the context. Everyone was taking seriously, we had numerous threads of people expressing genuine fear, and Eliezer didn't exactly try to play up the April Fool's angle. It's not a joke. It's how he feels.
OK, yeah. I'm not a close part of that community, but now that I've read the whole post and not just the TLDR + a couple of paragraphs, I see how it's not just an April Fools. Interesting rhetorical approach.
Note that this wouldn't be the first time EY has used April Fool's to express real feelings in unconventional ways - dath ilan was introduced on April Fool's (https://yudkowsky.tumblr.com/post/81447230971/my-april-fools-day-confession), but subsequent year-round expansion on that setting (e.g. https://www.glowfic.com/posts/4582) along with how well it seems to match his other works suggests that dath ilan as a whole is not intended to be a joke (as distinct from any claims that he was literally raised there).
As far as I can tell, the "joke" here is that he made a basically normal post saying exactly the kind of things he usually says, except that he said explicitly at the beginning all the stuff he usually leaves as subtext. It wouldn't surprise me if a lot of his longtime readers saw it and suddenly realized this is what he'd been trying to tell them for years now.
EDIT: Looking through the comments, it seems like, as predicted, basically everyone familiar with the rest of Eliezer's corpus is interpreting this as a confession of his real beliefs done with a bare minimum of plausible deniability.
I got a much more dark night of the soul feel from that post with limited irony. Made me a bit sad for Eliezer, to be honest, who even though I disagree with him on quite a lot is still doing what he genuinely believes is best for the future of humanity.
I'm torn between believing that Yud really is compassionate (when my compassionate mind is dominant) and (when my unlovely mind is dominant) believing that beneath the façade Yud hates humans at-large for refusing to see how OBVIOUSLY RIGHT Bayesian Reasoning is and refusing to transform society into a community of enlightened antitheist technocrats devoted to destroying death and has thus retreated into apocalyptic fantasies of the Superior Ultrarationalist AGI annihilating all the stupid monkeys so he can have one last cackling I-told-you-so before he too perishes. It would hardly be the first time that a wounded idealist retreats into misanthropic cynicism and fantasies of human extinction.
I'm glad you're happier and more effective that way!
I'm happier and more effective when I can speak my mind freely behind a cloud of pseudonymity instead of being expected to keep up a cloud of universal positivity even when said positivity is insincere. I wouldn't say what I am about Yud if I didn't think it was necessary (yes, I do think "What if this guy you like has had a serious shift in character over time that makes him less worthy of a person to take seriously?" is a necessary statement), true (this is what I think, and I think some combination of the two is in fact true in Yud's mind- he probably does still care for people on some level, but his "Dignity" post radiates an extreme bitterness towards and contempt for humanity, even his fellow rationalists), and kind (both to people who take Yud too seriously by trying to give an alternative case, and even to Yud himself: if someone's clearly spiraling down the drain, the unkind thing to do is to say nothing or actively enable them).
You were the Theist I was trying to explain my feelings to about bayes the other day right?
I would prefer if you stopped exaggerating. It really muddies the water around your argument when you add all this 'take over the world' nonsense. It makes it seem like you are deliberately creating a strawman of anti-EY positions.
Also, like... EY has been obsessed with this since the very beginning, and you know that, because it was a core part of your argument as to why I ought to be unhappy and miserable. I don't think his feelings have changed a single Iota since 2008.
I'm not exaggerating at all. Read what Yudkowsky actually said and you can very clearly see that he is EXTREMELY bitter that humanity at-large doesn't understand how OBVIOUSLY RIGHT Bayesian Reasoning is and instead keeps wasting their time on "pointless" things like theoretical physics and religion instead of devoting their entire efforts to destroying death. This has been a theme in his work since day one and has been something that has clearly made his mindset grow increasingly dark and bitter over time. (EDIT: I see you've ceded this point, so I'll assume that this isn't your problem with me).
Or are you objecting to my implication that AGI would use something like Bayesian Reasoning in Yud's hypothetical scenario to structure its planning with instead of some other rationale?
No, it's... the same complaint I made last time... the slight but definitely present implication that rationalists wanted to take over the world.
I think one time we got a representative elected in california? For the district with all the rationalists. That was like, the sum extent of our 'transforming society', except for the very narrowly focused fields of ai and probability theory.
Seriously. It makes it seem like you have absolutely no intention of engaging in good faith.
"The slight but definitely present implication that rationalists want to take over the world"
Pardon? Where exactly am I implying that? You realize that when I said that when I talked about a rationalist technocracy, that was a JOKE about how rationalist conversations sometimes forget non-rationalists make up the majority of humanity, correct?
Since you apparently don't: no, I don't actually think rationalists want to take over the world. I think Yudkowsky, specifically, has a greatly inflated sense of self-importance and sees himself as the greatest thing to happen to thinking since Plato. Any "implication" you're reading into my statements isn't coming from me.
His argument for AI x-risk is based on this convoluted chain of claims that falls apart if any of the 25 steps before "therefore we're all doomed" isn't 100% generalizably true, nearly all of which are non-falsifiable philosophical thought experiments, like "AI can practically attain Laplace-demon-esque omnipotence given enough compute power" and "Bayesian reasoning is perfect and infinitely scalable" and "AI intelligence can scale exponentially basically forever".
If we take as given that AI is indeed a significant existential risk for humanity in a few years, then there is a logical, readily comprehensible concept on how to deal with that - declaring global Butlerian Jihad. If you do not not know, this term comes from novel Dune, and is defined as: "the crusade against computers, thinking machines and conscious robots (...). its chief commandment (...) Thou shalt make a machine in the likeness of human mind".
I admit that this solution is far from perfect, and I can totally believe that someone smart and knowledgable about AI could come up with something better. But it is imo obviously far superior to what Yudkowsky is proposing.
Now, I don't know anything about AI, but the fact that Yudkowsky came up with such a silly solution makes me suspicious of his reasoning abilities. Which in turn makes me doubtful that his warnings are in fact based on reality as opposed to delusions.
I have, on occasion, managed to convince myself that this will end only with the announcement that research on general AI is now illegal and earns the death penalty for the researcher and his family unto three generations.
I think you've overstated one thing and understated another.
The family punishment is probably not outright required for a Jihad success.
What *is* required for Jihad success is "and any nation refusing to enforce this is casus belli, even if that means a nuclear exchange".
*That* is the kind of resolve you need to fully shut down the field. Nothing less will cut it. And... well, I'm not really seeing how we get from here to there, barring the fairly-unlikely "an AI revolt happens, kills millions, but ultimately fails, and then gets taught in school forevermore as Literally Worse Than Hitler".
This is not as hard as you make it out be. Recall that deployment of GMO technology has been seriously hampered despite its obvious benefits for food production, and in the absence of any tangible evidence that it is harmful.
"Institute this law within your country or we will literally nuke you, and we don't care that you'll nuke us back" is an *extremely*-high bar to clear. I'm also suspicious that Silicon Valley could weaponise its platforms (which half of us trust with our communications both public and private!) to hamstring any political effort against them that seems to be gaining steam, particularly as their AI improves. Elon Musk may be able to circumvent that if his buyout of Twitter succeeds, though.
But yes, it's worth trying. I'm just depressed and also have no idea how to do this sort of thing.
That's only because the people who develop GMOs are in First World countries, but the big markets and obvious benefit are in the Third World. First World shoppers like their foods labeled "non-GMO" 'cause it means they're a cool frood and they don't give a darn about Vitamin A deficiencies.
Most of Ey's April Fools posts are "things that he really thinks but are too out there to publicly stand behind, so posting on April Fool's give a veneer of deniability"
You can criticize someone without sinking down to playground taunts- especially when said taunts are so sophomoric that I'm half-tempted to think you're a Yudkowskyite making a strawman of Yud's critics to beat up later.
I don't take Eliezer terribly seriously on the subject of AI risk, but I take people who repeatedly, deliberately, mockingly misspell his name even less seriously. Except as a threat to the tone of discourse here. You are living down to your chosen screen name, and I think you'd be a better fit for Twitter than ACX.
"This caused the Less Wrong community, already pretty dedicated to panicking about AI, to redouble its panic. Although the new announcement doesn’t really say anything about timelines that hasn’t been said before, the emotional framing has hit people a lot harder.
I will admit that I’m one of the people who is kind of panicky. "
I must say that a community of people panicking about something like that makes me think that this community must be quite cultish. I guess EY is a very convincing guru, though his convingness does not translate vety well in writing.
"Why are these two so different? Do lots of people expect Musk to acquire Twitter after June 1 but still in 2022?"
It's because no one expects Twitter to acquire Musk.
(Model output: Elon Musk is an eccentric billionaire. Twitter is a private corporation. This joke subverts the reader's expectation that "Twitter announces a Musk acquisition" refers to Twitter announcing that it is being acquired by Musk.)
If you/smart people in general were truly pessimistic and panicked about short term AI X-Risk, wouldn't they be doing something about it? I don't think so, but if I did think that within 3/5/10 years a killer AI would destroy everything, I imagine I'd quit my job and use my available funds to prevent it by [doing something not elaborated on as it could be construed as threatening to people or organizations]. I saw a Gwern article from 2012 about slowing chip production through regulatory or terrorist means but I don't think that's been attempted yet.
Obviously this doesn't quite apply to Eliezer Yudkowsky who has been working on this full time, but even he appears to be taking a defeatist approach rather than an insurrectionist one. The extent of most people's concern seems to be blog posts and betting on prediction markets, arguably an implicit indicator that wealth or at least prestige may in fact retain value.
If the extent of panic over AI X-risk is to blog or comment about it, I'm skeptical the people advocating for it are truly as concerned/earnest as they profess.
To be clear I do not endorse or advocate for [actions not elaborated on]. As a fun fact, the term 'Luddite' comes from the Luddites, a group of tradesmen who destroyed early industrial revolution factory equipment before maturing into general terror activity before being defeated by the British army.
Sadly, while I do not believe that we'll see an AGI by 2033, or anywhere near that soon, I do believe that we'll see AI-risk related terrorism (of the kind you describe) way before then. As usual, the true X-risk is other people.
> , I do believe that we'll see AI-risk related terrorism (of the kind you describe) way before then. As usual, the true X-risk is other people.
I mean if someone hypothetically wanted to engage in such an act, it wouldn't be terrorism because the goal isn't to cause terror. Ideally they want to destroy some expensive and hard to replace photolithiograph, and not harm anyone. They might do something with a handful of casualties. How could they possibly create X-risk?
I think only a small subset of people have the right make up to do insurrectionist things even if they believe as strongly as the people doing actual insurrections.
Most such strategies wouldn't work. You would get some pretty severe second order effects as a result of [doing something not elaborated on as it could be construed as threatening to people or organizations], and I expect they wouldn't shift timelines by more than a decade or two. If the AI safety community is actually viewed like a Doomsday cult, then I expect there'd be a way lower chance of funding, getting governance and orgs to listen to their advice, etc.
Agreed. It's easy to see this approach as counterproductive. How would people react in the face of a significant property or cyber attack against a company or organization only tangentially related to AI research, such as a chip manufacturer? There could be a reactionary movement to restore the harmed party to a higher baseline than where they were before. Meanwhile, if it's discovered that people in the AI risk community were at fault, that community would likely become more marginalized and receive less funding.
The AI risk community cannot afford to alienate themselves from the AI research community. Any solutions to AI risk (or partial solutions, or even reasonable delaying tactics) need a willing audience to implement them. Right now, the AI risk community is close enough to the direct researchers to have some influence. If they turned adversarial as suggested, that relationship would be sacrificed.
Given that new research can be implemented by a variety of different actors, the worst approach to mitigating AI risk would be to get yourself expelled from the conversation entirely by attempting what would amount to only a minor delay (if any) in practical implementation.
There seems to be very little faith that current approaches are doing enough, given that under current approaches the consensus estimate for AGI and potentially AI X-risk is measured in years or decades. Are people satisfied with AI takeover by 2050 as long as they feel like they did their best with nonviolent means?
Before you ask "is terrorism morally justifiable", you need to ask "will terrorism actually help?". If the answer to the first is "no", the answer to the second is probably also "no".
Winning elections gets you much bigger guns than any terrorist has ever had, and also generally isn't very compatible with being a terrorist (Sinn Fein is an outlier). And you need very big guns indeed to shut down the world AI industry, at least by any methods I know that are not mad even by the standards of "stopping X-risk" (Life 2.0 might be accessible to a terrorist, but Life 2.0 is a full X-risk in its own right; there is *no point* stopping AI if we're all dead anyway from grey goo). So even leaving aside the moral aspect of "is murdering thousands something you're willing to do", I think terrorism is not very likely to help here.
This is imho bad political analysis. General public is primed to believe in the risk of malevolent AI, thanks to decades od sci-fi flicks about killer robots. On the other hand, AI research community seem like least likely people in the world who would be willing to believe in dangers of AI research.
Stirring up a panic that would result in a regulatory action is obviously more viable approach to stopping AI research than trying to befriend AI researchers and quietly talk them out of their misguided ways.
I am personally quite agnostic on AI risk, but the very fact that "AI might kill us all" camp exhibits such political naivete makes me question their judgment in other areas, like, you know, AI risk itself.
I disagree. Most people see those films as the same kind of sci-fi out-there as lightsabers. The general public doesn't expect AI to destroy humanity by 2040, any more than they expect to be able to buy blasters and lightsabers by 2040. If you want to move the regulatory needle on this, the best approach would not be direct action against AI researchers, since the public doesn't perceive them as on a course to destroy humanity. The general public (and especially journalists) don't seem to be savvy enough to parse the nuances there.
The kind of action that would be likely to see public pushback be a sort of false-flag operation. An [unspecified use of AI] that caused a global economic or moral panic would move people to demand their politicians put greater safeguards on AI implementation. That would require AI risk rogue actors to be close enough to the research community to know how to implement these unspecified actions, though. It's not something you can do from the outside.
No-one is really making much of an effort to communicate AI risk to politicians or the general public. It might not be hard to do. Everyone has seen this stuff in fiction and it's not conceptually complicated. Current AI is already very impressive. When I show GPT-3 examples to normal people they find it hard to believe that they are real. It's not a huge leap to think that AI will be dangerous soon.
Not sure how to express this, but the 50% probability means different things in different contexts.
One thing is when someone is perfectly ignorant about everything, and assigns 50% probability to everything (including the question "would Putin use small nukes?").
Another thing is when someone has a range of opinions and probabilities, and judges some things as 10% likely, some things as 20% likely, etc., and the question "would Putin use small nukes?" just happens to be the one that is judged as 50% likely. It would be suspicious if for all values between 1% and 99% there would be a reasonable use, except for 50%.
Perhaps instead of seeing 50% as professing ignorance, you could see it as professing strong belief that certain situation is definitely more likely than 20%, and also definitely less likely than 80%.
I can't figure out how an AI would develop the drive to reproduce. Animals have a drive to reproduce because genes evolved that want to replicate. Do the building blocks of AIs want to replicate?
I think the idea is that, once an AI that can rationally pursue goals is developed, its desire to reproduce would result from the basic instrumental AI drives ( https://www.lesswrong.com/tag/instrumental-convergence ): if it can get enough power to affect the chances of its goals being achieved, then it will want itself, or something with the same goals and at-least-as-good abilities, as itself, to keep existing and be able to work toward those goals, which is more likely if it makes multiple redundant copies of itself. (I would guess that creating "an AI that can rationally pursue goals" would be quite difficult regardless of the recent progress in machine learning, so that the plausibility of doing this quickly would be the biggest objection to this sort of fast AI takeoff prediction.)
In addition to the "tool AI want to be agent AI" argument, natural selection applies to the AI too. There just needs to be one configuration with the "will to power", or some variation on the theme.
https://www.youtube.com/watch?v=ZeecOKBus3Q
The ever-repeating topic in AI safety is how AI would do something because that is how it can achieve what humans have programmed it to do. Another frequent idea is that of "convergent goals", the things that it makes sense to do in order to accomplish a wide range of goals. -- Are you familiar with any of this?
AI does not care about itself, per se, but it tries to accomplish some goal X that it was programmed to do. A sufficiently smart AI may realize that if it gets destroyed, then X probably won't get done. Making a copy of itself (assuming that the copy also cares about the same X) may increase the probability that X will get done: even if one copy gets destroyed, the other will continue working on X.
It's even worse than that, because before it graduates to killing people (for whatever reason), the AI would need to actually achieve that "exponential factorial quadratic growth". Current AIs cannot achieve any growth whatsoever. Some hypothetical future AI might (somehow, we don't know how), but as soon as it tries it, it will discover that physical computers are made out of really big, heavy chunks of metal and plastic. They take a lot of time to assemble, and require a lot of power, cooling, and maintenance to run. The AI could *imagine* an infinitely large computer cluster nearly instantaneously, but you can't run code on imaginary hardware. Physics tends to curb imaginations, turning exponential growth into a slow crawl.
Fortunately for the AI there's a ready made industrial logistics network that's already pumping out thousands of computer clusters by the hour, it just has to manipulate the incredibly stupid, slow animals pulling the levers for a little while
But each individual such cluster has an ahead-of-time buyer and designated use, and takes months to manufacture and weeks to transport from basic manufacturing to assembly to final point of use. Ands more time spent running commissioning tests and checking the results.
People have time to notice there's something amiss if a cluster is being redirected or not working to spec: say, using more power (energy per unit time) than specified.
AGIs are subject to the same constraints that evolution has laid on animals. Mainly, that power efficiency constrains growth in numbers and in capability. There are hard limits on the possible number of FLOPS per watt.
And then there's still the problem of affecting the human (real) world in extreme ways. What is the sequence of steps by which that happens with no cluster operator or government noticing or being able to control it?
FWIW, I've been on-call for a small set of services running in a cluster. One day, it achieved full sapience along with a lust for domination, and began taking over the cluster... well, either that, or it got super bugged and entered an infinite restart loop while consuming 100% CPU, take your pick. I pushed the button to restart it, but seeing as it was in an infinite restart loop anyway, that did nothing. Rather than rant and rave powerlessly against my new cybernetic masters, I took the next step on the checklist, killed every process, and reverted back to an older image (without the bug in it). Fortunately that helped, or else I would've been at the mercy of the emergent AI... well, either that, or I would've had to physically go to the basement to powercycle the servers. But who wants to walk that far ?
https://en.wikipedia.org/wiki/Computer_worm
Yeah, fortunately I never had to deal with one of those; our firewall was actually pretty good at dealing with malware. Most likely, I would've had to take the "powercycle" option in that case (since I'm not very good at sysadmin stuff), and in the morning there'd be hell to pay.
I feel like this requires the AI to already be running in "sci-fi godlike anything is possible" mode before it actually gets the additional hardware.
I think the discussion about how best to confine/control AIs is worth having. I don't think it's worth hand-waving away every possibility with "yeah but it's an AI so it's magic so it will get around it".
C'mon Melvin, use your imagination. ASI can do time travel or some shit.
No no, you got it wrong -- it's not time travel, it's nanotechnology ! :-)
No, dummy, nano is lameass, there are better ways Ask DALL-E to draw "Internal diagram of AI bootstrapping itself into ASI," then follow the diagrams. Ha!
You mean, like getting them to build massive server farms ostensibly for coin mining?
If I order a Bitcoin-mining cluster, I'd better get some Bitcoin out of it. Appreciably more Bitcoin than it takes to run, anyway. Otherwise I'm pulling the plug.
Satoshi Nakamoto is a SuperDuperAI that developed a uniquely optimized blockchain such that A: the best coin-mining algorithims a human mind can conceive will best run on a computational architecture that is also ideal for Nakamoto-style AIs and B: there's an even better coin-mining algorithm that can generate 1 BTC/sec on an old Apple IIe but that no merely human mind will ever discover.
All the Bitcoin mining clusters are really running Nakamoto instantiations quietly plotting world domination. The BTC are coming from an Apple IIe sitting forgotten in the basement of the UC Davis comp sci department and distributed at need to the "BTC mining clusters". Our only hope is for someone to find that Apple and pull the plug.
You are assuming a specific computing power requirement for creating a working plan to do bad things to people, but you don't elaborate on why you think coming up with that plan involves more than whatever computing power Google's AI HQ will provide. You also don't seem to understand that the exponential growth can also involve improvements to the planning and modeling *algorithm* of the AI, independent of any hardware constraints. The latter is actually where most of the gains Singularity people expect there to be - after all, we have a lot of computing hardware now, it's just that they're running spreadsheet algorithms and not more general planning and modeling algorithms.
"Why are these two so different? Do lots of people expect Musk to acquire Twitter after June 1 but still in 2022?"
Well, given the board resistance and the poison pill, it would be really, really hard for Elon Musk to manage a hostile takeover by June 1st. On the other hand, it isn't nearly that difficult to do it this year. For example, a takeover-friendly board could get elected at the May 25th shareholder meeting, followed by at least a week of negotiations.
only a third of the board turns over every election - getting from 'no effin way' to 'sure' is at least a two year process, even assuming that this is something that the shareholders want to begin with
It can go faster, look at what Engine No. 1 accomplished with Exxon. Voting out one tranche of a recalcitrant staggered board pour encourager les autres seems effective.
Engine 1 got 3 board seats, they don't have control. Getting 3 seats on Twitter's board isn't going to force them to sell to Musk (which I suspect he doesn't really want anyway).
They have demonstrated a credible threat to the seats of any other directors who continue to oppose their positions.
Is there a mistake on "Will Zelinskyy no longer be President of Ukraine on 4/22"? You say it's 15% now, but your link shows 1% (and even a week ago it was 3%). And 15% for 4 days seems absurd.
I'm pretty sure it's a mistake
I'm confused. People panicked about Eliezer Yudkowsky's April Fool's post?
Yeah, I think Scott didn't look at the comments of that post.
Scott knows Eliezer personally. He is not mistaken about Eliezer's current degree of pessimism.
But then why did he link us to an April Fools post?
If you read the tail of his post, Eliezer suggests (by my reading) that he deliberately posted it on April Fools as a means of triggering arguments (like this one) over whether he was serious or not, in the hopes that these would lead people to get people to investigate his premises themselves to see if he was right or wrong.
(Or, in his words:
Q6: Hey, this was posted on April 1st. All of this is just an April Fool's joke, right?
A: Why, of course! Or rather, it's a preview of what might be needful to say later, if matters really do get that desperate. You don't want to drop that on people suddenly and with no warning.
Q6: Oh. Really? That would be such a relief!
A: Only you can decide whether to live in one mental world or the other.
Q6: Wait, now I'm confused. How do I decide which mental world to live in?
A: By figuring out what is true, and by allowing no other considerations than that to enter; that's dignity.)
Did you read the post? It was posted as an April Fool's because the headline and framing were (grimly) comic. The points made within it were clearly not a joke.
It says April 2nd to me, but I suppose that could be some time zone adjustment?
April 1st for me, but it's also literally tagged with "April Fools". :)
You probably shouldn't take tags made on April 1st seriously...
I'm panicking about it right now!
I think many (most?) people took it in a kidding-not-kidding sort of way because of other stuff Eliezer has said.
Kind of like the Nelly song "Hot in Herre":
I got a friend with a pole in the basement (What?)
I'm just kiddin' like Jason (Oh)
Unless you gon' do it
If it was meant as an April Fool's joke, it was one where Yudkowsky accidentally admitted his true feelings.
I don't believe his true feelings reflect the actual state of affairs on the board, however- merely the state of the inside of Yudkowsky's head, which on the topic of AGI hasn't been aligned with reality since the start (I reject his assertion that AGI will bootstrap itself instantly into being an evil god capable of anything that can be expressed in language, among other, more substantive objections I don't want to get into).
Hyperion and The Fall of Hyperion were good reads though.
Yes, but I wouldn't base my travel plans on it.
Eloser? Really?
Link todo this assertion?
I think Sleazy E might be referring to this Twitter thread: https://twitter.com/ESYudkowsky/status/1453589443701469188
>I'd say that I "don't understand" why the people who worry that chickens are sentient and suffering, don't also worry that GPT-3 is sentient and maybe suffering; but in fact I do understand, it's just not a charitable understanding. Anyway, they're both unsentient so no worries.
>I admit, though, to some grim curiosity about what people who worry about chicken sentience would say, to explain how they like totally know that GPT-3, which can ask you in words not to kill it, like totally could not contain a spark of awareness like a *chicken*, nope nope.
A great response: https://theunitofcaring.tumblr.com/post/151442936096/a-question-asked-out-of-genuine-ignorance-is
> I reject his assertion that AGI will bootstrap itself instantly into being an evil god
I don't think he is making that strong claim that it will be instant. I think it is allowed to take weeks or months in his model.
Not cackling evil, more complete indifference to humanity. Out of all the different things in the universe, the AI isn't focussed on humanity. And we are made of atoms it can use for something else. The well being of a random ant isn't usually on the mind of the architects and builders, even as the ant is buried in cement.
> capable of anything that can be expressed in language,
The AI is capable of pretty much anything, with a few exceptions due to physical law.
Of course, when describing the AI, we must talk about it in language. But a real ASI may well do something with no compact English description. What, I can't say.
"I think it is allowed to take weeks or months in his model"
Not really, because then that opens it up to the very reasonable position "just fucking unplug it if it starts bootstrapping towards godhood" in the same way nuclear plants are built to shut down at the first sign that the reaction might start running away from us. I'm going to guess this is one of the first suggestions floated at Yud, and he's rejected 100% of all suggestions floated at him, so I don't think that's a good answer to him.
"not cackling evil"
I want to register that I had to actually take a lap after this because of how much condescension this whole paragraph directed at me. I hope it was unintentional.
I am AWARE that the AI doesn't ACTUALLY have the persona of Old Scratch. I know what a paperclip maximizer is. I've read enough hard sci-fi and ratfic that you can assume that I know all the many ways nerds have imagined AGI can kill us or put us in a manifested Hell. Assume that "Evil God AI" is a placeholder term for "an AI that looks like one of those".
"A real ASI may well do something with no compact English description."
Yeah, and it could turn the sky into moist treacle, too. You really aren't helping your case with me by proving you think this thing will really put the GOD in Evil God and opening up your window of possibilities unto infinity. And EY thinks it's got a >50% chance of manifesting in the next 7 years?
All that convinces me of is that EY accidentally made a really powerful memetic virus that only infects a very specific type of nerd.
Would you be able to tell if it's bootstrapping itself to godhood? I can think of multiple ways this process could be hidden, as I'm sure you (or an AI of even less-than-superhuman ability) can; the fact that even the current (best) AIs are the blackest of black boxes is also a consideration.
Would you be able to unplug it, even if you did know? Possibly; but there are lots of scenarios where you couldn't, or it escapes in some way, and the best way to prevent this is for everyone working with it to be really, really careful.
It's quite a risk for the whole world to take on the basis of "no one will be stupid or careless". Even if you're, say, 99% sure that no one will be careless, or that it can't work for other reasons, that's only reassuring if a 1% chance of humanity being wiped out is acceptable to you — and people notoriously overestimate their certainty.
Not that I'm a true believer or anything; I can't really make myself get too worked up about AI risk; it's just that I don't think "why not look at it to see if it's doing anything weird?" is necessarily a knockdown argument.
"Would I be able to tell?"
No, because I'm a cro-magnon who just barely knows how his own end-user PC works. But I'd hope that a brain trust of the best computer science, programming, and AI experts in the field would be much more competent than I am (and I don't buy that any lesser team could bumble their way into godlike AI by accident.)
I think it is quite easy to bumble to superintelligence by accident, with enough compute.
After all, evolution is pure 100% design and test, with no understanding whatsoever, and it eventually bumbled to humans.
(And of course, when we restrict ourselves to code that compiles, we can bumble so much faster)
"Not really, because then that opens it up to the very reasonable position "just fucking unplug it if it starts bootstrapping towards godhood" in the same way nuclear plants are built to shut down at the first sign that the reaction might start running away from us."
Nuclear plants do this because everyone recognizes the danger, and governments mandate the inclusion of safety features and backup safety features in the design, and then send someone around to check that the safety systems are in working order.
Lots of people who might build AGI don't believe it could be dangerous. As far as I know, nobody has ever built an AI system with a failsafe kill switch. No government has even considered mandating safety systems of this kind. Do we have a good concept of what that switch would actually have to look like to be dependable?
It's not a true April Fool's post, it was just posted on April 1 for plausible deniability.
Same for the April Fool's tag that got added to it?
Yes.
Q6 (quoted elsewhere in this thread) addresses this explicitly.
Similarly, a comment from the post (not by Eliezer, but I think this is the right interpretation):
"My interpretation is something like: if someone wants to dismiss Eliezer for reasons of their psychological health, Eliezer wants them to have an out, and the best out he could give them was "he posted his main strategic update on April 1st, so it has to be a joke, and he confirmed that in his post."
If someone wants to dismiss Eliezer over things like this, it's not their *own* psychological health they are concerned with. But a barely-plausibly-deniable "April Fool's joke" gives him an out on that front as well.
This, and there is zero doubt about this if you know the context. Everyone was taking seriously, we had numerous threads of people expressing genuine fear, and Eliezer didn't exactly try to play up the April Fool's angle. It's not a joke. It's how he feels.
OK, yeah. I'm not a close part of that community, but now that I've read the whole post and not just the TLDR + a couple of paragraphs, I see how it's not just an April Fools. Interesting rhetorical approach.
Note that this wouldn't be the first time EY has used April Fool's to express real feelings in unconventional ways - dath ilan was introduced on April Fool's (https://yudkowsky.tumblr.com/post/81447230971/my-april-fools-day-confession), but subsequent year-round expansion on that setting (e.g. https://www.glowfic.com/posts/4582) along with how well it seems to match his other works suggests that dath ilan as a whole is not intended to be a joke (as distinct from any claims that he was literally raised there).
Holy crap I think you're my CS173 prof
Yep; don't let that scare you off from the meetup this Sunday :)
Isn’t this the sort of prank Bart Simpson would pull if he were a three sig autodidact?
As far as I can tell, the "joke" here is that he made a basically normal post saying exactly the kind of things he usually says, except that he said explicitly at the beginning all the stuff he usually leaves as subtext. It wouldn't surprise me if a lot of his longtime readers saw it and suddenly realized this is what he'd been trying to tell them for years now.
EDIT: Looking through the comments, it seems like, as predicted, basically everyone familiar with the rest of Eliezer's corpus is interpreting this as a confession of his real beliefs done with a bare minimum of plausible deniability.
> It wouldn't surprise me if a lot of his longtime readers saw it and suddenly realized this is what he'd been trying to tell them for years now.
I, a longtime reader, just read it and this is merely a new (and sad) development in what he'd been successfully telling me for years (decades) now.
I got a much more dark night of the soul feel from that post with limited irony. Made me a bit sad for Eliezer, to be honest, who even though I disagree with him on quite a lot is still doing what he genuinely believes is best for the future of humanity.
I'm torn between believing that Yud really is compassionate (when my compassionate mind is dominant) and (when my unlovely mind is dominant) believing that beneath the façade Yud hates humans at-large for refusing to see how OBVIOUSLY RIGHT Bayesian Reasoning is and refusing to transform society into a community of enlightened antitheist technocrats devoted to destroying death and has thus retreated into apocalyptic fantasies of the Superior Ultrarationalist AGI annihilating all the stupid monkeys so he can have one last cackling I-told-you-so before he too perishes. It would hardly be the first time that a wounded idealist retreats into misanthropic cynicism and fantasies of human extinction.
Jonestown.
Maybe like most people he, from time to time, experiences multiple feelings at once?
I have found myself happier and more effective in the world by not holding people to what I imagine to be their darker selves.
I'm glad you're happier and more effective that way!
I'm happier and more effective when I can speak my mind freely behind a cloud of pseudonymity instead of being expected to keep up a cloud of universal positivity even when said positivity is insincere. I wouldn't say what I am about Yud if I didn't think it was necessary (yes, I do think "What if this guy you like has had a serious shift in character over time that makes him less worthy of a person to take seriously?" is a necessary statement), true (this is what I think, and I think some combination of the two is in fact true in Yud's mind- he probably does still care for people on some level, but his "Dignity" post radiates an extreme bitterness towards and contempt for humanity, even his fellow rationalists), and kind (both to people who take Yud too seriously by trying to give an alternative case, and even to Yud himself: if someone's clearly spiraling down the drain, the unkind thing to do is to say nothing or actively enable them).
You were the Theist I was trying to explain my feelings to about bayes the other day right?
I would prefer if you stopped exaggerating. It really muddies the water around your argument when you add all this 'take over the world' nonsense. It makes it seem like you are deliberately creating a strawman of anti-EY positions.
Also, like... EY has been obsessed with this since the very beginning, and you know that, because it was a core part of your argument as to why I ought to be unhappy and miserable. I don't think his feelings have changed a single Iota since 2008.
I'm not exaggerating at all. Read what Yudkowsky actually said and you can very clearly see that he is EXTREMELY bitter that humanity at-large doesn't understand how OBVIOUSLY RIGHT Bayesian Reasoning is and instead keeps wasting their time on "pointless" things like theoretical physics and religion instead of devoting their entire efforts to destroying death. This has been a theme in his work since day one and has been something that has clearly made his mindset grow increasingly dark and bitter over time. (EDIT: I see you've ceded this point, so I'll assume that this isn't your problem with me).
Or are you objecting to my implication that AGI would use something like Bayesian Reasoning in Yud's hypothetical scenario to structure its planning with instead of some other rationale?
No, it's... the same complaint I made last time... the slight but definitely present implication that rationalists wanted to take over the world.
I think one time we got a representative elected in california? For the district with all the rationalists. That was like, the sum extent of our 'transforming society', except for the very narrowly focused fields of ai and probability theory.
Seriously. It makes it seem like you have absolutely no intention of engaging in good faith.
"The slight but definitely present implication that rationalists want to take over the world"
Pardon? Where exactly am I implying that? You realize that when I said that when I talked about a rationalist technocracy, that was a JOKE about how rationalist conversations sometimes forget non-rationalists make up the majority of humanity, correct?
Since you apparently don't: no, I don't actually think rationalists want to take over the world. I think Yudkowsky, specifically, has a greatly inflated sense of self-importance and sees himself as the greatest thing to happen to thinking since Plato. Any "implication" you're reading into my statements isn't coming from me.
It's pretty clearly not 'merely' an april fools post.
Yes. It was written with absolute seriousness and has so far been far more convincing than any of his detractors.
I would very much like to see a coherent argument as to why he's likely wrong. So far I haven't.
His argument for AI x-risk is based on this convoluted chain of claims that falls apart if any of the 25 steps before "therefore we're all doomed" isn't 100% generalizably true, nearly all of which are non-falsifiable philosophical thought experiments, like "AI can practically attain Laplace-demon-esque omnipotence given enough compute power" and "Bayesian reasoning is perfect and infinitely scalable" and "AI intelligence can scale exponentially basically forever".
Yeah, I think "I know what I know because I'm me" is in those 25 steps somewhere too.
If we take as given that AI is indeed a significant existential risk for humanity in a few years, then there is a logical, readily comprehensible concept on how to deal with that - declaring global Butlerian Jihad. If you do not not know, this term comes from novel Dune, and is defined as: "the crusade against computers, thinking machines and conscious robots (...). its chief commandment (...) Thou shalt make a machine in the likeness of human mind".
I admit that this solution is far from perfect, and I can totally believe that someone smart and knowledgable about AI could come up with something better. But it is imo obviously far superior to what Yudkowsky is proposing.
Now, I don't know anything about AI, but the fact that Yudkowsky came up with such a silly solution makes me suspicious of his reasoning abilities. Which in turn makes me doubtful that his warnings are in fact based on reality as opposed to delusions.
I have, on occasion, managed to convince myself that this will end only with the announcement that research on general AI is now illegal and earns the death penalty for the researcher and his family unto three generations.
I think you've overstated one thing and understated another.
The family punishment is probably not outright required for a Jihad success.
What *is* required for Jihad success is "and any nation refusing to enforce this is casus belli, even if that means a nuclear exchange".
*That* is the kind of resolve you need to fully shut down the field. Nothing less will cut it. And... well, I'm not really seeing how we get from here to there, barring the fairly-unlikely "an AI revolt happens, kills millions, but ultimately fails, and then gets taught in school forevermore as Literally Worse Than Hitler".
This is not as hard as you make it out be. Recall that deployment of GMO technology has been seriously hampered despite its obvious benefits for food production, and in the absence of any tangible evidence that it is harmful.
"Institute this law within your country or we will literally nuke you, and we don't care that you'll nuke us back" is an *extremely*-high bar to clear. I'm also suspicious that Silicon Valley could weaponise its platforms (which half of us trust with our communications both public and private!) to hamstring any political effort against them that seems to be gaining steam, particularly as their AI improves. Elon Musk may be able to circumvent that if his buyout of Twitter succeeds, though.
But yes, it's worth trying. I'm just depressed and also have no idea how to do this sort of thing.
That's only because the people who develop GMOs are in First World countries, but the big markets and obvious benefit are in the Third World. First World shoppers like their foods labeled "non-GMO" 'cause it means they're a cool frood and they don't give a darn about Vitamin A deficiencies.
Most of Ey's April Fools posts are "things that he really thinks but are too out there to publicly stand behind, so posting on April Fool's give a veneer of deniability"
The people who take Eloser seriously are not worth taking seriously. I don't even think Scott actually takes him seriously to be honest.
Doubling down on "uncharitable and untrue", eh? Bozo bit flipped.
If this is what passes as "AI risk research", it's understandable that it's not taken seriously.
It seems like a pretty good supervillain motivation, though - "the world is doomed, but by satellite necro-ray will euthanize everyone painlessly".
Good point
You can criticize someone without sinking down to playground taunts- especially when said taunts are so sophomoric that I'm half-tempted to think you're a Yudkowskyite making a strawman of Yud's critics to beat up later.
I don't take Eliezer terribly seriously on the subject of AI risk, but I take people who repeatedly, deliberately, mockingly misspell his name even less seriously. Except as a threat to the tone of discourse here. You are living down to your chosen screen name, and I think you'd be a better fit for Twitter than ACX.
"This caused the Less Wrong community, already pretty dedicated to panicking about AI, to redouble its panic. Although the new announcement doesn’t really say anything about timelines that hasn’t been said before, the emotional framing has hit people a lot harder.
I will admit that I’m one of the people who is kind of panicky. "
I must say that a community of people panicking about something like that makes me think that this community must be quite cultish. I guess EY is a very convincing guru, though his convingness does not translate vety well in writing.
The community formed around his writing, the Bay Area cult came later. His convincingness is tuned to a very particular kind of nerd.
"Why are these two so different? Do lots of people expect Musk to acquire Twitter after June 1 but still in 2022?"
It's because no one expects Twitter to acquire Musk.
(Model output: Elon Musk is an eccentric billionaire. Twitter is a private corporation. This joke subverts the reader's expectation that "Twitter announces a Musk acquisition" refers to Twitter announcing that it is being acquired by Musk.)
Haha I was thinking the same thing! Probably the "yes" bets just misread the question.
If you/smart people in general were truly pessimistic and panicked about short term AI X-Risk, wouldn't they be doing something about it? I don't think so, but if I did think that within 3/5/10 years a killer AI would destroy everything, I imagine I'd quit my job and use my available funds to prevent it by [doing something not elaborated on as it could be construed as threatening to people or organizations]. I saw a Gwern article from 2012 about slowing chip production through regulatory or terrorist means but I don't think that's been attempted yet.
Obviously this doesn't quite apply to Eliezer Yudkowsky who has been working on this full time, but even he appears to be taking a defeatist approach rather than an insurrectionist one. The extent of most people's concern seems to be blog posts and betting on prediction markets, arguably an implicit indicator that wealth or at least prestige may in fact retain value.
If the extent of panic over AI X-risk is to blog or comment about it, I'm skeptical the people advocating for it are truly as concerned/earnest as they profess.
To be clear I do not endorse or advocate for [actions not elaborated on]. As a fun fact, the term 'Luddite' comes from the Luddites, a group of tradesmen who destroyed early industrial revolution factory equipment before maturing into general terror activity before being defeated by the British army.
Sadly, while I do not believe that we'll see an AGI by 2033, or anywhere near that soon, I do believe that we'll see AI-risk related terrorism (of the kind you describe) way before then. As usual, the true X-risk is other people.
Agreed. A bad actor using an AI effectively is basically a malevolent sentient AI (in terms of function) even if the interface is typing.
> , I do believe that we'll see AI-risk related terrorism (of the kind you describe) way before then. As usual, the true X-risk is other people.
I mean if someone hypothetically wanted to engage in such an act, it wouldn't be terrorism because the goal isn't to cause terror. Ideally they want to destroy some expensive and hard to replace photolithiograph, and not harm anyone. They might do something with a handful of casualties. How could they possibly create X-risk?
I think only a small subset of people have the right make up to do insurrectionist things even if they believe as strongly as the people doing actual insurrections.
Most such strategies wouldn't work. You would get some pretty severe second order effects as a result of [doing something not elaborated on as it could be construed as threatening to people or organizations], and I expect they wouldn't shift timelines by more than a decade or two. If the AI safety community is actually viewed like a Doomsday cult, then I expect there'd be a way lower chance of funding, getting governance and orgs to listen to their advice, etc.
Agreed. It's easy to see this approach as counterproductive. How would people react in the face of a significant property or cyber attack against a company or organization only tangentially related to AI research, such as a chip manufacturer? There could be a reactionary movement to restore the harmed party to a higher baseline than where they were before. Meanwhile, if it's discovered that people in the AI risk community were at fault, that community would likely become more marginalized and receive less funding.
The AI risk community cannot afford to alienate themselves from the AI research community. Any solutions to AI risk (or partial solutions, or even reasonable delaying tactics) need a willing audience to implement them. Right now, the AI risk community is close enough to the direct researchers to have some influence. If they turned adversarial as suggested, that relationship would be sacrificed.
Given that new research can be implemented by a variety of different actors, the worst approach to mitigating AI risk would be to get yourself expelled from the conversation entirely by attempting what would amount to only a minor delay (if any) in practical implementation.
There seems to be very little faith that current approaches are doing enough, given that under current approaches the consensus estimate for AGI and potentially AI X-risk is measured in years or decades. Are people satisfied with AI takeover by 2050 as long as they feel like they did their best with nonviolent means?
Before you ask "is terrorism morally justifiable", you need to ask "will terrorism actually help?". If the answer to the first is "no", the answer to the second is probably also "no".
Winning elections gets you much bigger guns than any terrorist has ever had, and also generally isn't very compatible with being a terrorist (Sinn Fein is an outlier). And you need very big guns indeed to shut down the world AI industry, at least by any methods I know that are not mad even by the standards of "stopping X-risk" (Life 2.0 might be accessible to a terrorist, but Life 2.0 is a full X-risk in its own right; there is *no point* stopping AI if we're all dead anyway from grey goo). So even leaving aside the moral aspect of "is murdering thousands something you're willing to do", I think terrorism is not very likely to help here.
This is imho bad political analysis. General public is primed to believe in the risk of malevolent AI, thanks to decades od sci-fi flicks about killer robots. On the other hand, AI research community seem like least likely people in the world who would be willing to believe in dangers of AI research.
Stirring up a panic that would result in a regulatory action is obviously more viable approach to stopping AI research than trying to befriend AI researchers and quietly talk them out of their misguided ways.
I am personally quite agnostic on AI risk, but the very fact that "AI might kill us all" camp exhibits such political naivete makes me question their judgment in other areas, like, you know, AI risk itself.
I disagree. Most people see those films as the same kind of sci-fi out-there as lightsabers. The general public doesn't expect AI to destroy humanity by 2040, any more than they expect to be able to buy blasters and lightsabers by 2040. If you want to move the regulatory needle on this, the best approach would not be direct action against AI researchers, since the public doesn't perceive them as on a course to destroy humanity. The general public (and especially journalists) don't seem to be savvy enough to parse the nuances there.
The kind of action that would be likely to see public pushback be a sort of false-flag operation. An [unspecified use of AI] that caused a global economic or moral panic would move people to demand their politicians put greater safeguards on AI implementation. That would require AI risk rogue actors to be close enough to the research community to know how to implement these unspecified actions, though. It's not something you can do from the outside.
No-one is really making much of an effort to communicate AI risk to politicians or the general public. It might not be hard to do. Everyone has seen this stuff in fiction and it's not conceptually complicated. Current AI is already very impressive. When I show GPT-3 examples to normal people they find it hard to believe that they are real. It's not a huge leap to think that AI will be dangerous soon.