406 Comments

"Whatever you’re expecting you 'need self-awareness' in order to do, I bet non-self-aware computers can do it too. "

From very early in Vernor Vinge's "A Fire Upon the Deep":

" ... The omniscient view. Not self-aware really. Self-awareness is much overrated. Most automation works far better as part of a whole, and even if human powerful, it does not need to self-know. ..."

Vinge is writing science fiction, but still ...

Expand full comment

I would posit that humans largely aren't self aware in an absolute (not relative) sense - that they can observe some of their own behavior, and store certain verbalizations in memory without speaking them out loud, but this awareness includes only a tiny percentage of the mental processes that make up their "self". If anyone was truly self aware, implementing AGI would not be such a challenge.

Expand full comment

The full-novel version of "consciousness is incidental to, and maybe even parasitic on, intelligence" is Peter Watts' Blindsight, for my money one of the best hard SF novels of the 2000s. Full version: https://www.rifters.com/real/Blindsight.htm

Expand full comment

I hate to complain about a recommendation I agree with, but that's a bit of a spoiler. :-(

Also, he's right, Blindsight is awesome.

Expand full comment

Isn't the title pretty explicitly stating the book is about that.

Expand full comment

Are people here and Scott in the article using the term self-awareness interchangeably with consciousness?

It seems to me like they are but those two concepts are quite distinct to me.

But still, neither of them is required for making an AI dangerous.

Expand full comment

Exactly! Discussing consciousness is a complete red herring that just shows that the correspondent has no idea what they are talking about.

Expand full comment

Scott, this is kind of a big ask, but i think the only real way to understand the limitations and powers of current gen AI (and by extension, if there is a reasonable path forward from them to AGI), is to do a basic project with it or programming with one yourself. Hands-on work really, imo, really reveals much more information then any paper can convey. Physically seeing the algorithms and learning systems and poking at them directly explains things and conceptualized what papers struggle to explain in the abstract.

You talk about current AI systems as having "a blob of learning-ability", which is true, but not complete. We (and by we i mean AI developers) have a much deeper understanding of the specifics and nuances of what learning ability is, in the same way rocket engineers have a much deeper and more nuanced understanding of how a rocket system has "a blob of propulsive material". In my experience (which isn't cutting edge, to be fair, but was actual research at the higher level classes in college), our current blob of learning ability we can simulate has a large number of fundamental limitations on the things it can learn and understand, in the same way a compressed chunk of gunpowder has fundamental limitations on it's ability as a propulsive material. A compressed tube of gunpowder will never get you to space; you need to use much more complex fuel mixing system and oxygen and stuff for that (i am not a rocket scientist). In the same way, our current learning algorithms have strong limitations. They are rather brute force, relying on more raw memory and compression to fit information and understanding. It can only learn a highly specific way, and is brittle in training. Often, generalization abilities are not really a result of the learning system getting more capable, but of being able to jam more info in a highly specific way (pre-tuned by humans for the task type, seriously a lot of AI work is just people tweaking the AI system until it works and not reporting the failure). Which can accomplish amazing things, but will not get you to AGI. Proving this, of course, is impossible. I could be wrong! But that's why i suggest the hands-on learning process. I believe that will allow you to feel, directly, how the learning system is more a tube of gunpowder then a means to space.

Expand full comment
founding

The "fire alarm" post that Scott linked to at the very end of this post responds to this argument. (If you don't have time to read the whole thing, search for "Four: The future uses different tools".) Do you have a response to that?

Expand full comment

I think that's a good argument for making things easier to do, but not for achieving the breakthroughs required to get further, if i'm understanding it right. The future may have different tools to make training/setting up/creating an AI easier or even trivial. But that's still working within the "bounds" of what is possible, in the sense that future tools will just make things that are possible but very hard/very specific trivial and easily generalizable, but it won't make things that are impossible possible.

Which is a good counter argument to my claim about pre-tuning, but i don't think it addresses the larger part about how you can't use gunpowder to get into space and about the current weaknesses of our learning algorithms. This is more like increasing the amount of gunpowder you have, or more efficient methods for compacting it. You can make the learning algorithm maximally efficient with future tools, but that still won't be enough. Or at least, that is what i believe, given hands on experience.

Expand full comment

One response is that Point Four is simply not very true. There's any number of things that were immensely hard for entire teams in 2012 - and still definitely beyond the reach of a single comp-sci grad with a laptop working for a week.

Adam will not collect millions of samples for you, and automating that component has not been an easily solved problem. Sure, you can try to scrape the net - but high quality labeling is an entire branch of the economy for a reason. As they say, ML is not Kaggle. Other tools mentioned there, such as batch norm, are obviously great - but the main reason for their very existence is the scaling. They were developed to support ever larger models - so they couldn't be used to make the point about tools being by themselves such wonderful enablers. Tools have improved, but not as dramatically as that post indicates.

While we're there - I have *no* idea why those luminaries couldn't just say "no idea what's least impressive, but I'm fairly confident self-driving cars as a widely available service won't be a thing for two years though they might will be in ten". No idea when that conference was, but I've worked on self-driving ML at a world-class place for a while, and I'm fairly confident even now that fully autonomous self-driving cars won't be a wide-spread phenomenon in 2023. Yes, I know about that impressive demo. And about that other one. And about that third one. FWIW, I'm willing to say that we will have *some* autonomous driving in *some* contexts by then.

Expand full comment

Yeh but try use the AI it on a wet November evening in Donegal where the “slow - road works ahead sign” has fallen over and the new one way system hasn’t been updated to google or Apple maps.

There are probably thousands of scenarios not envisaged by the software writers of the automatic cars, or machine learned either because that needs real world usage.

Anyway I’ve never seen anybody talk about the legal issues here. If a car goes off the road because of a faulty brake the manufacturer is sued. If the car goes off the road because of human error then the manufacturer is fine. With auto cars it’s all the manufacturer.

Expand full comment

That is precisely in agreement with what I wrote, right?

Anyway, these kinds of scenarios are quite envisaged- just hard to address and exactly why I don’t think we’ll get to widespread autonomous driving in the wild in 2023. That and the legal aspect, sure.

Expand full comment

The “yes, but” was in response to the perfect demos you mentioned.

It’s good that they are thinking outside the perfect Californian weather - remember that Apple maps worked great in Cupertino.

I think that the experience learned will very much improve driving safety via software. Full autonomous I am dubious about.

Expand full comment
founding

I'm willing to 'bite the bullet' and argue that 'fully autonomous' self-driving cars, while FAR from perfect, might still be _better_ (on average) than the existing drivers of cars, i.e. humans.

Your example:

> Yeh but try use the AI it on a wet November evening in Donegal where the “slow - road works ahead sign” has fallen over and the new one way system hasn’t been updated to google or Apple maps.

Existing drivers – humans – _already_ make incredibly terrible (and dangerous, often fatally) mistakes.

I don't think the right comparison is to some hypothetical perfect driver under any possible conditions. It might be perfectly fine (and reasonable) for 'fully autonomous' self-driving cars to just refuse to drive under some conditions.

But it would still be a win – for humanity – for us to replace _worse_ human drivers with better AIs now, in the circumstances in which they would be better.

Expand full comment

Dear Lord, I think I would pay actual real money to watch real-time footage of the best self-driving car going up a boreen on a typical June afternoon where nobody has scarted the ditches because the council doesn't do that anymore and the farmers aren't going to pay for it until they have to, with the very high likelihood of the creamery lorry coming against you and the silage trailer zooming around the corner and once you get past those, the thing peters out into a sheep-track and you have a very unimpressed black-faced mountain ram sitting in the middle of the road looking at you.

Oh please please please Uber or Google or whoever, do it! Please! 🤣

No boreens, but nice views:

https://www.youtube.com/watch?v=HDDX3SrlvjQ

Expand full comment

> A compressed tube of gunpowder will never get you to space

It kind of does: https://en.wikipedia.org/wiki/Paris_Gun

(though I'm not sure if it's propelled by gunpowder or some other propellant that was available during WWI)

Expand full comment

You're looking for https://en.wikipedia.org/wiki/Space_gun

BTW, if 'a compressed tube of gunpowder' has never gotten anything 'to space' (https://en.wikipedia.org/wiki/Project_HARP ~57 years ago), than neither has Jeff Bezos or Richard Branson.

Expand full comment

Crap yeah, you're right i was thinking in terms of rockets, i completely forgot about these things. Fair play, my metaphor was wrong (although maybe if you change from "in space" to "in orbit"?). Also, i thought Jeff Bezos or Richard Branson's rockets used like, normal rocket propellent that requires oxygenation and other fancy stuff, not just gunpowder?

Expand full comment

Bezos & Branson did use conventional rockets, but only achieved a suborbital trajectory; you can do the same with a single powder charge. "HARP didn't go to space" requires using an unconventionally strict definition that precludes the first two as well.

In theory you could use a multiple-charge system to circularize an orbit, but at that point you're just using your powder as an inefficient propellant and the difference is a matter of engineering.

Expand full comment

Damnit ok my metaphor is dead in the water, i did say i wasn't a rocket scientist. I'll have to figure out a different one.

Expand full comment

As someone who has done many projects in the deep learning space, I have to say that I come to a very different conclusion than you. It is often evident that you are indeed *only* constrained by model size, and that you can empirically capture bigger and more nuanced abstractions with bigger model size. I make a toy model which fails to do a job; I make it ten times bigger, and it can now do the job. I regularly run up against the limitations of hardware when trying to build models.

We can already do amazing things with deep learning. Every year, we see more proof of the thesis that the main bottleneck to doing even more amazing things is simply model size. Every once in awhile, we see a whole new concept like policy learning, or transformers, or GANs, which gives you yet another tool. You connect the tools together, and get a step change in capability. You can see in your mind how you could connect various pieces together and get AGI, if only your model was big enough. Another way of saying this would be: we have all the pieces of AGI lying around already. They haven't yet been put together, and if they were, we probably don't have the compute to power the "learning blob." (But we may! I'm agnostic about how much compute is truly required.)

Expand full comment

You really think so? I mean, i have encountered problem that smaller models captured very poorly, and larger models were able to capture with seemingly high accuracy, but they were the the kinds of problems that you could still see the smaller models exhibit the basic level of behavior in. Like, do you think, given infinite compute power and infinite memory, with current day models, it would be possible to have an AI generate a functional and correct moderate to large size python program from a set of clear and consistent requirements?

I don't think it is! I mean, the best attempt we have, Github Copilot, doesn't even exhibit the smallest spark of that sort of understanding and knowledge required for that (and trust me i tested it out a good bit). I mean, i can't really prove it isn't, but i can't really prove that P=NP either, just that experience and knowledge has shown that it really REALLY probably isn't likely to be.

I think that the learning blob algorithms we have right now just are not capturing and storing information in a coherent enough way enough what AGI would required. Which, i do acknowledge is just a feeling and i could be wrong on. But i would be shocked, on the same level as a proof that P=NP, if it was otherwise.

Expand full comment

I do think so. But I am not saying that you can build an AGI if you just build a big enough transformer. I would instead build something more complex, in the specific sense of having many more distinct elements. Rather than write a long post nobody will read, I'll just said: start by looking an animal brains, and replace each piece with an existing algorithm that sort of does a similar thing. You would end up building the sort of thing that would teach itself how to generate or discover its own training data based on the sort of thing that it guessed that it was supposed to be trying to do, and following a Python code spec would be simply one implementation of that process. Something like that might not look like a big fat transformer anymore, but then again it might.

This sort of exchange has the risk of turning into burden-of-proof tennis. I don't have the spec for an AGI in hand, and you can't prove that a sufficiently big transformer isn't arbitrarily capable. What I can do is look at what animal brains do, note that we now have software architectures that do roughly all of the sorts of things you see in brains, and imagine how I might hook those algorithms together to build something brainlike. Brain mimicry is only one way of building an AGI, I suspect you could end up with an AGI architecture even simpler than that. (Brains are the way they are because of energy efficiency more than FLOPS maximization.)

I could also separately argue that maybe you can actually get a "giant blob" model to become an AGI. I didn't think something as simple as architecturally

simple as AlphaStar would be superhuman at StarCraft, but here we are. I lean more and more toward the perspective that our architectural inventiveness is going to be secondary to just adding more parameters and letting the model figure things out for itself, past a certain grain-size of problem.

Expand full comment

>But I am not saying that you can build an AGI if you just build a big enough transformer. I would instead build something more complex, in the specific sense of having many more distinct elements

Hmmm, i can see that actually, that's a more compelling possibility to me then throwing more training data/size at the problem. That would require a good enough understanding of generalized components of what a general intelligence has in order to replicate them, but i can see that being done in one fell swoop by someone in theory. (i have thought of similar myself tbh). If someone just can conceptualize the base components of what, when strung together, is necessary for a GI, then that can happen. And while i'm sure there are a lot of base components, we probably know a lot of them already. That is highly compelling.

>I could also separately argue that maybe you can actually get a "giant blob" model to become an AGI.

Given the above, that is possible, i suppose. But i think the structure of the above is so highly complex and requires such a vast search space (and additionally has no obvious gradient from not-working to working), that it doesn't seem as likely just by a blind network alone. But it might be doable!

Expand full comment
founding

Your right that "our current [AI] blob[s] of learning ability we can simulate [have] a large number of fundamental limitations on the things [they] can learn and understand", but the worry is that we our near-term future 'blobs' won't be similarly limited. That seems very reasonable! People are making rapid progress on doing exactly that, i.e. creating more capable blobs.

Expand full comment

I mean, that's possible, true. But it depends on how far away you think we are from AGI with our current learning blobs. Are we at like, pretty close and we can get right over with just one clever breakthrough? Or are we not even within 10 light years of it?

If the former, then yes near-term future blobs probably won't be similarly limited and will get to AGI! If the latter, near-term future blobs won't be similarly limited, but they will still be limited in different but still highly rigid and very strong ways that prevents them from getting to AGI. It'll be a blob that is 1 light year ahead, but still 9 light years away. The question is: where are we? And my experience makes me think we're still 10 light years away. Not provable in any way, just my experience.

I also want to clarify i am very much still in favor of researching AGI prevention and alignment measures. Nothing wrong with that! It's good to do. Just i wouldn't worry about AGI in the near future, from a personal standpoint.

Expand full comment
founding

I would personally love to be able to work on AGI safety/alignment, but I'm pretty sure I wouldn't be able to contribute meaningfully – not directly – given everything else going on in my life (e.g. my other responsibilities).

And I'm unsure about whether it's useful to 'worry' about it – tho that depends on what I think is meant by 'worry'! But I too don't think it's quite the same kind of impending disaster as, e.g. a large asteroid on a definite collision course with the Earth. But then I'm not worried about climate change in the near future, from a personal standpoint. (I think it's pretty obvious that we all will mostly just adapt to whatever happens, however we can.)

Expand full comment

I've done the basic programming. I'm a full time programmer. I've taken graduate level courses in machine learning. I don't think that current AI have a fundamental limitation that prevents them from being very very dangerous in agenty ways. We'll probably have to find some tweaks, like how we went from shallow neural networks to deep convolutional neural networks for image recognition, or from RNNs to LSTMs to Attention for language prediction, and that might take a decade, but it's not a fundamental limitation of current AI.

Expand full comment

No, i very much agree, i just wouldn't classify those things as "tweaks". Going from shallow to deep was a pretty significant breakthrough i think! And i think that we're going to need probably a dozen massive breakthroughs on an amazing level (like, on a higher level then those breakthroughs mentioned above) to get to a system that can scale to AGI. We will eventually get though breakthroughs, yes, in the same way we got breakthroughs in rocket propulsion technology. But I think each one is going to be hard fought and require a stroke of genius that will lead to a large scale reworking of the entire field when it's found. (like, we'll get 1 breakthrough, and the next 10 years will look like the last 10 years of AI scrambling and investigation and hype).

Is this provable? No, much in the same way it wasn't provable back in the day that you couldn't get into orbit just by strapping more tubes of gunpowder to your rocket (this metaphor is becoming very belabored i'm sorry). But that's just what my experience and knowledge have lead me to believe.

Expand full comment

Also i just realized you said "I don't think that current AI have a fundamental limitation that prevents them from being very very dangerous in agenty ways" and nothing about AGI. Whoops. Uh, then in that case yeah i totally agree. Current AI can act in very evil and agenty ways no problem, no debate there. Just that that has it's limits, and i don't think the worries about that match up with the worries about AGI Scott talked about here.

Expand full comment

I'd like to chime in as another person who's worked with modern machine learning. I am frustratingly familiar with the limitations (hello, RL).

And I have also seen tiny little tweaks, here and there, that do too much. Simple changes that aren't fully understood that make previously impossible things possible. Over and over, I see dense papers theorizing about the rigorous mathematical basis of something or other, and then someone goes "uhhh what if I clamp... this" and suddenly you've got a new SOTA.

This is not what a mature field of study looks like. The fruit is so low hanging that we're face down in the dirt, blindly reaching backward, and *STILL* finding order of magnitude improvements.

Seeing the kinds of generality being achieved with incredibly simple approaches, seeing frankly silly tweaks making such sweeping improvements... I was forced to update my predictions to be more aggressive.

And then GPT3 came out, with capabilities years ahead of my expectations. Scott's directionally correct.

Expand full comment

Hmm, yeah that is true. The fruit is certainly low hanging and plentiful. That's a good point.

Expand full comment

I think the element that's missing from the gunpowder and rock analogies is recursive self improvement. A big pile of gunpowder isn't going to invent fission bombs. A big/sophisticated enough neural network could potentially invent an even better neural network, and so on. (This isn't sufficient to get AGI, but it's one component.)

More pragmatically, the gunpowder analogy is overlooking fuel-air exclusives. As the saying goes, quantity has a quality all its own.

Expand full comment

why do you assume that a neural network that is capable of building improved neural networks is not sufficient to get to AGI?

Expand full comment

Right now, neural networks don't have agency. Maybe that agency could be developed by a recursive self improvement process, but that's far from a foregone conclusion.

Expand full comment

That's an argument that it might not be sufficient, not that it is not sufficient. Also, in this context I don't even understand what agency means or why you think its important.

Expand full comment

Since NNs don't currently have agency, the recursive self improvement process needs a human (or some other agent) to get started. If we all just sat here and did nothing, it wouldn't happen.

There's quite a bit of discussion of agency elsewhere in the comments on this post.

Expand full comment

Once a self-improvement process begins I don't see what other human intervention is necessary.

Expand full comment

"Hello, yes? This is Sheila in Accounts. We've noticed a large amount of invoices submitted from Scoggins, Scoggins and Blayne, Civil Engineers. Can you tell me who authorised this new building expansion? Sorry, I don't have the paperwork, these can't be processed until I get the paperwork. No, I've already stopped the bank payments and instructed the bank not to proceed with anymore it gets through. Well, if Mr. Mackintosh doesn't sign off on the authorisation, no payments can be made".

Sure, your self-improving AI could probably manoeuvre around that eventually, but it will have to deal with institutional inertia first, and somehow if the entire Accounts department is bypassed when it comes to paying out millions, the auditors will have a word to say on that.

Expand full comment

Agency means being able to set your own (sub-)goals. And they do have limited agency.

When it comes to working in the material world, agency would being able to run your own build-bots, and choose what tasks they operate on. (This could include contracting with outside contractors over the internet, of course.)

Expand full comment

Isn't it? An assumption that it's possible to set up a recursive self-improvement chain essentially requires that those iterations gain ever increasing autonomy to improve their design by whatever means, with an expanding capacity of interaction with the outside world. If that isn't agency then I don't know what is.

Expand full comment

Recursive self improvement can occur with access to only some means. Access to any possible means isn't necessarily required.

Expand full comment

But that's the whole meat of the AI x-risk paradigm. If the capacity growth start occuring ourside of your complete control and understanding, you can no longer be sure that access to other means would indefinitely remain off limits, especially if AI deems them desirable in the context of "instrumental convergence".

Expand full comment

What do you mean by agency? Do you think AlphaStar and OpenAI Five have agency?

Expand full comment

Within certain constraints - absolutely. The whole question of AI risk boils down to are humans smart enough to define those constraints effectively enough to prevent catastrophic results when AI gets to the point that it can expand its own capabilities.

Expand full comment

Typo: exclusives -> explosives

Expand full comment

I think the idea of recursive self improvement is one of the most overrated by non computer scientists. The current advances have mostly come through more data and assuming we scale in this fashion a smarter AI is not necessarily going to be able to keep ingesting more data (e.g. assuming we train GPT-N on X% of all knowledge in Y hours using Z bytes of data, GPT-N is not going to have a way to build GPT-N+1 in less than Y hours or Z bytes of space, nor will GPT-N necessarily be able to even scale X efficiently (learning all of Wikipedia is much easier than learning everything on the internet is much easier than learning non digital knowledge)). Even if we design a new approach to building a neural network, there is no reason to believe that there will be large areas of improvement available on that path.

Expand full comment

Reinforcement learning (or similar) would be the path used to achieve recursive self-improvement, GPT just sucks in data, it never evaluates whether it is 'good' or not.

Expand full comment

It also ignores pretty much everything known about complexity theory. Recursive self improvement is just God of the Gaps for nerds.

Expand full comment

You can't sail east to reach Asia, you'll just fall off the edge of the earth.

Expand full comment

It's one thing to just throw out low effort sarcasm instead of actually making an argument, but could you at least have the decency to not repeat popular myths about history when doing so?

Expand full comment

There is nothing about complexity theory that forgoes self-improving algorithms. But go along and assert to the contrary while adding nothing to the conversation.

Expand full comment

A tiny suggestion for fixing social media. or 2. a) Have human moderators that zing "bad" content and penalize the algorithm for getting zinged until it learns how not to get zinged.

b) Tax the ad revenue progressively by intensity of use. Some difficulty of defining "intensity" hours per day or per week? counts only if "related" page views?

Expand full comment

If you let human moderators zing "bad" content, then the algorithm will learn that vaccines are bad and Barack Obama was a Secret Muslim. If you pre-select the moderators and what can be learned, you've just recreated how algorithms are tweaked now.

Expand full comment

If Facebook is already zinging "bad" stuff, then they just need to turn up the penalty parameter in their their AI tweaking software

Expand full comment

If Facebook had a way to ensure that the “bad” content they’re zinging is actually bad, then the problem would be solved already, but we are not in that world.

The only reason that we are in the situation that we are with respect to Facebook moderation is that “have moderators zing the Actually Bad stuff (and only that)” is not an option on the table, regardless of whether algorithms are involved, or they are just doing it manually.

Expand full comment

The don't even have to zing ONLY "bad stuff" to improve things.

Expand full comment
founding

It seems like there's a potential crux here that Scott vaguely alluded to in a couple of these responses but didn't tackle quite as directly as I'd have liked: Can the current ML paradigm scale all the way up to AGI? (Or, more generally, to what Open Phil calls "transformative AI"?)

The response to Chris Thomas suggests that Scott thinks it can, since he sketches out a scenario where pretty much that happens. Meanwhile, pseudo-Dionysus seems to assume it can't, since he uses the relationship between gunpowder and nuclear weapons as a metaphor, and the techniques used to scale gunpowder weapons didn't in fact scale up to nukes; inventing nukes required multiple paradigm shifts and solving a lot of problems that the Byzantines were too confused to even begin to make progress on?

So is this the case for ML, or not? Seems hard to know with high confidence, since prediction is difficult, especially about the future. You can find plenty of really smart experts arguing both sides of this. It seems to be at least fashionable in safety-adjacent AI circles right now to claim that the current paradigm will indeed scale to transformative AI, and I do put some weight on that, and I think people (like me) who don't know what they're talking about should hesitate to dismiss that entirely.

On the other hand, just going by my own reasoning abilities, my guess is that the current paradigm will not scale to transformative AI, and it will require resolving some questions that we're still too confused about to make progress. My favorite argument for this position is https://srconstantin.wordpress.com/2017/02/21/strong-ai-isnt-here-yet/

I don't think people who believe this should rest easy, though! It seems to me that it's hard to predict in advance how many fundamental breakthrough insights might be needed, and they could happen at any time. The Lindy effect is not particularly on our side here since the field of AI is only about 70 years old; it would not be surprising to see a lot more fundamental breakthrough insights this century, and if they turn out to be the ones that enable transformative AI, and alignment turns out to be hard (a whole separate controversy that I don't want to get into here), and we didn't do the technical and strategic prep work to be ready to handle it, then we'll be in trouble.

(Disclaimer: I'm a rank amateur, and after nine years of reading blog posts about this subject I still don't have any defensible opinions at all.)

Expand full comment

Yes, it seems that there is a middle ground here: there seems to be some trick, or several tricks, that evolution has found but machine learning researchers haven't discovered yet. I think it's *not* scaling up something machine learning does already, but it's likely just a matter of inventing the right software architecture, and there's no reason to think that a group of smart ML researchers won't discover it, perhaps tomorrow or in fifty years. And once found, there's no reason to believe it won't scale.

I take GPT-3 demos as showing how existing techniques *don't* do this yet. It's an interesting imitation if you aren't paying attention, but falls apart if you really try to make sense of the output.

Expand full comment

Yeah but the worrying part about GPT-3 is:

I think GPT-3 is already more more knowledgeable and "smart" (let's say 'one-step smart', the sort of reasoning you can do at a glance) than any human alive. I think this because it'd kind of have to be to reach the quality of output that it has while also being dumb as a rock in other ways. So we may consider that "smartness overhead". If true, that suggests that once we find the secret sauce, takeoff will be very rapid.

(My two leading candidates are online learning and reflectivity as a side product of explainability research.)

Expand full comment

From the fact that GPT-3 is very effective at some tasks while being dumb as a rock in other ways, I'd reach the opposite conclusion.

This sort of 'smart along just a few directions' intelligence is also exhibited by non-AI computer programs, and your explanation makes no sense there.

Computers are dumb as sand in many ways, yet WolframAlpha will solve many complicated math problems and give you some human-friendly steps that lead to this solution. It almost looks creative how it explains what it does, but most of it is 'just' an expert system, arguably a complete dead-end for general intelligence.

Expand full comment

Yeah, but - the saying used to be that what's easy for computers is hard for humans, and what's easy for humans is hard for computers. Now I think GPT is ranging into areas where with its herculean effort, it can fake things that are easy for humans surprisingly well. I don't know if, say, a human sleepwalker could fake being conversationally awake as well as GPT-3 does, and that suggests to me that whatever thing we have that makes easy things easy for us, GPT has more of.

Expand full comment

If there a one thing that makes easy things easy for us, that GPT has more of, how do you explain its failure modes?

For instance, GPT is still imperfect at world modeling. It can write things that violate physics (GPT-2 happily had fire underwater, or other funny and clearly unintentional mistakes).

It doesn't always know how many body parts animals have, or the relative sizes of objects. It's bad at math.

And my point is very much *not* that GPT is bad, or unimpressive, or unintelligent.

But, from looking at the architecture, from looking at the failure modes of scaled-down GPT, it's plain to see that its skills are *built around* language. Then, world modeling emerges, for the sole purpose of making perplexity go down. Number manipulation, ever so slowly, becomes less disastrous. Perplexity twitches downwards.

Humans, on the other hand, are not stuck in a cave allegory made of text. We have many more inputs that teach us about the world, let us manipulate objects, and so these things are made easy for us long before we grow a propensity for writing endless amounts of convincingly pointless prose.

In other words, I think GPT-3's proficiency at immitating human writing is not a herculean effort, it is precisely the most natural thing for GPT-3 to be doing. Math, for GPT, is a herculean effort. And it sucks at it.

So when GPT-3 has visible failure modes in its primary task, I think we should conclude it's exactly as smart as it's observed to be.

Expand full comment

I seem to remember watching lots of cartoons as a kid that were equally imperfect at world modeling - fire underwater, animals with the wrong number of body parts, objects with the wrong relative size. I don't think underwater fire would happen in a cartoon series primarily set on land, like Scooby Doo, but it would not be uncommon in a series set entirely underwater, like SpongeBob, the way that Jetsons and Flintstones had futuristic/dinosaur things that didn't make sense as futuristic/dinosaur, but just as thematic copies of ordinary 1960s life.

Expand full comment

As I said, I think GPT-3 as an intelligence is "missing parts", or rather, I think the way GPT-3 works matches it to a particular part of the human mind, which is unreflective reaction. In other words, the first thing that comes to mind on considering a topic, in maybe the first 100 milliseconds or so, without awareness or consideration, assuming no filters and verbalizing every thought immediately and completely. A purely instinctive reaction, akin maybe to AlphaGo without tree search. A "feeling about the topic."

Expand full comment

I agree we're still at least one huge breakthrough away. Scott also seems to kind of agree, per

>But it’s possible that humans have a lot of inbuilt structures that make this easier/more-natural, and without those AIs won’t feel “agentic”. This is the thing I think is likeliest to require real paradigm-shifting advances instead of just steady progress.

Some OpenAI people seem to think scaling might take us all the way there, but I think the (vast?) majority of AI researchers agree there needs to be at least one novel paradigm-shifting development.

My even more amateurish, speculative, low-confidence opinion is that it may be a little like the classic half-joke for programming projects: "The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time."

------------------------

And to continue with the pattern of overly-labored metaphors:

I suspect scaling up neural networks and improving neural network techniques will get us very close to (or beyond) critical components of the human brain, but it may be like a thermonuclear fusion bomb. There's a critical balance required between the fission and fusion stages. No matter how powerful or sophisticated you make one stage, everything has to work in harmony, else you get something far weaker. The first stage is the first 90%, and the second (and potentially third) stage is the other 90%.

Alternatively, a super advanced AlphaAnything neural network (or ensemble of neural networks, or something) might be like a 50 ton antimatter hammer wielded by a gnat. The gnat's nimble and can slam it in the general direction of things and might easily level Australia, but it lacks complex self-direction and productive "agentic" attention beyond a few simple built-in reward/cost functions. (Find food, avoid obstacles/getting hit or eaten, reproduce.)

Or, instead of a hammer, it could be a neural network within a cyberbrain a la Ghost in the Shell, or some other kind of device that the gnat's brain somehow interfaces with with very low latency and that trains on the world around it and some of the gnat brain's signaling. It may become exponentially more effective at evading danger and finding food, perhaps by "intuitively" understanding physics and being able to predict outcomes much more accurately and quickly (similar to GitS "post-humans"), but that might be it. It has a very narrow set of motivations, so all of that immense predictive and analytical ability is never lifted to its true potential.

------------------------

Maybe this is where the nested prefrontal cortex layers and the attention and motivation stuff Scott talks about could come in. Some chunk of the powerful neural network could repurpose itself and take the role of directing the network and constructing a useful model of attention and motivation on top of the more base motivations. And the same might be true of humans. Especially right after birth; maybe those layers mostly start organizing and working in real-time rather than ahead of time, then eventually crystallize to a degree.

Open an adult human's skull and carefully remove certain parts of the prefrontal cortex (and maybe other areas?) and you might retain adult-like raw neural network-bequeathed skills, like being really good at throwing rocks or filing TPS reports or something, but with the attention and executive function of an infant or toddler. Something a little bit like this does seem to often happen when brain trauma damages the prefrontal cortex, I think.

------------------------

If Joscha Bach's hypotheses about consciousness and attention are right, then it's possible that the problem of artificially developing this system may overlap with the hard problem of consciousness. If so, that last 90% might take a very long time. Or maybe they could be right but it won't take that long to create and and the hard problem of consciousness will shockingly turn out to not be as hard as we thought. Or perhaps implementing consciousness and/or high-level generalizability will somehow be a lot easier than understanding it (e.g. make a blackbox AI do it), where you kind of "fake it till you make it" but can't show your work after you successfully make it.

(But in any case, the most likely answer is probably that his hypotheses simply aren't right, or are only a small piece of the puzzle.)

Expand full comment

FWIW, I think there are problems too complex for humans to understand. And that the boundary is pretty low (possibly as few as seven independent parameters). But also that *most* problems are simple enough to be handled within this bound.

OTOH, since we basically can't see the complex problems, we don't know whether they are important or not.

If this analysis is at all correct, then simple scaling up of the AI that included increased "stack depth"(metaphor) *might* be transformative. And we couldn't know until it was running. At which point its goals would determine the result. So its really important to get the goals correct NOW, which is basically impossible, because currently the AIs don't understand the existence of an external reality. But it can be worked on now, and multiple proposals evaluated and tried in toy systems. Perhaps a "good enough" result can be obtained.

Expand full comment

And why not fix spam calls by charging incoming calls just as some kinds of outgoing Call used to be charged. Say it's 10 cents per call credited to the answer's account. Not a big obstacle to normal personal calls, but it would make cold call spam unaffordable.

Expand full comment

The problem with that is that you need someone on the spammer's end (in particular, their telephone company) to co-operate in order to actually extract the money from them. However, the spammers and spammees are not co-located, and the spammers' governments have no real reason to mandate such a scheme.

Expand full comment

First my issue is not attempted fraud but unwanted cold calls; they make the telephone nearly useless. Second, it's just a different way to pay for your telephone service and does not require anyone's active cooperation.

Expand full comment

My point is that to make cold-call spam unaffordable, you have to find some way to make the spammer actually pay that 10c, which they naturally do not want to do (by assumption, it would make their business collapse).

Expand full comment

The way to do it is to charge for all calls. If a spammer or anyone else does not want to pay ATT/verizon. etc they do not have to allow them access to their circuits.

Expand full comment

It seems like this would have a very hard time getting a critical mass of phone companies agreeing in order to avoid the hassle of people with non-agreeing phone companies being unable to call you.

Expand full comment

How do you know whose account to charge? There's no authentication. The spam call from overseas pinky promises that it's coming from a number in your area and the system trusts it.

Expand full comment

A call originating overseas would be just like a call originating in the US. The caller pays a fe cents for making the call. The carrier charges the maker up front

Expand full comment

"AIs have to play zillions of games of chess to get good, but humans get good after only a few thousand games"

You can make an argument that for each chess position humans consider many possible moves that haven't actually happened in the games they play or analyze and that we can program a computer to also learn this way, without actually playing out zillions of games. It's just that it doesn't seem like the most straightforward way to go in order to get a high-rated program.

Expand full comment

this is essentially what the chess playing AIs do

Expand full comment

except they play out several scenarios all the way to end game

Expand full comment
founding

I think the evidence is that humans have a pretty general ability to 'chunk' things they're studying/analyzing/observing – something like a generic classification/categorization algorithm – and, AFAIK, there are no AI architectures that work like that.

I used to think that was an important, maybe crucial, missing ingredient in AI systems, but I'm becoming more and more skeptical that that's the case given the relentless advance of AI.

Expand full comment

> I don’t know, it would seem weird if this quickly-advancing technology being researched by incredibly smart people with billions of dollars in research funding from lots of megacorporations just reached some point and then stopped.

This wouldn't be weird, it's the normal course of all technologies. The usual course of technology development looks like a logistic curve: a long period in which we don't have it yet, a period of rapid growth (exponential-looking) as we learn about it and discoveries feed on each other, and then diminishing returns as we fully explore the problem space and reach the limits of what's possible in the domain. (The usual example here is aerospace. After sixty years in which we went from the Wright Flyer to jumbo jets, who would predict that another sixty years later the state of the art in aerospace would be basically identical to 1960s jets, but 30% more fuel-efficient?)

It seems like the 2010s have been in the high-growth period for AI/ML, just as the 40s were for aerospace and the 80s were for silicon. But it's still far too early to say where the asymptote of that particular logistic is. Perhaps it's somewhere above human-equivalent, or perhaps it's just a GPT-7 that can write newspaper articles but not much more. The latter outcome would not be especially surprising.

Expand full comment

To strengthen this point, it might be worth considering GPT-2 vs GPT-3. The key difference between the models is, rather openly, "merely" size - the latter is ~x17 times larger than the former, with no other essential changes. Does GPT-3 perform 17 times better than GPT-2? I cheerfully acknowledge that the question is a bad one - but not utterly without meaning. My intuitive, hard-to-quantify gut feeling would be that "x4" would be a better guess at whatever that is than "x17". It wouldn't be inconsistent with the ratio of improvement to scaling experienced in other domains. Whatever the key is (or more reasonably, are) to obtaining better generalization, long-term memory, "common sense" in the sense of reasonable priors, and other open problems that GPT-3 is still very much struggling with - it likely won't be just scaling. And it might or might not exist.

Expand full comment

Minor correction: not 17x bigger, but 115x bigger. GPT-3 had 175 billion parameters, while the largest version of GPT-2 had 1.5 billion.

Expand full comment

Darn! Sorry! I remembered the 175B part, and the ~10 ratio part for 2 over 1, but wrongly recalled that GPT-1 was the one with 1.5B :/

Expand full comment

Thank you! I came looking for this point.

Actually the prior on AI just stopping should be really, really high. I'm not a good Bayesian so I can't say how high but maybe it should be 1? We haven't had all that many decades of AI research in our society because computers are still pretty new, but we already had an event famously known as the "AI winter" where everyone was super optimistic and assuming that giant planet-sized brains were an inevitability, and then one day the hype caught up with it, the whole field just ran out of steam and research slowed to a trickle.

Surely if we're doing Bayesian reasoning, the chance of a second AI winter must be rated pretty highly? Especially given that there are some suspicious similarities with the prior wave of AI research, namely, a lot of sky-high claims combined with relatively few shipping products or at least relatively limited impact. Google Brain has done by far the best job of product-izing AI and getting it into the hands of real people, and that's praise-worthy, but outside of the assistant product the AI upgrades all seem to be incremental. Google Translate gets X% better, Google Speech Recognition gets Y% better, search results get Z% better and so on. They're nice to have but they aren't Industry 4.0 or whatever today's buzzword is.

Even at Google, there is a risk of another AI winter. DeepMind is extraordinarily expensive and has a noticeable lack of interest in building products. They're also the only major lab doing anything even approximating AGI research. If the political winds shifted at Google and DeepMind had its funding drastically cut for some reason, AGI research could easily enter another AI winter. Not only would the place writing most of the best papers be gone but it would send a strong signal to everyone else that it's not worth spending time on. Even if other parts of the industry kept optimizing pattern recognition NNs, agent-based learning would be dead because basically nobody except DeepMind and a few low impact university teams cares about that.

Expand full comment

I think you're somewhat overstating your case, with "relatively few shipping products". You barely ever touch a product that AI wasn't involved in. Now, much of it is not necessarily deep learning - but why is that bad? The intel chips in your laptop are AI-designed to an extent. The recommender system that offered you that next youtube video is AI-based. The ad that played mid-video was chosen by AI. You didn't buy it because it was annoying and shopped on amazon instead? The recommended products and the review summary were generated by AI. The financial transaction was ascertained as low-risk to be a fraud by an AI. The product was moved and loaded by a combination of AI planning, an actual robot and people obeying AI-generated instructions. It was then shipped by vehicles whose safety features are AI-based, using an AI-based pricing system. It was then delivered to you by people recommended for this delivery based on AI. You got an e-mail about it that was correctly not sent to spam unlike a million others that were filtered out - correctly - by an AI. Oh, it was the most popular house safety tool, Amazon Ring? Guess what powers that. Ah, sorry, it was a good old-fashioned toaster? I wonder what's behind the technology to detect defective products and save money for the factory. And when they're building a new factory - full of robots, incidentally - what's watching over construction men to ensure safety, increasingly? I could go on and on and on and on - but to summarize, you're not the customer for AI, usually. But everybody you are a customer of, is.

Expand full comment

I think a lot of your examples are conflating AI with algorithms in general.

I've actually built risk analysis systems and worked on the Gmail spam filter in the past. You could describe them as "AI" because they make probability based decisions but they were mostly just hand-coded logic and some statistics. Even non-neural conventional ML was responsible for only 1% of classifications when I worked on Gmail spam, the rest was all ordinary code + data tables with some basic stats like moving averages. These days they use more AI than they used to of course, but the system worked great before that. Just like the rest, it's incremental.

Now, that's Google where they're now putting neural nets into everything nearly as a moral imperative. It's an extreme outlier. In most organizations my experience has been that there's lots of talk about AI but relatively little usage. Also, the point at which "hand written logic+stats" blurs into "AI" is highly elastic. Like, you cite products being loaded and moved using "AI planning". Well, the algorithms for pathfinding are well known for decades and you can't really beat them using neural nets, so unless you mean something different to what I'm imagining when you say planning, I'd be very surprised by that. An Amazon warehouse with a bunch of Kivas running around is a highly controlled environment. The A* algorithm will kill that problem stone dead every single time, so why would anyone use an AI for that? It could only be slower and less reliable than conventional techniques.

Expand full comment

To clarify - I used "AI" as a shortcut for "Algorithms for which I personally know there has been major progress in the last few years". "Personally know" means "have personally worked on" or "have close friends who personally worked on and told me". I emphasize the last point to make clear that I don't rely on hype for these.

Those Kivas might run into any number of surprises, and used to. Safety and efficiency were improved by camera usage. The planning itself isn't quite AI, but combining it with camera input is non-trivial. A friend was on that team.

But let's go over my examples. The e-mail and youtube stuff - you acknowledge (though calling it an outlier). Of course, I haven't really gotten into the ad industry - but it utilizes deep learning heavily, e.g. for feature extraction from images.

Amazon famously started the whole AWS business for its internal uses, so of course it is another extreme outlier. A close friend was on the review summary team.

Financial transactions verification - this mostly uses random forests. But vanilla CART RFs won't do - it takes SOTA xgboost/catboost to be competitive here. A family member is doing that.

Ring does most of what it does using deep networks. I don't actually know anybody there personally, but I did some work for competitors in the past, sorta.

Do I need to justify vehicle safety features? Worked in that domain for a few years. 95% DL-based.

Defect detection- a friend works in a successful startup doing that. Same for construction safety. Done 100% based on deep networks.

So no, I don't think it's accurate to call all that "limited" or purely incremental. And the list was very incomplete. I'd say that's how a major shift feels from the inside.

Expand full comment

Hmm. Well, OK, you're using quite a different definition of AI than I think the one Scott is using.

Expand full comment

Hmm what quite a different definition could there be?..

If you think Scott is essentially equating AI with, say, reinforcement learning, then this is an incredibly narrow view. I find it hard to imagine he wouldn’t consider driver assistance systems as AI.

If either you or Scott equate AI with deep learning, then either you or Scott should revisit the actual structure of AlphaGo :)

If the specific example of Kivas still bothers you, feel free to set it aside. Though how a complex system combining cameras, other sensors, and sure, A*, is not an AI?

More important than the exact definition is whether *progress* in these examples is indicative of progress in AI.

Expand full comment
founding

> I think a lot of your examples are conflating AI with algorithms in general.

That's almost a meme at this point, i.e. that any new shiny algorithm is marketed as 'AI' until it's sufficiently understood well enough to be demoted to 'just an algorithm'.

Expand full comment

Let me repeat myself. The AI in question was *NOT* the A*. It was the robots detecting obstacles (humans, other robots, fallen objects etc.) using deep-learning powered computer vision and incorporating that into the planning. So no, it's not "stuff like A*". I don't want to quibble over definitions of AI, but if complex systems managing themselves using state-of-the-art computer vision isn't AI or progress in AI - what would count?!

Expand full comment
founding

> The AI in question was *NOT* the A*.

I don't know what this is referring to.

I was replying in particular to this:

> I've actually built risk analysis systems and worked on the Gmail spam filter in the past. You could describe them as "AI" because they make probability based decisions but they were mostly just hand-coded logic and some statistics.

And I don't exactly disagree, but I also don't think every 'AI algorithm' is 'intelligent', nor would that be a reasonable goal of AI research.

I had an 'old' AI textbook – from sometime in the mid '90s? There was some earlier neural network algorithms discussed but, at the time, I don't think anyone had been able to do anything really that impressive with it. Were/are any of _those_ algorithms 'intelligent'? Maybe, but I'd lean towards 'probably not' – even at contemporary scales.

But then, generally, I _expect_ AI to produce 'unintelligent algorithms' as they explore the space of 'intelligent behavior' and do something kind of like 'decompose' intelligence into a bunch of 'reductionist' components.

> ... if complex systems managing themselves using state-of-the-art computer vision isn't AI or progress in AI - what would count?!

I agree that the systems you mention ARE AI and a demonstration of AI progress.

I was commenting more on your seeming relegation of 'old AI' – things I definitely think of as _prior_ progress in AI – as 'not AI', and pointing out that that phenomena, whereby 'prior AI' commonly becomes 'not AI', is both common (and, to me, amusing).

Expand full comment
founding

Oh – "A*" actually _was_ referring to the pathfinding algorithm! I wasn't sure (as I hadn't read the entirety of your comment previously) – my bad!

But I still think A* _is_ ('old') AI and a (prior) demonstration of "progress in AI", even if it isn't – by itself – 'intelligent'.

Expand full comment

"You didn't buy it because it was annoying and shopped on amazon instead? The recommended products and the review summary were generated by AI."

Ah, so that's why the Amazon online sorting system has become really crappy now? I try searching for "tin openers" and if I go outside the "Featured" recommendations (e.g. sorting on price) suddenly the page includes dog's food bowls and seventy identical versions of the same blouse, except it's "red blouse", "blue blouse" and so on?

It really has degenerated from "I want product X - I type in name of X - it returns selection of X" to the "this is paid-for featured product/this is ninety crappy and unrelated Chinese goods".

Expand full comment

The Amazon algorithm system assumes that if you buy one item you’re starting a collection of said items. Buy a coat and you’re a coat collector. Buy a kettle and you’re now a dedicated collector of kettles. Sometimes they add in a toaster.

Expand full comment

It certainly is becoming more AI-based. Whether it also gets crappier is beyond me to say :) Anecdotally, for me it doesn’t but perhaps you’re not a typical user. Or perhaps you’re right. Or both.

Expand full comment

True, anecdotes are not data. But I have noticed over the past couple of years that I type in a search term and it comes back with results which are "No, that is definitely not a teapot, that's a lampstand. And I have no idea why the adult lingerie was in there".

Expand full comment

I was curious, so I tried "teapot" just now. Got 100% teapots, hundreds of items down the line :) And I lead a very boring life, so no adult lingerie was included.

I don't mean to doubt your experience - but that could have any number of explanations, and "Amazon is blindly using this new shiny AI to their detriment" is quite down the list, IMO. Perhaps the failures did become worse, but the average improved? Perhaps it is much better for typical users, which you might not be? Perhaps you are now further exposed to lampstands, and this increases the odds of you deciding to buy stuff not on your original list (for a generic "you")? After all, it's not about your satisfaction. It's about your money.

And then there's the fact that many, many products are not in fact entered into the system by Amazon, but by third parties. A bad description might very well account for poor recommendations - which will lead to those products not selling well, which is a bummer but not such a big deal for Amazon.

Finally, was the AI trying to flirt, with that lingerie? :)

Almost all of the above seem more likely to me than a famously data-obsessed, demonstrably successful company strategically ignoring a move to AI-based systems being disastrously bad for them, over two years of fantastic stock increases.

Expand full comment

Yeah: "You should extrapolate constant, linear progress" is a weird thing for Scott to hang a prior on given that that hardly ever happens.

Expand full comment

One interesting point about the air travel thing: I think everyone who has thought for a few minutes about the physics of air travel would naturally have predicted there would be a phase transition in travel speeds once we reached the speed of sound. But if we measure in terms of actual number of passengers that have traveled (which is probably partly a proxy for broader population and wealth, but partly a proxy for the price and quality of the travel experience) we find that within the United States it basically continued at a rate of doubling every decade, apart from a decade of stagnation after 9/11.

(I found data for 1950-1987 in this paper: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-2257.1992.tb00580.x

and data 1990-2020 here: https://www.statista.com/statistics/186184/passengers-boarded-by-us-air-carriers-since-1990/ )

Expand full comment

I can't tell whether the decade-long stagnation after 9/11 vindicates the claim that these trends stop in unpredictable ways, or should be seen as an external shock that is no challenge to the thesis that endogenous growth continues even as the most visible aspects of the technology (i.e., air speed) stall.

Expand full comment

Measuring the number of passengers is clearly goalpost shifting. It’s like using Moore’s law for computing progress and when that stagnates instead using the number of computers in use.

Expand full comment

It's definitely a different goalpost. But I think in terms of measuring the significance of flight, it's just as natural a goalpost to use as the speed of travel. I don't know if Moore's law works better in terms of size of processors, cost of processors, or speed of a given cost of computer, but it seems quite plausible that it might stall out in one of these metrics without stalling out in the others.

Expand full comment

It's not analogous, though. We're trying to predict growth in the "intelligent" seeming capabilities of software systems, not the number of deployed software systems. The birth of IOT devices probably shot that through the stratosphere more than any other event ever will, but many are among the most stupid pieces of software ever written.

Expand full comment

Yeah, I don't mean to suggest that one is clearly a better analogy than the other. My point was just that the standard clearest case of a technological trend that seemed to stall out is one that stalled in some ways and continued in others, and it's not obvious in advance which aspect is the one that is most relevant for the future changes we are interested in.

For instance, how much of the smartphone revolution we've been living through for the past decade would have happened basically the same if the phones themselves had stalled out at 2010-era capabilities but the numbers had kept increasing? How much would have happened if the numbers of phones had stalled out at 2010-era levels but the power of the individual phones had kept increasing? Reaching the power of GPS, a camera, and 3g network capabilities was probably essential to huge amounts of what happened (Uber/Lyft, Tinder/Grindr, the Arab Spring). It's not immediately obvious to me how much of the increasing power of phones since then has been significant - it's mainly been their ubiquity that matters. Though maybe there will be another transition in the capabilities, or maybe there are ways that I have underestimated the importance of the improving capacities (the way the traditional aerospace stagnation story misses the significance of the improvements in planes that did happen since then that enabled cheap vacation travel from Bratislava or Wuhan).

Expand full comment

The number of people who have ever flown isn't a technological trend, it's an economic one. There are certainly good reasons to care about economic trends, but they seem different in kind from technological advancement.

Likewise, an argument that AI will cap out in total capability not far above what we have now is decidedly not an argument that AI will be insignificant to society or the economy. Capabilities no greater than GPT-3, made fully ubiquitous and better-engineered, could be the basis of enormous economic changes. But that's a separate argument from whether and when we can expect human-equivalent AI.

Expand full comment

It seems like if you want practice at securing things against intelligent opponents, computer security is the place to be. And we aren't very good at that, are we? Ransomware attacks get worse every year.

For "hard" computer security (computers getting owned), I think the only hope is to make computers that are so simple that there are provably no bugs. I'm not sure anyone is doing serious work on that. Precursor [1] seems promising but probably could use more funding.

But beyond that there are financial and social attacks, and we are clearly helpless against them. Cryptocurrency shows that if even an unintelligent algorithm promises people riches then many people will enthusiastically take its side. (Though, it's sort of good for "hard" computer security, though, since it funds ransomware.)

[1] https://www.bunniestudios.com/blog/?p=5921

Expand full comment

We're actually fine at computer security in some cases, just not computer security in the realm of a public Internet that is supposed to allow arbitrary clients to connect anonymously, or arbitrary consumer devices that need to allow consumers to install and run arbitrary code.

But there are plenty of high-value military computer systems that have existed for over half a century as the target of much more sophisticated attackers than ransomware gangs, but have never been breached.

Expand full comment

Yes, there's always a tradeoff between convenience and security that's rarely acknowledged.

Expand full comment
founding

I wouldn't expect to know tho if any of those "high-value military computer systems" had ever been breached.

And I _would_ assume that, generally, the easiest way to breach them would be thru the humans with access to them, i.e. NOT by 'hacking' them directly from another computer with access to a connected network.

Expand full comment

I missed this one the first time around, but a more prosaic response than Scott's:

> Soon gunpowder weapons will be powerful enough to blow up an entire city! If everyone keeps using them, all the cities in the world will get destroyed, and it'll be the end of civilization. We need to form a Gunpowder Safety Committee to mitigate the risk of superexplosions."

https://en.wikipedia.org/wiki/Largest_artificial_non-nuclear_explosions

I guarantee you, if it looks like you have the makings of a new entry for that list, you'll get a visit from one of the many, many different Safety Committees. This is an example of a problem that was solved through a large, interlocking set of intentional mechanisms developed as a response to tragedy, not one that faded away on its own.

(I'll agree that nuclear belongs in a category of its own, but quibbling over whether those all count as "gunpowder" undercuts the hypothetical - that's a demand for chemical consistency beyond what our Greek engineer would have witnessed in the first place.)

Expand full comment

Good point, good link. If the Greeks had got fuel-air explosions out of Greek Fire we'd be in alt-history.

Expand full comment

Come up with an AI that can teach itself how to surf. That would be impressive.

Expand full comment

What would be impressive is not that an AI could teach itself to surf, but how an AI could "experience" the "zillions" of iterations in the real world. Not to mention, the real world is far more complicated than the sandbox environment of a board game or even video game. Starcraft behaves in the same way every time. A wave doesn't, let alone all of the other environmental factors related to the ocean and beach.

I think if we saw an AI (at least with current learning technology) trying to do a moderately complicated real-world task, we would all laugh and walk away. It's only in the world of the internet that an AI can look scary to humans.

Expand full comment

I agree we're not there yet, but it's important to remember that for humans a lot of those zillion iterations were our ancestors. We're all the product of some outer loop optimization problem. We don't start from scratch in any way. So using those zillion training loops to get something that can learn new tasks quickly is the right comparison. Again, we're not there yet but we are making progress.

Expand full comment

I think this is where the Boston Robotics "big dog" robots are really interesting, that have learned to walk and run on various kinds of surfaces. I seem to recall hearing that they got one that learned to fold laundry recently, which is an interestingly challenging task.

Expand full comment

The omniscient view. Not self-aware really. Self-awareness is much overrated. Most automation works far better as part of a whole, and even if human powerful, it does not need to self-know. ..."

Are you familiar with Julian Jaynes?

Expand full comment

"The trophy won't fit in my suitcase. I need a larger one."

- A super simple idea that we don't even think of as being ambiguous or difficult. But that's because we live in an actual world with suitcases and trophies. I'm not sure that if you skinner-boxed an actual human being in a room with a dictionary, infinite time to figure it out, and rewards for progress, they'd be able to figure it out.

It is absolutely true that the "learning blob" is wildly adaptable. But at the end of the day, it's a slave to its inputs.

That doesn't mean that AI isn't dangerous. As long as we have little insight into how it makes choices, it will remain possible that it's making choices using rationales we'd find repugnant. And as long as we keep giving algorithms more influence over the institutions that run much of our lives, they will have the capacity for great harm.

But we've been algorithming society since the industrial age. Computers do it harder, better, faster, stronger, but Comcast customer service doesn't need HAL to make day-to-day life a Kafkaesque nightmare for everyone including Comcast - they just need Goodhart. Goodhart will win without computers - it'll win faster with them, granted.

Expand full comment

I'm in the AGI is further off camp, but Winograd schemas are no longer good examples of tasks that the current ML paradigm won't solve. GPT-3 got 89% on Winograd and 77% on Winogrande in a few-shot setting.

Expand full comment

I think the reason skeptics insist on the AGI achieving consciousness is that this is the only way we know of for inferential reasoning, and brand-new ideas. Current forms of AI have zero ability to reason inferentially, and zero capability for coming up with new ideas. The only reason they can "learn" to do "new things" is because the capability of doing the new things was wired into them by their programmers. They need to find the most efficient path of doing the new things, to be sure, but this is basically a gigantic multi-dimensional curve-fitting problem, an exercise in deductive logic. From the strict comparison-to-human point of view, it's no more "learning" than my calculator "learns" when it finds the best slope for a collection of data via a least-squares algorithm, although I can understand the use of the shorthand.

We should just remember it *is* a shorthand, though, and that there is a profound difference* between an AI learning to play Go very well and a human child realizing that the noises coming out of an adult's mouth are abstract symbols for things and actions and meaning, and proceeding to decipher the code.

If you had an AI that could *invent* a new game, with new rules, based on its mere knowledge that games exist, and then proceed to become good at the new game -- a task human intelligence can accomplish with ease -- then you'd have a case for AI learning that was in the same universality class as human intelligence. But I don't know of any examples.

If you had an example of an AI that posed a question that was not thought of by it programmers, was indeed entirely outside the scope of their imagination of what the AI could or should do -- again, something humans do all the time -- then you'd have a case for an AI being capable of original creative thought. But again I know of no such examples.

*Without* creative original thought and inferential reasoning, it is by definition impossible for an AI to exceed the design principles it was given by humans, it is merely a very sophisticated and powerful machine. (The fact that we may not be clear on the details of how the machine is operating strike me as a trivial distinction: we design drugs all the time where we don't know the exact mechanism of action, and we built combustion engines long before we fully understood the chemistry of combustion. We *routinely* resort to phenomenology in our design processes elsewhere, I see no reason to be surprised by it with respect to computer programming, now that it is more mature.)

And if an AI cannot exceed its design principles, then it is not a *new* category of existential threat. It's just another way in which we can build machines -- like RBMK nuclear reactors, say, or thalidomide, or self-driving cars -- that are capable of killing us, if we are insufficiently careful about their control mechanisms. We can already build "Skynet," in the sense that we can build a massive computer program that controls all our nuclear weapons, and we can do it stupidly, so that some bug or other causes it to nuke us accidentally (from our point of view). But that's a long way from a brand-new type of threat, a life-form that can and does form the original intention of doing us harm, and comes up with novel and creative ways to do it.

-------------

* And I don't really see why you assume that difference is one of mere degree, as opposed to being a quantum leap, a not-conscious/conscious band gap across which one must jump all at once, and cannot evolve across gradually. If that were true, one would expect to see a smooth variation in consciousness among people, just as we see a smooth variation in height or weight: some people would be "more conscious" and some would be "less conscious." Through the use of drugs we should be able to simulate states of 25% conscious, or 75%, or maybe 105%. (And I don't mean "being awake" here, so that sleep counts as "not conscoius," I mean *self-aware*, the usual base definition of being a conscious creature.) How would we even define a state of being 75% as self-aware as someone else? So far as our common internal experience seems to go, being a conscious self-aware being is all-or-nothing, you either are or your aren't, it's a quantum switch. That doesn't really support the idea that it could gradually evolve.

Expand full comment

Maybe I should add that I don't doubt for a minute that an AGI *is* possible. That's because I'm a monist, and I don't think there is anything about *human* intelligence that doesn't derive mechanically from the workings of cells and molecules. Since human intelligence exists, for me that is a sufficient proof that consciousness can be built, at the very least out of proteins, and I have no particular reason to think also out of circuits on silicon.

However, I don't think we have the faintest idea of *how* to do that, and that if present progress is any guide, we are as far away from that accomplishment as were the Greeks from rational drug design once they hypothesized that all matter was made of atoms. Not decades, not even centuries, but millenia is the right timeframe, I think.

Expand full comment

>Not decades, not even centuries, but millenia is the right timeframe, I think.

I don't think we have any meaningful context from which to extrapolate what millennia of technological or scientific development looks like in a present or post-present day context. In a sense, we've had millennia with which to develop our current technology, but in another, very meaningful sense, we've only been engaged in concerted tech development for a few hundred years at best. We have individual companies today dedicating more concentrated person-hours to tech development than were likely engaged in such across the entire world 300 years ago, let alone a thousand. And those companies are networking with and building on developments from other similarly large companies, universities, etc. around the world. In some ways, our technology is inferior to what people from a century ago might have imagined we'd have today, but in many ways, it's expanded in ways they'd be unlikely even to imagine. That's just one century. Four centuries ago, relatively few people would even have thought of a question like "what will technology look like in four hundred years?" as meaningful to wonder about.

I think that unless we suppose that we'll have reached some hypothetical maximum technology state, wherein we are capable of manipulating what physical resources we have in any way that's theoretically possible within the laws of the universe, then we have essentially no meaningful ability to predict what our technology might look like at all in 1000 years, or even 300, any more than people could meaningfully predict what sort of technological problems we'd be working on today 1000 years ago.

Expand full comment

...but then you should probably change the phrasing of "it could gradually evolve", right? The one configuration of cells and molecules that achieved intelligence did precisely that.

A less nitpicky objection is that you're letting "creativity", "coming with new ideas" etc. do a lot of the heavy lifting. I have no idea what those mean. If you could give a good definition, then it could well be set as a goal to optimize for. More immediately - coming with new ideas accidentally is a thing, and it's a thing AI is very much doing. Sure, AlphaGo was just being a "sophisticated calculator" - but the results went beyond what was familiar to humans not only in mere playing strength but in patters of thought. If that isn't creativity, perhaps 'creativity' is a bad word to use in this discussion?

Expand full comment

(This was in response to the additional post, not the OP)

Expand full comment

> there is a profound difference between an AI learning to play Go very well and a human child realizing that the noises coming out of an adult's mouth are abstract symbols for things and actions and meaning, and proceeding to decipher the code.

Is there? I'm not sure there is. The Facebook BaBI tasks involved a set of simple auto-generated word puzzles of the following form:

"Mary is in the office. Mary moves to the hallway. Where is Mary now?"

The puzzles are basic but test many different kinds of reasoning ability. You haven't heard much about BaBI because it didn't last long - soon after the challenge was set, Facebook themselves built an AI that could read and answer them. Importantly this AI was self-training. It had no built-in knowledge of language. It learned to answer the questions by spotting patterns in the training set and thus worked when the puzzles were written in any language, including a randomly generated language in which all the words were scrambled.

So - AI has been capable of "learning to talk" from first principles for quite some years now. The limitation on it is, of course, that children need far more sensory input to become "intelligent" than just massive exposure to speech, and such AIs don't have that.

Expand full comment

> a life-form that can and does form the original intention of doing us harm, and comes up with novel and creative ways to do it

Intention and novelty/creativity don't have anything to do with it, they're red herrings. The concern I've seen by AI risk people mainly revolves around huge optimization power + misaligned-by-default objective functions (because human values are complex and fragile).

The right metaphor isn't sentient software "out to get us", it's just powerful planners. No intention required: https://intelligence.org/2015/08/18/powerful-planners-not-sentient-software/

The Eliezer quote that “the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else” comes to mind. Same way we have no qualms over cutting down trees or whatever.

More on fragility of value: https://intelligenceexplosion.com/2012/value-is-complex-and-fragile/

The 2016 paper 'Concrete problems in AI safety' by Google Brain / OpenAI / Stanford / Berkeley researchers lists a bunch of problems we can work on right now, as counterpoint to Andrew Ng's strawmannish "worrying about Mars overpopulation" sentiment: https://arxiv.org/abs/1606.06565

Expand full comment

I think a significant fraction of people trying to think about AGI are tripped up by the following:

Humans have this neat cognitive hack for understanding other humans (and sorta-kinda-humans, such as pets). If you tried to understand your friend Tom in the way an engineer understands a car, as a bunch of simple parts that interact in systematic ways to generate larger-scale behaviors, it would be quite difficult. But since you happen to BE a human, you can use yourself as a starting point, and imagine Tom as a modified version of you.

You don't really *understand* yourself, either. But you have a working model that you can run simulations against, which lets you predict how you'll behave in various hypothetical situations with a reasonable degree of accuracy. And then you can tweak those simulations to predict Tom, too (with less accuracy, but still enough to be useful).

I think a lot of people look at the computers of today, and they understand those computers in the normal way that engineers understand cars and planes and elevators and air conditioners. Then they imagine AGI, and they apply the "modified version of myself" hack to understand that.

And those models doesn't feel like models, they just feel like how the world is. ( https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside )

This tends to produce a couple of common illusions.

First, you may compare your two mental models (of today's computers vs future AGI) and notice an obvious, vast, yet difficult-to-characterize difference between them. (That difference is approximately "every property that you did NOT consciously enumerate, and which therefore took on the default value of whatever thing you based that model on".)

That feeling of a vast-yet-ineffable difference exists in the map, not the territory.

There might ALSO be a vast difference in the territory! But you can't CONCLUDE that just from the fact that you MODELED one of them as a machine and the other as an agent. To determine that an actual gulf exists, you should be looking for specific, concrete differences that you can explain in technical language, not feelings of ineffable vastness.

If you used words like "self-aware", "conscious", "sentient", "volition", etc., I would consider that a warning flag that your thinking here may be murky. (Why would an AGI need any of those?)

Second, if you think of AGI like a modified version of yourself (the way you normally think about your coworkers and your pets), it's super easy to do a Typical Mind Fallacy and assume the AGI would be much more similar to you than the evidence warrants. People do this all the time when modeling other people; modeling hypothetical future AGI is much more difficult; it would be astonishing if people were NOT doing this all over the place.

I think this is the source of most objections along the lines of "why would a superintelligent agent spend its life doing something dumb like making paperclips?" People imagine human-like motives and biases without questioning whether that's a safe assumption.

(Of course YOU, dear reader, are far too intelligent to make such mistakes. I'm talking about those other people.)

Expand full comment

> There might ALSO be a vast difference in the territory! But you can't CONCLUDE that just from the fact that you MODELED one of them as a machine and the other as an agent. To determine that an actual gulf exists, you should be looking for specific, concrete differences that you can explain in technical language, not feelings of ineffable vastness.

Agreed, but that goes both ways. Nebulous concepts such as "self-awareness" aside, the truth is that we humans can argue about abstract concepts on a forum; drive a car (even stick !), learn to speak new languages, and do a myriad other things. We can also learn to perform a wide (though obviously finite) array of tasks. No modern AI comes even close to doing all of that, and thus far any attempts to make it do that have met with spectacular failure.

Sure, there might not be a difference in territory, and maybe it's all in the map, and we're just trying hard enough. But maybe the radical differences in performance between humans and algorithms really are due to a radical difference in architecture -- and we can't tell what that is, because we don't know how humans work. In fact, this would appear to be the more parsimonious assumption.

Expand full comment

*just not trying

Expand full comment
founding

I'm not accusing you of doing this, but I find it very common that people compare a 'hypothetical AI' with something like 'the superset of all amazing skills and capabilities of every human ever'.

Lots of people don't in fact seem capable of 'arguing about abstract concepts', many struggle with learning to drive a stick shift car (tho most probably just avoid doing so because it's unnecessary). Many people struggle to learn a new language, particularly after the natural 'language learning window', and many struggle to do a "myriad other things" for any particular thing in that myriad.

(I'd consider any particular evidence, that people _could_ gather, about general human capabilities to be weak given that, for any particular 'thing', it'd be pretty hard to know whether the research subjects just 'didn't want to learn the thing' without, somehow, 'forcing' them to try to learn it.)

I do think we _might_ be missing some necessary architectural tricks with AI. I'm pretty sure attention and motivation, in particular, are active areas of AI research.

But I also hope that we continue to better understand how our own 'natural intelligence' works!

Expand full comment

I mean, if you say "human-level AI" it seems to carry the implicit qualifier "at the level of a median or high-performing human".

We already have "human-level AI", if you allow a comparison with someone who's in a permanent coma.

Expand full comment
founding

We already have _super_-human AI, compared to the _best_ humans – in some specific (narrow) domains.

There is no generic "median or high-performing human" across _all_ domains.

There are tho quite a few domains where probably _most_ humans perform better than _any_ AIs.

It's a very uneven 'landscape' of comparison.

Expand full comment

1) Would AI of below-(human)-average intelligence be a threat?

2) If we don’t know very much about how natural human intelligence works (and we don’t), how would we know if we were getting nearer to duplicating it in a machine?

Expand full comment
founding

1) Sure!

2) I think the current paradigm of treating 'test subjects' (i.e. AIs AND humans) as 'black boxes' and strictly testing 'functional performance' is perfectly reasonable. It seems pretty obvious to me that the best AIs really are better at chess and Go than (almost) any human. It doesn't seem necessary to also know how either AIs or humans actually play those games. Similarly, car driving seems pretty amenable to comparisons of functional performance, e.g. are injuries and fatalities (and property damage) statistically different between AIs and humans?

That written, I'm personally very interested in better understanding of how both AI and human intelligence works – and that's an active area of research anyways, fortunately.

Expand full comment

Have you updated your list of examples recently?

> we humans can argue about abstract concepts on a forum

Bruce Schneier is issuing public warnings about bots coming to dominate online political discourse:

https://www.schneier.com/essays/archives/2021/05/grassroots-bot-campaigns-are-coming-governments-dont-have-a-plan-to-stop-them.html

https://www.schneier.com/essays/archives/2020/01/bots_are_destroying_.html

> drive a car

I've heard stats that there are over 1000 self-driving cars on the roads in the US today.

> learn to speak new languages

We've got automated translation, voice recognition, and voice synthesis. (Though we're still struggling with semantics outside of constrained domains.)

Around 20 years ago, I did a cursory survey of the field, found that researchers seemed to be stuck on basic essential problems like computer vision and natural language that they had been working on for decades without obvious progress, and figured AGI wasn't coming any time soon. But since then, there have been some fairly impressive advances in those areas, too! They're not "solved" by any means, but they're good enough to fake it in some contexts, and continuing to improve.

Lots of stuff that I used to mentally classify as "wicked problems that we might not ever really solve" is suddenly moving pretty fast.

It's become a lot harder to name something that an ordinary human can do that computers haven't at least made significant progress towards.

Expand full comment

> Bruce Schneier is issuing public warnings about bots coming to dominate online political discourse...

Yes, and so far the AI attempts are laughably bad, and instantly recognizable as such. I know I'm not the best writer in the world, but would you believe that I am and AI ? Be honest.

> I've heard stats that there are over 1000 self-driving cars on the roads in the US today.

Yes, there are, but they require special hardware with lots of extra sensors; and still, their driving record is decidedly unsafe -- although they do reasonably well on freeways. Don't get me wrong, it's still a monumental achievement, but it's a long shot from the average human's ability to hop behind the wheel of any car and drive it off the lot.

> We've got automated translation, voice recognition, and voice synthesis. (Though we're still struggling with semantics outside of constrained domains.)

Yes, and again, they are almost laughably bad. There have been tremendous improvements in the past 20 years, which have elevated these tools from "totally useless" to "marginally useful in constrained domains". Again, this is a monumental achievement, but nowhere near AGI-level.

Expand full comment

*that I am an AI

Expand full comment

In that first Bruce Schneier essay I linked, he writes about how researchers submitted a bunch of AI-written comments to a government request for input on a Medicaid issue, and the administrators accepted them as legitimate until the researchers came clean.

Self-driving cars are good enough that they are being used *commercially* (Waymo).

Google Translate is usually basically understandable, for essentially any topic.

All of these are at a point TODAY where you might seriously consider buying them in some uncommon-but-plausible circumstance where you would otherwise have hired a professional human being.

The move from "totally useless" to "marginally useful in constrained domains" could also be described as moving from "pure science fiction" to "just needs some fine-tuning". Characterizing that as "not even close" and "spectacular failure" seems pretty dubious to me. Do you have a specific threshold in mind at which you would be impressed?

Expand full comment

Personally I wish we'd table the long-term, strong AI topic since, as I commented on the original Acemoglu post, these conversations are just going in circles. Do you yourself honestly feel like your understanding of this issue is progressing in any way? Or that your plan for action has changed in any way? You're at 50-50 on the possibility of true AI by 2100. So after all this, you still have no idea. And neither do I. And that's hardly changed in years. We aren't accomplishing anything here. Four of the last eight posts have been AI-related. Sorry to keep being such a pooh-pooher, but I really appreciate your writing and I feel like it just goes to waste on this. I'd love for the focus here to be on more worthwhile topics.

Expand full comment
author

I think there's still some issue where to me, if there's a 50-50 chance of the world being profoundly changed / destroyed within a few decades, that's the most important thing and we should be talking about it all the time, whereas to other people it seems like "well we don't know anything, let's forget about it". I feel like this meta-level conversation is still an important one to have and I don't know how to have it without it also involving the object-level conversation.

Expand full comment

I get where you're coming from. But there's still the issue of not having any clue what to do even IF the world is going to be destroyed in a few decades. I'm reminded of a traditional prayer: "Grant me the serenity to accept the things I cannot change, courage to change the things I can, and wisdom to know the difference."

Perhaps the fact that we're having the meta-level conversation now should serve as a promising sign that we're at least heading where I want us to.

Expand full comment

^ Similarly, I have trouble seeing what the goal is here.

This all seems to be a debate about *whether* we should worry, instead of what we would do if everyone agreed to worry.

We already developed one world-killing technology. Did public chatter about the risks of nuclear weapons save the world? Or change any of the outcomes at all? It seems to me that we're still here because the people with their fingers on the triggers were also invested in the world not being destroyed.

If we ever so much as come up with theoretical way of building this AI, every single person on earth is going to think about the risk long before it is completed. What is this scenario where they are upset that they didn't think about the risks even earlier? It's a shame we didn't worry about this 20 years earlier, or else we would've... what? Not equipped a million Amazon delivery robots with box cutters?

Expand full comment

Not every person on earth knew nuclear bombs were a possibility when they were first built. We learned about them after Hiroshima, and long after the Trinity test that, if the calculations had been wrong, would have killed every living thing on the planet and left Earth a lifeless rock.

Do you know how far Google is from a dangerous AI? Do you think the US Congress knows?

Do you think the US Congress *ought* to know? Because that's something we can work on right now.

Expand full comment

Exactly. We could very well find ourselves on the eve of apocalypse wishing we'd acted differently, but hindsight is 20/20. Aside from hardcore Unabomber Manifesto-style de-technologization (which seems preposterously out of the question, both in terms of desirability and feasibility), as of now it doesn't seem to be the case that we will have actually had any way of knowing which changes we should've made. And the experts haven't reached much of a consensus on that either, as far as I can tell.

One critique I'll make of your nuclear weapons analogy is that we don't have as good a reason to believe that good intentions like that will be able to prevent an AI apocalypse. The most plausible doom scenario to me is just that the technologies start behaving unpredictably at a point where we can't undo their own self-advancement, much as we might want to. It would be more like if, when we produced some critical number of nukes, they rather surprisingly attained some weird emergent property of all going off at once. Even under a more predictable doom scenario, like bad actors or irresponsible custodians, there are strong barriers preventing rogue individuals or groups from privately creating nuclear weapons, which seems less likely to be the case for world-destroying AI.

Expand full comment
author

If you think the world will be destroyed in a few decades, but you don't know what to do about it, I think the thing to do is to figure out what to do. At least figure out if you can figure out what to do.

Expand full comment

Several very smart people have tried to figure this out for several years and as far as I'm aware they don't seem to have produced anything very useful, isn't this evidence we can't figure it out, for now? (That is to say, the state of the field is not ripe enough to produce progress on the question)

Expand full comment

Some problems are hard enough that it takes very smart people more than a few years to solve them. But that doesn't mean that they can't be solved by having more very smart people work on them for longer.

Two examples that come to mind are Fermat's Last Theorem and the Poincare Conjecture, which took 358(!) and 102 years to prove, respectively.

Expand full comment

Those are examples of the opposite phenomenon. A mathematician working in the 19th century won't solve fermat's last theorem by thinking about fermat's last theorem, you need all sorts of a priori unrelated "technology" (commutative algebra, schemes, etc) to solve it. Something much the same is the case with the poincaré's conjecture, from what I understand.

Expand full comment

Are you familiar with every research agenda to solve AI Alignment and pessimistic about all of them? What about Debate? Microscope AI?

This sounds to me like you're thinking of Miri as the only people who work on AI alignment, when in fact they're only one pretty tiny center. They're also unrepresentative in how pessimistic they are about the difficulty of the problem.

Expand full comment

Perhaps inventing AGI is precisely the thing to do, so that something of human civilization survives on this planet once we cook ourselves. Maybe the solution to the Fermi paradox is that other civilizations keep emerging life in isolation until it either figures out that a carbon-silicon transition is unavoidable or kills itself.

I'm thinking about Asimov's story in which R. Daneel Olivaw formulates the Zeroth Law of Robotics, "A robot may not harm humanity, or, by inaction, allow humanity to come to harm", and what a robot might do in accordance with that law when it realizes that humanity is objectively suicidal. One strategy might be to kill off 90% of us immediately, because we're so far gone that it's the only way to allow the Earth's ecology to recover. Then rule with a literally iron fist for a couple of thousand years until humanity's matured enough to understand the concept of exponential growth and develop a culture capable of modifying its genetic program of exploiting every available resource. Life has to pursue that evolutionary strategy in its early days if it's not going to be extinguished on a hostile planet, but it's maladaptive once an intelligent species is consuming the plurality of net primary production. The end game would be for the robotic overlords to arrange their own overthrow and destruction, with a severe taboo against computers of all kinds, and let humanity go into the future equipped with a more effective sense of self-restraint and self-reliance and a much stronger sense that we're all in this thing together.

Oh well. What would we do without wishful thinking?

Expand full comment

As I see it, we have figured out (to a sufficient degree) that we can't figure out what to do for now. Not me at least, and from what I can tell, not the more-informed commenters here either. It's an endless back-and-forth of thought experiments which fail to reveal to us which way things will actually go. It's frustrating and definitely a little scary, but I don't see the prescriptions for action changing much until the true experts come to more of a consensus on a solution. And I hope that day comes soon.

Expand full comment

I think the question is what it takes to have "the serenity to accept the things I cannot change". The world changed drastically in many ways over the past fifteen years. Is there something my 2007 self could have done differently to prepare himself for the fact that a decade hence, he could go to any city in the developed world, and use a device in his pocket to find a good place to eat, and get there via bikeshare or scooter, and flirt with a nearby local, but that politics in every democracy would be corrupted by a vast flood of misinformation? And is there something my 2019 self could have done differently to prepare himself for the fact that a year later, he would be stuck in a boring suburb with no reasonable possibility of travel?

I think there's every reason to think that some major change in the next couple decades, whether through AI or something else in the vicinity, is going to be quite a bit bigger than the smartphone revolution, and probably even bigger than the pandemic. I'm very glad that on March 1, 2020, I went to the grocery store and bought a big supply of toilet paper and non-perishable food, but there are several other things I wish I had done to better prepare my work-from-home setup. Maybe there are things that would have been better to do to prepare for the smartphone revolution, and maybe others we can think of for the AI revolution.

Expand full comment

Hindsight is 20/20. I wish I'd invested in GME and Dogecoin on a particular day, but I had no way of knowing that would be a good idea at the time. I don't think you had a strong reason pre-pandemic to prepare a work-from-home setup, but of course it's easy to say not that you wish you did. I hope that some experts working in the AI field will be able to come up with more consensus prescriptions for action, but I've been following this topic for about 5 years now, and it pretty much feels the same now as it did then. What did those 5 years of discussion accomplish? Maybe it got more people interested in the field of AI safety? But how should we have any confidence that *that* is good in itself? So far they don't have much to show for it. Maybe we've just wasted talent and money. Or maybe we just accelerated the inevitable.

Expand full comment

I think a lot of that is right.

I think a more interesting question is whether I could have thought in the first or second week of March that I should upgrade my work-from-home set up or buy home weight equipment, instead of waiting until the end of March and discovering that many products had weeks or months long backlog.

Expand full comment

I think one must distinguish between the prepper-style actions of individuals, and the benefits of a "societal discourse" about this topic at this stage.

Expand full comment

Presumably the issue there is people's prior on other people saying "the end is nigh" being reliable is quite low. History and the world are full of people claiming that there's going to be certain doom from this presently near-invisible / very complicated phenomenon , and the doom is always just far enough in the future that if the prediction turns out to be wrong, nobody making it will care because they'll be dead or retired.

Over time people learn to tune these things out because usually:

a. They're not actionable

b. The probabilities being claimed for them are often unfalsifiable.

Expand full comment

A 50/50 chance of Singularity in the next 30 years is not just a little high; it is so absurdly high as to border on unimaginable. Of course, I am well aware that the same inferential gulf exists between me and e.g. your average Rapture-believing Christian, and I have no more hope of convincing him than I have of convincing you. There's no evidence I can provide that would dissuade the Christian, because his prior is so high that any new evidence is always going to be more consistent with the "god exists" proposition than the alternative.

Expand full comment

You sound a bit too confident, and it sounds like maybe you haven't engaged with the better arguments for AI concerns. Also I'm not sure if I'm actually at 50-50. It's more like "I continue to have no fucking clue," so 50-50 seems the best representation of that. It's just really really hard to assess. So many unknown unknowns. So yeah, then you want to rest of your priors that people have been predicting doom since the dawn of man, but then you reckon with the AI arguments a bit more, and they seem a lot more plausible than past predictions, and I just keep going back and forth like that. It's not some supernatural faith-based revelation thing like the Rapture, though. Regardless, nothing is changing. What to do about it remains as unclear as ever.

Expand full comment

There's a difference between saying "I've considered all the facts carefully and made a well-informed judgment that there's a 50-50 chance of the world being destroyed" vs. saying "I have no clue how to evaluate this situation so I'm just going to assume it has a 50-50 chance of destroying the world". The former is very important if true, the latter is just a variant of Pascal's wager and fails for the same reasons.

Expand full comment

Agreed absolutely.

Besides, shouldn't the raw prior on destroying the world start off quite low, given a). the age of the world, and b). the track record of doom prophets thus far ?

Expand full comment

I think the age of the world is pretty irrelevant in assessing AI risk, just as it would be with nuclear risks.

Expand full comment

I disagree. The prior on "global thermonuclear war destroys world" should still start off very low. But then, you can factor in all the available evidence, i.e. all the actual, physical nukes that we have detonated for real, as well as the projected number of such nukes currently existing in the world. This raises the probability to a level of concern.

Back in the 50s and 60s, nuclear war seemed to be inevitable; it's possible that "50/50 chance of armageddon in the next 10 years" was a realistic estimate at the time. Today, the probability of that happening is a lot lower.

We have nothing like that kind of evidence for AI. Not even close. At best, we have the equivalent of the beginnings of atomic theory (minus quantum physics), except it's really more like "atomic hypothesis"; perhaps something similar to the Ancient Greek idea of "indivisible elements" (which is where the word "atom" comes from).

Expand full comment

I still don't see how the first 4.5 billion years of Earth history provide you any shred of confidence that a technology which has existed for less than a century does not have the capacity to turn into something apocalyptic. What probability would you give for world-destroying AI happening by 2100?

Expand full comment

Well, the humanity arguably gained the ability to destroy the world only sometime in the past century, and while it hasn't tried to exercise it yet there were some close calls. Also, there's a general agreement that so far technological progress continues, which implies enhancement of said destructive capability, whereas it's much more doubtful whether the "sanity waterline" rises fast enough to continue preventing its deployment.

Of course, it doesn't follow fom this that there's a 50/50 chance that AI kills us all, but I'd say that there's a good reason to think that the total risk of the end of the world is significantly different from what it was a couple of centuries ago.

Expand full comment

Yes, of course -- but the risk of armageddon doesn't *start* at 50/50. Rather, the prior starts off quite low, and with each bit of available evidence (collected over decades if not centuries), it goes up a notch. We have no such evidence for AI, and yet Scott proposes that we start at 50/50. That's just irrational, IMO.

Expand full comment

Well, the whole point of the original Yudkowskian sequences was to build a framework in which it makes sense to take hypothetical future AI very seriously. For all his faults he didn't claim that we should start from 50/50, so I'm not sure why Scott said this.

Expand full comment

I'd be willing to lend you money now in exchange for everything you own in a few decades. If the world gets destroyed, I can't collect anything and you got a free lunch.

Expand full comment

Can I get in on that deal ? We could split Scott's belongings 50/50. Assuming the world doesn't end. Which I'm sure it will, any day now, so no worries !

Expand full comment

I can save you the trouble and tell you now that there's a 100% chance of the world being profoundly changed. You can just look at the past to figure that one out.

Expand full comment

> OpenAI’s programs can now write essays, compose music, and generate pictures, not because they had three parallel amazing teams working on writing/music/art AIs, but because they took a blob of learning ability and figured out how to direct it at writing/music/art, and they were able to get giant digital corpuses of text / music / pictures to train it.

One thing outsiders might not understand is how huge a role "figured out how to direct it" plays in AI software. In Silicon Valley everyone and their intern will tell you they're doing AI, but there are very few problems you can just naively point a neural network at and get decent results (board games / video games are the exception here, not the rule). For everything else, you need to do ton of data cleanup--it's >90% of the work involved-- and that means hard-coding a bunch of knowledge and assumptions about the problem space into the system. The heuristics from that effort tend to also do most of the heavy lifting as far as "understanding" the problem is concerned. I've worked at one startup (and heard stories of several more) where the actual machine-learning part was largely a fig leaf to attract talent and funding.

So here's another scale we might judge AI progress on: How complicated a problem space is it actually dealing with? At one extreme would be something trivial like a thermostat, and at the other-- the requirement for AGI "takeoff"-- would be, say, the daily experience of working as a programmer. Currently AI outperforms top humans only at tasks with very clean representations, like Go or Starcraft. Further up the scale are tasks like character recognition, which is a noisier but still pretty well-defined problem space (there's a single correct answer out of a small constant number of possibilities) and which computers handle well but not perfectly. Somewhere beyond that you get to text and image generation, much more open ended but still with outputs that are somewhat constrained and quantifiable. In those cases the state of the art is significantly, obviously worse than even mediocre human work.

My wild guess is that games are 5-10% of the way to AGI-complexity-level problems, recognition tasks are 20-30% of the way there, and generation tasks are 40-60% of the way there, which would suggest we're looking at a timescale of centuries rather than decades before AGI is within reach.

Expand full comment

Many fantastic points being made here - but let me just add that board/ video games are also very much not "naively point an AI and solve". It took a lot more than good neural networks for AphaGo, and much of that was not at all about networks in fact. Excellent engineering, data cleanup, progress in algorithms that aren't "proper AI" and what not were needed.

Expand full comment

Excellent engineering is often overlooked in AI. AlphaFold2 did a lot of little things right that the competitors didn't.

But I'm fairly confident this is nothing that wouldn't have been solved with another 10x more data and compute.

One thing we learned from AlphaZero is that it could do just as well or better than AlphaGo at Go, while having much less task-specific engineering. AlphaZero also plays very strong Chess and Shogi, and all it took was generously soaking the matrices in oodles of Google compute.

We don't have anything close to a "point any input at it, don't bother cleaning" architecture. But we are getting architectures that are more polyvalent. GPT-3 can do zero-shot learning. Transformers have been applied to anything and everything, with admirable success.

I will cheerfully predict that the bitter lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html) will stay relevant for many years to come.

Isn't it fun to spend 6 months beating SOTA by 0.3% with some careful refinements, only for the next OpenAI or Google paper to triple the dataset and quintuple the parameters?

Expand full comment

> For everything else, you need to do ton of data cleanup--it's >90% of the work involved-- and that means hard-coding a bunch of knowledge and assumptions about the problem space into the system.

This is an excellent point, and evolution already did this for us by natural selection, ie. the constraints of self-propagating competitive agents in a real world governed by natural laws.

But assuming "data cleanup" is the only key missing, this then reduces the AGI problem to learning how to feed the same inputs to existing learning algorithms and scaling the parameter space 2 orders of magntitude. If anything, that makes AGI even closer than expected...

Expand full comment

About 20 years ago, I saw a conference by Douglas Hofstadter at École polytechnique about computer-generated music. It wrote fake-Beethoven, fake-Chopin, etc. We were subjected to a blind-test, and IIRC for one of the pieces about half the public was fooled.

But in the course of the conference, it became apparent that the computer didn't output these pieces after getting the whole works of the composers as machine-readable music sheets. The authors had to analyze the works to split them into musical phrases and words: this was the real input to the program. And they had to orchestrate the output, but I suspect that is not the real blocker.

It was still quite impressive, though.

Expand full comment

Humans don't just make art, humans invented art. And humans invented games. Programs which can beat all humans at chess and go are amazing, but I might wait to be impressed until a program invents a game that a lot of people want to play.

So far as I know, programs can make art that people can't tell from human-created art, but haven't created anything which has become popular, even for a little while.

Other than that, I've had a sharp lesson about my habit of not actually reading long words. I was wondering for paragraphs and paragraphs how the discussion of the Alzheimer's drug had turned into a discussion of AI risk.

Expand full comment

Regarding art specifically, this problem is being worked at from both sides: AI is getting better, and humans are getting worse. It's not too difficult to make an AI that generates e.g. a bunch of realistic-looking paint splatters, and calls it "art".

Expand full comment

To be fair, some style transfer results could pass for decent Impressionist paintings.

Expand full comment
founding

>> Regarding art specifically ... humans are getting worse

You should probably notice that you are confused if you consider that seriously.

Expand full comment

I think this reflects a mistaken understanding of how the skill of "noticing confusion" works. "Humans are getting worse at art" is an observation which might be mistaken, but isn't obviously at odds with plausible models of human culture or development. A lot of people give credence to the notion that average standards of living in developed countries are going down. Some models predict that this shouldn't happen, but these are crude models and people don't necessarily assign much confidence to them being true.

Humans getting worse at art, given changing incentive structures or cultural landscape, isn't something people should obviously find surprising. If they observe that, it's not a clear sign that something is wrong with their models.

Expand full comment

Admittedly, the quality of painting and sculpture is a subjective metric, so if this is what you're implying than you are correct. Still, it is objectively easier to write a software program that will render a black square, as compared to rendering e.g. the crew of a sailing ship fighting to save their vessel in stormy waters.

Expand full comment

I always thought it was going to be a collection of AIs which can direct each other. One can see the power that comes in the example of the Learner + Strategy narrow AI/algorithms.

If 1 + 1 = 2...then might 1 + 1 + 1 = 3 or 1 + 2 = 3? Or perhaps at some point it is 2 controller AIs x 3 problem AIs = 6 AI power instead of only 5 AI power.

Just like in the body we have layers of systems that interact. Certainly a sufficiently well organised collection of AIs serving various function, coordination, strategy, and learner roles could become extremely powerful. It may well take a sophisticated group of humans to put it together, but I don't see why that could not happen.

The blood and plasma fill the veins and these get pumped by the heart and the entire cardiovascular system receives instructions through hormones such as adrenaline to go faster or various parasympathetic instructions to slow down.

You can do the same with the kidneys and neural nets, etc. and at some point you have a human being. I'm inclined to agree with Scott that we're on a path now and just need to keep going...at some point we hit sufficiently complex and multifunctional AIs that they are better than many humans or better than any human at more things than any one person could ever do.

Some combination of AIs to recognise problems, sort them into categories, assign tasks to those systems, and implement solutions could work to create a very complex being.

Basically we keep thinking that AI is the brain only...but that's not a great analogy. There needs to be many body like parts and I'm not talking about robotics. But many functional AIs.

Just imagine we had a third AI to the very simple Learner + Strategy AI. This AI is a 'Is it a game?' AI or a Categoriser AI.

So now we have Categoriser + Learner + Strategy. This Categoriser is like a water filter that stops stupid junk from going into our Learner.

Here's a book....Categoriser says this is not a game! rejection.

Here's a board game....Categoriser says this is a game! Accept - Learner learns...Strategy developed.

Game 2 - Categoriser ...is a game...accept - learner learns - Strategy 2 is developed.

Game 1 is presented again - Categoriser see this - Strategy 1 is deployed...Learner accepts new data.

Something like this can work. That way our Learner doesn't keel over in confusion or get clogged up with lots of useless information from a book which is not a boardgame.

This could be a single node within a series of nodes. We connect up lots of these systems with various indepnedent heirarchies of AI and bam...over time it add a lot of functionality.

We don't need a super general AI which can become super generalised to figure out how to play Go or Chess or Starcraft...we already have these. Why not simply have one AI that turns on another AI?

If we can brute force solve enough problems and over time get slightly and moderately better at creating general AI, then we can get towards some arbitrary point in space where we can say we have an AGI.

Over time and with enough testing of which AI to wire to another AI we'll likely discover synergistic effects. As in my opener above where 2 x 3 instead of 2 + 3 AIs occurs.

Now three children stacked on top of each other in a trenchcoat isn't' an adult per se. But if those kids get really good at doing a lot of things which adults do, then they can 'pass' or at least achieve certain things for us.

Throw on some robotics AI....and you can get some interesting smarter robots vs the dumb robots of today. We don't need to teach 'Spot' from Boston Dynamics how to do everything with a single AI, but can use a combination set to be the brains to drive the thing. Hopefully not straight into picking up a gun and going to war, but that'll probably happen.

But the more useful function of being able to navigate a sidewalk or neighbourhood to deliver parcels from driverless delivery trucks for a truly hands free warehouse to front door experience. If Tesla can get a car to drive, hopefully we can get a robot doggie to deliver parcels without bumping into or hurting anyone, even calling an ambulance for anyone it finds in trouble someday.

Who knows what a philosopher will call such a being, the multi-chain AI, not the robo-doggie. Is it an AGI, is it a risk to humanity, is it narrow or broad? Who cares when considering the greedy pigs who'll try to make money from it without a care or thought in their minds about abstract or future focused concepts outside of increasing their net worth...side question: are they sentient? The main question will be, is it useful? if it is, then someone will make and sell it or try to.

So yea...I see no problem with 2 x 3 = 6 AI being how a series of properly connected AI could operate. So as we move forward in a stepwise direction, we'll get increasingly complex AI.

Maybe the Categoriser is hooked up to the Learn + Strategy line for boardgames...but will redirect textual information to GPT4 or GPT5 or whatever successor GPT gets developed to improve its database of things to learn from. That could be a (1 x 2) + 1 scenario or even (1 + 2) + (1 +1) chain. The future notation or existing notation I'm unaware of will address now to denote AI chains to estimate their overall complexity.

Expand full comment

This may well be true - but firstly, let me point out that it's not a novel thought that escaped the world of AI (not saying you claim otherwise). Hierarchical AI in various forms of shapes has been an idea for ages. FWIW, literally my first ML research in 2015 was about a combination of a Controller and Experts. But secondly, practice shows that such complex schemas designed by poor humans get beaten by sufficiently clever end-to-end approaches. If you have a Controller AI that decides what Expert AI to activate, and you train those separately - consider a single model that does both, explicitly or not, and you'll probably get better results. For a concrete example - training attention models separately, and then classifying candidate detections, works out less well as end-to-end training of both components.

Expand full comment

Thanks. I'm sure AI stacking and some way of trying to measure AI performance are existing concepts, I'm not sure I've seen a generalised AI complexity notation before, but I've not read everything there is to read either :). Is it fair or unfair to stack two super simple AIs and call that a 2 vs 1 single function AI that's a lot more capable?

That'd be a challenge to work out, but in terms of design schematics going back to some arbitrary point of counting how many legos went into the castle, that's one possible approach.

I'd just say that anything can be done badly and how the sausage gets made doesn't matter to the end user. Human talent in connecting AIs so far can't really be used as a limiter to say that the way we have created better AI systems so far using 'integration is better vs connection is worse' will remain the case going forward. Forward is the key word.

When you have a driverless car it doesn't matter how separate or integrated the AIs are in terms of why anyone wants AIs which can drive cars. As far as I know Tesla has a redundancy system of one totally separate AI checking the other....that's a real world AI chain with meaning and purpose where an integration would make things less functional. What goes on within those 2 AIs in terms of how integrated they are, I'm not sure, but that's at least one example of why you wouldn't want to integrate the entire system.

I'd also wonder what the difference is in a practical way to anyone using the AI conglomerate? There may well be specific better or worse practices for a fiberglass custom builder to create patches to repair sailing vessels...but for the people who do the sailing those techniques and method choices are only measured in the durability, sailing performance, and watertightness etc. that they care about.

These skills matter and we couldn't fix boats without them, but if they were not useful boat fixing skills or useful for some other fiberglass application, then they're orphan techniques that may or may not be useful someday. I'm not doing any kind of ML research and have a low preference for these sort of details and maybe I'm biased in that way, but the ultra-majority of people using AI will be in this camp, so if it is a bias, it is worth considering.

When I look at a humanoid robot...i see legs, arms, etc. Could it be argued the whole thing is connected and just an integrated end to end collection of parts which only resemble arms and legs? Sure, but then it is just semantics and no one using that robot to carry groceries would care or be confused if you talked about the arms or legs of the robot.

If you find that Learner + Strategy is 1+1 and useful together....integrating them to just call it (1) system is not correct in my arbitrarily proposed system of AI stacking measurement. I'd say the integration doesn't change my proposed AI power/efficiency rating system and doing mathematical simplification or retaining the elbow/+ sign isn't too important.

I'd probably agree with you too that at soem point 1 + 1 + 1 + 1 + 1 +1 is clunky and it is easier to just say 6 instead. But a 6 function AI made up from clever integrations of 6 different AIs is a 6 in my mind, not a new type of 1.

If a programmer leaves it as 1 + 1 or combines them and uses a 2 within their broader AI power/capability calculations, it doesn't change or mean much to me. I'm sure that integration is a skill in and of itself and is probably a difficult thing to do. But it'd be a mistake to call those two combined AIs as just a new single AI in terms of power ratings.

Our forearm is connected to our upper arm with the elbow to make an arm. An arm is clearly more useful than just a forearm or just an upper arm or a lone bloody elbow. So I'd avoid the idea of 1 + 1 = 1 and opt for the 1 + 1 = 2 idea.

Then if we can connect or integrate (whatever works better) many such AIs we'll achieve greater and greater complexity. Will we be better a building bridges or elbows?

I have no idea, will we connect or integrate or a combination of those two systems to build better solutions? I have no prediction or reason to make a prediction about what will work and I'm sure researchers are trying both ways that fit into my metaphors and things I'm unaware of which don't fit into the connector or integration framework.

And I think that adds up to meeting Scott's idea of a functional AI using a practical approach rather than a philosophical one. As in the case of a non-swimming nuclear submarine which navigates through long distances of ocean water just fine. If we have a sufficiently multi-capable AI fit together using duct tape or complex elbows, then at some point it will cross into a grey zone of 'is this an AGI?'

Expand full comment

I'm certainly not claiming confidently that you're wrong - in fact, my very first words were "this may well be true". But for practical purposes, the distinction is very much not a semantic one. It's one of design philosophy, and end-to-end is better by end result, complexity, and other metrics you might care about across the board. For now. Will it change? Who knows. But, precisely because AI is sometimes smarter at narrow tasks, we believe in delegating the organization of information flow to the learning process. I doubt that enforcing our ideas about correct stacking will beat letting the natural training process decide, in most cases.

Expand full comment

AI was worse at most things until it was better. I'm with Cups and Mugs in thinking that the next leap is having a different kind of AI that can organize other specialist AIs. I found Scott's Learner+Strategy schema useful and imagined a Chooser that is able to find interesting problems to solve as the next step in the direction of generalization. Perhaps we'd later add an AI that is narrowly focused on understanding what the rest of the AIs are thinking, in a parallel to attention schema theory, as a way to inform the Chooser and help it make better decisions.

Expand full comment

So, it’s 2035, and there is yet another general learner, and it’s even more capable than the previous one, and there are news about yet another drone war, and there is no tangible progress from AGI risk prevention community, but there are even more research labs and agents working towards AGI, and Metaculus predictions are scary, and you have the same feeling about the future that you had in February 2020 - about impotence and adversity of regulators, and events following their own logic.

But this time you cannot stockpile and self-isolate, and this time the disaster is much worse than in 2020. So then you ask yourself what you could have done differently in order to prevent this? Maybe waiting for some smart guys from MIRI to come up with panacea was not the best plan of action? Maybe another slightly funnier Instagram filter was just not worth it? Maybe designing better covid drug was not the problem you should have worked on?

And when the time come, will you just sit and watch how events unfold? And shouldn’t you have started acting earlier, and not in 2035, when the chances of positive outcome are much smaller, and actions much more radical?

Expand full comment