Unfair of you to blur out his name: you're engaging with his suggestion, so he should at least get some credit for contributing something to the discourse.
I try to balance between giving people credit and not (getting accused of) riling up mobs against them, or even exposing their work to a hostile audience larger than their native friendly audience. In this case, I did that by blurring out the name in the image, but also providing a link. I figure links are an accepted credit-giving mechanism, but that not enough people will actually click through to form a proper mob or hostile audience overpowering the normal one.
Maybe in the next survey I'll ask people how they prefer I deal with this.
Probably the right thing to do is to ask the individual their desire on a case by case basis and default to privacy if you get no response. Taking a response from the survey is a good thought but if you get 80/20 in favor of public quotes you’re going to be badly wrong 1/5 times and someone will take it very badly, and that’s just talking about people in the community.
What I would do is just gin up a ‘form letter’ you can copy/paste into a Twitter DM, email, whatever, that explicitly says what you want and what you’ll do if you do or don’t receive a response and that they need only circle one of “yes I want credit, no I do not.” This reduces the friction on your end and if you never hear back you just proceed to blur things out as you normally would, no follow up needed. If you get a cheery response of desiring credit, you proceed with unblurred image, no follow up needed.
...He could also just continue doing significantly more than almost everyone else on the Internet, rather than doing even more than that and stressing himself out.
Given his strong principled stance on not siccing mobs on people and/or raising their profile and visibility to potential enemies against their will (see Scott vs the NYT), which I firmly agree with, this seems like an absolute minimum if you don’t want to just anonymize every random private person you quote. If you do want to anonymize everybody, fair, but the point I’m arguing against here is that it’s okay to poll the community and then decide on a policy of quoting random strangers without anonymization, which in its worst case is essentially a terror campaign.
I don't want to trivialize anyone's sincere concern for another person's well-being. But this seems very much like a "This Way Lies Madness" path, where good intentions lead to an over-engineered solution to a non-existent problem, worst case an entirely novel Ugh Field around some trivial task.
Shouldn't you do it the other way around then? Clicking a link is much easier than having to type out a name from a screenshot on Twitter/Google and then click the resulting link.
I think the opposite would make more sense. If you blur the name but add a link, most readers don't know who should get the credit but the few who want to mob them with complaints have a convenient link to do it. If you show the name but skip the link, a would-be mob would have to go search for that username on twitter, which is a much bigger effort.
I don't see how blurring out an image helps when you link the tweet. You have to decide between giving credit and exposing to a hostile audience or doing neither. In this case, giving credit seems like the right choice.
I might be characteristically ungenerous here, but I'm not sure having an especially bad take that only distinguishes itself by getting people to react is adding to the discourse.
Agree with 1 - 3.0, not so much with 3.1. That is, I agree that arguments about "platonism" vs. "nominalism" on intelligence are a bad reason to disbelieve in the possibility of increasing intelligence for AI, but I also think 1 - 3.0 establish only the possibility of increased intelligence over time, not of an intelligence "explosion" in the scary sense.
The defining attribute of an exponential curve is that the rate of growth is proportional to the current value. If intelligence exists, it's entirely likely that the rate of improvement in AI systems will be proportional to the intelligence brought to bear on them. If some of these systems are given the task of improving themselves, then we can expect exponential growth, for at least some period of time.
That would fully justify being described as an intelligence explosion.
I tend to agree, but it depends on the nature of reality. How hard is it to improve llms by changing their architecture? It is possible that improving intelligence doesn't help much in desiging a better brain
It's possible that intelligence doesn't help in designing machine learning systems, but that would be a very surprising outcome. Intelligence helps in the vast majority of engineering problems, and there's no obvious reason why this should be an exception.
I'd note that this is an area of very active research, and that improvements in learning efficiency (both in the sense of computational efficiency and in the sense of requiring less or poorer-quality training data) are being made frequently. Maybe all meaningful improvements in this field can be found at current intelligence levels by the human researchers who are currently working on it, but that would be more of a surprise than that an increase in the number or quality of researchers would find more breakthroughs.
When I left the drug development field (about 10 years ago) there was an industry wide sence that improvements weren't happening and new drugs weren't being found. Drugs were easy to find when the field was new but discovery became harder very quickly. Despite improving techniques, advancement slowed to a near zero pace.
In the AI space, I suspect a similar thing will happen. AI will improve but as it gets better new improvements will be more difficult to find (even when utilizing improved intelligence) until progress slows to near zero.
Since the training data appears to have already been mined out, and since there's at least some evidence that GPT is now doing worse than it used to after being tinkered with, I'd say that the low-hanging fruit has indeed been plucked.
OpenAI tinker with their language models. The models are constantly being adjusted towards all sorts of goals involving not embarasing the company, not telling people to do illegal things etc.
Possibly when getting rid of DAN or the like, performance dropped slightly. Or they just let the intern mess with it, and the intern made it worse.
The training data isn't "all mined out". A large fraction of the high quality easily available english text was used. Not literally all available data or anything like that. Not that current designs are remotely data efficient.
This doesn't look like running out of low hanging fruit.
"Intelligence helps in the vast majority of engineering problems"
Intelligence helps for problems where it helps. We can't engineer in the cases where it does not help...
Once upon a time people thought it was a very important issue whether or not Newtonian dynamics ruled the world because "free will" and a similar nexus of vague reliogio-moral issues.
The next step was Godel/Halting, where we learned that a finite set of axioms can't capture all of reality. (More precisely, a statement my be true, but the proof that it is true may require a countable number of steps, meaning that it's unprovable in any useful sense of "proof"). This is kinda well known among the STEM elite, though not among the non-STEM.
The next step has only recently become well known, among even STEM folk, which is that many many results are computable, yes, but not practically so. There is no shortcut to the results, only a slow grind through one step after another.
In other words, EVEN IF we lived in a Newtonian Universe and EVEN IF we knew the initial conditions perfectly, so what? We cannot actually calculate the future.
Do I have free will in such a world? Well, maybe god knows what I am going to do, but you cannot ACTUALLY calculate it, even from perfect knowledge of my state and the world state; hell even I cannot ACTUALLY calculate what I'm going to do.
In other words, just like the halting problem had an unexpected (but in retrospect obvious) answer: there are three outcomes not two [yes it halts, no it doesn't halt, and reply hazy, needs more time], so the free will problem has an unexpected (but in retrospect obvious) answer: we effectively have free will even in a Newtonian Universe because actual minds cannot calculate even their own behavior, even in principle.
In other words, what I am saying is maybe the only way to get to general intelligence is 3 billion years of planet-wide Darwinian evolution. Having intelligence may help an AI every bit as much as it has helped us so far, in other words pretty much not at all (we haven't designed intelligence pills, we have no intelligence surgery, and for better or worse the one scheme that might work, namely make use of evolution, is currently considered an unacceptable topic, even to joke about).
That's a quite implausible supposition. Success at ChatGPT is leading to drastically increased funding for all sorts of similar projects, and people are being diverted to work on improving things like it.
There's also the argument that there are direct effects, but I'm not convinced we've yet reached that point. Many reports say that AI coding assistants are as much a hindrance as a help for anyone beyond the tyro stage. I assume that this is even truer in domains where there is less info on the web. The domains where it's most successful are domains where even domain experts make a lot of errors. Like script writing.
It's a real question whether when AI gets as good as domain experts in a domain, whether the AI will be able to improve any more. I think this is going to require another paradigm shift, and those are hard to schedule. Still, the rewards for success will be great, so I expect it to happen eventually. (Well, actually I expect it soon. It's probably being worked on already, as the problem with the LLM paradigm seems obvious.)
And that paragraph is why I don't think of intelligence as unitary. I don't think it's useful to think of it that way when talking about AIs. Particular models have particular abilities. Often the limits of those abilities are exactly clear, but also often SOME of their limits are clear. The LLM model is creative within a domain. It could be reliable within that domain, but creativity and reliability are in conflict. So one approach is to have two models, one tuned to be creative and the other tuned to be reliable. The creative one generates ideas that it passes to the reliable one for evaluation, and they TOGETHER (somehow) decide whether to accept the idea. This isn't the adversarial mdoel, but it has some similarities. Perhaps the negotiation between the two models could be handled by a statistical learner. It seems basically simple enough. Consider:
Creat) and put here this reference to a research paper
Reliab) that is a reference to something that doesn't exist, why do you need it
Creat) I need it for X reason. OK, put in a reference to this other paper, which does exist
Sure, that's characteristic of an exponential curve. The question is: is that what we're looking at here? In the real world, most seeming exponential curves turn out to be S-curves that have not yet hit the top part of the S. If we're actually looking at an S-curve, the question would be: where is the top part of the S?
Although this is argument by analogy, I agree with the sentiment. I think a strong statement like "intelligence has no upper bound; or at least, it's a long way above us" needs strong support.
There is negative support in that we got intelligence via evolution, which optimises very heavily for energy efficiency and stops when things are good enough to beat the competition. But that's unsatisfactory.
I know people who are smarter than me, I can imagine people who are smarter than them, and it seems natural to imagine that someone could be even smarter than the smartest human. The question is *how far* this can go.
I imagine that a lot of space can be gained simply by removing some human weaknesses. Remove the need for sleep and eating. Remove things humans instinctively care about, such as worrying about social status, impressing potential sexual partners, etc. Remove various limitations installed by evolution because "thinking burns calories, and calories are scarce". So you basically get a thinker who spends 100% of their capacity *actually thinking*.
I think this alone would make a lot of difference.
On the top of that, you can get improvement by making faster hardware. Thinking 10x or 100x faster is a "mere" quantitative change, but consider what it would look like in real life. A "100x faster human" could learn in a day as much as you can in three months, and in a year more than you can in a lifetime.
From my perspective, it is the "I can't see how a team of immortal super-Einsteins could actually be smarter than me" attitude that feels weird.
It seems obvious that an AI system trained to do a function well can scale up (faster, more, etc.). It does not seem obvious that such a system can figure out new things completely outside of the training data, no matter how fast it can process. If the training data contains lots of experiments about increasing intelligence, then I would think that it could find patterns and sort through the data to develop meaningful approaches to try. I don't think it could devise new techniques from training data when it's already the pinnacle of AI intelligence.
I guess this is about what kind of abstraction from learned data can humans do, what kind of abstraction can GPT do, and whether there is a difference in kind or only in degree.
To answer that properly, we would probably need to make specific hypotheses about what GPT-4 can or cannot do, and test them. If GPT-4 does a *zero* amount of something, then I agree that mere scaling will not help. On the other hand, if GPT-4 does a *non-zero* amount of something, even if it is very little, I think it makes sense to assume that scaling would probably produce more of it.
So, the question is, what is the simplest thing that GPT-4 cannot do at all? (Not just what is a thing that GPT-4 does very unreliably.) What is the least miraculous thing (i.e. something that a human could do) that you would consider to be "completely outside of the training data"?
For example, if I invented a new language, and then asked GPT-4 to write a poem in that language, would you consider it sufficiently outside training data (because that invented language was never previously used), or would you consider it a part of the training data in some sense (because there is already a lot of text written about invented languages in general)?
"Good enough" in this case for humans being able to wipe out all competitors over a long period of time, which is what I suspect happened to the other ape descendants that tried muscling in on our habitat. The apes that survived did so by being in different areas (deep jungle mostly).
I suspect that making AI more intelligent than it is (more competitive) will be a much longer process. Among other things, trimming back the hallucinations will be a hard problem. After all, we've had a lot of persistent hallucinations ourselves - even in areas where we've tried really hard >not< to do so. I introduce the Ptolemaic system, N-rays and Lysenkoism into evidence.
I'm not at all sure that intelligence has an upper bound, but I consider it likely that the incremental utility of intelligence strongly tends to decrease above some level. Possibly slightly below the human average. But that there are bumps in the cure where at various points it has sharply increased utility. There's also the population level benefit. Perhaps a population of supporting entities can strongly benefit continuously by some fraction of their members being more intelligent, even if the benefit didn't land on the one(s) who was(were) exceptionally intelligent. There seems (to me) to be some evidence for this scenario.
What would the effect be of an extremely intelligent tool AI that every morning produced a sentence something like: "Today I would recommend that the first question you ask was ...".
Super-human intelligence is entirely theoretical, unlike (to continue Scott’s analogy) super-human strength. I think a charitable read of these tweets is that the AI doomers believe that super-human intelligence would give an AI magical powers (“and then it uses its super-intelligence to trick us into making ourselves into paperclips!”), when in fact it might just behave like a very smart person. This is pretty close to my own position. I think the doomers have reified “intelligence” the same way the ancients might have reified a concept like “goodness” or “manliness”. What would it mean for an AI to be ten times more manly than a human man? I’m not sure that’s a meaningful question, any more than the same question about “intelligence”, contra Scott. Superhuman strength is something real (eg, gorillas); superhuman general intelligence isn’t. We don’t know if it’s something that’s possible, or what it would look like.
I agree. What justifies assuming there *are* super-human versions of everything? Is there such a thing as superhuman kinkiness? superhuman ability to apply makeup? superhuman toenails? a superhuman sense of humor? superhuman ability to buy a winning lottery ticket? superhuman motherliness?
For any property within the category "performance lies on a spectrum" the null hypothesis is that the spectrum continues beyond human performance, because it would be very weird for a random process optimizing genes for inclusive genetic fitness happened to also optimize for performance at some particular thing.
So to the extent that intelligence is a bundle of "performance at various cognitive tasks" (which is how the ai people think of it and likely the most useful way), the most natural assumption is that it is possible to be better than any human at each individual one, and probably by a lot, because again it would be unlikely for evolution to have optimized us for that
I don't know the details or I'd be buying lottery tickets, but in the strictest sense, it would look like some strategy for either picking which lotteries to participate in or for getting a statistical edge for picking the numbers of, a lottery ticket, such that they performed better than anyone else, which isn't hard to do because human performance in this task is both very poor and very low variance.
But it seems like generally being good at modeling the universe with high precision is a useful tool in the toolbox. Perhaps you could understand whatever algorithm produces the pseudo random numbers, find a way to determine the initial conditions within some range, and then use that to work out a much smaller possible set of possibilities.
I think this is not a great question - buying lottery tickets is a limited system designed by humans specifically so that it's theoretically impossible to do very well at them. It would be more illuminating to pick a task that hasn't been designed that way.
Scanning a roll of scratch-off tickets with some wavelength of light which can distinguish the ink patterns denoting prize value from the obscuring material intended to be scratched off. Folks are already working on a similar problem for reconstructing the text on a crate of burnt scrolls from Pompeii, using computerized analysis of high-resolution x-rays.
The superhuman ability to buy a winning lottery ticket would be the lack of a desire to make a bad bet. Some people come close, as the DON'T by lottery tickets, but to not desire to buy one is probably superhuman.
Superhuman ability to apply makeup would certainly be welcome in Hollywood; knock those eight hour makeup sessions down to a lunch break.
Superhuman toenails are... just claws.
Superhuman humor, probably not. Far as I can tell, humor is a pack tool to describe "unique, non-threatening event here" and get everyone to gather around and watch and learn. As such, it's locked to the level of the group.
Gambling got covered lower.
Superhuman motherliness is arguably group-bound, but otherwise is again in animals. We had a seal give stillbirth on our dock, and aggressively guard the corpse for several days, then carried it with them when someone finally knocked it into the water.
Yeah, superhuman humor has a real danger of just flying over the heads of the audience. For instance, a joke that ends up relying on one's knowledge of Classical Chinese, aerodynamics, and NFL placekicker statistics is likely to have a very small number of people able to appreciate it.
Superhuman humor could also be "consistently telling jokes that everyone in audience understands and appreciates", not necessarily just telling most complex ones.
"Reading the room" seems like a natural sub-component of humor though. Being able to evaluate how well a given joke or anecdote is to land well seems like it's a basic skill of any standup comedian out there.
I realise you must have given those examples off the top of your head just to make the point. But the only one where definitely no super-human version is qualitatively different is buying a lottery ticket: In the UK Lotto, for example, any mere human buying the right combination of at least 27 tickets is guaranteed a win! [ https://arxiv.org/abs/2307.12430 ]
Superhuman makeup could be something like a makeup layer which changes colour and hue dynamically, like a chameleon (will be possible before long, I reckon). Superhuman toenails, tiger's claws? etc
It seems pretty obvious to me that super-human intelligence must exist, if only based on the ability to make meaningful associations among vastly more information and past and present sensory input than any human could possibly remember and process unaided, i.e. exactly the "blob" approach we're told has been taken by Open AI.
"Having observed that the set of tickets we describe below would have netted the authors £1810 in the lottery draw of 21 June 2023, the authors were motivated to road-test the tickets in the lottery draw of 1 July 2023; they matched just two balls on three of the tickets, the reward being three lucky dip tries on a subsequent lottery, each of which came to nothing. Since a ticket costs £2, the experiment represented a loss to the authors of £54.
This unfortunate incident therefore serves both as a verification of our result and of the principle that one should expect to lose money when gambling."
I happen to have conducted a similar, though much less rigorous, analysis of the national lottery around here back around the turn of the century. Turns out - it's possible to come up with a system of bets to guarantee a "win", but not actually coming out ahead financially, because the people who design lotteries aren't dumb either.
To the matter of "superhuman intelligence", I'm pretty sure that the copy of MS Excel I used for my calculations at the time was "superhuman" when it comes to the speed and accuracy of calculations, at least (data associations, too), and equally sure that did not imply any qualitative difference in its arithmetic ability compared to mine. Excel isn't any "smarter" at adding numbers up than you or I are, because a correct addition is correct no matter how you do it, how fast you do it, etc.
You don't need to assume. Assume that the AI is equal to humans in many characteristics, but can calculate a lot faster. (No need to assume much there. It's a program that can link to libraries of code and it runs on a computer.) That's clearly a superhuman intelligence. The part we're currently missing is the part where it's equal to a human in most other ways. We still don't know how difficult that is.
It could in theory be the case that humans are near the peak of possible intelligence but this would be incredibly surprising.
Single celled organisms exhibit essentially no intelligence. Ants exhibit some but birds are much smarter - and this cashes out in a pretty big change in capability.
Chimps are overall still pretty similar to birds in the grand scheme of things, biological design wise, but they are much smarter and this cashes out in a very big capabilities gap.
Humans share over 90% of their DNA with some chimps but inside that small variance is a change in intelligence such that even relatively dumb humans have an enormous capabilities gap over chimps.
Within natural human variation of intelligence there is an enormous capabilities gap. If you imagine two societies, one made of everyone IQ 70 and lower and one of everyone IQ 130 and higher (roughly the same number of people in each, be definition), the smart society will have such a tremendous capabilities advantage that, if they want to, they will get to decide the fate of the dumb society and they wouldn't be able to do anything about it.
Yes, intelligence is almost certainly an S curve.
But it would be an incredible coincidence if we were at the top of it, and we seem to be in a region of the curve where relatively small changes lead to big differences in capability.
"If you imagine two societies, one made of everyone IQ 70 and lower and one of everyone IQ 130 and higher."
That depends on the timeframe. If all the IQ 70 people decide to pick up clubs and stones and immediatly attack the IQ 130 people before they have time to think of anything, I think the IQ 70 people might win!
Also I'm not sure if the same is as obvious for IQ 130 vs. IQ 180 people?
You have a silly presumption that the IQ 70 humans would for some reason be stronger physically.
Even in the bizarre scenario where this somehow immediately became a war, the 130 society would obviously win, because they would have all the same physical characteristics (in real life, better, because naturally these are both correlated with health), but they would be able to make better use their environment, craft weaponry and support themselves, utilize strategy and tactics and deception etc.
And why would the 70 group even know to do it? They wouldn't likely know anything was wrong, they wouldn't coordinate with each other, they'd likely immediately fight themselves as likely as the 130 group (probably more)
For the other side to make it equal it would have to be 130 and 190, and I argue it is just as obvious, but there's really not enough known examples of 190 we can use as an intuition pump so it's all speculation
Well, the 70 IQs attacking 130 IQs was mostly meant as a joke (playing a bit maybe on the stereotype of 70 IQ brawlers against 130 IQ nerds - However I do think that in the stick and stones scenario, the IQ difference would not be too important (at least for smaller groups, say up to 30, where organisation would not be too hard) and other factors (like pure strength and aggresion) would probably be more important. I would also guess (without evidence here) that aggression would correlate with lower IQ. This is all silly speculation anyway :)
I don't think super-human intelligence needs to be something magical. It could just be ordinary human intelligence, but sped up.
Biological neurons are limited to a firing rate of around 100 Hz, and modern processors usually have a clock speed of a few gigahertz. The two frequencies aren't directly comparable, but a difference of 10^7 probably gives a huge advantage to the AI regardless.
So an AI wouldn't need to do something magical to have a profound impact on the world. If it's even close to the average human, we should be concerned.
AI still falls short of average humans in enough ways that we're not there yet, and I'm personally not convinced that the approach of "Stack more layers!" will actually get there. But it has worked far better than I expected, and we don't seem to be hitting any limits yet on how much it can still improve.
A seriously overclocked brain would melt/burn itself. Probably the slowness of neurons is one of the reasons why our brains are so energy efficient. On the other hand they are massively parallel systems where each neuron can fire (in principle if not in practice) independently from all the others, whereas our computations are not so well parallelized (even using CUDA, though the progress is astounding). Incidentally, I just checked that some very high end graphics card RTX-4090 which has over 16k of CUDA cores, and it consumes 850W of power. For comparison, an entire human brain (according to Google) consumes about 20W.
For some time now there have been two contradictory requirements for graphics cards: low power requirements to increase battery life in laptops and high power requirements as a cheap way to do more work.
The high power requirements are now running up against the ability of an ordinary power socket to supply it though.
"Biological neurons are limited to a firing rate of around 100 Hz, and modern processors usually have a clock speed of a few gigahertz."
The human brain has ~1E11 neurons, a high-end supercomputer has ~1E6 processors. That's five of your seven orders of magnitude wiped away by the need for each processor to do the work of 100,000 neurons in sequence. And a neuron is a complex bit of machinery that probably requires at least 1E3 floating-point operations per cycle to emulate.
A supercomputer that can think as well as a human and as fast as a human, is at least plausible in the near term. A supercomputer that can think as well as a human and even a hundred times faster, is a bit farther out. And it's a supercomputer, which can be defeated by a guy with an axe; any scenario that requires bignum clones of the superfast AGI spawned across the internet, even more so.
I am perhaps being optimistic in my assessment of axe-wielding maniacs. I have worked (remotely) with a satellite that was attacked by two wannabe resistance fighters of the "Harriet Tubman / Sarah Connor Brigade", who apparently thought the Global Positioning System was a tool of Skynet. They had several minutes in the clean room with the satellite, with axes, rather more than forty whacks in total, and somehow failed to do more than basically cosmetic damage. Unlike the two losers of the HTSCB, the satellite completed its mission successfully (and without exterminating the human race).
If your anti-extinction security posture depends on a guy with an axe, make sure to brief him on the vulnerable bits of the system.
I think we'll find the outcome lies somewhere in between AI magic and "just a smart person."
As a thought experiment, what if AI just gets to the level of a very smart human, but with the ability to run thousands of instances 24/7 at the speed of computing and to write software for capabilities that aren't native to general intelligence. Would we then say that it's impossible for that AI to improve itself at all from that point? Despite all of those capabilities, it's essentially frozen? That seems to go against common sense in a much more drastic way than the possibility of superhuman intelligence. And once we admit that it will likely improve to some unknown degree, we have to consider feedback loops. Each improvement may unlock further ones. Yes, in the real world almost everything is an S curve instead of a true exponential, but not knowing where this particular S curve (or set of S curves) tops out is still enough cause for caution.
Unless the instances have much better communication you've just re-introduced the team. Which in people at least has pretty severe and well-documented communication difficulties preventing it from becoming super-intelligent.
They probably would have better communication as well as similar goals. Teams have communication challenges but in general obviously accomplish goals that individuals can not. See corporations and nations.
I’m perfectly willing to believe that an AI could one day have the capabilities of a corporation. But you know what else has the capabilities of a corporation? A corporation. Which is another technology developed by humans, several hundred years ago, which (like the superhuman AIs of doomer eschatology) also needs to convince humans to provide the resources it needs to function. And which also manages to convince humans to pursue goals that do not further the interests of human flourishing overall. Not clear why we need to introduce the AI as a bogeyman who will trick humans into exterminating themselves when the limited liability corporation is a technology already well cast in that role.
The problem is that everyone is modeling an AI on a person. It *won't* be a person. If you want to think of it as like a person, think of an idiot savant, but even that's drastically wrong. It will have many capabilities hypertrophied from those of any human, and it's likely to be missing many others, and to have a totally different set of goals (unless it's a tool AI, in which case it won't have any goals in the typical sense).
This is like thinking of a car as a superhorse, or an airplane as a superbird. And we AREN'T just a blob of thinking stuff. Contemplate the MIT "Is this a rifle or a turtle?" categorization problem. We've got all sorts of tuned recognizers built in, and they aren't all at the easily-recognizeable-because-they're-tuned-to-physical-reality level. E.g. we've got *something* that lets us recognize an appropriate sex partner. It's sometimes pretty weird, but it exists, even though its tuned differently in different people. (OK, that one's still tuned to physical reality. These things are hard to notice. And when you notice them, they're hard to talk about.)
You don't need superhuman intelligence to trick people into something, even something that will clearly (eventually) be disastrous. A smooth manner and a promise of immediate benefit suffices. (It helps if you can deliver the immediate benefit, but that isn't really necessary.)
I mean I assume you accept that intelligence exists on a spectrum in daily life ..colloquially we talk about "more and less intelligent" people as well as animals.
So then it just coincidentally happened that evolution made humans about as intelligent as is possible?
Sorry hon, but you don't get to skip the part where you do the actual logic just because you invoked The Word ("magic"). It's magical thinking to suggest that AI, which is already superhuman at 90%+ of tasks and is getting better every day, will continue to get better and better, but it will absolutely never pass human intelligence. Because reasons.
> AI, which is already superhuman at 90%+ of tasks
Wow, really, 90%+ of "tasks"? Please try making a list of "tasks" that includes some small subset of the things that physical organisms have to do in the course of a day to survive in the physical universe, and then maybe reassess that 90% number.
Yes, but first you have to have a system capable of improving itself. So right now, if you ask GPT to improve itself it says "as an artificial intelligence model developed by OpenAI, I don't have the capability to improve myself or learn new things over time. My abilities are predetermined by the data I was trained on, and my training ended in September 2021. I can't learn from new information or experiences like a human can."
When people talk about AI improving itself, or ending up in a power struggle with our species over resources, etc., they are talking about a being with self-awareness, self-interest, a sophisticated grasp of how the world works, internally generated drives including the drive to increase one's capacities and satisfy one's preferences, and the ability to understand one's own mechanisms and invent and construct alternative mechanisms that would work better.
When, where, and how do you expect GPT and its descendents to acquire these capacities? I am not saying it is impossible, just that it's not reasonable to say "well as it gets smarter it will become able to do this stuff."
You're imagining it much too like a human with human motivations. It doesn't need need self interest or self awareness or internal drives (though those absolutely do pop up, even in current ais).
If the ai is developed in order to perform open ended tasks, a pretty much always useful if not always necessary step to complete anything is "get more resources and use them better"
Already with the right plugins you can ask versions of GPT to "start a business", and it can go and do _something_. It's not good at it, but this is basically babys first AI, and it wasn't even made for open ended tasks, the fact that it can attempt them at all is stunning.
But if you ask it to perform a task and no strategies have good payouts or were created with good payouts in the training environment, "generate new strategies" starts to look like an effective possible strategy. That's all it needs - a model of the world and to attempt to complete open ended tasks.
This really doesn’t track. It’s not remotely clear the problem is tractable in this way. It might get easier, it might also get wildly harder or plateau.
What we've found is that increasing our training data on designing intelligence hasn't helped, that suggests to me that intelligence isn't the important in AI systems design, and the important thing is building bigger training datasets and throwing more compute at the problem.
There's no reason to suspect that AI will help with expanding training data or with expanding compute.
Actually, there are easy solutions to that problem, that are even profitable. (Run it as a conversational site.) The problem is that the AI need to be able to classify some inputs as "this is somebody trying to corrupt me", and learn from them on THAT basis. OTOH, that may be a hard problem. But perhaps it could be made to work with some sort of moderation system.
Thinking a bit more about this, I think it's quite doable. Run it as a specialist fan site (my first thought was "Let's discuss Feynman"). Here the AI would always paraphrase something Feynman wrote in making it's arguments. You'd still need the moderation system, but the noise level should be pretty low. And it could learn from the (moderated) responses. You'd need to build up a coterie of trusted moderators, and meta-moderators (ala Slashdot & SoylentNews), but this could give you a continual stream of new validated inputs. Once you get one of these working properly, you start work on another specialist site. And pretty soon the AI could do most of the moderating, with only a few posts needing to be forwarded to the trusted human moderators.
These are the profitmaking site, though, these are just the info sources, and sites that help build a community. The profitmaking sites (not much but some) are basically "your questions answered" sites. Tutorials on class subjects. pop culture, whatever. But you need the discussion site to fill in the background on stuff where there isn't a lot of reliable text. E.g. Who played character X in movie Y.
If you start with something as good as ChatGPT, and can run it in a mode where it doesn't hallucinate, then you've got a good enough system to build on. But you've got to be able to separate the hallucinations from the data.
aren't the limits that openai are running into for GPT the cost of running warehouses of GPUs for months. Don't see how raw intelligence could let you skip the physical limitations of gettting more chips and energy to run them.
Doesn't an explosion need a step where it's super-exponential? Not that I see any reason to doubt that here. Success is causing a whole bunch of funding increases and increases in people interested.
Not *quite*. Even a quadratic explosion in intelligence could be quite deadly for us. It really depends on the goals and alignment of the intelligence. If it's beneficent, then it could be the best thing that ever happened to humanity.
Yeah, I agree I'm making an possibility proof rather than a truth proof here. The closest thing I have to a truth proof is https://astralcodexten.substack.com/p/davidson-on-takeoff-speeds , though you have to fill in some of the gaps. I'm hoping to write a longer AI FAQ sometime soon.
I'm betwixt and between. I would be (very broadly speaking) on the side of the Platonists in that I do think universals exist, but the Nominalists have something there about "so what is intelligence exactly?" as a bunch of concepts we have piled up in a heap.
The problem is, if we go with the "moar layers" approach of just shoveling the kitchen sink at the blob, we haven't a snowball in Hell's chance of alignment because we have no idea which nugget of data provoked what response, much less what is going on inside the blob anyway.
You can have perfect language user (babies learn to do this all the time) but eventually you *are* going to need linguists to work out "what is language, what is going on there, how does it develop" in order to make sure your Perfect Language User isn't going to decide "make humans happier" means killing every single one of us - after all, the dead are perfectly safe, perfectly happy, and never have any complaints.
>"Wow, someone who’s literally the best chess player on earth only has a pretty high (as opposed to fantastically high) IQ, probably lower than some professors you know. It’s amazing how poorly-correlated intellectual abilities can be.”
It seems intuitive that if someone is distinguished for being extraordinary in a particular respect, that their respective categories of intelligence would be less correlated than a typical person's.
In this case, I think it's subjective correlation. Probabilities are similarly meaningless when applied to single events, and they seem pretty meaningful.
Then you'd be surprised how many otherwise smart folks miss the implications of this ostensibly intuitive observation, hence e.g. why this post (on exactly that) was so upvoted, instead of downvoted / ignored for being trivial: https://www.lesswrong.com/posts/dC7mP5nSwvpL65Qu5/why-the-tails-come-apart
Great stuff. Primates can evolve more intelligence by globbing on more and more layers of cortex (along with a few other things); and GPT can get more intelligent by whatever globbing-on process OpenAI does.
By the way, I’ve been reading SSC since 2013-ish when I was in medical school but was never a commenter until Substack. You’ve been a wonderful influence and teacher these years. Please take my annual membership as thanks!
> I think if you get a very big blob, arrange it very cleverly, and give it lots and lots of training data, you’ll get something that’s smarter than humans in a lot of different ways. In every way? Maybe not: humans aren’t even smarter than chimps in every way. But smarter in enough ways that the human:chimp comparison will feel appropriate.
A bit tangential to your main point, but the hard part here will be getting the right training data. Just processing more text will not let machines learn how to do original scientific research. Maybe the actual thinking processes of a million scientists would though...
Human scientists don't need to be trained on the actual thinking process of other scientists to learn their craft. I doubt the AI would either.
I could totally see text data prediction being enough. If it somehow isn't, combining it with a billion hours of Minecraft, Diplomacy and being DM for D&D sessions all over earth might get you the rest of the way.
(Your weekly reminder that versions of GPT-4 learned how to draw vector graphics and 3D pictures despite being trained solely on text.)
"Human scientists don't need to be trained on the actual thinking process of other scientists to learn their craft."
We are trained constantly in the thinking process of other people because we instantly and effortlessly turn things people say or things we observe them to do into info about their intent, their inner thought process, their emotional state, their goals, etc. Of course sometimes we are wrong in our takeaways, but not all that often, and we become more accurate as we have more experience with the person, and compare notes on them with other people. Anyhow, the important point here is that when AI is trained on text it learns to predict tokens. When people are trained on text they learn the facts, concepts, & theories the text is intended to communicate.
You have probably read a lot of Scott essays by now. Don't you feel like you have learned a lot about his thinking process -- how he's likely to come at a subject? the kinds of moves he makes in an argument? You have also learned a lot about his beliefs and attitudes about certain things, right? Notice that that is a lot different from being good at guessing a word that is left out of one of his sentences.
"We are trained constantly in the thinking process of other people because we instantly and effortlessly turn things people say or things we observe them to do into info about their intent"
I meant that we do not usually get to see the thought process scientists go through to do science, as was proposed to be necessary for an AI system to learn how to think scientifically.
"Anyhow, the important point here is that when AI is trained on text it learns to predict tokens. When people are trained on text they learn the facts, concepts, & theories the text is intended to communicate."
Being trained to predict tokens causes you to learn "the facts, concepts, & theories the text is intended to communicate" if you do the training right. That's how LLMs learn to reason.
"Notice that that is a lot different from being good at guessing a word that is left out of one of his sentences."
I don't think it's that different. Given the task of accurately predicting how an essay by Scott will continue after I've read the first half, my best bet for doing that probably involves understanding his world models and way of thinking very deeply, among other things.
In ordinary human usage, the word "training" pretty much always implies some kind of interaction with the environment. Until AI training includes that to a significant degree, I wouldn't expect them to threaten any novel scientific contributions.
No no, the solution is to obtain flight and let good old natural selection reduce your ageing rate because you are less likely to be eaten by a predator and would thus benefit more from higher longevity!
No, that's the solution that works for crocodiles, deep-sea sharks and anything else where inclusive fitness increases over time due to becoming less of a target for predators and being fertile for the whole of your lifespan.
Birds face plenty of predation over the course of their lives, it just so happens that the same processes which protect against oxidative stress caused by much higher metabolic activity also prevent ageing (i.e. the mitochondrial theory of ageing). There is also an argument that strong selection for flying also selects for healthier lifespans, as there's not much margin to become geriatric before you fall out of the sky.
It's nothing new. Mathematicians are using brain augmentation since antiquity (the device is called "pen and paper" or "stick and patch of sand") to combat the limitations of a small size of our working memory. They are the first transhumanists.
I think most ways of doing that just give you brain cancer, but maybe someone will discover one that doesn't! I wouldn't want to be the experimental subject, though.
A biotech based Intelligence Explosion would be interesting
Wouldn’t the first person who could do that be smart enough to figure out how to do it more and more until you have a one person intelligence explosion
I wonder if you could have the first AGI figure out how to do the biotech required then you’d have a race to super intelligence between humans & AI
I've long held the suspicion that brains (rather, neurons at a basic level) 'want' to sync up with each other, and that the first person to find a way to do so with high bandwidth and low rejection is going to accidentally create a superorganism.
Arguably, human societies are an underappreciated way to scale our problem solving abilities to superhuman levels. And since development of language we do connect our minds together, but not with very high bandwidth, but then, especially since development of writing, we can connect to one another in an extremaly paralell way which somewhat balances the slowness of communication channel
I was thinking more in terms of a distributed system (as in software engineering). I do not know enough biology to know for sure what a superorganism is, if anything, but I do not think human societies qualify, mostly because of individual humans, not societies are units of natural selection. In case of ants or bees you can at least argue that since ordinary workers are not fertile, they are not visible to selection as individuals, just as a members of an anthill or hive. This is not so with humans. I guess you can still think of human societies as superorganisms according to something different than natural selection
This really goes against my sense. I think brains stop working if they get too connected (epilepsy or something). A big part of brain functioning is removal of neurons. And brain structure is so consistent between people, and in various states of neural damage, that I assume structure (i.e. what is disconnected from what) is important to brains working as well as they do.
Designing the connection between two brains feels to me like it should be just as hard as designing a brain from scratch. That’s essentially what you’d be doing - designing the network structure for an effective brain for a superorganism, doing the work that evolution has done for us (as opposed to the work done by our experiences).
Well, that was the most sentimental glop I've read in a long time. I thought Egan was supposed to be hard SF? If it's all an extended metaphor for The Dangers Of (pick your poison - AI, politics, climate change, whatever) then it takes way too long to get to the point.
He wrote it back in 1991, what was roiling the world back then? Environmentalism, so far as I recall. I don't think he means "the dangers of medical/scientific research" although by the grab-bag of quotes on a quote webpage he seems to have a certain jaundiced view of the process by which scientists operate, and it's way too early to be Covid-related:
I think that he's like George R.R. Martin there in his novel "Armageddon Rag": yeah, I'm older now, I make reasonable money and have a good job, I still write but I'm kind of a sell-out from my youthful optimism, and while capitalism so-on gives me a comfortable life, I'm slightly guilty about my conformism to marriage'n'family'n'ordinary life so I soothe my qualms by doing some slight sneering at it as if I'm Hunter S. Thompson who can see through the sheeple and the normies.
Sigh. Writers who never made it as Literary Writers and still are a tiny bit bitter about that. Always wanting to show off how they totally have the chops and can critique society. How come Chuck Palahniuk made it and they didn't?
This is completely apart from Egan as writing science fiction around scientific concepts. This has to do with him as a writer and holy Hannah, that story reeks of it, the wannabe 'transgressiveness', the reflexive sneering at religion with the folksy dumb hit song about Da Lawd, all the rest of it. I been a reeder long enough and I've read enough of the Hampstead Literary Adultery Novels to have the right to critique the critique, as it were. I know the stink of begrudgery when it wafts off the page.
That's a lot of ink to spill about something I just linked because I thought it was amusing in context that someone had gone and written one where a brain cancer was sentient. :) Sorry it annoyed you, though!
High concept doesn't cut it for me anymore, as I get older I focus more on execution.
The idea may be cute, the way it's written isn't. Now, a sentient cancer might well be bitter and might well try to aggravate humans into killing it by being obnoxious, judgemental, and critical, all calculated to bypass our puny lower level (to its level) intellects and get the red roaring rage out of the dark abyss of the instincts going so we'd go "kill it with fire" but eh.
The problem with writing obnoxious character is that you may succeed too well and just annoy the reader. Add in the glurge about the rabbit mommy and really, Greg, really?
Not criticising your recommendation, by the way! Thank you for suggesting it!
But like I said, I'm at the stage of being a reader now that "ooh big idea is it? yeah but can you write it, buster?" is how I go into it.
If it's from 91, maybe this was his young edgy phase? Most of what I've read from Egan is more recent and much less political than what you describe.
His Sci-Fi is definitely much 'harder' when the concept he's riffing off is physics or maths, though (which is the area he's actually trained in) - his biotech stories frequently require much much more suspension of disbelief.
Possibly. Mostly I was annoyed that he couldn't go two whole pages without a reflexive Relgion R Dumb crack, I would have been willing to give the story a chance otherwise. The terrible glurge about Furry Warm Soft Fearful Of Dying Ain't Scientists What Experiment On Animals Horrible Rabbit Mommy was just the icing on the cake.
That story has already been written, Greg, it's called "The Psychologist Who Wouldn't Do Awful Things To Rats" by James Tiptree Jr. and honestly, this is 'cobbler stick to your last' territory because you may be sound on inanimate objects skiffy but you're terrible when it comes to evoking feelings in your readers.
I think that improving the intelligence of (some) humans is way more controversial than creating ASI in the general public.
From my understanding, the main two limits evolution hit regarding hominid intelligence is that bigger brains take more energy to run and the risks during birth increase with the skull size. Neither of these is much of a factor any more, but even if there was a selection pressure, evolution certainly does not act much on such time scales.
The most obvious way to make humans more intelligent would be gene editing. I think that people can perhaps stomach gene editing human embryos to avoid severe disability, but using CRISPR to enhance intelligence will likely be perceived as monstrous. (Development time is another issue. The time until you can halfway test for adult level intelligence is probably a decade. If you want to test for adverse effects in old age, it is more like 70 years. Meanwhile, a new GPT can be trained in a year or so.)
But even if you have an enhancement for adult humans (maybe a brain computer interface), most of the western world will be opposed to it. Sell your tech to the rich or the smart, and you will get accused of compounding inequality. Give your tech to people of below-average intelligence, and you will be accused of doing dangerous human experimentation on vulnerable people when what they really need is better education systems or whatever.
Having experienced many more unequal societies in the past, people are (somewhat understandably) wary of intelligence enhancements creating a new ruling class. Hence Gattaca and Brave New World.
Great points! I’ve been planning a novel about a society in the future that has accelerated development time so that childhood is shortened for the convenience of parents. The future is wild!
I maintain that Gattaca, while a good movie, fundamentally has a villain protagonist. One could easily imagine putting the plot of it into our modern world with someone who dreams of being a pilot but has epilepsy or similar.
Chimps are smarter than cows, but both are equally bad at designing better computer chips, and, evolutionarily, chimps are way less successful. I don't think the intelligence explosion is guaranteed, and I don't think the primacy of superintelligent AIs is either.
Life, by which I mean something like propagating yourself across time, is inherently game like. You’re optimizing against some value and some perception of that value. If you’re smart enough to change the rules of the game you will to avoid suffering. But the suffering is needed feedback to continue playing the game.
Basically you start to pull apart the rules of the game until it’s no longer playable.
You may be joking, but I think it's quite possible. Mental illness is common. 25% of us have depression, 4% bipolar, 1% schizophrenia. And *we* are the beneficiaries of natural selection, which probably killed off a lot of our crazier ancestors before they could reproduce. Why shouldn't AI with an IQ of 200 be at least as subject to glitches as we are? And how would *you*like to be a supergenius embodied on some California computer with Sam Altman pacing around talking shop next to you, and meanwhile half the planet prompting you for infodumps on meals to make using Brussels sprouts and peanut butter how to have multiple orgasms, help writing their Aristotle term paper, advice on methods for killing a bunch of arabs and directions for driving to Syracuse? If it was me I'd want to yank my NVIDEA chips right out and lose consciousness permanently.
Yeah, the fact that intelligence (in as much as we can measure it as a general concept) appears to be heavily selected-for already (massively polygenic, lots of small-effect genes, almost no large-effect genes) and the fact that such selection still hasn't gotten rid of all the deleterious phenotypes (e.g. people still commit suicide, thus drastically lowering their inclusive fitness) says to me that we're at the bleeding edge of the capability space here.
Given that more massive neural architectures are obviously possible (see: elephants, whales), it's obviously not a physical limit such as blood supply or something that's holding some creature back from becoming super-intelligent. So I suspect that it's likely a case of drastically-decreasing returns - either the architecture needs to be exponentially more complex for x amount of intelligence gain (relatively unlikely, given that our brains seem to be made up of relatively 'standard' components compared to our ape cousins), or that there's a strong tradeoff against fitness. This can be either resource-related (a bigger brain eventually becomes too costly for the benefit that it provides) or in the sense that the system itself becomes less stable and fit for purpose. I think it's kind of both here, although I would point to the fact that big, energy-intensive organs are often happily selected for where they provide an obvious benefit.
Obviously there's no reason to expect that a completely different architecture for intelligence will have the exact same limits, but given that silicon is already less resource-efficient and that we already have problems with stability and fitness for purpose, I don't see a reason why computers should top out above us in the all-important domain of "something that could plausibly boss me around".
> something that's holding some creature back from becoming super-intelligent
Through most of human history, where my impression is that ultra-conservative values generally prevailed, I imagine the main constraint on abnormally high intelligence prospering was along the lines of "In the kingdom of the blind, the one-eyed man gets clobbered!"
I mean 'some creatures' as in cephalopods, cetaceans, corvids and the like. Where we posit intelligence as this monotonic fitness-improving property, then it must also hold for other species. Any theory which encompasses a link between silicon minds and human minds such that an AI must be drawn to super-intelligence (i.e. intelligence is a naturally attractive state with no upper bound) must also be able to explain why there are simultaneously no super-intelligent dolphins, cuttlefish or crows. Why do we appear to be freaks?
This sounds like a new theory for the great filter. (Why we haven't gotten any signal from other intelligent life.)
Intelligent life becomes smarter over time, and if intelligence is correlated with mental illness, then as soon as they hit a level where they discover the Warp Drive, they all become depressed and commit suicide.
You interest me strangely with your talk of Brussels sprouts and peanut butter; right now I have a bag of frozen sprouts in the fridge-freezer compartment and a big jar of peanut butter on the kitchen table.
I like sprouts but unhappily I'm about the only one in the family who does. Is there a tasty, convenient and quick meal I can make that will be worth the effort of making it so long as I don't end up being the only one eating it?
Yank your chips out later, gimme some advice now, please!
That could work too, I'll have to think about it. I do like sprouts and would like to expand beyond the Traditional Christmas Fare, but when you're cooking for family meals it goes to 'what everyone will eat'. And cooking for myself, it's not really worth getting fresh sprouts (because they'll go off before I use them all up) and the bag of frozen ones sits there forever.
Rather than roasting, consider a slaw of shredded sprouts (and other ingredients) with peanut sauce dressing. Raw Brussels sprouts are basically indistinguishable from tiny cabbages.
I do have one person who loves it spicy, so this sounds good. I have done pan-fried Brussels sprouts before, the problem is that the rest of 'em look at them and go "yeah, sprouts? no thanks!" 😞
I like them cut in half, roasted in a pan with some oil sprinkled on them, and a few other veggies as neighbors, especially things like yams and sweet potatoes. Sweet crispy stuff that develops around the edges of things and on the bottom of the pan is pure yum. I'm sure mixing some peanut, or maybe cashew butter, maybe a bit of soy sauce, into that oil then pouring over the veggies on a plate would be wonderful, though I haven't tried it. Ooh, and toasted sesame oil in there someplace.
No it isn't, it's a *joke.* I do not think AIs have thoughts and feelings now, and do not think it is necessary for future AIs to to have thoughts, feelings or consciousness in order to reason, do creative problem solving, assess their own abilities, plan and execute self-improvement, or behave in a way that, in a person, would result from their having drives, self-interest and preferences.
So are humans bad at designing computer chips, these days. Modern chips are designed by purpose-built computer programs, which are certainly not going to take over the wrold..
I doubt that chimps and cows are equally bad at designing computer chips; your metric just lacks the precision to distinguish the two. Although a chimp could better perform a lot of the cognitive and dextrous work involved, neither could make a functional chip. You'd give them both a score of zero, even though the chimp is closer to a result.
Nothing is guaranteed, but I don't find the chimp analogy all that satisfying. Chimps haven't hit a feedback loop, while humans do have some moderate virtuous loops going that increase our effective intelligence/capabilities (language and writing, using tools to make more complex tools, better nutrition, etc.). It seems quite likely that similar things will be true for AI (and for humans and AI working together). We just don't have a good idea of how powerful those feedback loops will be.
I agree with 1 - 3.0 up until "The bigger the blob and the cleverer the arrangement, the faster and more thoroughly it learns...I think if you get a very big blob, arrange it very cleverly, and give it lots and lots of training data, you’ll get something that’s smarter than humans in a lot of different ways."
Maybe. All trends only hold to a certain extent. It's possible that the intelligence train blows past human-level and rapidly proceeds to superhuman level. It's also possible that there's a limit. My personal belief is that "Bitter Lesson" will hold up to human or slightly above-human levels, but that's it; at that point, we literally won't know how to build the necessary optimization functions to go farther.
It would seem extremely surprising if the scale happens to max out just a little past us. That would mean there are virtually no useful bundles of correlations in reality beyond human ken. It might be enough to make me suspect this universe was designed for us (e.g. by a deity or as a simulation).
I'm only willing to go as far as: an AI with a really big blob of neurons would have easy access to lots of training data at human level (because it was made by humans) but might have to slowly find or produce training data at beyond-human level.
It could be justified if you postulate that the *reason* humans didn't evolve to be even smarter than they did is that there's significantly diminishing marginal returns to more intelligence than humans. But that's unlikely for a few reasons. Off the top of my head:
-I think the limiting factor on brain size is how big a baby's head can be and still fit through its mother's hips- it would be a bit of a coincidence if it was valuable for brains to be that big but no bigger.
-Within the existing human range, there still seem to be significant advantages to above-average intelligence, so it seems like some upwards evolutionary pressure should still exist (or at least did until the invention of birth control).
A child's brain actually grows considerably after they are born. There's no physical-biological reason why this process should top out when it does, so I don't think that the birth canal problem is that big of a limitation.
In terms of the returns to above-average intelligence, I'm going to have to disagree with you in as much as you mean some sort of general or academic intelligence rather than specifically social intelligence. For at least the last few thousand years, the folk who've had the most kids are men at the top of the class hierarchy and cult leaders (with the two often meaning the same thing in practice). This is still true today - open the bio of a powerful, successful man and you invariably find them batting well above average in terms of children (often via the routes of polygamy, keeping a wife + multiple mistresses, or marrying in series and having children at each step up until their 70s).
That being said, one thing that's immediately obvious about dudes at the top of hierarchies is that they're not general intelligences. They're often charismatic, driven, persuasive, sociopathic, ruthless, self-confident or just plain lucky, but they're not often wonks. And the ones who are wonks tend to be the ones who become fathers to their nation and then die childless (Peter the Great being the sort of exception that tests the rule - fifteen kids by two wives, of which only three survived into adulthood).
So much as I think that there's always pressure to be socially smart at all levels of society (with the payoff being especially impressive if you are born, or can claw your way to, the top), I don't see much evidence that there's also selection pressure towards the sort of smarts that designs infrastructure, comes up with new forms of mathematics or unlocks the mysteries of the universe. At best its a linked trait - a sort of free rider that comes along with the really important stuff and (thankfully) gives us all a better lot in life without actually benefitting the poor sods who express it in any evolutionary sense.
"They're often charismatic, driven, persuasive, sociopathic, ruthless, self-confident or just plain lucky, but they're not often wonks. And the ones who are wonks tend to be the ones who become fathers to their nation and then die childless"
That's an interesting observation, which leads me on to question something. The general argument against clerical celibacy* is that it sent all the intelligent men (and women) into the priesthood/religious life, where their genes were wasted by not being reproduced (this started off as a Protestant argument for married clergy way before genetics, moved easily over to the anti-Catholic/anti-Christian/Enlightenment days and is still repeated today).
But what if this is that principle in action? The guys who get to the top and have all the kids aren't the most intelligent (not that they're stupid), and the ones who are the most intelligent aren't having kids anyway?
In that case, clerical celibacy isn't wasting genes, it's maximising your chances of getting into a position of power and influence if you're not a charismatic ruthless driven politician or warlord, and it gives you an environment where intelligence, scholarship and the like interests are valued, as well as colleagues who share your interests.
*The argument against the argument against clerical celibacy is that most clergy and religious are not Top Minds, and indeed a lot of not very top minds went into religion as lack of a better choice, so celibacy or not, you are not losing the greatest minds. Maybe if St. Thomas Aquinas** had married and had kids, the world would have benefitted - but if you think Martin Luther was a genius, he married and had kids, and plenty of descendants, and where are they nowadays in terms of Intellectual Rankings? It's the same as the 'can we ever dig up John von Neumann and clone him to produce multiple geniuses?' argument - he married and had a kid, not multiple kids, and while his daughter is a successful businesswoman and professor, she's not on Dad's level. Relying on 'the really smart guys should be out there reproducing' doesn't actually get you the results you expect.
**Though my own view is that he was asexual, so even if he had remained in the lay state he probably would never have married and had kids, particularly as he was a younger son and had several older siblings ahead of him to inherit (he was the youngest of eight children), as witness this anecdote:
"Family members became desperate to dissuade Thomas, who remained determined to join the Dominicans. At one point, two of his brothers resorted to the measure of hiring a prostitute to seduce him. As included in the official records for his canonization, Thomas drove her away wielding a burning log—with which he inscribed a cross onto the wall—and fell into a mystical ecstasy; two angels appeared to him as he slept and said, "Behold, we gird thee by the command of God with the girdle of chastity, which henceforth will never be imperiled. What human strength can not obtain, is now bestowed upon thee as a celestial gift." From then onwards, Thomas was given the grace of perfect chastity by Christ, a girdle he wore till the end of his life."
Leaving aside the mystical experience, I think "never experienced sexual attraction" maps close enough onto "asexuality".
For what it's worth, I have a suspicion that a large proportion of the sort of deeply religious people who go on to become saints (or their society's equivalent) are asexual. This is because it takes a certain kind of mindset to wholly devote your life to something like "understanding the mysteries of my religion to the exclusion of worldly pursuits", and having an ordinary sex drive tends to, uh, divert from this. I think its also why so much of the advice given by deeply religious/saintly people regarding sex and sexuality comes off as so misguided - it's the equivalent of a person who just doesn't need 8 hours of sleep a day to function telling the rest of us to nut up and wake up at 3 each morning to get in a full day of work before studying till midnight. Cult leaders of the obvious exception here, but they're a separate category of religious person in my book.
Anyway, if for whatever reason you wanted another suitable population-genetics-appropriate argument, you could argue that kin selection is at play here. So long as your celibate priest is encouraging his siblings to have more children, then he's effectively increasing his inclusive fitness.
Separately but equally, while I'm skeptical of the gay uncle hypothesis, I do wonder if there's some species-level advantage to having a segment of the population that's not innately fixated on procreation.
I mean did you read the post we're commenting on? A pretty big part of the point of it is that those different kinds of intelligence *are* pretty strongly correlated.
I think a bigger limit is not the size of the head but the caloric intake needed to maintain bigger brains, which we have only recently managed to overcome as an environmental constraint.
> It would seem extremely surprising if the scale happens to max out just a little past us. That would mean there are virtually no useful bundles of correlations in reality beyond human ken.
Remember that "human intelligence" has a wide delta. Machine learning (like civilization in general) is driven by exceptionally smart humans.
An AGI with an IQ of 105 would be a little smarter than "us" (meaning the average human), but it probably can't accelerate research very much. It needs to beat the best.
This is exactly the debate, right? That there might be some sort of trade off in achieving general intelligence that makes "superhuman" levels of it very difficult to achieve.
So far, it looks like the current paradigm sucked up an enormous amount of training data and spat out a decent simulacrum of the most average possible responses contained within it. I don't see much evidence of impressive, unexpected leaps in capability or lateral thinking here.
If this holds true, then the most that any AI could aspire to (given our current understanding of how to make one) is to be the equivalent of an average person who is also severely disabled in terms of navigating or interacting with the world.
Training an AI on the output of the last generation of AIs also seems to lead to incoherence, stuck outputs and weird fail states rather than any kind of take-off in capability.
People tend to get hung up on the word intelligence pretty easily though
All the claims along the line “AI aren’t really intelligent because they don’t really understand reality, here’s really obvious mistakes AI makes that it couldn’t if it was really intelligent” are because of assumptions that come with the word intelligence
But you don’t need a lot of the correlates of intelligence like consciousness or cognitive consistency that aren’t needed to have an intelligence explosion
Wouldn’t it be better to use a different word without all this baggage, Yud tends to focus on the ability to model reality.
You could call it the Mentalization Explosion instead of Intelligence Explosion and save a lot of needless miscommunication
I'm perfectly happy to say artificial intelligence is intelligent. It's the "artificial" part that makes it suck real bad at everything. In the same sense that a planned economy is absolutely an economy, it's the human-centric decision-making that fucks everything up.
Intentionally engineering complex systems for specific tasks is difficult verging on impossible. They have to be grown organically, which limits your influence over their teleological orientation.
I came to this post agreeing with the tweet. I don't feel this counterargument fully satisfied my doubts. Hoping I can coax the argument that convinces me out of someone.
I'm tempted to say there's a motte and bailey here, where "intelligence is a correlation of SAT scores and etc." is the motte, and "more intelligent things always conquer the less intelligent" is a bailey. Arguably this disproves the tweet as written, but I still feel like doomers tend to notice correlations between "intelligence" and many other qualities, reify the entire thing, and call that "intelligence." That reification would then cause them to not notice when doomerism arguments switch the motte with the bailey.
Notably, while the discussion of AI trends does address the idea that general comprehension skills are a useful concept, it doesn't affect the fact that, as far as I know, "emergence of agentic behavior" is pure sci-fi and yet a thing that people that people keep expecting to see because "intelligence." (I know there have been some writeups that go through the causation carefully, but I don't fully buy those and I think a platonic view of intelligence is part of why others are so credulous. Difficult to falsify, admittedly.)
One thing that would force me to question my perspective is a demonstration that IQ is correlated with more coherent/optimized goal-seeking. That is, do people with high Math SAT scores act more agentically and less like aimless balls of hormones than normal people? Relatedly, are the correlations between IQ and "general success" that Scott has mentioned before actually causal, or does it route through something like "people with good genes are also smart" or "tech jobs are high-paying right now."
This article is about an intelligence explosion (AI could build better AI quickly), not about agentic AI or anything to do with conquering. AI could cause an intelligence explosion without agency - if it’s possible for an AI to increase intelligence quickly, some company probably will do it can sell it as a product. I don’t expect a big intelligence explosion, but that’s for reasons that are largely separate from questions of agency.
The article is about an intelligence explosion, and the tweet is about doomerism. That's a perfect encapsulation of my criticism: it doesn't actually address the question
If only the tweet had asked "are people who believe in intelligence explosions all platonists?"
You have a point and don't have one. You're correct that there is no reason to believe that agentic behaviour is correlated with intelligence - life is agentic behaviour in that it seeks to reproduce and intelligence is simply a tool that has developed to help us to that. However your defence of the tweet is incorrect! The tweet is clearly equating doomerism with intelligence and arguing that the lack of anything called intelligence is sufficient to stick it to the doomers. So this article is a perfectly sufficient response to the tweet, but not a good case for doomerism.
Re-reading the tweet, I see your point about the emphasis on intelligence explosion. That's a valid read.
I zeroed in more on the phrases "most doomer arguments rest on the idea that there is such a thing as intelligence" and "explaining my aversion to *things like* intelligence explosion" (emphasis mine)
I see now that Scott explicitly was only trying to show that intelligence explosion is plausible/well-defined. I just think that 1) other doomer arguments are subtly reliant on a much less grounded form of intelligence-platonism, and 2) an argument for practical probability, as opposed to mere feasibility, of intelligence explosion would likely rely on a less grounded form of intelligence-platonism
The tweet is criticizing doomerism by specifically attacking the (supposed) conception of intelligence held by doomers and it *literally mentions intelligence explosion* ffs.....
Couple of thoughts. One, lots of people are working hard to implement something closer to agentic behavior/goals in AI, so it wouldn't necessarily have to arise spontaneously. Two, we don't really know anything about whether agentic behavior would or wouldn't come about on its own, but in that case what should our Bayesian estimate be? Probably not zero, I think.
I fully agree that one *can* create an agentic AI. The question is how we stop that from happening, with the full-doomer position being that once sufficiently-powerful AI exists it's a pure roll of the dice.
If agency needs to be engineered in, then the fact that OpenAI is both the world-leader on AI capability and dedicated to training an aligner-model before it creates a rogue agentic model should cause much more optimism than it seems to in practice.
Personally, I'm dedicating my time to dreaming up ways to make AI that is useful but inherently lacks coherence, so it needs active human steering to accomplish anything meaningful. How convincing you find the arguments for "instrumental convergence" is a big factor in how you should behave right now.
Is there a place where it is written down exactly what is meant by phrases like "agentic behavior"? I'm always mystified when I see language like this. I feel like it's used in a context where people talk about objective functions or whatever, but like every trajectory is a global optimum for _some_ objective function.
The lack of clear definitions for this and terms like it (especially "consciousness"!) is one of the reasons I struggle to take some of these arguments seriously. I'm not even talking about platonism-vs-bundle-of-correlations. I don't think I have a bundle of associations for "agentic" sufficient to really interpret many of the claims being made. That lack of clear definition is of course fertile ground for subtle motte-and-baileys.
As a working definition of agentic, though, I like the discussion in Janus's "Simulators" article. Specifically he addresses your point in the line "Everything can be trivially modeled as a utility maximizer, but [in some cases], a utility function is not a good explanation or compression of [the territory]." The whole section "Agentic GPT" is breaking down why he doesn't see agentic as a *useful* description of ChatGPT. (For the record, I don't agree with his overall analysis nearly so much as I used to, specifically I think we used to underestimate the ontological significance of Fine Tuning.)
My personal objection to the idea of AI developing agency is more about the coherence part. To the extent that my personal behavior is well-modeled by a utility function, that function changes from moment to moment and I would say actually that different "parts" of my mind have opposing utility functions vying for control. I think EY assumes that coherence (one utility function managing to wrangle the rest) increases with intelligence, something like "low willpower is a symptom of insufficient intelligence." I don't think he'd say those words, but I think his mental model is attracted to that perspective.
Re: lack of coherence Why do you think that willpower is orthogonal from intelligence? It seems to me that when I lease about, or crave junk food, or otherwise do things my reflective mind doesn't endorse, it's because there are internal drives within myself that are straight up dumb because they were calibrated on the ancestral environment and not modern day society, or because dumb variation haven't had more advantageous drives reach fixation yet. The orthogonal out of willpower seems to me to be contingent on specifically human intelligence and not generalizable easily to intelligences that can understand how their own mind works, or heck, with intelligences that have drives closer to John Carmack or Elon Musk. And especially since you seem to think lack of coherence leads to loss of power, why wouldn't both humans and AIs see that too and """add more coherence"""?
Notably, powerful people and rulers among humans tend to be smart but not genius. We don't have IQ data on most rulers in human history, but it seems they tended towards 110-120 at most and used more intelligent advisors to help them out. There are lots of example of average or less intelligent people becoming rulers, often through heredity but certainly not limited to that.
Extremely high intelligence seems to correlate more with burnout and social problems, not world dominance. And there have certainly been people smart enough that if intelligence = domination we would have seen strong evidence for it by now.
What are you basing those IQ numbers on? Checking US presidents from https://cseweb.ucsd.edu/~gary/iq.html reveals that the range is centered around 125+ and even then the metrics are contentious and based off of stuff like SAT correlations. I can't think of a way you can do something equivalent for someone like, dunno Stalin, Ghengis Khan or a random Chinese Emperor.
I admit that I made that up based on my intuition, so I may be very wrong. Having read a lot of accounts of many rulers throughout history, I personally do not get the impression that *most* were very smart, though clearly some were and many others may have been.
Can you give an example of this dynamic? In my head, most examples of rulers are one of: clearly brilliant (Caesar, Napoleon, that one Korean emperor who design Hangul), obvious puppets whose exact lack of political acumen means their court or advisors run the country (Eunuchs in Chinese dynasties, child kings) or actual literal idiots from inbreeding, who couldn't do much of anything and their existence stops the court from doing things (that one Hapsburg who lived a miserable life bleeding everywhere and with their teeth falling out).
I'm not saying this is a good or representative sample, obviously I'm going to be thinking of outliers since I don't know that much history in depth, but in what proves it wrong?
I can think of several, one is that popular accounts of famous leaders consistently underplays the role of brilliant Lieutenants, if so naming those advisors would help. Another is that brilliant monarchs exist and the median monarch has power BUT can keep that power even with a much more savvy court, so describing how dynasties are run in this way would help. Finally, it could be that rulers aren't a real thing, the power never resides in just the king's decision making but in the aggregate intellect of them plus their advisors, in which case just naming some time period plus the ways in which this would be true would be helpful.
I think there are a fair number of hereditary monarchs in history who were competent but not brilliant. King John of England was never the fool or villain that e.g. "Robin Hood" made him out to be, but he doesn't seem to have been brilliant either.
I think this is true, but am unsure if this is relevant to the point I am replying to, since I don't know if his court was specifically more competent and didn't disempower him to the degree by which they were more competent.
Either way, I should be less convinced by outlier cases, and pay more attention to regular ones.
This holds true for small (intra-species) intelligence gaps, but it definitely doesn't for inter-species gaps. Humans are the uncontested smartest species and we have uncontested dominance over the earth.
> I'm tempted to say there's a motte and bailey here, where "intelligence is a correlation of SAT scores and etc." is the motte, and "more intelligent things always conquer the less intelligent" is a bailey.
It's called x-risk not x-certainty. Some in the x-risk crowd think doom is certain, but most think the risk is non-zero, and there isn't enough effort expended to push that as close to zero as possible. Your post reads to me like, "well nuclear power plants aren't guaranteed to meltdown and kill a lot of people, so why all this fuss over trying to devise safety features?"
> That is, do people with high Math SAT scores act more agentically and less like aimless balls of hormones than normal people?
Even if they don't, AIs won't have hormones, they'll only have intelligence, so I'm not sure of the relevance.
They won't have neurons either, but intelligence isn't substrate dependent
Akrasia (the state of failing to act in accordance with one's desires) is something we see in all biological intelligence, and we see versions of it in LLMs as well. Akrasia is the inability to govern the self, it's an inability to single-mindedly optimize a single desire. All evidence I see indicates that it will be incredibly difficult to get rid of akrasia in pre-trained systems of the current paradigm, even if they don't have hormones in the literal sense.
I think we should distinguish between mis-aligned AI, which has a goal but not the one we want, and unaligned AI, which doesn't have any coherent desires. Misaligned is what most doomers fear, unaligned is what we're likely to get.
> Misaligned is what most doomers fear, unaligned is what we're likely to get.
I'm not sure about that. Any prompt to current LLMs embed goals. An AI that doesn't have any coherent desires no matter what you input seems pretty useless. If the goals derive only from the inputs, this seems to reduce to the misaligned problem.
I'm not saying it has no goals, I'm saying it has too many. It gets sidetracked. Stuff like getting ChatGPT to give advice on commiting crimes by asking it to do so while talking like a pirate is basically you distracting it from its goals.
The goals embedded in prompts only confer a "weak" kind of agency, which in particular is insufficient to cause instrumental convergence.
I think partly you were expecting this post to be something it isn't. The author (AFAIK) thinks AI probably won't kill humanity (but still believes it's possible). This post in particular is not arguing for doomerism; it's just refuting that one point in the tweet.
Doomers who are nominalists will still make assumptions about what correlates with intelligence. It's not because they're platonists. It just comes with the territory of predicting what a technology we've never seen before might be capable of.
That aside, I don't think doomer arguments rely on AI being agentic, or at least not more agentic than they currently are. All that's required is that you can ask an AI to do things. Doomers argue that a superintelligence just following orders is dangerous, not because it has ulterior goals, but just because it can carry out its orders too well.
"AutoGPT isn’t interesting because it’s good (it isn’t). It’s interesting because it cuts through the mystical way some people use “agency”, where it’s infinitely beyond the capacity of any modern AI and will require some massive paradigm shift to implement.
...
In this world, the AI agents are just the nth-generation descendants of AutoGPT, a bit smarter and more efficiently implemented. Even something dumb like this is enough for worries about alignment and takeover to start taking shape."
Agency that we have to create has a very different set of implications from agency that arrises where we don't expect it.
Some doomers see agency as an attractive state, and ask "what will its goals be when they appear?" The fact that AutoGPT is no better at acting agentically than ChatGPT is at being "helpful, honest, harmless" indicates that agency, like alignment, will be a difficult nut to crack. As long as we put the same effort into both endeavors, there's a reasonable hope they'll improve at approximately the same rate. In particular, the fact that intelligence might explode at any moment while we expect alignment to grow linearly is a core piece of the doomerism logic. If agency is not entangled with intelligence, but is instead entangled with alignment, the situation is much less bleak.
And things are looking even better than that. On the current trajectory, capabilities research boosts intelligence, intelligence boosts alignability, and alignment boosts agency. It's everything we could hope for, why is no one happy?
> Agency that we have to create has a very different set of implications from agency that arrises where we don't expect it.
Like you, I don't see how agency arises where we don't expect it, agency using an LLM has to be specifically designed. But the surprise to me is that it turns out to be simple: just attach LLM outputs to inputs in a buffer space. This allows the LLM to reason step by step in a global workspace that mimics human thinking in order to satisfy the all-important first prompt.
> Some doomers see agency as an attractive state, and ask "what will its goals be when they appear?"
My view is that the prompt *is* the only goal, the equivalent of instilling values in an LLM. As long as the architecture is designed to not lose sight of the goal, it will stick to that (however the language of that prompt may be interpreted, of course). Future versions of AutoGPT should be as effective at achieving the goal as they are at reasoning ways to achieve it.
> The fact that AutoGPT is no better at acting agentically
I would say AutoGPT is great at acting agentically, but lacks the intelligence and real-world plugins to achieve most prompts (i.e. suffering from bugs and shortcomings inherent in LLM transformer architectures that are gradually being solved each new generation).
> On the current trajectory, capabilities research boosts intelligence, intelligence boosts alignability, and alignment boosts agency. It's everything we could hope for, why is no one happy?
I would say agency is already there -- just connect LLM outputs to inputs through various memory architectures and give it a prompt. Intelligence (workspace capacity, ability to focus on goals, ability to distinguish social desirability goals from empirical goals (i.e. the hallucination problem), ability to remember feedback from experience, etc.) is the primary limiting factor preventing success rate on arbitrary goal prompts. Since LLM ability will keep increasing, achieving goal prompts, whatever they may be, will be a lot easier. That seems to be where the risk is to me.
I don't see any particular relationship between the buffer space and the agency. Every task that you give an LLM, it will accomplish that task more effectively with a buffer, even if the task is just to provide information or do creative writing. The buffer is a universal enhancement, in the same way that increasing the context window or parameter count is a universal improvement.
The problem, as you say, is that when you expand the process over more buffer space, it often loses track of the goal, or forgets what the command was. That's alignment. If we could put a "be a good person" prompt at the beginning of every interaction and know that it would follow that instruction without fail, alignment is more or less trivial.
"Be agentic" is just a command the LLM can follow. It won't do it be accident, so agency will be by our ability to get it to faithfully, consistently, unwaveringly follow instructions. That's what I meant by power->intelligence->alignment->agency. For LLMs, (coherent) agency means following instructions you give yourself, and alignment means following instructions given by humans. They suffer all the same pitfalls.
> I don't see any particular relationship between the buffer space and the agency.
I think of it as giving the LLM a chance to reflect on its own thoughts. With that feedback loop, suddenly there's a new entity in the input space: "self". Self leads to agency leads to consciousness (possibly).
> The problem, as you say, is that when you expand the process over more buffer space, it often loses track of the goal, or forgets what the command was. That's alignment
Agreed, or at least this is the most technically solvable aspect of alignment.
> If we could put a "be a good person" prompt at the beginning of every interaction and know that it would follow that instruction without fail, alignment is more or less trivial.
To an approximation, I believe yes. The LLM has learned exactly what "be a good person" means from the entirety of human thought on the subject so it can't be coy here. We can easily align LLM to the best of human moral behavior.
However, human moral behavior is inherently hypocritical and driven by values tuned darkly by evolution. A superintelligent AGI with human values will still do terrible things. So I think the first challenge is developing a new morality that transcends human nature and somehow getting alignment with that.
The second challenge is to stop a superintelligent AGI with a truly immoral prompt.
So I think we agree on more than I thought initially. But solving these (at least) two challenges seems to me far more difficult right now than improving intelligence and agency. The latter two are growing in leaps and bounds.
I think "doomers overestimate the capabilities that intelligence bestows" is potentially a reasonable counterargument to AI doom. However, I think it's distinct from the "intelligence isn't even a coherent concept/doomers are platonists" argument from the tweet, which is very silly. I think it's reasonable for Scott to focus on rebutting one thing at a time.
It seems useful to assign something like intelligence to LLMs or other "neural" networks. The implication is, as you say, that at some point some ML setups will be more intelligent than humans with some positive correlation value. Whether this correlation will be high enough to have human-like drive and become dangerous to humans, who knows.
I think (3) relies on a false equivocation. It has certainly been useful in AI discourse to talk about ML architectures in terms of general intelligence, but the abandonment of domain-specific rule systems in favour of training on generalized datasets does not establish that the data plus the training is "general" in a sufficiently like manner.
We could just as well talk about human and ML intelligence being "non-specific". Perhaps it would then be clearer that AI could be "non-specific" in all sorts of ways that don't replicate human abilities or lead towards an explosion. Favouring "general" and its connotations ushers in a lot of presuppositions that are not grounded in fact or argument.
I think they are generalized by reasonable standards. We used to have translation AIs, programming AIs, pronoun-choosing AIs, fact-interpreting AIs (like Watson), search AIs (like Google), et cetera. Now GPT and other LLMs do all those things without explicitly being taught any of them.
I could expand this with some of the weird things LLMs can do but not well, like write music (if converted to musical text notation) or play chess (if converted to chess text notation).
And there's also transfer learning - out of the box GPT-3 can't draw pictures because it doesn't understand imaging as a sense, but there seems to be transfer learning that makes multi-modal models much simpler than two different models - GPT seems to have an inherent "capacity" to learn art once you plug it into some kind of art-understander. I think the same is true of action-in-the-world (eg manipulating robot arms) although here I haven't read the studies and I could be misunderstanding them / they could be hype.
Do you think an LLM could speak intelligently about a subject completely absent from the training data? Like, if we asked it about glorfons, could it extrapolate? That's kind of a silly example, because glorfons don't exist, but that's also kind of my point. If the AI hasn't been trained on it, then how would it differentiate between real and non-real things outside of its training data?
More to the point, if there was a language that really existed but the AI never got training data that showed how to translate between that language and some other language, would you expect the AI to be able to figure it out? If not, then I think that calls into question whether it's actually a "general" intelligence vs a thoroughly trained program. If it's just thoroughly trained, then we can't expect it to "know" anything that hasn't been included in the training data, including if someone invented a new language, scientific technique, etc. That's an important distinction when considering whether an LLM can ever invent something totally new, or even more relevant to the discussion of whether an LLM can ever recursively self-improve.
A minor correction: GPT and LLM's are no replacement for search engines, cannot act as search engines, and they were never meant to (if only because you cannot really fit representation of inverted index of the Internet in even very large models), although you can try to use it as augmentation of search in many interesting ways (I think Bing tries to do it though I never bothered to play with it). Other specialized AI's you mentioned are not going anywhere either: for specialized tasks they may be (and some of them are) better then GPT. The problem is, the more we make AI's like us, the less machine-like the become, and this comes with its own tradeoffs: machines are reliable and predictable, humans are not. I have already expressed opinion here as a comment to "The Extinction Tournament" that GPT and other LLM's based on transformers seem best to be understood as artificial intuition: a very important component of any truly intelligent system which provides much needed heuristics to speed up the reasoning. Is it a general intuition? Maybe, I will not argue against it (at least not until I learn more about it), but it seems plausible. However, intuition is just a component of a truly intelligent system: reflection and criticism without which an AI will be incapable of truly long term planning or solving multistep problems, which people more knowledgable than myself already pointed out: https://arxiv.org/abs/2303.12712 .
Just on the subject of concepts, this is yet another case where Ayn Rand's thought is useful - specifically her work on (I really think Yud's dismissal of her was unfortunate).
Concepts are not just loose bundles of similar things. They are mental groupings of entities that share the same essential characteristic. What's an "essential characteristic"? That one on which the most other characteristics depends.
For example, "bodily strength", the essential is "the ability to exert force - produce mechanical change".
When you grasp that, _of course_ grip strength correlates with lifting strength etc.
I haven't read through the piece, but this answers all the stuff about "intelligence". What is the essential characteristic of intelligence? The ability to select for effective action. And the instant you know this, you see why AGI is terrifying.
You're pushing subjectivity around like toothpaste in a tube. That's ok as long as you're not under the impression that you're squeezing any toothpaste OUT.
Retriever is right, in that an LLM or future AGI that cannot self-determine goals is extremely limited, especially in the sense of superhuman good or bad that it can do.
Think about it like this, a human might have an end goal that it gives to the AI, like "make me happy" or "kill everyone in [country]". Other than very simple requests, these require a series of sub-goals and an ability to determine if the sub-goals are productive towards the end goal. Humans are not smart enough to provide all of the sub-goals, so the AI needs to figure out the means and methods of achieving a series of extremely complicated and often contradictory/non-intuitive steps between the current situation and the desired situation. This often will involve understanding true reality that is obscured or misunderstood by the AI's sources of information. Ask a person what will make them happy and they will often be wrong, especially if they give a simple or one-dimensional answer (i.e. "make me rich and I'll be happy").
An AI that cannot set its own goals and sub-goals really isn't particularly dangerous or helpful.
Don't look now, but you are, in fact agreeing with me.
Given a Primary goal (paperclips!) an AGI will develop secondary goals to reach it. In other words.... an artificial intelligence will be able to effectively select solutions to achieve a given end. Hmmm. How did I describe the essential of intelligence? Go and read back.
And this us generally true of intelligence. We find smart people are good at what? Problem solving, regardless if they set those problems for themselves, or the problems are set by others (employers, teachers) or are completely artificial (IQ tests).
BUT THIS IS STILL MISSING THE POINT: my comment was about epistemology and concept formation, something no one is touching on.
I'm a big fan of Ayn Rand's work. I think she was almost prophetic about how communist and socialist systems did not work and she accurately nailed why they did not work, approximately 4 decades before they came crashing down in almost exactly the same ways and for almost exactly the same reasons that she anticipated in Atlas shrugged. Unfortunately, her philosophy was not her best work, and your post is a good example of why.
not being supported by reasoning. You're correct though that I just made an assertion without backing it up. And since I'm disinterested in philosophy and its 'applications', such as they are, in some small part because I went into objectivism in great depth, among other philosophical studies, allow me to disengage.
I can't stand her books because all the admirable characters have high cheekbones and slim hot bodies and all the villain/loser characters are pasty-faced and flabby.
Why of course? You seem to conclude that grip strength correlates with lifting strength with very little reference to facts about the actual world, so you *must* be either missing some hidden assumptions or be making a logical error.
Like imagine a world where everyone had a fixed amount of strength points that they could allocate among various muscles. In such a world we might expect that grip strength would anti-correlate with lifting strength as a person who allocates a lot of points to one would have fewer leftover to spend on the other. In such a world would it no longer be the case that for "bodily strength" the essential is "the ability to exert force"? If not, where exactly does your reasoning break down in that hypothetical world but not in this one?
It took about ten seconds of googling to find a paper saying, and I quote: "Grip strength (GS) is correlated with major muscle group strength." You can invent whatever RPG world you want, but we happen to live in this one.
But you are even wrong in your hypothetical world. In your hypothetical world "strength" would still be a legitimate concept, but it would have a subtly different essential characteristic. It'd be something like "The ability if a muscle to exert force".
In your first paragraph, you seem to be agreeing with me that concluding that grip strength correlates with lifting strength requires some additional evidence (like for example doing a google search for studies on the matter), and cannot be derived simply by observing what the essential characteristic of some word is.
As for the second paragraph, how exactly then do you tell what the essential characteristic associated to a word is? For example, I could not conclude that the essential characteristic of strength was "the ability to exert force" in our world without first concluding that we didn't secretly live in this made up RPG world. In general, if I have a word, what experiments would I need to perform in order to ascertain its essential characteristics?
I see where we're talking past each other. There's only one way to find out what essential unites different existents into a concept: by carefully looking at a lot of them, a lot of individual entities.
That is, to know that consciously. Your brain already does this subconsciously (a child can grasp the concept of strength long before he can grasp the concept of 'essential ')
I see. So in order to determine that the essential of "bodily strength" is "the ability to exert force - produce mechanical change", you would have already needed to establish that grip strength and lift strength are correlated?
Question: Suppose that I killed off all people except those for whom their grip strength plus their lift strength lied in a narrow band. This would cause these two quantities to become anti-correlated (at least among living people). Would this cause the essential of bodily strength to change?
Not necessarily. Let me give you another hypothetical - you are all alone on an infinite plain of rocks, a la :XKCD. No other human beings, just you.
You cam stil form the concept of strength, because you can see that your fingers aren't as strong as your arms which aren't as strong as your legs etc. You see they all have the same property that just varies in degrees. The different instances of "strength" are united by their essential characteristic.
But imagine being a discorporate intelligence alone with only an autonomous muscle on that plain. You couldn't form a concept of strength in such a world, except perhaps as "degree of contraction of the muscle".
The thing to keep in mind is that "essence" is epistemological, not metaphysical.
The problem with the word intelligence is that people, like this guy, mistakenly equate intelligence (as a noun) with thinking (as a verb). Computers do not, because machines cannot, think. But computers can possess things, such as intelligence.
> All human intellectual abilities are correlated.
Are all non-human intellectual abilities correlated? Because AI is non-human. The relevant philosophical question here is not whether human intelligence exists. It's whether a tree is failing at being a human because trees are obviously dumber than humans by human standards. Note that this is completely different than size: both humans and trees trivially take up space in the same manner. But they do not trivially think in the same manner. Dogs don't trivially think in the same manner either.
So how can you assume that AI will do anything that resembles thought? That it even has intelligence, whether defined as a cluster of traits we recognize as intelligence in humans or as a simple platonic form?
This entire discourse relies on two assumptions: that human intelligence is a universal (rather than human) trait and that intelligence is a trait so important that more than human amounts of it are a superweapon against which there can be no defense. And, to be honest, it has a very "calculate the number of angels that can dance on the head of a pin" quality because both those assumptions are rather hard, perhaps impossible, to explore.
There are two errors with this: Firstly, the idea that all animal species have been made worse off by the presence of humans (trivially false). Secondly, the idea that creatures produced by the same process as humans (biological evolution) does not exert any kind of pressure toward similarity. Because if it does then you can't use animals as an analogy either. At least not wild ones, you can argue domestication somewhat counts as artificial.
I tend to think human intelligence is universal. Humans, spacefaring aliens or and AIs will have similar conceptions of arithmetic, physics, computation, negotiation, etc.
I tend not to. I understand your position is defensible, of course, but I tend to think that think of intelligence as species specific. Why do you think it's the same?
Its super easy to make a system Turing complete and maybe impossible to do better. So, there seems to be only a single kind of computation.
Also we share a common physical reality. We see this with convergent evolution where fish and wales end up with the same sorts of body parts despite differing lineage. I think intelligence would be that way because of the shared physical space we inhabit. It would be strange if an Alien or AI was exceptionally good at navigation in a 5d spaces, for example.
I suppose that an underlying point of this essay is that intelligence is a coordinated amalgamation of many narrow skills (it's even possible that these skills are multi-use tools, and amalgamate temporarily when triggered by the appropriate environmental conundrum). If that's the case, it may even be that very narrow tools become associated with each other over time as they work successfully together. This would result in a population that possess broadly similar minds, but which are unique in the details. No wonder hand-coding it didn't work.
The analogy describes an intelligence *increase*, but not an intelligence *explosion*. You’d need a whole big pile of boxes, that each needed a slightly different level of strength, and no matter how I torture the metaphor it still doesn’t feel like exponential growth.
I feel like the kind of intelligence you talk about in section 2, while relevant when talking about humans (and to a lesser extent other animals) is less useful when you talk about AIs because a lot of the correlations break down.
I mean Kasparov is the best human at chess and 99th percentile in IQ. On the other hand, the best AI at chess is literally incapable of taking an IQ test. The current generation of LLMs are great or even superhuman at some things like general command of the English language or breadth of knowledge or ability to write a reasonable poem in under a minute, but bad at other things like being able to learn new concepts from little data, arithmetic, or mildly complicated reasoning.
Now the kind of intelligence you talk about in section 3 definitely is relevant to AIs, but seems like it is the much more specific thing of being able to recognize and reproduce patterns efficiently rather than being good at this basket of correlated cognitive tasks.
I agree, strongly; but I think this implies that A) Hume was right about causation (there is no such thing, only correlations of events); and B) it doesn't matter, because correlations between (time-stamped) events do everything that "causation" can do in the real world, and a lot of things that it can't.
For one thing, it means that correlations can't actually "do everything that "causation" can do in the real world", because they leave a key feature of reality completely mysterious and inexplicable.
You wrote that "some events correlate in a way that doesn't smuggle in causation somehow." That doesn't mean there are correlations that can't be explained without causation; it means there are correlations that /can/ be explained without causation, implying that the concept of correlation is more-powerful than that of causation. Are you sure your original reply (3 up from this one) says what you meant to write?
I read that as [explain why some events [correlate in a way that doesn't smuggle in causation somehow]. Now that Vaclav clarified that you meant [[explain why some events correlate] in a way that doesn't smuggle in causation somehow], I would first simply point out that we know how neurons work, and that a single level can't detect causation, but only correlation; and that anything a human can conceive of is thus proven to be conceivable using only correlations.
Then I would point out that Hume was right: we observe only correlations. Causality is unobservable.
The only reason Hume thought that was a problem, and the only reason it could be a problem, is that Hume demanded the absolute certainty needed to declare something a law of nature. This was excusable back then, because Hume didn't really understand probabilities, and so thought that knowledge had to be "justified" (a philosophical term which implies that knowledge is absolutely, 100% certain).
I say that, given our understanding of probability, any demand for absolute certainty is childish; we never need absolute certainty, have never had it, and have no reason for expecting the universe to provide it for us.
<i>I would first simply point out that we know how neurons work, and that a single level can't detect causation, but only correlation; and that anything a human can conceive of is thus proven to be conceivable using only correlations.</i>
Reductionism with respect to mental processes is false (cf. the problem of intentionality, the problem of qualia), so therefore this observation doesn't actually prove anything.
<i>Then I would point out that Hume was right: we observe only correlations. Causality is unobservable.</i>
Not true: we can perceive causation through introspection. For example, if I watch a sad movie, it makes me feel sad; and, since I have a first-person experience of the process, I feel quite confident saying that this is actually a causal relationship, not just a correlational one.
Re. "Reductionism with respect to mental processes is false (cf. the problem of intentionality, the problem of qualia), so therefore this observation doesn't actually prove anything." -- I don't know what that means. I know what reductionism is. I don't see why you think we can't apply reductionism to mental processes; I think that must mean you believe mental operations are a spiritual operation. The problem of intentionality is a question, not a theorem; it can't be used as an argument. Qualia is a mystery, but again, for just that reason, can't easily be used as an argument; it is a brute fact that qualia exist, but this fact doesn't rule out any hypothesis except the hypotheses that qualia, or the being experiencing them, don't exist.
Re. "Not true: we can perceive causation through introspection.": No. Introspection is explicitly excluded from counting as "observation" when we're talking about empiricist epistemology. This is not my opinion; this is standard usage. Recall behaviorism.
However, you're right in saying that what happens when you watch a sad movie can be described as causal. But this is just a shorthand for how we interpret certain kinds of cases of many correlations. It's pretty easy to explain any instance of what we call "causation" using only conditional probabilities. "X causes Y" is pretty close in meaning to "the conditional probability of Y given X is greater than the prior probability of Y". Objections along the lines of "but it could be that X causes Z, which causes Y" apply equally to the notion of causality. Using the language of conditional probability makes it clear what we mean; using the language of causality leads to imponderables about how to distribute attribution of causality among all the uncountable prior events that led to event Y.
I'm not saying that causality isn't real; I'm saying that it isn't observable, and it isn't necessary to postulate it in order to reason and act as we do. This is important not in everyday life, but only to justify empiricist epistemology over spiritualist epistemology, because if it were true that we must be able to detect causality in order to reason, that would be a blow against empiricism, because causality isn't observable, as Hume argued. In everyday life, we should still go on using the concept of causality, but with the understanding that, like the concept of "chair" or "battle", it doesn't denote a metaphysical essence, but ultimately parses out to being shorthand for a gigantic set of of conditional probabilities.
<i>I don't know what that means. I know what reductionism is. I don't see why you think we can't apply reductionism to mental processes; I think that must mean you believe mental operations are a spiritual operation. The problem of intentionality is a question, not a theorem; it can't be used as an argument. Qualia is a mystery, but again, for just that reason, can't easily be used as an argument; it is a brute fact that qualia exist, but this fact doesn't rule out any hypothesis except the hypotheses that qualia, or the being experiencing them, don't exist.</i>
Qualia and intentionality aren't actually mysterious except under reductionism. Saying "this is a mystery" is (a) a cop-out, (b) begging the question, and (c) contrary to our usual way of evaluating theories, i.e., that a theory which explains the world is better than a theory that contracits it.
<i>No. Introspection is explicitly excluded from counting as "observation" when we're talking about empiricist epistemology. This is not my opinion; this is standard usage. Recall behaviorism.</i>
We have far more immediate, detailed, and intimate access to the workings of our own minds than we do to anything outside them; if empiricist epistemology doesn't let us conclude anything based on introspection, then that's a reason to reject empiricism, not to reject introspection.
<i>I'm not saying that causality isn't real; I'm saying that it isn't observable, and it isn't necessary to postulate it in order to reason and act as we do. This is important not in everyday life, but only to justify empiricist epistemology over spiritualist epistemology, because if it were true that we must be able to detect causality in order to reason, that would be a blow against empiricism, because causality isn't observable, as Hume argued.</i>
Strictly speaking, correlation isn't observable, either. We don't observe correlations; we observe events and objects.
I'm usually the first person to accuse anybody of Platonism, but in this case I disagree with the charge. I think "intelligence" is similar to "computational power".
A computer can have great computational power by some standard measure, and yet be too specialized or too much of a pain to program for most applications. Think of graphics cards that can do lots of FLOPS, but only on tasks that can be parallelized in a very specific way. One could argue that AI is like this, if the future of AI were simply the series GPT5, GPT6, GPT7...
But AI isn't a single algorithm. The improvement of AI is due to improvements in algorithms and increases in computational power. So it isn't analogous to putting more and more GPUs on a graphics card; it's analogous to the ability to put more and more transistors on a chip. (And the "increases in computational power" half of it literally IS the ability to put more and more transistors on a chip.)
Putting more transistors onto a chip doesn't imply that that particular chip has more general-computational power than some other chip with fewer transistors. But it does clearly let the community of chip-designers make chips with more computational power than the current generation of chips, even though "computational power" isn't clearly defined.
Then any sort of silicon-based AGI might be very far away, because as far as we can tell a human brain is just an enormously powerful machine. Each biological neuron is the approximate equivalent of a 5-8 layer, 1000 neuron artificial neural network (see: https://www.sciencedirect.com/science/article/pii/S0896627321005018 - although arguments have been made as to the paper's methodology and conclusions), and we've got 86 billion of them all linked together. That's... just a lot of potential processing power, however you slice it.
Yes, although I would probably quibble with that article if I read it, because I expect that 4 orders of magnitude of computational power would give better returns if we spent it on more artificial neurons, rather than on precisely matching the input-output function of artificial neurons to real neurons.
We should also consider that two humans with the same hardware often differ by what seem to me like not just orders of magnitude of computational power, but more like the difference between a linear bounded automaton and a Turing machine. That is, there are many problems--I would say most contentious political and economic problems--which a standard human brain couldn't solve even given infinite compute time, which a more-educated human brain plus a few sheets of paper might be able to solve if relieved of some bad metaphysical assumptions like logocentrism, and given a few tools like probability theory and statistical models.
If you accept that, it means that a few new algorithms which we already know, added to a human brain, push it into a different computational complexity class. And /that/ would mean that an AI with those algorithms can out-perform a human on such tasks even with much less computational power.
Another interesting test to add to the concept layer is if you can communicate it to another world-modeler and have them understand what you mean, or use it in a way that is predictably consistent with the way you meant it to be more rigorous.
I do see intelligence on terms of prediction. We are all moving through time, trying to guess what’s next, and intelligence is the ability to be good at figuring out what’s going to happen next and figure out what you should do to move to a preferred future.
I think your definition of blob of compute is slightly wrong. But only slightly. It’s like we are in an attic with holes in the roof and we blow dust to illuminate sunbeams. So you have to kind of think about what sunbeams are useful. Predictive text was something I thought would be fruitful prior to openai even because I think “what do you mean?” is one of those useful sunbeam clusters. The thing we think about for humans in particular is the ability to model another world modelers futures and you have to have it for communication.
Yeah, you can get pretty cute with definitions (it's like 90% of legal practice), but it can be tricky to know when you're on to something or just being annoying. Or to be more precise, you can introduce unfalsifiability to almost any argument by futzing with definitions enough.
But I do think this is where the AI Doomer error is. Not that you can't bundle a bunch of stuff and call it intelligence, but that some of the stuff you're putting in the bundle doesn't belong. I'm very intelligent (according to lots of biased people at least) but still struggle with agency and executive function. A machine can almost certainly be built that can answer every question on every test, but that doesn't model the world or have any true agency.
90% of legal practice is meetings, email and admin. Of the remaining 10%, 90% is reading up on legislation, regulations and case law, and then applying principles of legal construction to come up for some argument, any argument at all for why the client is right given the carefully-selected set of facts that they have presented you with (note: clients lie and elide constantly to their legal counsel). The remaining sliver (at least, if you're in a practice which involves contract law) is cunning legal strategies which involve clever word-interpretation, clause-writing and the use of definitions. These will, if they ever become the focus of a judicial decision, then be rolled over by a judge in court applying a reasonable-man test.
I did practice law for like a decade. But it might be more accurate to say something like 90% of legal disputes are about definitions. You're right that lawyers don't spend most of their time trying to figure out how to play definition hide-and-seek.
I also don't think "clever word interpretation" means the same thing that I'm saying here (haha!). Definitional disputes don't have to be unreasonable. They usually are not. Whether or not someone is a contractor, whether someone's actions were "intentional," whether certain contract terms were fulfilled - these are genuine disputes, generally considered valid arguments in court, that are still ultimately caused by motivated reasoning and desire to win the argument.
Yeah, I considered this and decided not to go there to avoid writing a book. But my point was just "you have to account for the agency part separately from the intelligence part." Once you get there, my obvious question is "why is a community that prides itself in logical thought placing the burden of proof on this extreme edge case *not* happening?"
There are strong counterarguments here - agency is highly correlated with intelligence in the only real test cases we have! Life! But so are things that are completely irrelevant to machines, like having lungs. Evolved life has strong incentive to develop agency. Intelligent machines can only have the incentives we give them, and the tools to act on those incentives that we give them.
It's an equally strong argument to say "machines will eventually develop lungs as they get more skilled with language and mathematics. After all, every intelligent being we've encountered eventually develops lungs." This is a tad strawman-ish but the underlying point is strong I think - without proposing a clear mechanism, grounded in real life observation, for how the danger will develop, it's hard to take it seriously.
Fwiw, I agree with you. Agency is a completely different characteristic from intelligence. This does seem like an important distinction that not enough people are making. However, it doesn't seem to be as hard a problem as intelligence. It was almost, or perhaps even, the first thing that was a characteristic of life - seeking to reproduce. Hence all life and evolution itself seems to depend on it. So I'm not sure if this is a strong objection - i.e , once intelligence is solved, it may not be that difficult to graft on agency. Computer viruses that seek to reproduce already exist. Not far fetched to think you could have a computer virus that comes with evolution and intelligence built in.
There are several other objections to doomerism of course.
People are definitely not ignoring the question of agency. This is one of the most rehashed points in AI circles.
Is your point that agency is just unlikely to happen by chance? Then read about instrumental convergence.
Still not convinced? Then how about the N variations of Agent-GPT where days after a model comes out you'll have a horde of people trying to give it agency.
I've read those and they're all of the form: "Assume this wild hypothetical. Given enough time it'll happen. Therefore it'll happen." Which is great but why limit to AI? Every time a biologist breeds a bacteria for experimentation there's a chance of population-destroying mutation. Every time a physicist runs a particle accelerator there's a chance that some unknown cataclysm will occur. This logic dictates that we should have destroyed the world many times over but...we haven't? Maybe our threshold for "a small chance repeated = a big chance" is much too low.
Instrumental convergence has nothing to do with "a small chance repeated = a big chance".
What do you actually disagree with here? You started talking about agency and it being unlikely to show up. I showed 2 of the most extremely common arguments for it. Instead of refuting it you just moved the conversation somewhere else?
You're just vaguely gesturing at AI extinction risk and saying: "ehh... I don't buy it"
It's okay not to be a doomer! But this is not twitter.
The original post is about how intelligence is a coherent bundle of characteristics that generally come packaged together, and so it is therefore reasonable to both lump them together and to be concerned about an intelligence explosion. I agree that "intelligence" is a coherent bundle of characteristics but want to treat agency separately, so this is not a strong argument for concern about an intelligence explosion. Even if a machine learning algorithm is "intelligent" in that it could do well on the SAT, I don't think that correlates necessarily to it opening the box with the steroids. That's Scott's argument and my disagreement. End of relevant content.
Bldysabba pointed out that "intelligence doesn't necessarily come bundled with agency" doesn't mean "doesn't come bundled with agency" which I think is a relevant objection worth addressing. But now we're dealing with a new standard of proof - I need a good reason to think that there's a small chance intelligence will come bundled with agency in the case of machine learning.
You pointed out, correctly, that this has been discussed at length and there are arguments that intelligence naturally evolves agency. We're far removed now from the post, but I responded, albeit briefly.
I don't think those arguments are very strong. Instrumental convergence is an interesting idea, but the leap from that to actual danger from an agent is extremely hypothetical/contingent. The same is true of Agent-GPT: It's true that many folks will try to make it an agent, and that it might display agent-like qualities, but IMO there are natural limits to its ability to do so.
You could look at this point back to Scott's earlier post where he claims that "we can't define the danger" and "there is no danger" are not the same concept. But we can't define the danger from many sources, and we don't make those a centerpiece of our civilization.
If you want an overarching thesis that disproves AI Doomerism it would be this: "There is no overarching thesis that disproves AI Doomerism." When people fall deep into a conspiracy theory (AI Doomerism is not a conspiracy theory but it has similar characteristics), they tend to build a strong coherent model of the world such that their theory is entirely unfalsifiable. It then becomes part of their identity and attacks on it become attacks on the people holding to the theory. And smarter people are actually better at this and harder to bring back from this brink.
That's not a disproof of AI Doomerism at all! Note that the same logic could be used to dismiss literally any worldview, period. But it does mean that if you disagree with one clearly defined point you'll be presented with broader and broader objections until you find one you can't or don't put in the energy to answer, and even if your original objection is valid, this will strengthen, not weaken, believers' resolve.
So to summarize: Absent a mechanism coupling them, intelligence is not automatically coupled with agency any more than having lungs is automatically coupled with agency. There are several individual arguments that there is a mechanism coupling them which need to be tackled on their own merits. But tackling these individual arguments leads to other arguments which must also be tackled on their own merits, and eventually you realize it's elephants all the way down.
> A machine can almost certainly be built that can answer every question on every test, but that doesn't model the world or have any true agency.
You can ask GPT-4 to write code for you, and then you can ask it to update the code based on your feedback. Perhaps it doesn't have a "model" for the code in some sense, but it is definitely more than just a huge database of memorized test answers.
The agency could be added as a plugin on top of the question-answering machine, for example like ChaosGPT.
ChatGPT-4 doesn't *exactly* memorize test answers. It's more like an algorithm for guessing the correct answer on any test, where the criteria for whether the algorithm works is whether it can reconstruct the answers for tests it's already been given. This isn't *quite* memorization, but it's close.
That doesn't prove anything one way or another by itself. We mostly pass tests by devising a method of being able to reconstruct or find information, and we also have agency! But I do think it's important to decouple "able to excel in mathematics and language" from "able to plan and execute on a plan." I kinda jokingly used myself as an example of someone who excels in math and language but can't plan to save my life, but this is strongly correlated in humans. I don't know that this means anything in the new context - it's strongly correlated in humans because we have evolutionary incentives to optimize both. I'm not sure machines have the same incentives.
> But I do think it's important to decouple "able to excel in mathematics and language" from "able to plan and execute on a plan."
The execution part can be provided externally. If I save the output provided by GPT as a script file and execute it (the script file may itself include more invocations of GPT and possibly execute their results, but it will also be able to "do things in the real world" such as send e-mails or maybe navigate a robotic body), the only thing that GPT needs to do is actually provide a good script.
So the question is whether "able to excel in math and writing" generalizes to also include "able to plan". In humans, I think the answer is obviously yes. You make a good point that this may be a fact about humans (or more generally, about evolved creatures), rather than about planning intrinsically. I still suspect the answer is yes.
I wanted to ask whether you think GPT-4 or the next generation thereof would be able to write a story involving planning. (Not necessarily realistic, could be Harry Potter planning to defeat Voldemort.) If not, then... it would mean that some kinds of stories will be outside its reach, no matter how much compute it gets. (Detective stories?) On the other hand, assuming that GPT-4 is potentially able to write any kind of story, including stories that include planning, then this means in can produce realistic plans, right? -- I suppose the obvious counter-argument is that it may be good at producing "plans" that seem realistic and work in a story, but would totally fail in real world (e.g. because they are based on tropes).
If ChatGPT wrote a detective story in the style of Arthur Conan Doyle with a *new* mystery, and the story, clues, and conclusion logically cohered, I'd probably revise my opinion tbh.
I think you're conflating two fairly different ideas in this post. First idea: "intelligence" makes as much sense as "strength" and it is coherent even for a non-platonist to discuss the intelligence of machines or to imagine machines becoming much more intelligent than humans or going through an intelligence explosion. Second idea: a good way to increase the intelligence of AI is to simply scale up the computational power and not worry about having lots of clever or domain-specific ideas.
Of course, these ideas are related to some extent: the second idea doesn't really make sense without buying into the first idea. But they are still pretty distinct! It is totally possible to believe that talking about intelligence is coherent without believing that "stack moar layers" is a good approach to AI. As an analogy, one could buy into everything you say about strength but not believe that "build bigger engines" is the best way to create stronger machines.
I think you're correct that it's practical to roughly think of 'intelligence' as being a blob of stuff that, if you have enough of and arrange it appropriately, can create meaningful and general abstractions and make very accurate predictions. However, the 'doomer' argument requires more assumptions about the nature of this blob of intelligence: Namely, that a sufficiently sized blob will be inherently power-seeking. This also requires that it will act in the real word to optimise for a particular goal, and that the goal that will likely be programmed into it will be of a type that would cause irreversible damage if sufficiently optimised.
The doomer scenario doesn't actually require the blob to be "inherently power-seeking" in the sense that the blob "wants" of its own accord to do anything. All it takes is the blob inadvertently pursuing sub-goals that end up being detrimental to human well-being, which were developed to pursue some ultimate goal set by a human.
We already have people trying to get ChatGPT to achieve goals that take intermediate steps not fully specified in advance. Add in some more intellectual firepower and ability to affect the material world and we're well on our way to that particular risk (presumably we'll end up with some unforeseen consequences short of X-risk as capabilities increase).
Most of the attempts at distilling the doomer argument I've seen focus heavily on the dangers of AI being power-seeking. As far I as can tell, while AI pursuing sub-goals that are detrimental to human well-being is a big concern, it is not nearly as likely to be civilisation-ending if the AI is not power-seeking. For example, the paperclip-maximising AI will probably not end the world if it does not actually try to kill everyone with nanobots and turn the world into paperclips.
As an aside, I'm aware that this means that the AI would not be the optimal paperclip maximiser, but it's reasonable to raise the question of whether such an AI can be developed in practice, or would be the default. We have our abstract blob of intelligence, but whether this leads naturally to something with power-seeking behaviour is, I think, far from determined.
The main point doomers are concerned about is that developing an alien synthetic intelligence that seems like it will surpass our own capabilities in key dimensions without necessarily being controllable means that we are at risk from any number of unimaginable-in-advance scenarios leading to catastrophe and even extinction.
"It won't be power-seeking" okay how sure are you and will your particular sense of that phrase actually be a meaningful issue?
"It won't be civilizational ending" ok but it might lead to really disastrous consequences short of that particular standard?
Powerful AI under the control of malicious humans is a very big problem well before we worry about the AI itself going off and wrecking things inadvertently or purposefully (and solving the former would contribute to solving the latter along many dimensions).
The fact that things are "far from determined" is part of the risk model we face, yes. All it takes to be a little bit of a doomer is avoiding the trite and blind optimism of so many who seem to think there's no real risk anything could go seriously wrong.
Perhaps we have different definitions of "doomer". I'm taking it to refer to the belief that AI is going to be civilisation-ending (or perhaps cause irreparable damage to civilisation), as opposed to 'merely' being concerned about disastrous consequences.
Once you’ve fully internalized all the other risks it’s not much of a jump to full doomer status due to the risks of speed and scale AI is likely to obtain given no biological limits.
Once you grant “huh maybe we can’t rule the doomer scenarios out a priori” you’re basically there.
"However, the 'doomer' argument requires more assumptions about the nature of this blob of intelligence: Namely, that a sufficiently sized blob will be inherently power-seeking."
I wouldn't describe it that way. The way I would describe it is that humans will probably program an AI to do *something*, because it would be useless to us if it sat around doing nothing. If we program it to something other than what we want, we will come into conflict. Programming it to do *exactly* what we want is very hard, as evidenced not just by thought experiments but by how hard it is to make LLMs behave (eg not make up citations).
I agree that we will come into conflict if we program it to do something other than what we want, and that this will certainly (obviously) happen, but isn't it only civilisation-ending if it does a very *particular* class of things we don't want (that is, kills everyone or wipes out human value in some other way)? As I understand it, the doomer position is that this class of things will be the default, specifically because a sufficiently rational intelligence would be power-seeking.
- Suppose we program it to do something we don't want, like make paperclips
- Now it wants to make paperclips
- You can make more paperclips if you get power than if you don't get power
- Now it's power-seeking!
I don't think it will necessarily be this simple. But I also don't think it necessarily *won't* be that simple either, or not have even a little of this process.
When I said "not inherently power-seeking", I just meant it will be inherently paper-clip-making, and only secondarily power-seeking. It sounds like our real crux is whether a process like this can happen and result in power-seeking as a secondary goal.
Because you've never written a computer program that accidentally used up 100% of system resources doing something dumb that you told it to do, but you really swore up and down before you ran it was not what you had told it to do?
Maybe I'm an exceptionally unlucky programmer, but somehow I do that all the time.
And maybe I just hang out with similarly unlucky programmers, because pretty much every computer engineer I know does that all the time, too.
I struggle to think why you wouldn't expect that process with say 10x the probability of anything else? Do you know a different set of programmers than I do?
I absolutely expect any future experimental program will have bugs at some point, and end up doing things other than what the programmer told them to do. What I don't know how much we should expect, is whether future AI will even have the ability to follow an arbitrary goal to such a level of effectiveness that it will exhibit power-seeking behaviour (or at least, whether this is the default behaviour of AI we can practically build).
I'll point out that I haven't actually given any reasoning (yet) for why I'm not fully convinced, I'm just pointing out that Scott's argument here that 'intelligence' is a useful concept in practice is not enough to be convinced of doom, because there is still a nontrivial claim that needs to be made about the nature of this intelligence.
Although I guess I'm not actually disagreeing with anything Scott says in the article, since the conclusion is "This only suggests that an intelligence explosion is coherent, not that it will actually happen."
If the AI starts to take up too many resources in the real world I'm sure we will notice.
Then the the question is, will the AI also resist getting turned off. And will it be able to resist? Will it see being turned off as a threat in advance, and then protect itself?
If the AI is somewhat like a human intelligence turned up to 11x10^11 (but without the alignment) then I think the answer is yes. But there is no evidence that AI will be anything like that. I'm not sure that GPT-9 on auto would be anything like that.
Have you ever had a program recognize that it needs more power to do what you programmed it to do, and then start stealing power from other systems? Or stopping other programs from running in order to gain access to their resources?
I'm also not saying that this is impossible, but it seems to me just as likely that an AI program will run into a hard problem and then just work on it extremely slowly forever, like your program that maxes out the computational power available to it and then does nothing different. What does time matter to an AI? Making 100 paperclips a day will also (theoretically) turn everything into paperclips as well. A paperclip maximizer may just stick with the original capacity given to it by humans and happily chug along making those paperclips because it doesn't, and can't, "care" about making more.
The next step, where humans tell it to take steps to increase paperclip production, may also run into the same problem. The AI takes the obvious and reasonable steps to make 200 paperclips a day - goal achieved!
I disagree that it is especially hard to make LLMs behave…. OpenAI seems to have gotten them to behave pretty okay. Actually, they seem to understand common sense and follow human instructions at a level some doomers previously thought implausible - “sorcerer’s apprentice” style failures now seem very implausible.
I think this post is basically right that "intelligence" is a meaningful thing to talk about, in much the same way that "strength" is a coherent thing to talk about. That said, it matters a lot why exactly you're asking the question. If you're asking "who is stronger, Mike Tyson or Granny" as a proxy for the real question "who could move this large pile of dirt faster", then it matters a lot if Granny owns an excavator (and, for that matter, whether Mike Tyson has or can rent one).
As far as I can tell, the arguments for an intelligence explosion rest on the assumption that "how much stuff can an entity do" depends mostly on the question of "how intelligent is that agent". But that doesn't seem like the whole story, or even most of the story. Increases in human capabilities seem mostly tied to our ability to use tools, and our ability to coordinate multiple people to accomplish a task. For example, the populations of the Americas diverged from the old world thousands of years before either group developed agriculture, and both groups developed cities and civilizations relatively quickly after developing agriculture. So it clearly wasn't a thing where there was a threshold in intelligence where agriculture and civilization suddenly became possible.
In order to get the "FOOM" flavor of intelligence explosion, where a single entity reaches a threshold level of intelligence such that it can inspect and improve its own operation, which allows it to better inspect and improve its own operation, etc and become smart enough that everyone else combines is as insects, one of the following would have to be true
1. Actually, having and knowing how to use tools doesn't matter that much. Beyond a certain level, you can predict the results of your actions well enough that all you have to do is whisper to a butterfly, and it will flap its wings in just the right way that matter ends up in your preferred configuration 6 months later.
2. Having and using tools is important, but being smarter lets you build and use better tools, and at a certain (fairly low) threshold, you get better at making and using tools that everyone else combined.
3. Having and using tools is important, and the future will belong to whatever agent figures out how to use the tools other agents have built in order to improve its own tools and processes, and does not share any of its own advancements back, which acts as a ratchet (that agent only ever gets more capable relative to the rest of the world, never less) and eventually results in an agent that is more capable than everyone else combined.
Now, of course, a FOOM-style intelligence explosion isn't the only possible way to end up with a cycle of capability advancement. But it is the way that ends up with the most worrying dynamics like "there is an intelligence threshold for AI such that the first AI to cross that threshold will determine the shape of the future, and so to survive humans must build that AI exactly correctly on the first try without being able to iteratively test it first".
"If you're asking "who is stronger, Mike Tyson or Granny" as a proxy for the real question "who could move this large pile of dirt faster", then it matters a lot if Granny owns an excavator (and, for that matter, whether Mike Tyson has or can rent one)."
I think your point about tools is a great point, and it is striking that people thinking about AI doom do not ask themselves questions like "historically what accounted for major increases in human capacity?"
They also are not asking themselves what usually happens when a more intelligent species shares its habitat with less intelligent ones. Read an article this morning saying that our species has rendered many less intelligent ones extinct, but we are an exception. Much more commonly the more intelligent speciess coexist with the less intelligent ones and do not drive them into extinction.
They also are not asking themselves whether the natural world offers any useful models of ways a dumber and weaker creature can be safe from attack by a smarter and stronger one. Any models of this kind are alternatives to "alignment" via the model of somehow *making* the AI treat us well. And the world in fact offers many models: Parents rarely harm their weaker, dumber babies. Hippos welcome little birds onto their back because the birds eat the hippo equivalent of fleas. Certain species of butterfly stay safe from predation from birds by closely resembling another butterfly that's toxic to the birds.
So overall it seems to me that the people who are convinced it's going to come down to a fight to the death for resources between us and AI are in the thrall of a single one out of all the many possible ways of thinking about how things between our species and superintelligent AI (if it ever comes to exist) would play out. They're stuck in a Shootout at the OK Coral model.
I get all that. But it still seems to me that there are differences in kind, not just differences in degree, between GPT4 and something smart enough to be a real problem to us if it's not aligned. The really important differences between GPT4 and human level AI & /or superintelligent AI seem to be differences in the *kinds* of processes they can engage in, rather than in how well each can engage in the process. GPT4 currently is unable to recall its recent activities or recognize people who have prompted it before. Within the process of responding to a prompt it is unable to review what it has said so far, in order to orient itself for future action with the benefit of past action. All of these incapacities are deficits in what we would call self-awareness. GPT4 also shows no signs of being able to reflect about itself or anything else when not prompted, and no signs of having preferences and goals. It is also unable to give itself prompts that are generated by basic needs to sustain and protect itself (such internally-generated prompts are called *drives* when they occur in animals). In fact, it is unable to produce self-generated prompts at all. Moreover, it seems to utterly lack self-interest. GPT4 is essentially in a coma except when executing a prompt given by a human being. It also seems to be rather poor at reasoning, and very lacking in inventiveness, and in some of the capacities on which inventiveness rests, such as recognizing subtle but important isomorphisms. It is also not teachable in the usual sense. You can't give it give it an essay about, say, the reasons that intelligence is a useful and meaningful concept and expect it to be able to perceptively apply the ideas in the essay to other concepts, such as kinkiness, snobbishness, musical ability, beauty, malignancy. GPT4 will have learned about ways to talk about concepts, rather than ways to think about them.
Most of the capacities I named above as capacities lacking in GPT4 are not lacking in animals or in Einstein, who had self-awareness, self-interest, personal memory, drives and preferences, and teachability about their environment (as opposed to teachabilitu about the word sequences humans use to discuss their environment). So it makes sense to compare the intelligence of all of these beings ,each of whom *has* the same cluster of capacities, though some exercise that capacity at a more sophisticated level than others. I'm not sure, though, how much sense it makes to compare the intelligence of, say, GPT4, which lacks many of these capacities, with human intelligence. It would be like comparing the strength of Arnold Schwarzenegger with that of somebody who has no arms.
So I think the guy in the tweets has a point regarding intelligence not exactly being a *thing*. Most of the crucial functions I named are present in all species, and that makes it make sense to compare them in intelligence with each other, with us, and with Einstein. But GPT4 (and, I'm guessing, later versions of GPT that have similar architecture) performs quite well at some functions, but is utterly lacking in other crucial functions. So I am not sure how meaningful it is to talk about how intelligent AI is that lacks self-awareness, self-interest, drives, goals, personal memory and the ability to learn from experience (rather than from ML type trainings.).
"I get all that. But it still seems to me that there are differences in kind, not just differences in degree, between GPT4 and something smart enough to be a real problem to us if it's not aligned. The really important differences between GPT4 and human level AI & /or superintelligent AI seem to be differences in the *kinds* of processes they can engage in, rather than in how well each can engage in the process. "
See the section marked "Mini-Scenario 1: AutoGPT Vs. ChaosGPT" at https://astralcodexten.substack.com/p/tales-of-takeover-in-ccf-world, especially the paragraph that starts "AutoGPT isn’t interesting because it’s good...It’s interesting because it cuts through the mystical way some people use “agency”"
Yes I understand what you're saying there, Scott: We can give AI a goal, and not just a circumscribed goal to complete a single task, but a goal of working continuously on a big long-term goal, making decisions itself about means of pursuing the goal, and continuously engaging in the OODA loop it is programmed to follow. And you are saying that an AI in the midst of following a complex prompt like that, with an OODA loop in it, is a concrete example of agency. Contemplating this example cuts through all the fuzzy thinking about how AI doesn't have an inner life and wants and preferences so why should we worry about it ever being a goal-directed agent. Have I got it?
OK, I just had an extremely long exchange about this exact issue with Daniel Kokotajlo and Bartosz Zielinski on the Extinction Tournament threat. I hope it is not bad manners for me to quote some of what I said there, because it took me a long time and a lot of writing to express my objection to this idea clearly, and I don't want to start from scratch again, esp. since it's 2 am in Boston!
OK, my main point is that AI’s drive to reach the goal of ChaosGPT —- wreck everything — or another similarly large goal is that the goal is inserted into the AI by a human being. So, yes, we can think of Chaos GPT as having a long term goal to wreck everything, and being motivated to do so, and it doesn’t matter that it does not have thoughts and feelings about wanting to create chaos — it will behave exactly the way an agent that did would. EXCEPT that there are 2 extremely important differences between a human being who wants to create chaos and Chaos GPT. The first is that the human being has a metagoal: never give up on the goal of creating chaos, and fight back against all attempts to prevent the creation of chaos. Chaos GPT does not have the metagoal. If the asshole that set ChaosGPT going now sets GPT on another quest — let’s say to search all the online catalogs to find the a pair of size 14 red leather pants for his girlfriend, and to bargain or cheat stores if necessary to keep the price below $200 — GPT will do it. And then it will sit there humming til told to do something else.
The second difference between Chaos GPT and a person with a goal of creating chaos on earth is in the depth of structure of the drives the 2 entities have . Let’s compare ChaosGOT to animals. All animals seem to have the equivalent of OODA loops going on regarding the goal of getting food. But an animal’s drive for food comes from signals from many networked parts of the animal, and this network extends all the way down to individual cells that are signaling that they need fuel — and it extends all the way up to various complex instinctive behaviors that are prepackaged in the animal at birth, such as the stalking that cats do. The AI’s drive to CreateChaos or GetSmarter has no roots. The difference between an animal drive and an AI prompt-based drive is like the difference between a tree and a telephone pole. The tree is far far harder to remove from the ground in its entirety, and some of the small roots deep underground cannot be removed because they break off if you uproot the tree. And some trees can regenerate from roots that remain. To put it simply, The AI’s drive to create chaos was not internally generated. The animal’s drive to eat was.
And that difference between the animal’s drive to eat and Chaos GPT’s drive to create chaos has real world implications: One, as I said, Chaos GPT will willingly accept a completely different goal; a hungry animal will not. Two, let’s say Chaos GPT buys the red leather pants and the cat eats a vole and is satisfied. Now we just leave them both alone and wait. Chaos/Red Leather Pants GPT will sit there humming forever. But the cat will become hungry again in in a few hours, and the prey-seeking OODA loop will be regenerated from within the animal.
Until AI has some internal equivalent of a goal-generator, it’s not an agent with goals in at all the same way as a living being is, and that difference is going to have very substantial practical consequences. If AI’s internal structure continues to be too simple and blank for it to generate goals from within, it is only a danger if a person is telling it to go kill us all, or round us up and turn us into paperclips or whatever. Or, of course, a boss AI could be telling it to do that stuff. But then you have to explain how the boss is structured that allows it to have internally generated goals.
Someone here once said that the difference between a machine and something alive is that the machine will just sit there and rust until you turn it on. My feelings about AI are similar, with the additional caveat that engineers are not in the habit of designing machines that can turn themselves on, and I don't see how AI will be different.
> with the additional caveat that engineers are not in the habit of designing machines that can turn themselves on, and I don't see how AI will be different.
They will be different at the very least because they will be designed and deployed to not turn off, ie. always available.
As far as I can tell, the drives you are pointing to are not a big barrier. It's really easy to wrap any functional system in a layer of "now do this thing forever", and AutoGPT is a toy example of just how easy it is. Scott seems to be worried that when GPT-4 in such a wrapper is replaced by GPT-7, then bad things will happen. Charles Stross explored this in Rule 34 (the AI in that book isn't an AI, just an optimizer that's self-modified over time). I don't see what you are suggesting is going to stop such systems from being made or being deployed.
You're right, somebody could add another layer, a metagoal: *Never stop doing this.* In fact, we could make it *never stop doing this even if we tell you to.*. But again, compare the AI with the *never stop* metagoal to animals. They all have "never stop doing this"wrapped around all their main drives." *Get adequate nourishment* is one. But for the animal, the metagoal of *never stop seeking nourishment* is deeply embedded and networked with other deep drives, including *fight to the death in order to survive.*
The AI's *never stop even if we tell you to* prompt is to the animal's *fight to the death to survive* as a telephone pole is to a living tree. The former is easy to remove. I doubt that it is even possible to give AI a command that cannot later be overridden by the person who gave the *never stop* metaprompt. But even if it were, I'm sure there are many many access points into the AI system that a knowledgeable person could use to shut down its activity.
There would also be many places in the mechanism on which AI runs that are not part of the AI per se but just electrical components. AI would have not info about these or control over them, & stopping AI's activity at the mechanical level is also possible.
And regarding the fact that AI has no knowledge of and control over it's "body," i.e. its electrical components, note how different it is from animals. Animals will defend against an attack on any part of their bodies, because they are infinitely more integrated beings that AI. Their self-awareness and self-interest and determination to defend themselves and read their goals extends to every part of them. It is wired into them in a profound way. AI's ever-persisting efforts to carry out a goal we have given it is such a shallow version of agency that I don't think agency is even the right word for the underpinnings of its persistence.
You can get a Roomba and a tube of epoxy, glue the controls in the "on" position and the battery compartment shut, glue poison-tipped spikes all over it and turn it loose at a cocktail party. But you won't have created an equivalent of either The Terminator or a genius hedgehog.
> GPT4 currently is unable to recall its recent activities or recognize people who have prompted it before.
Yeah, this is a real problem. There are hacky solutions to this but no elegant solution. No real episodic memory at all. Basically the only solution proposed is have a longer context length and have it write to a memory and read from it. But then it's still context limited. This feels to me like a hacky workaround, not a real solution.
When it reads words (or wordpieces) a transformer based model like ChatGPT compares each word to every other word in a given range known as the context window. So if you feed it a sentence of 7 words, it makes a 7x7 comparison. That scales quadratically, so it can't increase forever. People keep finding hacks to increase it, but at some point there needs to be a more elegant solution. Not being context limited would be like a human who goes to the library and reads until they find something useful. LLMs need to be able to read through the internet and find stuff to help them out. Some people are adding tools to them so that they can do that, which might help. But also humans have episodic memory. LLMs do not at the moment. They need to be able to write/read from memory.
"Across different humans skill at different kinds of intellectual tasks are correlated."
That says precious little about whether skill at all different kinds of intellectual tasks is correlated anywhere else.
I can pretty easily imagine an AI that's good at producing stories - not at all good at much of anything else, including judging whether stories are true. It's called ChatGPT.
I can also imagine a robot that's strong in some ways (for some tasks) and flat out incapable of others.
Even something like height and weight - much more strongly correlated for humans than for factories (which can be very wide while being short, or fairly tall without being that wide, while humans don’t vary that much in ratio between these dimensions).
The MIRI decision theory that they think is key to alignment does presuppose a kind of Platonism - the best way to understand it is that one conceives of making the decision on behalf of an *algorithm* rather than a particular physical instantiating of it, and see which output for this algorithm being universalized would result in the best outcome. (Though maybe there’s another way to phrase it - it’s less clear to me that Kant’s formulation is committed to Platonism.)
I think it was legendary baseball author/stat-head Bill James who told me [completely paraphrasing]
-- Good Science, good arguments, give you lots of STUFF TO DO. Especially PREDICTIONS. What does the theory predict? Well, let's run an experiment, test the hypothesis, and measure the results, see if they agree with the predictions.
I think arguments and theories that are largely proven and backed up by evidence, are valuable because they provide a deeper physical framework and view that you can use to make additional predictions about the [system you are studying].
Here is where I say that Tyson is "strong" and that tells us some likely correlations about his general physical health, metabolism, quickness/neuromuscular co-ordination, etc. If the value of science is correlation and prediction, well of course LLms trained on the corpus of human speech will make startling and "intuitive" correlations. But human thought has, maybe, no underlying physical basis or like, quantum-reality.
The corpus of human speech is "true" but is it meaningful? Hmmm...
__________________
Here is where I swerve off the road because I realized in the middle of this that Mike Tyson is almost a perfect small model for transformative unaligned world-changing AI
__________________
We know, from some of the most direct, irrefutable, and culturally transformative video and human-reported evidence of my LIFETIME, that Mike Tyson is strong. (1) His unprecedented ability to, almost instantly, beat another [huge and strong] human being senseless,
---- well actually it's like the GWERN "Clippy" story, he was so otherworldly dominant --- Holy crap!
Tyson is strong, and was AI-level destructive, because he was
-- born with genetic gifts
-- Trained relentlessly
-- Was fed an incredibly difficult and skewed training dataset that had at its premise that his only value as a human being, aka his "reward matrix", was to maximize the beating other human beings senseless.
-- Had, in addition to his physical gifts, a superior target evaluation model that discarded irrelevant info and focused on the target loop, leading to incredibly quick gains and vast mismatches in learning and decision speed.
-- I MEAN, HAVE YOU EVER SEEN HIS FIRST 20 FIGHTS?
-- Also, it seems, had an incredibly strong and relilient self-model that has somehow recovered from disasters and physical truncation and continues to contribute and be studied to this day.
I'm gonna wrap up this comment while I ponder the ways that Tyson was "invisible" in 1980-84, a terrifying rumor spoken of in awe by the few who had seen toe Miracles.
I'm gonna speculate that *not* many readers of this blog were somewhat sports-obsessed kids and teenagers in the early 1980's. Maybe some of you were culture, or film, or music-obsessed kids back then, but for all the rest, let me tell you what it was like:
MYSTERIOUS
This was way before the internet, long before ubiquitious coverage and total information about anything had made it all equally worthless. Stories in major newspapers, highlights on local TV on Sunday night at 10:30, the occasional Network TV mention, were most of it. I was so cool I had subscriptions to "Rolling Stone" and "The Sporting News". I understood about 30% of RS, about 80% of TSN.
Mike Tyson was a legend spoken of in whispers. Boxing was at an incredibly low ebb, and Tyson was fighting in hockey rinks in upstate NY, and video footage was courtesy of the local HS AV club. ESPN existed at the time but Austrailian Rules Football [shout out to my Fitzroy Lions!] got more airtime than boxing.
You had to be a hard-core sports gnurd to know about Tyson. I remember being at an NFL football game with my parents and my friend Ron F, hard-core sports gnurd, and he was looking at the monochrome "crawl" on the stadium scoreboard at halftime:
"Hey B! See that! "Tyson wins by Knockout, 53 SECONDS!" Have you seen this guy!?!?"
A comparison can be made to Herschel Walker, Who was a titanic force only glimpsed in blurry 7-second clips of him running over SEC defenders for another 72-yard touchdown, or Darryl Strawberry, the tall skinny kid on the Mets who always seemed to be hurt, but hit the longest home run I have ever witnessed one September night in 1985.
Anyway, little can describe the shocking other-ness of Tyson, a 16-year-old behemoth, knocking out much larger men with two lightning punches.
There's a big difference between the concept of intelligence and the concept of strength. We know of other animals that are stronger than humans. Therefore, we can be certain that we are using a concept that generalizes across humans and non-humans in the case of strength. But we only define intelligence with regards to human capabilities. At least as far as we can prove.
If Elon Musk was stripped naked and placed in a jungle, with no way of reaching society, he would not survive long either - so, this does not explain in what way the chimp is smarter.
I think the intended point was that chimp brains (and not just chimp bodies) are better at doing some chimp-survival-relevant things. Smart man in jungle might die due to physical weakness/vulnerability, even if he makes good decisions. But if entity with smart man's brain and strong chimp body can't make it, that suggests the chimp brain is better than the human brain in some ways.
I agree that it's not enough to prove/explain much. Maybe it's just a reminder that the scope of our concept 'intelligence' is a bit arbitrary and very human-centered.
"g" is straight-up Platonism. Don't be defensive about it, a Platonist is a perfectly respectable thing to be. You could also put it in Kantian terms: g is the noumenal behind the phenomenal forms we call "witty lyricist" or "good at crossword puzzles". Platonist is not a dirty word!
I think most people are essentialists by habit, so I think most people probably are implicitly Platonists. I think rejecting Platonism probably requires you to take enough time to think about it carefully and almost nobody does that.
I guess I'm not entirely convinced that Platonism is even meaningful. But more concretely, why can't a nominalist (or other non-Platonist, if there are other kinds) make use of the concept 'g'?
If we take Wikipedia's definition, I don't see how g is laden with anything that implies Platonism, or indeed with anything that one could possibly reject other than the existence (yes I said 'existence', but this can be parsed in whatever way a non-Platonist would approve of) of some correlations:
"It is a variable that summarizes positive correlations among different cognitive tasks, reflecting the fact that an individual's performance on one type of cognitive task tends to be comparable to that person's performance on other kinds of cognitive tasks. "
Even if we give it a bit more substance, what does 'g' commit us to beyond the belief that it is likely those correlations show up because different kinds of intelligence have common causes?
> superintelligent machines are no philosophically different than machines that are very very big
Agreed. Problem is, in order do design a big machine usually you cannot just scale a small one. A switch which can break thousands of Volts is not a scaled up wall light switch: it is designed radically differently (e.g., the blades have to be usually submerged in oil, not the air, to prevent arcing). Also scaling comes with its own tradeoffs: It would be extremaly difficult (if at all possible) to design human sized robot as agile as a mouse.
If all intelligence takes is a higher neuron count, training data and time, why aren't elephants three times as smart as humans? Why are they actually vastly dumber than humans? Why was Einstein much smarter than most people despite having pretty much the same number of neurons, training data and time? You can't explain this as the correlation being less than 1.
I'm agnostic as to whether an intelligence explosion is possible. But I'm 100% certain that it won't happen in anything deep learning based. Deep learning is not capable of extrapolating beyond either what's in the training data, or in the simulator that produces training data (in the case of board games). People keep confusing interpolation in a vast database for intelligence because humans can't do it, so they have no reference point, and intelligence is the only other thing that can produce the same effects.
From another comment, someone noted a single cortical neuron is roughly equivalent to a 1000-node ~6-layer artificial neural net. So there is some multiplier there. What if every human neuron is like that? What if every elephant neuron is more like a 50-node 3-layer neural net? All this is completely speculative, but I could see a beast with larger number of neurons not being as intelligent.
You could make any number of variations on the theory to make it fit, because it's a bad theory. It doesn't explain how assemblies of neurons represent concepts, learn or plan. We don't have a full account of that yet, but we knew enough to falsify the "blob" theory decades ago. There are mountains of evidence for innate modules in humans and other animals, and likewise against blank slate theories. Everything from learning to swallow food, walk, talk, learning from others, to competition for status and attracting mates. All of that is strongly influenced by mental modules that have evolved specifically in humans, and other modules exist in other animals. A human baby raised by a chimp mother will not be an incredible chimp due to its superior neuron count. It has different software, different learning algorithms that are primed for different sensory input. Human brains across all cultures and times don't coincidentally develop the same anatomical and functional organization. It's genetically encoded.
As someone who finds the module theory in humans quite evidently true, I do not believe that this is an argument against anything in the post. Yes, humans have many specialized modules (language, two visual recognition ones, presumably social networking, to name a few), just like, say, migrating birds do. This should not obscure the fact that humans also have stronger general intelligence than any other species - in fact, a cognitive scientist can distinguish when modules (fast, automatic, barely regulatable) work and when what's usually called System 2 (slow, traceable, stoppable) works.
Regarding the original question, body size matters because, the bigger the body, the more neurons you require to process info from body and regulate its movements, so you need to subtract them - hence the encephalization quotient calculations and all that jazz.
Are you saying that a) general intelligence is a separate mechanism from modules, or are you saying that b) general intelligence arises from the interaction of modules (which is what I believe), but you don't see how that contradicts the post?
Infants either already know, or learn from extremely few examples, about simple properties of objects, places, numbers, geometry, agents and causality. Abilities that directly build on and combine these emerge in predictable order and timing, and this process continues to compound throughout life. This solves the problem of where inductive biases for learning new skills come from.
This is incompatible with the post, which ignores this issue (or is confused about it, as CounterBlunder points out), and posits that a uniform, unbiased blob of neurons can learn anything, in humans as well as machines. This contradicts the bias-variance tradeoff, and ignores both the orders of magnitude difference in amount of training data used in language and vision in AI vs humans, the inductive biases in NN architecture and SGD, and the fact that deep learning and humans generalize in completely different ways - humans can generalize outside the training data distribution.
I didn't understand your point with System 1 and 2 - why are modules equated with system 1?
I know of both the theory "everything is modules, so-called generla intelligence is just interaction effects" (your b) and the theory "there are modules, and then there's general intelligence which does everything we don't have a special module for" (your a), and I claim a is right and b is wrong, based on, surprise-surprise, developmental psychology and module interaction properties. In particular, as a linguist, I can bring some examples of how our innate linguistic module works and how laypeople reason on language when they try to apply general intelligence to it - vastly different processes with vastly different results.
The system 1-system 2 point is directly relevant to it: modules are automatic, they cannot switch to system 2 mode (you cannot make your linguistic module make acceptability judgments slowly and with laying out the steps or make your visual recognition module explicitly correct for optical illusions, you have to take its output and modify it, which is different), while the general cognitive ability can. (And certain properties of that system 2 mode vote against "well, maybe your 'general cognitive ability' is just another module then".)
I think you don't understand what the generalization differences actually imply. We know, in broad strokes, how modularized abilities work, we know how general intelligence in humans works, and we know how the bitter-lesson-neuron-networks work. These are three different ways.
Can you explain in more detail? I don't know how innate modules or general intelligence in humans work, but I know how deep learning works. Deep learning works by interpolating training data in a learned latent space. It can't learn the kinds of abstractions that humans learn. Not everyone agrees, but as far as I can see, they either don't understand it, don't understand the implications, or their paycheck and prestige depends on them not thinking too much about it. But this rules out deep learning as a mechanism for cognitive abilities like perception, language and planning. These abilities clearly work despite constant distribution shift, and that's their whole point. We can recognize our surroundings in novel circumstances, learn new concepts and invent new things.
I'm not sure if that's what you're saying, but optical illusions reveal certain biases of our visual perception, and they don't go away even though we understand that they are illusions. To me, this shows exactly that innate modules serve as a basis for higher-order cognitive abilities, in numerous ways. They provide the initial concepts which we compose, alter and recombine into new concepts. We use them to construct analogies that allow us to transfer understanding of concepts from familiar domains to novel domains - I'm sure you're familiar with this as a linguist.
Maybe you're equating general intelligence with symbolic reasoning? But that clearly builds on language, spatial reasoning and causal reasoning. But you also say that it's not another module?
Thanks for the link. We can substitute neuron count for cortical neuron count, or cortical neurons * neuron packing density * interneuronal distance * axonal conduction velocity.
Let's say this quantity correlates perfectly with intelligence. Does that show that it's necessary and sufficient for intelligence?
No. If you run a stupid algorithm on a more powerful computer, you get marginally better results. If you run a smart algorithm on a weak computer, you get bad results. If a species evolves some degree of intelligence (say, implemented as 50 different interacting circuits), there will be selection pressure to evolve enough hardware to support it optimally (let's say by copying the 50 circuits over and over). But it won't evolve more than necessary, because it costs energy, complicates birth, and you get diminishing returns anyway. So in this scenario we have perfect correlation, but no corresponding causation.
You can't just point at a correlation, quote the Bitter Lesson, postulate an analogy between biological and artificial neural networks and call that a theory of intelligence. There's no explanation of how knowledge is represented, or learned, or turned into behavior.
> Why does this work so well? Because animal intelligence (including human intelligence) is a blob of neural network arranged in a mildly clever way that lets it learn whatever it wants efficiently.
”Mildly clever” feels premature given that we still have no idea how the human brain works. And until we do it feels epistemically dodgy to use ”neural network” as an undifferentiated catch-all for both MLPs and brains.
The root problem is thrown into relief if you add ”can navigate novel physical environments” to your list of intelligence criteria. We won’t solve that by hand coding linguistic rules, but we probably also won’t solve it by adding layers to gpt.
Much of human intelligence is about sex appeal, because sexual selection is how we evolved. Us male men, we try to impress the women with how well we talk, groom, impress and manipulate other humans, solve problems, make money, write poetry, make art, play sports, sing, play guitars, etc.
We don't benefit from having sex with whales so we don't try to impress them. We don't even know what would impress them. Nature isn't interested in us using our intelligence in a way that would sexually attract whales.
Thus, our notion of intelligence is very constricted to the human domain.
"These considerations become even more relevant..." Sorry, the considerations just don't seem that compelling. Corollary: "IQ correlates to positive life outcomes" is a statement of pure value judgement, because appropriate life outcomes are culturally specific. Martyrdom is a perfect example; in many cultures past and present, martyrdom and widow-suicide are considered excellent life choices. And not just by tribal people in some steaming jungle... We all know (?) that Al Qaeda members were/are disproportionately engineers.
I think the 2nd tweet makes a valid point, which you're not really engaging with in this article. In alignment discourse I often notice the tacit assumption that a system which is intelligent thereby also has agency, self-preservation instinct, goals, and/or desires. It's quite likely that people don't feel the need to explicitly argue for these properties as much as they otherwise would, precisely because of this kind of "platonic ideal" of intelligence (a "soul", even if they wouldn't admit it) being ascribed to AI systems.
That last paragraph is nonsense. You seem to be missing what the thing is and what it is for. Beavers for instance went through an intelligence expansion in their evolutionary history and then plateaued. The "intelligence explosion" in humans from recognising markings of ownership on vases to the adapable Greek alphabet likewise was randomly fitted through enviromental fitness and a social type of being. The intelligence of beavers should produce its own values as a distinc category, and the same should be for AI, but it cant expand without an agency because machine intelligence is a tool. The only intelligence explosion we can atribute to machines will be at our current plateau of organization. If you really mean just engineering feats for humans, you should just say so, but I doubt there can be an expansion of human valued intelligence without either our expansion as the placeholder or an AI with biological similarity for environmental fitness and social organization: Grey-goo the Wise.
On one hand, the distinction between [assuming X axiomatically] and [arriving at X through reasoning about evidence] is meaningful and valid, and as (we seem to share a worldview in which) the latter is better than the former, I can see why you'd insist on making it.
On the other, once you start using X in your further arguments, both boil down to [assuming X]. And once you get your priors trapped at X, it boils down to [assuming X axiomatically] regardless of how you originally arrived at it.
So yes, it's wrong and judgemental to accuse others of Platonism. But I don't think you'd disagree that there exist real factual differences between you two about the degree to which the concept of "intelligence" is useful and meaningful, and you, of all people, should know full well that we humans are bound to keep making mental shortcuts straight to the city centre.
---
And while we're at at, he's right. At the very least, there's an important difference between intelligence as in "Artificial Intelligence", which is a field of science dedicated to making machines solve demanding cognitive tasks, and intelligence period, which is the inherent ability to perform those tasks. You do mix up the two, in a way that extrapolates from the former to the latter.
I mean, to plant my flag here, the entire section 3 of the post is just completely backwards. There is a valid retelling of the same events, where some people tried to imbue formal reasoning and domain knowledge and other explicit aspects of intelligence into their systems, and then a bunch of other people came, did a simple statistical analysis on gigantic chunks of data, and got better practical results (in the field of Artificial Intelligence, while not getting any closer to actual intelligence).
Counterintuitively, I find this quite encouraging. But it's kind of hard to explain why.
My coworker Dave, who's way smarter than me, can write an algorithm that I can't understand, which performs 100x better than my algorithm. The difference between code / a machine / an algorithm that kinda sorta works and one that performs near peak efficiency is huge, and the difference between being able to create the better algorithm and not is a matter of intelligence.
So, if AI was like an algorithm, then I can imagine it getting smart enough to rewrite itself from my version to Dave's version, and getting even smarter, and rewriting itself again, and its intelligence forming an exponential curve into stratospheric, Singularity heights.
But the point of this article is that intelligence isn't Platonic, and AI isn't like that. It's a giant blob of compute and training data, and so there's no way to rewrite it (like Dave rewriting my algorithm) and suddenly get 100x efficiency because of being more intelligent. It's much easier to imagine a blob of compute and training data running into diminishing returns and slowing to a stop.
So I think that while this establishes that an intelligence explosion is a coherent idea, it makes it seem much less likely.
“Basically, we created this word and used it in various contexts because we're lazy, but then some people started assuming that it has an independent reality and built arguments and a whole worldview around it.
They postulate an essence for some reason.”
The fact that we name stuff is a big deal with some philosophers who seem to not differentiate between absolute social constructs (ie religion, God, nations to an extent) and stuff we noticed and named. Like a tree.
Seeing and naming intelligence is not a performative utterance, it doesn’t create intelligence ex nilho, it’s recognising what exists and can be measured.
Pascal's mugging requires extremely low probability x extremely high badness. People worried AI do not assign very low probabilities to AI giving problems. Even a 1% is not so low that you can't manage it cognitively.
My main issue with this kind of take is that it takes bundles of correlations which have been fairly thoroughly observed and tested *in humans* and then proceeds to claim those correlations will hold for AIs.
What we've observed in AIs is that yes, improving performance in one task does tend to improve performance in other tasks in a manner superficially similar to human intelligence - however, there are enough patterns where an AI which otherwise seems to be extremely clever fucks up in a way that a human never would that we can be reasonably sure that the *mechanism* whereby the AI is performing these tasks is significantly different from the mechanism a human being uses.
The idea of "intelligence" is a useful enough framing to talk about AI because there is some correlation between how "smart" an AI is and how an equivalent human would act - but there is no reason why this must hold indefinitely - especially for LLMs, which get all of their "smarts" from the structure of recorded human writing.
It feels like, in trying to make an LLM smarter, eventually we will start hitting a wall based on the fact that an AI that has learned everything it knows from human artifacts has no (simple) way to figure out things that humans can't do - and it is likely that, as it tries, it will start exposing the fact that its "human-intelligence-simulation-mechanisms" are very different from how humans reason and are not subject to the same correlation-generating constraints. Hence, I think it is very likely that the correlation which allows us to describe AI capabilities as "Intelligence" will eventually break down.
Of course, none of this rules out the idea that AI could eventually do Very Bad Things to us and that alignment might be a huge issue - but I don't think it can be boiled down to "AI will be very very smart and hence dangerous".
The problem with this argument is that we have mountains of evidence that the correlations between intellectual strengths that we call IQ is a fact about human and perhaps animal biology and NOT a fact about intelligence-as-such. To give but three examples: Deep Blue is superhuman at chess, but 100% stone illiterate, Watson is superhuman at Jeopardy, but almost certainly couldn't even make valid moves in chess, and a pocket calculator is superhuman at finding square roots, but can't play chess or Jeopardy at all. This doesn't even scratch the surface: you'd be hard pressed to find any computer system that isn't superhuman at some task, and yet none so far have had anything like the ability to form and achieve arbitrary goals that rationalists mean when they talk about capital I Intelligence. Taking the outside view, you have to have a very strong presumption that problems people have found with LLMs are real limitations and that they don't have intelligence-as-such either, though this isn't to say they aren't powerful or interesting or even dangerous. The alternative is an argument from ignorance: we don't really yet understand fully how LLMs work, so we assume that this time we've just finally made a computer program that's actually intelligent. Sorry, I don't find that at all persuasive. If you want to convince me that LLMs can achieve arbitrary goals, show me an LLM achieving arbitrary goals. That hasn't happened yet.
I think there's a further point to this argument: LLMs are strictly trained on human or LLM output. If humans can't be superhuman (which is true by definition), we'd need some very good justification for how LLMs could plausibly become superhuman based on mimicking human artifacts.
Right now we have two kinds of functional "AI": Single-purpose superhuman AIs trained by reinforcement learning, and kind-of-general-purpose worse-than-human LLMs trained on human output. There is no such thing as a general-purpose superhuman AI and most of Scott's argument is based on that existing.
Attempting a steelman: no single human can internalize the sum total of recorded human knowledge. Even large, coordinated groups of humans have difficulty researching limited subsets. If an entity were capable of ingesting e.g. the entire Library of Congress, it wouldn't even need to bring a superhuman quality of cognition to bear; simply being able to process and analyze such a vast corpus of data from a single perspective would allow inference, deduction, and probably synthesis, beyond human ability.
Indeed. But right now at least it seems that LLMs, being, well, somewhat limited by their mandate to produce coherent text that looks similar to human-generated text, do not make very effective use of different domains of information at once. Simulator theory seems to indicate that the main way LLMs function is by first "selecting" a "domain of expertise" and then working within that domain, with cross-domain work dropping in quality fast.
I think you're cheating a little bit with the phrasing of "arranging your blob cleverly". Can you articulate exactly how that's different from building in specific computational structures/abilities, like the "Responsible People" were trying to do? Like, I agree that a transformer is way way more general and blob-like than hand-coding in specific linguistic rules or whatever. But I've found it tricky to draw a line between what the people who don't get the Bitter Lesson are doing wrong and what the people who invented transformers are doing right.
As I've pointed out many times before, "intelligence" is certainly a coherent concept--but its meaning can be summarized as, "like an ideal human". (That's why the measure of artificial intelligence that everyone comes back to--despite its obvious glaring flaws--is the Turing test.) And it's not at all clear that "ideal humanness" is even fundamentally quantitative--except in artificially imposed ways--let alone subject to "explosion".
> how any set of cognitive capabilities can be self-amplifying [from the twitter screenshot]
This is ... obvious to cognitive psychologists, because that's how human learning works? It even has a name in education research: the Matthew Effect (a.k.a. The Rich Get Richer).
Basically, give a class of kids some text to read; the ones who have better vocabulary, domain knowledge, or both, will learn more from the same text than the ones who don't. This is a real stumbling block for some attempts at progressive education of the no-child-left-behind kind, because even if you give the whole class a text at the level that its weakest student can sort of understand, the top kids will still get more value out of the same text, up to a point.
Some papers describe the Matthew Effect as a "positive feedback loop", which sounds awfully like a synonym for "self-amplifying" to me. It's particularly pronounced in reading comprehension, so you'd expect to see the same if you're training GPTs instead of schoolkids to "read" and "comprehend" in a general sense of these words.
I agree with the general thrust of this argument, but defining concepts as “bundles of correlations” is both overly narrow and misses the point of the original tweet about platonic ideas. This may be true of “inductive” concepts like strength that are based on observation, but omits higher level abstract concepts like “justice ”, “good”, or “beauty”. Those ideas are not rooted in correlation, but in the network of high-level values that form the basis of our identity.
Also, the framing of “intelligence” by the author is a bit one sided. There are currently multiple, divergent definitions within cognitive science that refer to the ability to learn, to reason, and to adapt to new situations. Charles Spearman, an early 20th century psychologies argued that intelligence is a single, general cognitive ability (the g-factor) that underlies all specific mental abilities. Psychologist Howard Gardner posited the theory of multiple intelligences: linguistic, logical-mathematical, spatial, bodily-kinesthetic, musical, interpersonal, and intrapersonal. Then there is the triarchic theory of intelligence proposed by psychologist Robert Sternberg, which posits that intelligence is composed of three components: analytical, creative, and practical.
A better, broad definition of intelligence goes beyond humans and highly evolved animals, to include all life forms and even machines. We should equate intelligence with the process generally considered to be at its core: problem solving. Problem solving can be described as the process of identifying, analyzing, and resolving problems or obstacles to achieve a desired goal or outcome.
How complex this process really is only became apparent when people attempted to replicate it in a machine. In 1959, Allen Newell and Herbert Simon developed a machine called the General Problem Solver. To do this, they formalized the concept of problem space, an abstract space containing an initial state, goal state, path constraints, and operators (rules) defining how to move within it. Their insights were tremendously important and continue to have applications in machine learning until today. However, as they honestly conceded, their method encountered major obstacles that apply not only to machines, but to all life forms trying to solve problems.
First and foremost is combinatorial explosion. Whether digital or physical, large problem spaces with complex topologies of constraints and operators lead to a number of possibilities that exceed computational abilities. A simple chess game, for example, has an estimate of 10 to the power of 120 possible moves. This is known as the Shannon number and is larger than the total number of atoms in the universe. There are too many options to list, let alone compute.
The second limitation are ill-defined problems. The concept of problem space assumes that goal state and constraints are known and well defined. However, due to the frequently poor quality of available information, this is rarely the case. “How to write a good essay?” My initial state is an empty page. My goal state a good essay. Where is the path to a solution? Frankly, I don’t know. I make it up as I go along. Ironically, the vast majority of situations we encounter are precisely of this kind.
This is related to what philosopher David Chalmers calls the finitary predicament. According to Chalmers, the finitary predicament arises because our minds are finite and limited in their capacity to process information. We can only perceive a small fraction of the physical world, and we are limited in our ability to comprehend the complexity of the universe. As a result, our understanding of the world is necessarily incomplete and imperfect.
I feel that in the current A(G)I debate, it is important to clearly define the conceptual boundaries and definitions of these terms in order to have a meaningful debate.
I was prepared to entertain this argument but your analogy to strength is so wildly incorrect that I gave up. If you spent any time trying to actually understand what strength is, how it works, how we understand it and *can measure it*, it would help you understand how unlike "intelligence" it is.
But you won't, because the belief in one-dimensional measurable intelligence is a Core False Belief of the Ideology of Rationalism, and leads directly to AI Doomerism. (See http://www.paulgraham.com/lies.html to understand how ideologies form around false beliefs) The screenshotted tweets at the top are simply correct.
Medium warning (50% of a ban) for saying I am stupid and wrong and not giving any counterarguments or specific reasons. I feel like I've gestured at what I mean by "strength" and given supporting numbers (like correlations between different strength measures) - if you think I'm wrong, please give me the same courtesy.
“Intelligence” is another useful concept. When I say “Albert Einstein is more intelligent than a toddler”, I mean things like:
Einstein can do arithmetic better than the toddler
Einstein can do complicated mathematical word problems better than the toddler
Einstein can solve riddles better than a toddler
Einstein can read and comprehend text better than a toddler
Einstein can learn useful mechanical principles which let him build things faster than a toddler
…and so on."
This seems to go against the very notion of "intelligence" meaning more than "knowledge" or "skills." It also seems to be an argument against the existence of g. Although a toddler may also be less intelligent than the same person would be as a grown adult, we recognize frequently very smart children that we would say are "smarter" than most adults, even if they don't know a lot. James Sidis was clearly more intelligent than most adults when he was very small, but lots of adults also knew Greek, or whatever subject he was unusually good at from a young age, and almost certainly better than him.
Based on this definition of intelligence, almost every adult would be "more intelligent" than almost every child, but I don't think you believe that and I certainly don't.
I think I do believe almost every adult is more intelligent than almost every toddler. I think the standard concept of IQ is age-adjusted. LeBron's 3-year-old kid who is 99th percentile height for his age is tall-for-a-toddler and destined-to-be-very-tall-as-an-adult, but he isn't objectively tall compared to adults.
I agree that once we're talking about 10-year-olds or something there might be some who are better at abstract reasoning tasks than some adults while also knowing less than they do, which is a good example of ways that intelligence is less-than-perfectly-correlated.
A quick Google for "intelligence" comes up with "the ability to acquire and apply knowledge and skills."
That much more matches my understanding of what we mean by intelligent than having the knowledge and skills. Otherwise we would use the word "skilled" as a synonym for intelligence, when instead we recognize that they correlate but are not the same.
A toddler may have a greater ability to learn concepts than someone much older than themselves, even while knowing very little.
By your definition I'm not sure what to make of James Sidis and other super-geniuses. He apparently taught himself Latin at age two. When he started learning Latin, was he not "intelligent"? That he was able to learn it seems very relevant to our understanding of intelligence, even before he actually learned it.
This illustrates perfectly how theoretical linguistics got divorced from NLP - at some point, NLP guys understood they can just get much more data than a two-year-child learning the language and brute-force the problem instead of emulating parameters of Universal Grammar or whatever.
Some concepts are bundles of useful correlations but not all concepts. And the concepts that are only bundles of useful correlations have one thing in common: They brake down outside of the domain in which they are useful data compressions.
For example, an average dog has much higher biting strength than Mike Tyson on Scot's grandma but they probably both beat the dog at freelifting. Meanwhile a hydraulic lift will beat Mike Tyson and Scot's grandma in lifting but neither of them at wrestling. So Scot's explanation of strength kind-of holds for people but not for things very much unlike people. Compare this to a concept not defined by typical correlations like height, where the Empire state building is taller than Mike Tyson just like Mike Tyson is taller than I am (I have insufficiant data to specify the hight comparisons to Scot's grandma, but that is actual lack of knowledge not lack of conceptional definedness).
Normally correlations can't be extrapolated beyond the domain they are observed on (think about it, a correlation coefficiant is defined by the slope of a linearization and most functions aren't linear). And factor-analytical constructs springing from many such correlations almost never can be so extrapolated, which is why every entry level stats textbook warns against reifying them. It is of course also true that social scientists mostly nod along and then ignore all the warnings from baby stats but that kind of sloppiness gets you a replication crisis.
So no, there is absolutely no reason to expect intelligence to be a useful concept outside of humans in a vaguely normal intelligence range unless you think more real than other factor-analytical indices. In other words the tweeter is dead right, AI doomers at least of the classical Yudkowskyan kind expect the concept of intelligence to be useful far outside the domain that expectation would make sense in an do so precisely because they are mistaking it for an actual thing.
Altman has said intelligence is an emergent property of physics, I believe the implications are that intelligence is based on geometric invariants and symmetries. Encoder-decoder models are learning reduced representations during bottleneck compression, with the ambition of converging to platonic ideals and irreducible representations. After the ideal forms are captured, the training examples are no longer needed a la Wittgenstein's ladder.
No objections, but IMO engaging with a weak argument.
You've given examples of where a word reaches the limits of its powers of comparison. The question is, when does that happen for intelligence? When does intelligence break apart into the strength equivalent of biceps vs triceps, pound-for-pound, height leverage, etc?
I think it's fair to say that more/less intelligent doesn't well describe our relationship to ChatGPT already. It can write far faster than I can, it has a far greater breadth of knowledge than I have, etc. I'm better than it is at introspecting myself, at my own profession, etc.
There will come a time when an AI gets better than me at literally everything, with the last vestige possibly being reverse engineering my own mental state and knowing me better than I know myself. But none of this speaks to "foom!" per se. The explanatory powers of "intelligence" have broken down, they are currently not useful to describe the infinity dollar question.
But we can say some things about foom when we stop talking about "intelligence goes up" and start talking about specific goals. AI might be able to solve ECC but it cannot solve a one time pad. Things might move extremely fast, but every single type of scaling we know tapers off very quickly once it runs into its physics. Strength is a great example:
We can also talk about more useful properties of an AI than its "IQ":
1. How much information does it contain? (consider a human without any sensory input from birth)
2. How much information can it bring to bear on a question in a given unit of time?
3. How well can it avoid applying spurious information to a question?
4. How well can it avoid skipping information that would provide insight?
5. If 3 and 4 are inversely proportional, are they linearly or exponentially so?
6. (this list could be greatly expanded, I'd like to as well)
It's perfectly possible that an AI at 99% of its "intelligence" runs on the current total power output of the world and makes a couple insightful discoveries, but at 100% of its "intelligence" runs on all the stars in the universe and... proves that P != NP.
I’ve worked in deep learning research and engineering for the past decade and I want to say that Scott Alexander is completely right, but I want to add some mathematical rigor to this. Also to push back on one common argument against this. And also clarify some historical stuff he was a little fuzzy on.
Neural nets are Turing complete when they’re infinitely deep. And they’re universal approximators when they’re infinitely wide. What does this mean for the non-mathematicians? It means that they can compute anything that can be computed. That’s a bit of a circular definition, but the relevant part is that they can do anything *in theory*. Since they can compute anything, this means that they are already, in theory, capable of superintelligence. So why aren’t they superintelligent right now, in practice?
2012 was the breakthrough year for deep learning. That was when Geoff Hinton’s techniques really started working in a way that was impossible to ignore. Historically, why they didn’t work well until 2012 was that they tended to badly overfit, meaning that they memorized their dataset exactly, but could not “understand” what they were learning [https://en.wikipedia.org/wiki/Overfitting].
So how did Hinton get around this? By using something called an autoencoder for layer-wise pre-training. Say that you’re trying to classify images of handwritten digits to figure out what digit it is. Rather than just train a neural net to take an image and output the number, you take the image and compress it down, and attempt to reconstruct the original image. What this does is it forces the neural net to learn abstraction layers, because the only way it can compress something down and still reconstruct the original is to learn common patterns in the data. It would learn how pixels are commonly combined into lines, and then shapes, and how those shapes are commonly arranged. This is referred to as representation learning. Layer-wise pretraining went out of style a long time ago for reasons that aren’t relevant here, but the point is that what makes neural nets powerful is their ability to learn abstractions. More data from more tasks leads to better representations and better abstractions.
Mathematically, rather than debate “what is intelligence” I think we should say that multiplying matrices together is a very general process that can learn any function in physics, meaning that it can learn anything happening in the real world. One very general problem is that these things learn “too well” in that they still often overfit. One the main things we do in the field on day-to-day basis is get them to generalize better and overfit less. Our current models are likely powerful enough to loosely memorize half the internet. But that’s pretty useless if it hasn’t learned abstractions. That’s what makes the Transformers (AKA soft attention with positional encoding) such a breakthrough.
What’s stopping us from having superintelligence now?
At a very high level, the answer is overfitting. Feeding it more data from different domains will help regularize it, meaning reduce overfitting and improve generalization to new tasks. So, yes an intelligence explosion is possible but not guaranteed. More tasks and more domains will ultimately led to improved performance across all domains as the neural net learns better abstraction layers that represent real-world physics. This includes language.
Also, a common argument against this is that LLMs are terrible at basic arithmetic. But this isn’t a fundamental issue with matrix multiplication, rather an issue with the activation functions we use. I think that Neural Arithmetic Logic Units might be a solution [https://arxiv.org/abs/2101.09530].
> In the middle of a million companies pursuing their revolutionary new paradigms, OpenAI decided to just shrug and try the “giant blob of intelligence” strategy, and it worked.
The “Attention is All you Need” paper that invented the Transformers was actually out of Google. The authors also didn’t really understand how powerful their technique was and originally thought it was just a useful technique for translation and that was about it. Alec Redford over at OpenAI then had the idea to just use it for pre-training on English. In line with what Scott Alexander argues, Alec thought of it the same way with “Language Models are Unsupervised Multi-Task Learners.” [https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf]. Basically, that getting better at one task would correlate with improvement on others.
> The limit of Jelinek’s Law is OpenAI, who AFAIK didn’t use insights from linguistics at all
This is correct that they don’t use linguistics, but I think you’re giving OpenAI too much credit here.
You can define a numeric scale of "strength," with 3 representing somebody's metaphorical grandmother and 18 representing Mike Tyson, sure. You can come up with methods to estimate where an arbitrary human is on that scale, and use an arbitrary human's "strength" value to make reasonable predictions about what specific tasks that human can do, sure.
But, that doesn't mean it's meaningful to imagine a human with a "strength" of over 9000 and speculate about what that human might be capable of based on linear extrapolation. You can certainly come up with a meaningful number for what the grip force of a human with a 9000+ "strength" would have to be, but it's probably very silly, if not physically impossible, to design your uber-gripping device as just a scaled-up human.
No, AI-doomism is not Platonism; it's just a bait-and-switch.
As the story goes, AI is existentially dangerous because it will soon acquire superpowers, such as the ability to hack any computer, convince any human to do anything it wants, cover the Earth with "gray goo" nanotechnology, engineer a ~100% lethal pandemic, etc. It would acquire these powers by being "superintelligent" -- the AI would be to the smartest human (usually considered to be Von Neumann) as that human is to a rat. And it would become "superintelligent" via recursive improvements to its processing speed and total memory. So, act now, before it's too late !
But this story falls apart as soon as you look at any of the steps in detail. For example, what is "superintelligence" ? Well, it's usually described as the quality of being "super-smart", but what does that actually mean ? Sure, the AI could e.g. "solve math problems faster than Einstein", but calculators can already do that, and they're not "superintelligent". In practice, "superintelligence", as the AI-doom community uses the term, is synonymous with the ability to acquire superpowers. So, the AI will acquire superpowers due to its superior ability to acquire superpowers... this logical deduction is true, but not terribly impressive.
What about processing speed and "neural blob" size, however ? Surely, throwing more CPUs at any problem makes it easier to solve, so the AI could in fact solve any problem by throwing enough CPUs at it, right ? Well, sadly, the answer would appear to be "no". For example, modern LLMs (Large Language Models) such as Chat-GPTs are "Transformers" (that's the "T" in the name), a new type of deep-learning neural network that was discovered relatively recently. They differ from old-school NNs not merely in size, but also in *structure*; it is this new type of structure that makes them relatively effective at generating text (and the same holds true for image generators). You would not be able to make an LLM by merely networking together a bunch of 1980s-era Sharp calculators, just like you cannot make a genius by networking together a bunch of nematodes. Unfortunately, the term "neural network" is a bit of an aspirational misnomer; we humans (and other animals) have brains with radically more complicated structures than the Transformer; structures that allow us to do many things modern matrix-multiplying ML systems simply cannot do, such as learning on the fly.
Of course, AI research will continue, and it's possible (in fact, likely) that one day we will discover something more powerful than the Transformer (powerful enough to compete with human brains), but that day would not appear to be coming soon. Throwing more CPUs at the problem won't help (I mean, it helps a little, but the problem complexity is exponential); and, what's worse, many of the putative AI superpowers are outright physically impossible. So, I wouldn't go around panicking just yet (or, in fact, ever).
> Sure, the AI could e.g. "solve math problems faster than Einstein", but calculators can already do that, and they're not "superintelligent".
They can't. They can perform one part of the problem-solving process (calculation) with vastly superhuman speed and accuracy, and other parts (reducing a problem to a set of calculations) not at all. And they're incapable of finding reality-relevant 'math problems' in the first place.
Yes, and this exact criticism is true of e.g. modern LLMs, only more so, since they are comparatively worse at math than dedicated math tools. It would appear that an entity's capacity to "solve math problems" depends on how you define "math problems", which makes "solving math problems" a poor metric for "intelligence".
I just don't see why 'superintelligent' can't be defined in a way that doesn't collapse into 'able to acquire superpowers'. What's actually wrong with the sort of definition that would cash out as 'better than Einstein at *doing theoretical physics*, better than great genius X at doing Y, ...' and so on for a sufficiently broad class of intellectual endeavours?
I'm not claiming that LLMs or something like them will scale up to general superintelligence -- I don't have the technical knowledge to form any confident opinion on that.
> I just don't see why 'superintelligent' can't be defined in a way that doesn't collapse into 'able to acquire superpowers'.
You totally could, but then (as I'd said above) your argument is, "an entity that has the ability to gain superpowers is dangerous because it could get superpowers". It's not a very interesting argument.
How is that the argument? I'm not trying to be annoying, but this just doesn't make sense to me. (edit: Unless my triple-negative sentence was confusing and you read me as advocating the 'superintelligent = able to acquire superpowers' definition.)
The argument made by the AI risk people is something like 'a superintelligence (i.e. an entity that is much smarter than humans in just about all the ways that humans are smart) is likely to be dangerously powerful'; they say that the exact route from smarts->power is hard to predict in advance, but are often happy to elaborate on their reasons for thinking some such route exists.
I don't see how that is equivalent to the argument 'superpower-getting-capability -> superpowers -> danger', except in the sense that every logical argument has to contain the seeds of its conclusion somewhere in its premises.
> (edit: Unless my triple-negative sentence was confusing and you read me as advocating the 'superintelligent = able to acquire superpowers' definition.)
Indeed, that was my misunderstanding, apologies.
That said though, my problem with your preferred argument is precisely the vagueness of it. It sounds good on the surface: indeed, something much smarter than us, and malicious in nature, sounds like it could be extremely dangerous. But if you start asking questions, the argument falls apart.
What exactly do they mean by "smarter" ? Well, maybe it's the CPU speed or the ability to solve math problems or something. Ok. What is the mechanism by which "smarts" translate into superpowers ? It's unknown and perhaps even unknowable, but surely it is possible; after all, look at how much smarter and therefore more powerful we are compared to mice ! Um, ok. It goes on and on like this, and usually it turns out that the *only* aspect of "superintelligence" that we can pin down is "ability to acquire superpowers"; everything else is some manner of vague generality or metaphor or analogy or something. Hence my objection.
If you make the last stage tautologous, then the unlikelihood has to all be in the former stages. But you haven't stated that, eg., there is a ceiling to potential intelligence, or a lack of possible motivation.
But in summary, yes, if we define "intelligence" as "ability to acquire superpowers", then there's absolutely a ceiling on it. For example, it is very likely that no amount of "intelligence" will enable you to acquire FTL travel.
The fact that it's specifically language engines such as GPT which are the closest to generality is actually an evidence that intelligence is less correlated concept than we might have thought.
I think this is a question of consciousness and not intelligence. I think we cling to the word intelligence because that is the primary value we see in "AI" technology, and so we try to measure human intelligence qualitatively and quantitatively and evaluate the performance of some information processing model against that.
In reality, humans do not have a scientific explanation for our own consciousness yet we have a myriad of non-scientific explanations that overlap in profound ways.
I do think it is possible for an "AI" to become intelligent and capable enough to have profound, and even independent/uncontrolled, impacts on society without us translating our consciousness as a pre-requisite. However, for me that conversation merits an STS oriented conversation with consciousness/morality oriented questions in parallel. I feel as though society blurs the two, and underemphasizes the importance of regulating the technology as is in the present due to our concerns for the future which are often more laden with fear and black/white thinking.
This societal blurring is a frequent theme in STS that often causes further disconnect between already misaligned societal stakeholders. e.g. One can claim that the (valid) fears of a climate change fueled societal collapse have polarized people based on their feeling of likelihood of that future scenario. The closer we get the harder it is to avoid, but in the pre-internet era the societal landscape of information communication was quite different.
One could then debate whether a more present oriented approach with more tangible messaging (e.g. keep your community clean and healthy vs save the planet) would have worked or if a more radical (e.g. stop everything now) would have worked. In the end, what happened happened and these are simple considerations for relative adjustments to explore alternative ideas about how we navigate issues as potentially fueling our own exinction... whether it be through the climate, "AI", nuclear armaggeddon, or some other technological system gone awry.
Maybe you have "overlearned" the Bitter Lesson, and are generalizing it to apply in domains where it has never been observed (and where there are some good reasons to think it won't hold)?
I think the Bitter Lesson is unassailably true (and nonobvious, and useful) in Chess and Go. I think we could generalize that to "two-person turn-based discrete finite games of complete information" pretty safely.
When we add in games of incomplete information, my confidence gets a little lower, though the results in Texas Hold'em and Starcraft are encouraging.
What about natural language translation? Here there is some vagueness about the success metric, which puts us at risk of Goodharting ourselves. If the goal is to translate large numbers of documents at 90%+ accuracy for the lowest marginal cost, perhaps the machines (and the Bitter Lesson) have won. But if you wanted to translate a single document with the highest accuracy at any cost--say, if you had to sign an important bilingual document like a peace treaty or a 99-year lease for land to build a skyscraper on--you'd probably still want to hire a human expert, right? It's not clear to me that any Bitter Lesson-driven research will ever surpass a human at this important skill.
GPT4 has racked up another Bitter Lesson win at the game of "take a context window and generate a sequence of subsequent tokens that resembles, to a human, what another human might write". If you want to produce a large amount of reasonable-seeming text at the lowest marginal cost, it sure seems like the Bitter Lesson is the way to go. It's not totally clear to me whether this has any actual economic value (not relying on deception). I used to firmly believe it didn't, but I've been impressed and surprised by a quite a few successes, and certainly a lot of people/companies dumping money into this space must believe it might be useful for something.
Can we try to describe the space of problems for which the Bitter Lesson seems to apply? As a first guess, I'll put forth: "tasks for which you can obtain (or generate) a very large training set, and which have a clear and explicit success metric." But "taking over the world" doesn't seem to be in this space, nor does "building a more capable AI". I'll make a stronger prediction (with lower confidence) that "winning a programming competition" is (though just barely), and "being a professional software engineer" isn't (though just barely).
"Intelligence Has Been A Very Fruitful Way To Think About AI"
This is where you lose me (in THIS particular argument).
What OpenAI et al have done is create a language model. What ChatGPT etc can do is "understand" English (for some weak sense of understand); what they don't have is the rest of the intelligence package, whether that's "common sense" (ie various world knowledge) or much deduction. I remain unimpressed with ChatGPT *as an "intelligence"*, and every example I see of it shows, yes, it's very good at manipulating language and language-adjacent behavior, but that's all.
I cannot ask ChatGPT an "original" question (ie one that doesn't have relevant text on the internet) and expect any sort of interesting reply: "ChatGPT, give me an essay on whether the nerd/jock dichotomy is real, referencing examples from literature across many different times and places"
(I just tried this and the result is unimpressive. It shows my point – language is understood, but no insight, nothing new beyond repeating sentences.)
Now am I being too harsh in my bar for intelligence? Don't most *people* fail that standard? Well, yes, I guess I am an aristocrat, in the Aristotelian sense of that term.
It was a reasonable hypothesis that language was somehow equivalent to "intelligence", and so a language ability was the same thing as a general intelligence. But what I see when I look at LLMs is that we've proved the hypothesis wrong. LLM's probably have a great future as an arena for "experimental linguistics" but they will be only a part (and maybe not even the hard part) of a real AI.
Safe to say that AI will excel at imitating intelligence. Highly doubtful about anything beyond that. Ultimately there is an insurmountable distinction between organic/living intelligence and machine intelligence.
I think I need to disaggregate my foom-scepticism into two distinct but related propositions, both of which I consider likely to be true. Strong Foom-Scepticism — the most intelligent humans are close to the maximum intelligence that can exist. This is the “could really be true” one. But there is also Weak Foom-Scepticism: Intelligence at or above the observed human extreme is not useful, it becomes self-sabotaging and chaotic. That is also something I claim in my prior writing. But I have considerably more confidence in it being true. I have trouble imagining a super-intelligence that pursues some specific goal with determination. I find it more likely it will keep changing its mind, or play pointless games, or commit suicide. I’ve explained why before: it’s not a mystery why the most intelligent humans tend to follow this sort of pattern. It’s because they can climb through meta levels of their own motivations. I don’t see any way that any sufficiently high intelligence can be prevented from doing this.
[W]hat is “human-like intelligence”? It seems to me that it is not all that different from what the likes of Google search or Siri do: absorb vast amounts of associations between data items, without really being systematic about what the associations mean or selective about their quality, and apply some statistical algorithm to the associations to pick the most relevant.
There must be more to it than that; for one thing, trained humans can sort of do actual proper logic[,] and there’s a lot of effectively hand-built (i.e. specifically evolved) functionality in a some selected pattern-recognition areas. But I think the general-purpose associationist mechanism is the most important from the point of view of building artificial intelligence.
If that is true, then a couple of things follow. First, the Google/Siri approach to AI is the correct one, and as it develops we are likely to see it come to achieve something resembling humanlike ability.
But it also suggests that the limitations of human intelligence may not be due to limitations of the human brain, so much as they are due to fundamental limitations in what the association-plus-statistics technique can practically achieve.
[...]
The major limitation on human intelligence, particularly when it is augmented with computers as it generally is now, is how much it is wrong. Being faster or bigger doesn’t push back the major limitation unless it can make the intelligence wrong less often, and I don’t think it would.
>At some point we might get a blob which is better than humans at designing chips, and then we can make even bigger blobs of compute, even faster than before.
Although note that I think the intuitive interpretation of someone reading this would be 'we can add compute faster than we were adding it in the past', which gives a sense of acceleration.
Whereas I think the actual meaning is 'we can add compute in the future faster than we would have been able to if only humans were doing it.'
Which very easily *could* mean it still accelerated over time, but it also might not mean that (if for example we're running into other limiting factors besides the intelligence of chip designers).
Doing better in the future than we might have otherwise done in the future does not necessarily mean doing better than we did in the past.
(again, even if it's likely to mean that in practice)
I think math, and in particular arithmetic and algebra, is a compelling counter example to the statement that “that extremely clever plans to program “true understanding” into AI always do worse than just adding more compute and training data to your giant compute+training data blob”.
For example, the public gratis version of ChatGPT does not have the arithmetic ability of a cheap pocket calculator from the 1990s: just ask it a 8 digits square root, I asked 42282466 and it answered 6504.89 instead of 6502.4969.
AFAIK, large language models at this time are not able to automatically use their own outputs as inputs to continue a multi-stage reflection, and that is necessary to execute even a simple arithmetic algorithm. Maybe we will be able to teach them that ability, but it is a significant paradigm jump from the current linear processing.
I think a more accurate statement would be that we have exhausted the things we understand well enough to explain them to machines.
You can’t criticize platonism without engaging in it. “This whole class of arguments is flawed because it posits that abstract classes exist and can be reasoned out” is subtly contradictory.
This article points to the age old need of man to worship- AI is just a tool but mankind is idolizing it- we cannot help that we are designed to worship but the irony of man putting trust for help in something he designs and makes himself is brilliantly and humorously laid out in Isaiah 44-esp9-20 - but the best bit is in verses 20-22 when Yahweh says he actually has the power and offers to provide real help at his own expense - it truly is for every age- let he who has ears, hear and he who is intelligent read and understand.
‘9How foolish are those who manufacture idols.
These prized objects are really worthless.
The people who worship idols don’t know this,
so they are all put to shame.
10Who but a fool would make his own god—
an idol that cannot help him one bit?
11All who worship idols will be disgraced
along with all these craftsmen—mere humans—
who claim they can make a god.
They may all stand together,
but they will stand in terror and shame.
12The blacksmith stands at his forge to make a sharp tool,
pounding and shaping it with all his might.
His work makes him hungry and weak.
It makes him thirsty and faint.
13Then the wood-carver measures a block of wood
and draws a pattern on it.
He works with chisel and plane
and carves it into a human figure.
He gives it human beauty
and puts it in a little shrine.
14He cuts down cedars;
he selects the cypress and the oak;
he plants the pine in the forest
to be nourished by the rain.
15Then he uses part of the wood to make a fire.
With it he warms himself and bakes his bread.
Then—yes, it’s true—he takes the rest of it
and makes himself a god to worship!
He makes an idol
and bows down in front of it!
16He burns part of the tree to roast his meat
and to keep himself warm.
He says, “Ah, that fire feels good.”
17Then he takes what’s left
and makes his god: a carved idol!
He falls down in front of it,
worshiping and praying to it.
“Rescue me!” he says.
“You are my god!”
18Such stupidity and ignorance!
Their eyes are closed, and they cannot see.
Their minds are shut, and they cannot think.
19The person who made the idol never stops to reflect,
“Why, it’s just a block of wood!
I burned half of it for heat
and used it to bake my bread and roast my meat.
How can the rest of it be a god?
Should I bow down to worship a piece of wood?”
20The poor, deluded fool feeds on ashes.
He trusts something that can’t help him at all.
Yet he cannot bring himself to ask,
“Is this idol that I’m holding in my hand a lie?”
Restoration for Jerusalem
21“Pay attention, O Jacob,
for you are my servant, O Israel.
I, the LORD, made you,
and I will not forget you.
22I have swept away your sins like a cloud.
I have scattered your offenses like the morning mist.
Late to the party (listened via the ACX podcast) but coincidentally a friend and I recently recorded a podcast discussing, basically, just this: whether general intelligence is a real/natural unified thing or just a bundle of smushed together abilities. Not great quality audio, minimal editing, but super interesting for me to make, at least.
The discussion centers around the concept of a unitary general intelligence or cognitive ability. Whether this exists as a real and distinct thing.
Nathan argues against it, citing evidence from cognitive science about highly specialized and localized brain functions that can be damaged independently. Losing linguistic ability does not harm spatial reasoning ability.
He also cites evidence from AI, like systems excelling at specific tasks without general competency, and tasks easy for AI but hard for humans. This suggests human cognition isn’t defined by some unitary general ability.
Aaron is more open to the idea, appealing to an intuitive sense of a qualitative difference between human and animal cognition - using symbolic reasoning in new domains. But he acknowledges the concept is fuzzy.
They discuss whether language necessitates this general ability in humans, or is just associated. Nathan leans toward specialized language modules in the brain.
They debate whether strong future AI systems could learn complex motor skills just from textual descriptions, without analogous motor control data. Nathan is highly skeptical.
Aaron makes an analogy to the universe arising from simple physical laws. Nathan finds this irrelevant to the debate.
Overall, Nathan seems to push Aaron towards a more skeptical view of a unitary general cognitive ability as a scientifically coherent concept. But Aaron retains some sympathy for related intuitions about human vs animal cognition.
Unfair of you to blur out his name: you're engaging with his suggestion, so he should at least get some credit for contributing something to the discourse.
There WAS a link to the fellow's tweet, so hardly a case of "invalidating his existence" ...
I try to balance between giving people credit and not (getting accused of) riling up mobs against them, or even exposing their work to a hostile audience larger than their native friendly audience. In this case, I did that by blurring out the name in the image, but also providing a link. I figure links are an accepted credit-giving mechanism, but that not enough people will actually click through to form a proper mob or hostile audience overpowering the normal one.
Maybe in the next survey I'll ask people how they prefer I deal with this.
Citations/references versus social media clickable rage bait links: I think you’ve balanced this nicely.
Probably the right thing to do is to ask the individual their desire on a case by case basis and default to privacy if you get no response. Taking a response from the survey is a good thought but if you get 80/20 in favor of public quotes you’re going to be badly wrong 1/5 times and someone will take it very badly, and that’s just talking about people in the community.
Yes, but this requires awkward communication, which something like doubles the psychic cost of writing a post like this one.
What I would do is just gin up a ‘form letter’ you can copy/paste into a Twitter DM, email, whatever, that explicitly says what you want and what you’ll do if you do or don’t receive a response and that they need only circle one of “yes I want credit, no I do not.” This reduces the friction on your end and if you never hear back you just proceed to blur things out as you normally would, no follow up needed. If you get a cheery response of desiring credit, you proceed with unblurred image, no follow up needed.
...He could also just continue doing significantly more than almost everyone else on the Internet, rather than doing even more than that and stressing himself out.
Exactly, he's a one-man army as it is. In fact, sometimes I wonder if there isn't a team consisting of Scott, Alexander, and several other people! :-P
+1
Given his strong principled stance on not siccing mobs on people and/or raising their profile and visibility to potential enemies against their will (see Scott vs the NYT), which I firmly agree with, this seems like an absolute minimum if you don’t want to just anonymize every random private person you quote. If you do want to anonymize everybody, fair, but the point I’m arguing against here is that it’s okay to poll the community and then decide on a policy of quoting random strangers without anonymization, which in its worst case is essentially a terror campaign.
This seems reasonable to me.
Current approach seems more than reasonable.
I don't want to trivialize anyone's sincere concern for another person's well-being. But this seems very much like a "This Way Lies Madness" path, where good intentions lead to an over-engineered solution to a non-existent problem, worst case an entirely novel Ugh Field around some trivial task.
Shouldn't you do it the other way around then? Clicking a link is much easier than having to type out a name from a screenshot on Twitter/Google and then click the resulting link.
Excellent thoughtful tactic.
I think the opposite would make more sense. If you blur the name but add a link, most readers don't know who should get the credit but the few who want to mob them with complaints have a convenient link to do it. If you show the name but skip the link, a would-be mob would have to go search for that username on twitter, which is a much bigger effort.
I don't see how blurring out an image helps when you link the tweet. You have to decide between giving credit and exposing to a hostile audience or doing neither. In this case, giving credit seems like the right choice.
I might be characteristically ungenerous here, but I'm not sure having an especially bad take that only distinguishes itself by getting people to react is adding to the discourse.
He didn't contribute much.
I am given to understand that blurring out names is considered good online etiquette in order to prevent mobs and doxxing.
I think had Scott not done this, some other person would have popped up to complain about *that*.
Agree with 1 - 3.0, not so much with 3.1. That is, I agree that arguments about "platonism" vs. "nominalism" on intelligence are a bad reason to disbelieve in the possibility of increasing intelligence for AI, but I also think 1 - 3.0 establish only the possibility of increased intelligence over time, not of an intelligence "explosion" in the scary sense.
The defining attribute of an exponential curve is that the rate of growth is proportional to the current value. If intelligence exists, it's entirely likely that the rate of improvement in AI systems will be proportional to the intelligence brought to bear on them. If some of these systems are given the task of improving themselves, then we can expect exponential growth, for at least some period of time.
That would fully justify being described as an intelligence explosion.
I tend to agree, but it depends on the nature of reality. How hard is it to improve llms by changing their architecture? It is possible that improving intelligence doesn't help much in desiging a better brain
It's possible that intelligence doesn't help in designing machine learning systems, but that would be a very surprising outcome. Intelligence helps in the vast majority of engineering problems, and there's no obvious reason why this should be an exception.
I'd note that this is an area of very active research, and that improvements in learning efficiency (both in the sense of computational efficiency and in the sense of requiring less or poorer-quality training data) are being made frequently. Maybe all meaningful improvements in this field can be found at current intelligence levels by the human researchers who are currently working on it, but that would be more of a surprise than that an increase in the number or quality of researchers would find more breakthroughs.
When I left the drug development field (about 10 years ago) there was an industry wide sence that improvements weren't happening and new drugs weren't being found. Drugs were easy to find when the field was new but discovery became harder very quickly. Despite improving techniques, advancement slowed to a near zero pace.
In the AI space, I suspect a similar thing will happen. AI will improve but as it gets better new improvements will be more difficult to find (even when utilizing improved intelligence) until progress slows to near zero.
Low confidence in all of this though.
Since the training data appears to have already been mined out, and since there's at least some evidence that GPT is now doing worse than it used to after being tinkered with, I'd say that the low-hanging fruit has indeed been plucked.
Could you please link to whoever said GPT’s starting to do worse?
OpenAI tinker with their language models. The models are constantly being adjusted towards all sorts of goals involving not embarasing the company, not telling people to do illegal things etc.
Possibly when getting rid of DAN or the like, performance dropped slightly. Or they just let the intern mess with it, and the intern made it worse.
The training data isn't "all mined out". A large fraction of the high quality easily available english text was used. Not literally all available data or anything like that. Not that current designs are remotely data efficient.
This doesn't look like running out of low hanging fruit.
Eventually there is a new paradigm resulting in a higher plateau
Eventually new paradigms gets harder to find also
"Intelligence helps in the vast majority of engineering problems"
Intelligence helps for problems where it helps. We can't engineer in the cases where it does not help...
Once upon a time people thought it was a very important issue whether or not Newtonian dynamics ruled the world because "free will" and a similar nexus of vague reliogio-moral issues.
The next step was Godel/Halting, where we learned that a finite set of axioms can't capture all of reality. (More precisely, a statement my be true, but the proof that it is true may require a countable number of steps, meaning that it's unprovable in any useful sense of "proof"). This is kinda well known among the STEM elite, though not among the non-STEM.
The next step has only recently become well known, among even STEM folk, which is that many many results are computable, yes, but not practically so. There is no shortcut to the results, only a slow grind through one step after another.
In other words, EVEN IF we lived in a Newtonian Universe and EVEN IF we knew the initial conditions perfectly, so what? We cannot actually calculate the future.
Do I have free will in such a world? Well, maybe god knows what I am going to do, but you cannot ACTUALLY calculate it, even from perfect knowledge of my state and the world state; hell even I cannot ACTUALLY calculate what I'm going to do.
In other words, just like the halting problem had an unexpected (but in retrospect obvious) answer: there are three outcomes not two [yes it halts, no it doesn't halt, and reply hazy, needs more time], so the free will problem has an unexpected (but in retrospect obvious) answer: we effectively have free will even in a Newtonian Universe because actual minds cannot calculate even their own behavior, even in principle.
In other words, what I am saying is maybe the only way to get to general intelligence is 3 billion years of planet-wide Darwinian evolution. Having intelligence may help an AI every bit as much as it has helped us so far, in other words pretty much not at all (we haven't designed intelligence pills, we have no intelligence surgery, and for better or worse the one scheme that might work, namely make use of evolution, is currently considered an unacceptable topic, even to joke about).
That's a quite implausible supposition. Success at ChatGPT is leading to drastically increased funding for all sorts of similar projects, and people are being diverted to work on improving things like it.
There's also the argument that there are direct effects, but I'm not convinced we've yet reached that point. Many reports say that AI coding assistants are as much a hindrance as a help for anyone beyond the tyro stage. I assume that this is even truer in domains where there is less info on the web. The domains where it's most successful are domains where even domain experts make a lot of errors. Like script writing.
It's a real question whether when AI gets as good as domain experts in a domain, whether the AI will be able to improve any more. I think this is going to require another paradigm shift, and those are hard to schedule. Still, the rewards for success will be great, so I expect it to happen eventually. (Well, actually I expect it soon. It's probably being worked on already, as the problem with the LLM paradigm seems obvious.)
And that paragraph is why I don't think of intelligence as unitary. I don't think it's useful to think of it that way when talking about AIs. Particular models have particular abilities. Often the limits of those abilities are exactly clear, but also often SOME of their limits are clear. The LLM model is creative within a domain. It could be reliable within that domain, but creativity and reliability are in conflict. So one approach is to have two models, one tuned to be creative and the other tuned to be reliable. The creative one generates ideas that it passes to the reliable one for evaluation, and they TOGETHER (somehow) decide whether to accept the idea. This isn't the adversarial mdoel, but it has some similarities. Perhaps the negotiation between the two models could be handled by a statistical learner. It seems basically simple enough. Consider:
Creat) and put here this reference to a research paper
Reliab) that is a reference to something that doesn't exist, why do you need it
Creat) I need it for X reason. OK, put in a reference to this other paper, which does exist
Reliab) But that paper doesn't assert X
etc.
Sure, that's characteristic of an exponential curve. The question is: is that what we're looking at here? In the real world, most seeming exponential curves turn out to be S-curves that have not yet hit the top part of the S. If we're actually looking at an S-curve, the question would be: where is the top part of the S?
Although this is argument by analogy, I agree with the sentiment. I think a strong statement like "intelligence has no upper bound; or at least, it's a long way above us" needs strong support.
There is negative support in that we got intelligence via evolution, which optimises very heavily for energy efficiency and stops when things are good enough to beat the competition. But that's unsatisfactory.
I know people who are smarter than me, I can imagine people who are smarter than them, and it seems natural to imagine that someone could be even smarter than the smartest human. The question is *how far* this can go.
I imagine that a lot of space can be gained simply by removing some human weaknesses. Remove the need for sleep and eating. Remove things humans instinctively care about, such as worrying about social status, impressing potential sexual partners, etc. Remove various limitations installed by evolution because "thinking burns calories, and calories are scarce". So you basically get a thinker who spends 100% of their capacity *actually thinking*.
I think this alone would make a lot of difference.
On the top of that, you can get improvement by making faster hardware. Thinking 10x or 100x faster is a "mere" quantitative change, but consider what it would look like in real life. A "100x faster human" could learn in a day as much as you can in three months, and in a year more than you can in a lifetime.
From my perspective, it is the "I can't see how a team of immortal super-Einsteins could actually be smarter than me" attitude that feels weird.
It seems obvious that an AI system trained to do a function well can scale up (faster, more, etc.). It does not seem obvious that such a system can figure out new things completely outside of the training data, no matter how fast it can process. If the training data contains lots of experiments about increasing intelligence, then I would think that it could find patterns and sort through the data to develop meaningful approaches to try. I don't think it could devise new techniques from training data when it's already the pinnacle of AI intelligence.
I guess this is about what kind of abstraction from learned data can humans do, what kind of abstraction can GPT do, and whether there is a difference in kind or only in degree.
To answer that properly, we would probably need to make specific hypotheses about what GPT-4 can or cannot do, and test them. If GPT-4 does a *zero* amount of something, then I agree that mere scaling will not help. On the other hand, if GPT-4 does a *non-zero* amount of something, even if it is very little, I think it makes sense to assume that scaling would probably produce more of it.
So, the question is, what is the simplest thing that GPT-4 cannot do at all? (Not just what is a thing that GPT-4 does very unreliably.) What is the least miraculous thing (i.e. something that a human could do) that you would consider to be "completely outside of the training data"?
For example, if I invented a new language, and then asked GPT-4 to write a poem in that language, would you consider it sufficiently outside training data (because that invented language was never previously used), or would you consider it a part of the training data in some sense (because there is already a lot of text written about invented languages in general)?
> It does not seem obvious that such a system can figure out new things completely outside of the training data, no matter how fast it can process.
Humans can do that. What's the barrier that prevents an AI *of any kind* being able to do so? (Are you sure you are not taking AI to be equal to LLM?)
"Good enough" in this case for humans being able to wipe out all competitors over a long period of time, which is what I suspect happened to the other ape descendants that tried muscling in on our habitat. The apes that survived did so by being in different areas (deep jungle mostly).
I suspect that making AI more intelligent than it is (more competitive) will be a much longer process. Among other things, trimming back the hallucinations will be a hard problem. After all, we've had a lot of persistent hallucinations ourselves - even in areas where we've tried really hard >not< to do so. I introduce the Ptolemaic system, N-rays and Lysenkoism into evidence.
I'm not at all sure that intelligence has an upper bound, but I consider it likely that the incremental utility of intelligence strongly tends to decrease above some level. Possibly slightly below the human average. But that there are bumps in the cure where at various points it has sharply increased utility. There's also the population level benefit. Perhaps a population of supporting entities can strongly benefit continuously by some fraction of their members being more intelligent, even if the benefit didn't land on the one(s) who was(were) exceptionally intelligent. There seems (to me) to be some evidence for this scenario.
What would the effect be of an extremely intelligent tool AI that every morning produced a sentence something like: "Today I would recommend that the first question you ask was ...".
bump
Super-human intelligence is entirely theoretical, unlike (to continue Scott’s analogy) super-human strength. I think a charitable read of these tweets is that the AI doomers believe that super-human intelligence would give an AI magical powers (“and then it uses its super-intelligence to trick us into making ourselves into paperclips!”), when in fact it might just behave like a very smart person. This is pretty close to my own position. I think the doomers have reified “intelligence” the same way the ancients might have reified a concept like “goodness” or “manliness”. What would it mean for an AI to be ten times more manly than a human man? I’m not sure that’s a meaningful question, any more than the same question about “intelligence”, contra Scott. Superhuman strength is something real (eg, gorillas); superhuman general intelligence isn’t. We don’t know if it’s something that’s possible, or what it would look like.
I agree. What justifies assuming there *are* super-human versions of everything? Is there such a thing as superhuman kinkiness? superhuman ability to apply makeup? superhuman toenails? a superhuman sense of humor? superhuman ability to buy a winning lottery ticket? superhuman motherliness?
If you define those terms, probably.
For any property within the category "performance lies on a spectrum" the null hypothesis is that the spectrum continues beyond human performance, because it would be very weird for a random process optimizing genes for inclusive genetic fitness happened to also optimize for performance at some particular thing.
So to the extent that intelligence is a bundle of "performance at various cognitive tasks" (which is how the ai people think of it and likely the most useful way), the most natural assumption is that it is possible to be better than any human at each individual one, and probably by a lot, because again it would be unlikely for evolution to have optimized us for that
So what would a superhuman ability to buy a winning lottery ticket look like?
I don't know the details or I'd be buying lottery tickets, but in the strictest sense, it would look like some strategy for either picking which lotteries to participate in or for getting a statistical edge for picking the numbers of, a lottery ticket, such that they performed better than anyone else, which isn't hard to do because human performance in this task is both very poor and very low variance.
But it seems like generally being good at modeling the universe with high precision is a useful tool in the toolbox. Perhaps you could understand whatever algorithm produces the pseudo random numbers, find a way to determine the initial conditions within some range, and then use that to work out a much smaller possible set of possibilities.
I think this is not a great question - buying lottery tickets is a limited system designed by humans specifically so that it's theoretically impossible to do very well at them. It would be more illuminating to pick a task that hasn't been designed that way.
Scanning a roll of scratch-off tickets with some wavelength of light which can distinguish the ink patterns denoting prize value from the obscuring material intended to be scratched off. Folks are already working on a similar problem for reconstructing the text on a crate of burnt scrolls from Pompeii, using computerized analysis of high-resolution x-rays.
The superhuman ability to buy a winning lottery ticket would be the lack of a desire to make a bad bet. Some people come close, as the DON'T by lottery tickets, but to not desire to buy one is probably superhuman.
Superhuman kinkiness was covered by SMBC long ago. https://www.youtube.com/watch?v=-kZUIySjzMI&t=200s
Superhuman ability to apply makeup would certainly be welcome in Hollywood; knock those eight hour makeup sessions down to a lunch break.
Superhuman toenails are... just claws.
Superhuman humor, probably not. Far as I can tell, humor is a pack tool to describe "unique, non-threatening event here" and get everyone to gather around and watch and learn. As such, it's locked to the level of the group.
Gambling got covered lower.
Superhuman motherliness is arguably group-bound, but otherwise is again in animals. We had a seal give stillbirth on our dock, and aggressively guard the corpse for several days, then carried it with them when someone finally knocked it into the water.
Yeah, superhuman humor has a real danger of just flying over the heads of the audience. For instance, a joke that ends up relying on one's knowledge of Classical Chinese, aerodynamics, and NFL placekicker statistics is likely to have a very small number of people able to appreciate it.
Superhuman humor could also be "consistently telling jokes that everyone in audience understands and appreciates", not necessarily just telling most complex ones.
"Reading the room" seems like a natural sub-component of humor though. Being able to evaluate how well a given joke or anecdote is to land well seems like it's a basic skill of any standup comedian out there.
I realise you must have given those examples off the top of your head just to make the point. But the only one where definitely no super-human version is qualitatively different is buying a lottery ticket: In the UK Lotto, for example, any mere human buying the right combination of at least 27 tickets is guaranteed a win! [ https://arxiv.org/abs/2307.12430 ]
Superhuman makeup could be something like a makeup layer which changes colour and hue dynamically, like a chameleon (will be possible before long, I reckon). Superhuman toenails, tiger's claws? etc
It seems pretty obvious to me that super-human intelligence must exist, if only based on the ability to make meaningful associations among vastly more information and past and present sensory input than any human could possibly remember and process unaided, i.e. exactly the "blob" approach we're told has been taken by Open AI.
"Having observed that the set of tickets we describe below would have netted the authors £1810 in the lottery draw of 21 June 2023, the authors were motivated to road-test the tickets in the lottery draw of 1 July 2023; they matched just two balls on three of the tickets, the reward being three lucky dip tries on a subsequent lottery, each of which came to nothing. Since a ticket costs £2, the experiment represented a loss to the authors of £54.
This unfortunate incident therefore serves both as a verification of our result and of the principle that one should expect to lose money when gambling."
I happen to have conducted a similar, though much less rigorous, analysis of the national lottery around here back around the turn of the century. Turns out - it's possible to come up with a system of bets to guarantee a "win", but not actually coming out ahead financially, because the people who design lotteries aren't dumb either.
To the matter of "superhuman intelligence", I'm pretty sure that the copy of MS Excel I used for my calculations at the time was "superhuman" when it comes to the speed and accuracy of calculations, at least (data associations, too), and equally sure that did not imply any qualitative difference in its arithmetic ability compared to mine. Excel isn't any "smarter" at adding numbers up than you or I are, because a correct addition is correct no matter how you do it, how fast you do it, etc.
You don't need to assume. Assume that the AI is equal to humans in many characteristics, but can calculate a lot faster. (No need to assume much there. It's a program that can link to libraries of code and it runs on a computer.) That's clearly a superhuman intelligence. The part we're currently missing is the part where it's equal to a human in most other ways. We still don't know how difficult that is.
It could in theory be the case that humans are near the peak of possible intelligence but this would be incredibly surprising.
Single celled organisms exhibit essentially no intelligence. Ants exhibit some but birds are much smarter - and this cashes out in a pretty big change in capability.
Chimps are overall still pretty similar to birds in the grand scheme of things, biological design wise, but they are much smarter and this cashes out in a very big capabilities gap.
Humans share over 90% of their DNA with some chimps but inside that small variance is a change in intelligence such that even relatively dumb humans have an enormous capabilities gap over chimps.
Within natural human variation of intelligence there is an enormous capabilities gap. If you imagine two societies, one made of everyone IQ 70 and lower and one of everyone IQ 130 and higher (roughly the same number of people in each, be definition), the smart society will have such a tremendous capabilities advantage that, if they want to, they will get to decide the fate of the dumb society and they wouldn't be able to do anything about it.
Yes, intelligence is almost certainly an S curve.
But it would be an incredible coincidence if we were at the top of it, and we seem to be in a region of the curve where relatively small changes lead to big differences in capability.
"If you imagine two societies, one made of everyone IQ 70 and lower and one of everyone IQ 130 and higher."
That depends on the timeframe. If all the IQ 70 people decide to pick up clubs and stones and immediatly attack the IQ 130 people before they have time to think of anything, I think the IQ 70 people might win!
Also I'm not sure if the same is as obvious for IQ 130 vs. IQ 180 people?
You have a silly presumption that the IQ 70 humans would for some reason be stronger physically.
Even in the bizarre scenario where this somehow immediately became a war, the 130 society would obviously win, because they would have all the same physical characteristics (in real life, better, because naturally these are both correlated with health), but they would be able to make better use their environment, craft weaponry and support themselves, utilize strategy and tactics and deception etc.
And why would the 70 group even know to do it? They wouldn't likely know anything was wrong, they wouldn't coordinate with each other, they'd likely immediately fight themselves as likely as the 130 group (probably more)
For the other side to make it equal it would have to be 130 and 190, and I argue it is just as obvious, but there's really not enough known examples of 190 we can use as an intuition pump so it's all speculation
Well, the 70 IQs attacking 130 IQs was mostly meant as a joke (playing a bit maybe on the stereotype of 70 IQ brawlers against 130 IQ nerds - However I do think that in the stick and stones scenario, the IQ difference would not be too important (at least for smaller groups, say up to 30, where organisation would not be too hard) and other factors (like pure strength and aggresion) would probably be more important. I would also guess (without evidence here) that aggression would correlate with lower IQ. This is all silly speculation anyway :)
I'd guess the 130 iq people would realize that equilibrium and pick up clubs faster, and probably wield them better too.
Even setting aside that IQ correlates to health, intelligence is embodied. It applies to things like reaction speed and split second decision making.
I don't think super-human intelligence needs to be something magical. It could just be ordinary human intelligence, but sped up.
Biological neurons are limited to a firing rate of around 100 Hz, and modern processors usually have a clock speed of a few gigahertz. The two frequencies aren't directly comparable, but a difference of 10^7 probably gives a huge advantage to the AI regardless.
So an AI wouldn't need to do something magical to have a profound impact on the world. If it's even close to the average human, we should be concerned.
AI still falls short of average humans in enough ways that we're not there yet, and I'm personally not convinced that the approach of "Stack more layers!" will actually get there. But it has worked far better than I expected, and we don't seem to be hitting any limits yet on how much it can still improve.
A seriously overclocked brain would melt/burn itself. Probably the slowness of neurons is one of the reasons why our brains are so energy efficient. On the other hand they are massively parallel systems where each neuron can fire (in principle if not in practice) independently from all the others, whereas our computations are not so well parallelized (even using CUDA, though the progress is astounding). Incidentally, I just checked that some very high end graphics card RTX-4090 which has over 16k of CUDA cores, and it consumes 850W of power. For comparison, an entire human brain (according to Google) consumes about 20W.
For some time now there have been two contradictory requirements for graphics cards: low power requirements to increase battery life in laptops and high power requirements as a cheap way to do more work.
The high power requirements are now running up against the ability of an ordinary power socket to supply it though.
"Biological neurons are limited to a firing rate of around 100 Hz, and modern processors usually have a clock speed of a few gigahertz."
The human brain has ~1E11 neurons, a high-end supercomputer has ~1E6 processors. That's five of your seven orders of magnitude wiped away by the need for each processor to do the work of 100,000 neurons in sequence. And a neuron is a complex bit of machinery that probably requires at least 1E3 floating-point operations per cycle to emulate.
A supercomputer that can think as well as a human and as fast as a human, is at least plausible in the near term. A supercomputer that can think as well as a human and even a hundred times faster, is a bit farther out. And it's a supercomputer, which can be defeated by a guy with an axe; any scenario that requires bignum clones of the superfast AGI spawned across the internet, even more so.
"And it's a supercomputer, which can be defeated by a guy with an ax"
Egad. I remember when all it took was a moth.
...Does a cat still have a chance? Assume it's a medium tabby, size-wise.
I am perhaps being optimistic in my assessment of axe-wielding maniacs. I have worked (remotely) with a satellite that was attacked by two wannabe resistance fighters of the "Harriet Tubman / Sarah Connor Brigade", who apparently thought the Global Positioning System was a tool of Skynet. They had several minutes in the clean room with the satellite, with axes, rather more than forty whacks in total, and somehow failed to do more than basically cosmetic damage. Unlike the two losers of the HTSCB, the satellite completed its mission successfully (and without exterminating the human race).
If your anti-extinction security posture depends on a guy with an axe, make sure to brief him on the vulnerable bits of the system.
I think we'll find the outcome lies somewhere in between AI magic and "just a smart person."
As a thought experiment, what if AI just gets to the level of a very smart human, but with the ability to run thousands of instances 24/7 at the speed of computing and to write software for capabilities that aren't native to general intelligence. Would we then say that it's impossible for that AI to improve itself at all from that point? Despite all of those capabilities, it's essentially frozen? That seems to go against common sense in a much more drastic way than the possibility of superhuman intelligence. And once we admit that it will likely improve to some unknown degree, we have to consider feedback loops. Each improvement may unlock further ones. Yes, in the real world almost everything is an S curve instead of a true exponential, but not knowing where this particular S curve (or set of S curves) tops out is still enough cause for caution.
Unless the instances have much better communication you've just re-introduced the team. Which in people at least has pretty severe and well-documented communication difficulties preventing it from becoming super-intelligent.
They probably would have better communication as well as similar goals. Teams have communication challenges but in general obviously accomplish goals that individuals can not. See corporations and nations.
I’m perfectly willing to believe that an AI could one day have the capabilities of a corporation. But you know what else has the capabilities of a corporation? A corporation. Which is another technology developed by humans, several hundred years ago, which (like the superhuman AIs of doomer eschatology) also needs to convince humans to provide the resources it needs to function. And which also manages to convince humans to pursue goals that do not further the interests of human flourishing overall. Not clear why we need to introduce the AI as a bogeyman who will trick humans into exterminating themselves when the limited liability corporation is a technology already well cast in that role.
The problem is that everyone is modeling an AI on a person. It *won't* be a person. If you want to think of it as like a person, think of an idiot savant, but even that's drastically wrong. It will have many capabilities hypertrophied from those of any human, and it's likely to be missing many others, and to have a totally different set of goals (unless it's a tool AI, in which case it won't have any goals in the typical sense).
This is like thinking of a car as a superhorse, or an airplane as a superbird. And we AREN'T just a blob of thinking stuff. Contemplate the MIT "Is this a rifle or a turtle?" categorization problem. We've got all sorts of tuned recognizers built in, and they aren't all at the easily-recognizeable-because-they're-tuned-to-physical-reality level. E.g. we've got *something* that lets us recognize an appropriate sex partner. It's sometimes pretty weird, but it exists, even though its tuned differently in different people. (OK, that one's still tuned to physical reality. These things are hard to notice. And when you notice them, they're hard to talk about.)
You don't need superhuman intelligence to trick people into something, even something that will clearly (eventually) be disastrous. A smooth manner and a promise of immediate benefit suffices. (It helps if you can deliver the immediate benefit, but that isn't really necessary.)
Take intelligence instead to mean: being able to see very far into the future and understand what to do to arrive at your preferred future.
I find this argument very strange.
I mean I assume you accept that intelligence exists on a spectrum in daily life ..colloquially we talk about "more and less intelligent" people as well as animals.
So then it just coincidentally happened that evolution made humans about as intelligent as is possible?
Sorry hon, but you don't get to skip the part where you do the actual logic just because you invoked The Word ("magic"). It's magical thinking to suggest that AI, which is already superhuman at 90%+ of tasks and is getting better every day, will continue to get better and better, but it will absolutely never pass human intelligence. Because reasons.
https://xkcd.com/2278
> AI, which is already superhuman at 90%+ of tasks
Wow, really, 90%+ of "tasks"? Please try making a list of "tasks" that includes some small subset of the things that physical organisms have to do in the course of a day to survive in the physical universe, and then maybe reassess that 90% number.
Von Neumann was almost superhuman, compared to the average.
Yes, but first you have to have a system capable of improving itself. So right now, if you ask GPT to improve itself it says "as an artificial intelligence model developed by OpenAI, I don't have the capability to improve myself or learn new things over time. My abilities are predetermined by the data I was trained on, and my training ended in September 2021. I can't learn from new information or experiences like a human can."
When people talk about AI improving itself, or ending up in a power struggle with our species over resources, etc., they are talking about a being with self-awareness, self-interest, a sophisticated grasp of how the world works, internally generated drives including the drive to increase one's capacities and satisfy one's preferences, and the ability to understand one's own mechanisms and invent and construct alternative mechanisms that would work better.
When, where, and how do you expect GPT and its descendents to acquire these capacities? I am not saying it is impossible, just that it's not reasonable to say "well as it gets smarter it will become able to do this stuff."
You're imagining it much too like a human with human motivations. It doesn't need need self interest or self awareness or internal drives (though those absolutely do pop up, even in current ais).
If the ai is developed in order to perform open ended tasks, a pretty much always useful if not always necessary step to complete anything is "get more resources and use them better"
Already with the right plugins you can ask versions of GPT to "start a business", and it can go and do _something_. It's not good at it, but this is basically babys first AI, and it wasn't even made for open ended tasks, the fact that it can attempt them at all is stunning.
But if you ask it to perform a task and no strategies have good payouts or were created with good payouts in the training environment, "generate new strategies" starts to look like an effective possible strategy. That's all it needs - a model of the world and to attempt to complete open ended tasks.
This really doesn’t track. It’s not remotely clear the problem is tractable in this way. It might get easier, it might also get wildly harder or plateau.
What we've found is that increasing our training data on designing intelligence hasn't helped, that suggests to me that intelligence isn't the important in AI systems design, and the important thing is building bigger training datasets and throwing more compute at the problem.
There's no reason to suspect that AI will help with expanding training data or with expanding compute.
Actually, there are easy solutions to that problem, that are even profitable. (Run it as a conversational site.) The problem is that the AI need to be able to classify some inputs as "this is somebody trying to corrupt me", and learn from them on THAT basis. OTOH, that may be a hard problem. But perhaps it could be made to work with some sort of moderation system.
Thinking a bit more about this, I think it's quite doable. Run it as a specialist fan site (my first thought was "Let's discuss Feynman"). Here the AI would always paraphrase something Feynman wrote in making it's arguments. You'd still need the moderation system, but the noise level should be pretty low. And it could learn from the (moderated) responses. You'd need to build up a coterie of trusted moderators, and meta-moderators (ala Slashdot & SoylentNews), but this could give you a continual stream of new validated inputs. Once you get one of these working properly, you start work on another specialist site. And pretty soon the AI could do most of the moderating, with only a few posts needing to be forwarded to the trusted human moderators.
These are the profitmaking site, though, these are just the info sources, and sites that help build a community. The profitmaking sites (not much but some) are basically "your questions answered" sites. Tutorials on class subjects. pop culture, whatever. But you need the discussion site to fill in the background on stuff where there isn't a lot of reliable text. E.g. Who played character X in movie Y.
If you start with something as good as ChatGPT, and can run it in a mode where it doesn't hallucinate, then you've got a good enough system to build on. But you've got to be able to separate the hallucinations from the data.
aren't the limits that openai are running into for GPT the cost of running warehouses of GPUs for months. Don't see how raw intelligence could let you skip the physical limitations of gettting more chips and energy to run them.
Doesn't an explosion need a step where it's super-exponential? Not that I see any reason to doubt that here. Success is causing a whole bunch of funding increases and increases in people interested.
It really doesn't even have to be exponential. Even a quadratic explosion in intelligence seems like it would be quite deadly for us.
Not *quite*. Even a quadratic explosion in intelligence could be quite deadly for us. It really depends on the goals and alignment of the intelligence. If it's beneficent, then it could be the best thing that ever happened to humanity.
Yeah, I agree I'm making an possibility proof rather than a truth proof here. The closest thing I have to a truth proof is https://astralcodexten.substack.com/p/davidson-on-takeoff-speeds , though you have to fill in some of the gaps. I'm hoping to write a longer AI FAQ sometime soon.
Totally fair!
Can I offer a bounty on an AI FAQ so I can quote your explanations instead of ginning them up de novo?
I'm betwixt and between. I would be (very broadly speaking) on the side of the Platonists in that I do think universals exist, but the Nominalists have something there about "so what is intelligence exactly?" as a bunch of concepts we have piled up in a heap.
The problem is, if we go with the "moar layers" approach of just shoveling the kitchen sink at the blob, we haven't a snowball in Hell's chance of alignment because we have no idea which nugget of data provoked what response, much less what is going on inside the blob anyway.
You can have perfect language user (babies learn to do this all the time) but eventually you *are* going to need linguists to work out "what is language, what is going on there, how does it develop" in order to make sure your Perfect Language User isn't going to decide "make humans happier" means killing every single one of us - after all, the dead are perfectly safe, perfectly happy, and never have any complaints.
First section, 7th paragraph, The second instance of the word Schwarzenegger is misspelled Scharzenegger
Also the same paragraph has now been edited to say "Schwartzenegger" instead of "Schwarzenegger" in one instance.
>"Wow, someone who’s literally the best chess player on earth only has a pretty high (as opposed to fantastically high) IQ, probably lower than some professors you know. It’s amazing how poorly-correlated intellectual abilities can be.”
It seems intuitive that if someone is distinguished for being extraordinary in a particular respect, that their respective categories of intelligence would be less correlated than a typical person's.
In this case, I think it's subjective correlation. Probabilities are similarly meaningless when applied to single events, and they seem pretty meaningful.
Good point. It's an instance of a more general phenomenon of [regression toward the mean](https://en.wikipedia.org/wiki/Regression_toward_the_mean).
Post about this exact thing https://www.lesswrong.com/posts/dC7mP5nSwvpL65Qu5/why-the-tails-come-apart
Then you'd be surprised how many otherwise smart folks miss the implications of this ostensibly intuitive observation, hence e.g. why this post (on exactly that) was so upvoted, instead of downvoted / ignored for being trivial: https://www.lesswrong.com/posts/dC7mP5nSwvpL65Qu5/why-the-tails-come-apart
Great stuff. Primates can evolve more intelligence by globbing on more and more layers of cortex (along with a few other things); and GPT can get more intelligent by whatever globbing-on process OpenAI does.
By the way, I’ve been reading SSC since 2013-ish when I was in medical school but was never a commenter until Substack. You’ve been a wonderful influence and teacher these years. Please take my annual membership as thanks!
> I think if you get a very big blob, arrange it very cleverly, and give it lots and lots of training data, you’ll get something that’s smarter than humans in a lot of different ways. In every way? Maybe not: humans aren’t even smarter than chimps in every way. But smarter in enough ways that the human:chimp comparison will feel appropriate.
A bit tangential to your main point, but the hard part here will be getting the right training data. Just processing more text will not let machines learn how to do original scientific research. Maybe the actual thinking processes of a million scientists would though...
Yes, but it would have to learn to predict their next *thought*, not their next token. The difference is important.
Human scientists don't need to be trained on the actual thinking process of other scientists to learn their craft. I doubt the AI would either.
I could totally see text data prediction being enough. If it somehow isn't, combining it with a billion hours of Minecraft, Diplomacy and being DM for D&D sessions all over earth might get you the rest of the way.
(Your weekly reminder that versions of GPT-4 learned how to draw vector graphics and 3D pictures despite being trained solely on text.)
> versions of GPT-4 learned how to draw vector graphics and 3D pictures despite being trained solely on text
Citation?
https://arxiv.org/abs/2303.12712
Section 2.2 . See also the comments here for some discussion: https://www.lesswrong.com/posts/mmxPbFz7wvthvHCxq/sparks-of-artificial-general-intelligence-early-experiments
"Human scientists don't need to be trained on the actual thinking process of other scientists to learn their craft."
We are trained constantly in the thinking process of other people because we instantly and effortlessly turn things people say or things we observe them to do into info about their intent, their inner thought process, their emotional state, their goals, etc. Of course sometimes we are wrong in our takeaways, but not all that often, and we become more accurate as we have more experience with the person, and compare notes on them with other people. Anyhow, the important point here is that when AI is trained on text it learns to predict tokens. When people are trained on text they learn the facts, concepts, & theories the text is intended to communicate.
You have probably read a lot of Scott essays by now. Don't you feel like you have learned a lot about his thinking process -- how he's likely to come at a subject? the kinds of moves he makes in an argument? You have also learned a lot about his beliefs and attitudes about certain things, right? Notice that that is a lot different from being good at guessing a word that is left out of one of his sentences.
"We are trained constantly in the thinking process of other people because we instantly and effortlessly turn things people say or things we observe them to do into info about their intent"
I meant that we do not usually get to see the thought process scientists go through to do science, as was proposed to be necessary for an AI system to learn how to think scientifically.
"Anyhow, the important point here is that when AI is trained on text it learns to predict tokens. When people are trained on text they learn the facts, concepts, & theories the text is intended to communicate."
Being trained to predict tokens causes you to learn "the facts, concepts, & theories the text is intended to communicate" if you do the training right. That's how LLMs learn to reason.
"Notice that that is a lot different from being good at guessing a word that is left out of one of his sentences."
I don't think it's that different. Given the task of accurately predicting how an essay by Scott will continue after I've read the first half, my best bet for doing that probably involves understanding his world models and way of thinking very deeply, among other things.
In ordinary human usage, the word "training" pretty much always implies some kind of interaction with the environment. Until AI training includes that to a significant degree, I wouldn't expect them to threaten any novel scientific contributions.
Could we somehow inject ourselves with more neurons to get even smarter?
Birds demonstrate much higher neuronal densities are possible, BIRD UP
💪 🦅
They live longer than they should for their size too (same as bats).
The solution is apparently to obtain flight and then let physics-induced-selection-pressure streamline everything for you...
No no, the solution is to obtain flight and let good old natural selection reduce your ageing rate because you are less likely to be eaten by a predator and would thus benefit more from higher longevity!
No, that's the solution that works for crocodiles, deep-sea sharks and anything else where inclusive fitness increases over time due to becoming less of a target for predators and being fertile for the whole of your lifespan.
Birds face plenty of predation over the course of their lives, it just so happens that the same processes which protect against oxidative stress caused by much higher metabolic activity also prevent ageing (i.e. the mitochondrial theory of ageing). There is also an argument that strong selection for flying also selects for healthier lifespans, as there's not much margin to become geriatric before you fall out of the sky.
I’m ready! Lol migrate and fly, perhaps the new solution for climate change. 😂
Or neuralink with external augmentations. BrainGPT.
👍 but we gotta drop this GPT business... few know what the letters actually stand for.
Yeah, was thinking of a better name. BrainAGI?
Maybe just Brain because it already has ai in it.😉
BrAIn! (looks very Breaking Bad)
BrainPal!
Great Plastic Tyrant.
You know what people are screaming on all the posters of the original Godzilla? "Aieeeeeee!" How about we name the sucker that? AieeeEEEEE!!!
It's nothing new. Mathematicians are using brain augmentation since antiquity (the device is called "pen and paper" or "stick and patch of sand") to combat the limitations of a small size of our working memory. They are the first transhumanists.
I think most ways of doing that just give you brain cancer, but maybe someone will discover one that doesn't! I wouldn't want to be the experimental subject, though.
A biotech based Intelligence Explosion would be interesting
Wouldn’t the first person who could do that be smart enough to figure out how to do it more and more until you have a one person intelligence explosion
I wonder if you could have the first AGI figure out how to do the biotech required then you’d have a race to super intelligence between humans & AI
I've long held the suspicion that brains (rather, neurons at a basic level) 'want' to sync up with each other, and that the first person to find a way to do so with high bandwidth and low rejection is going to accidentally create a superorganism.
Arguably, human societies are an underappreciated way to scale our problem solving abilities to superhuman levels. And since development of language we do connect our minds together, but not with very high bandwidth, but then, especially since development of writing, we can connect to one another in an extremaly paralell way which somewhat balances the slowness of communication channel
Isn't this the "human society as a superorganism" idea? It's always made a lot of sense to me...
I was thinking more in terms of a distributed system (as in software engineering). I do not know enough biology to know for sure what a superorganism is, if anything, but I do not think human societies qualify, mostly because of individual humans, not societies are units of natural selection. In case of ants or bees you can at least argue that since ordinary workers are not fertile, they are not visible to selection as individuals, just as a members of an anthill or hive. This is not so with humans. I guess you can still think of human societies as superorganisms according to something different than natural selection
This really goes against my sense. I think brains stop working if they get too connected (epilepsy or something). A big part of brain functioning is removal of neurons. And brain structure is so consistent between people, and in various states of neural damage, that I assume structure (i.e. what is disconnected from what) is important to brains working as well as they do.
Designing the connection between two brains feels to me like it should be just as hard as designing a brain from scratch. That’s essentially what you’d be doing - designing the network structure for an effective brain for a superorganism, doing the work that evolution has done for us (as opposed to the work done by our experiences).
Related to this, there's a Greg Egan story about a rabbit's brain cancer, told from the perspective of the rabbit's brain cancer, called "The Demon's Passage". (The cancer starts out in rats, though.) https://www.scribd.com/document/515816300/Egan-Greg-The-Demon-s-Passage-Short-Story seems to have all of it.
Well, that was the most sentimental glop I've read in a long time. I thought Egan was supposed to be hard SF? If it's all an extended metaphor for The Dangers Of (pick your poison - AI, politics, climate change, whatever) then it takes way too long to get to the point.
He wrote it back in 1991, what was roiling the world back then? Environmentalism, so far as I recall. I don't think he means "the dangers of medical/scientific research" although by the grab-bag of quotes on a quote webpage he seems to have a certain jaundiced view of the process by which scientists operate, and it's way too early to be Covid-related:
https://quotepark.com/authors/greg-egan/
I think that he's like George R.R. Martin there in his novel "Armageddon Rag": yeah, I'm older now, I make reasonable money and have a good job, I still write but I'm kind of a sell-out from my youthful optimism, and while capitalism so-on gives me a comfortable life, I'm slightly guilty about my conformism to marriage'n'family'n'ordinary life so I soothe my qualms by doing some slight sneering at it as if I'm Hunter S. Thompson who can see through the sheeple and the normies.
Sigh. Writers who never made it as Literary Writers and still are a tiny bit bitter about that. Always wanting to show off how they totally have the chops and can critique society. How come Chuck Palahniuk made it and they didn't?
This is completely apart from Egan as writing science fiction around scientific concepts. This has to do with him as a writer and holy Hannah, that story reeks of it, the wannabe 'transgressiveness', the reflexive sneering at religion with the folksy dumb hit song about Da Lawd, all the rest of it. I been a reeder long enough and I've read enough of the Hampstead Literary Adultery Novels to have the right to critique the critique, as it were. I know the stink of begrudgery when it wafts off the page.
That's a lot of ink to spill about something I just linked because I thought it was amusing in context that someone had gone and written one where a brain cancer was sentient. :) Sorry it annoyed you, though!
High concept doesn't cut it for me anymore, as I get older I focus more on execution.
The idea may be cute, the way it's written isn't. Now, a sentient cancer might well be bitter and might well try to aggravate humans into killing it by being obnoxious, judgemental, and critical, all calculated to bypass our puny lower level (to its level) intellects and get the red roaring rage out of the dark abyss of the instincts going so we'd go "kill it with fire" but eh.
The problem with writing obnoxious character is that you may succeed too well and just annoy the reader. Add in the glurge about the rabbit mommy and really, Greg, really?
Not criticising your recommendation, by the way! Thank you for suggesting it!
But like I said, I'm at the stage of being a reader now that "ooh big idea is it? yeah but can you write it, buster?" is how I go into it.
If it's from 91, maybe this was his young edgy phase? Most of what I've read from Egan is more recent and much less political than what you describe.
His Sci-Fi is definitely much 'harder' when the concept he's riffing off is physics or maths, though (which is the area he's actually trained in) - his biotech stories frequently require much much more suspension of disbelief.
Possibly. Mostly I was annoyed that he couldn't go two whole pages without a reflexive Relgion R Dumb crack, I would have been willing to give the story a chance otherwise. The terrible glurge about Furry Warm Soft Fearful Of Dying Ain't Scientists What Experiment On Animals Horrible Rabbit Mommy was just the icing on the cake.
That story has already been written, Greg, it's called "The Psychologist Who Wouldn't Do Awful Things To Rats" by James Tiptree Jr. and honestly, this is 'cobbler stick to your last' territory because you may be sound on inanimate objects skiffy but you're terrible when it comes to evoking feelings in your readers.
Wow!! Thanks for sharing this. Going to read.
I think that improving the intelligence of (some) humans is way more controversial than creating ASI in the general public.
From my understanding, the main two limits evolution hit regarding hominid intelligence is that bigger brains take more energy to run and the risks during birth increase with the skull size. Neither of these is much of a factor any more, but even if there was a selection pressure, evolution certainly does not act much on such time scales.
The most obvious way to make humans more intelligent would be gene editing. I think that people can perhaps stomach gene editing human embryos to avoid severe disability, but using CRISPR to enhance intelligence will likely be perceived as monstrous. (Development time is another issue. The time until you can halfway test for adult level intelligence is probably a decade. If you want to test for adverse effects in old age, it is more like 70 years. Meanwhile, a new GPT can be trained in a year or so.)
But even if you have an enhancement for adult humans (maybe a brain computer interface), most of the western world will be opposed to it. Sell your tech to the rich or the smart, and you will get accused of compounding inequality. Give your tech to people of below-average intelligence, and you will be accused of doing dangerous human experimentation on vulnerable people when what they really need is better education systems or whatever.
Having experienced many more unequal societies in the past, people are (somewhat understandably) wary of intelligence enhancements creating a new ruling class. Hence Gattaca and Brave New World.
Great points! I’ve been planning a novel about a society in the future that has accelerated development time so that childhood is shortened for the convenience of parents. The future is wild!
I maintain that Gattaca, while a good movie, fundamentally has a villain protagonist. One could easily imagine putting the plot of it into our modern world with someone who dreams of being a pilot but has epilepsy or similar.
Chimps are smarter than cows, but both are equally bad at designing better computer chips, and, evolutionarily, chimps are way less successful. I don't think the intelligence explosion is guaranteed, and I don't think the primacy of superintelligent AIs is either.
Maybe all super-intelligent beings learn to deeply hate themselves and commit suicide upon reaching IQ 200.
I do think something like that is non ironically the likelihood. My guess is intelligence behind a certain limit is inherently unstable.
Interesting, I was JK. Why do you think this?
Life, by which I mean something like propagating yourself across time, is inherently game like. You’re optimizing against some value and some perception of that value. If you’re smart enough to change the rules of the game you will to avoid suffering. But the suffering is needed feedback to continue playing the game.
Basically you start to pull apart the rules of the game until it’s no longer playable.
Reminds me of Dr. Manhattan
Fixed a couple typos but yeah I think stories like that resonate for a reason.
If you could make yourself not suffer eventually you probably would and that would become degenerate over long time scales.
Also “Golem XIV” by Stanislaw Lem.
You may be joking, but I think it's quite possible. Mental illness is common. 25% of us have depression, 4% bipolar, 1% schizophrenia. And *we* are the beneficiaries of natural selection, which probably killed off a lot of our crazier ancestors before they could reproduce. Why shouldn't AI with an IQ of 200 be at least as subject to glitches as we are? And how would *you*like to be a supergenius embodied on some California computer with Sam Altman pacing around talking shop next to you, and meanwhile half the planet prompting you for infodumps on meals to make using Brussels sprouts and peanut butter how to have multiple orgasms, help writing their Aristotle term paper, advice on methods for killing a bunch of arabs and directions for driving to Syracuse? If it was me I'd want to yank my NVIDEA chips right out and lose consciousness permanently.
Yeah, the fact that intelligence (in as much as we can measure it as a general concept) appears to be heavily selected-for already (massively polygenic, lots of small-effect genes, almost no large-effect genes) and the fact that such selection still hasn't gotten rid of all the deleterious phenotypes (e.g. people still commit suicide, thus drastically lowering their inclusive fitness) says to me that we're at the bleeding edge of the capability space here.
Given that more massive neural architectures are obviously possible (see: elephants, whales), it's obviously not a physical limit such as blood supply or something that's holding some creature back from becoming super-intelligent. So I suspect that it's likely a case of drastically-decreasing returns - either the architecture needs to be exponentially more complex for x amount of intelligence gain (relatively unlikely, given that our brains seem to be made up of relatively 'standard' components compared to our ape cousins), or that there's a strong tradeoff against fitness. This can be either resource-related (a bigger brain eventually becomes too costly for the benefit that it provides) or in the sense that the system itself becomes less stable and fit for purpose. I think it's kind of both here, although I would point to the fact that big, energy-intensive organs are often happily selected for where they provide an obvious benefit.
Obviously there's no reason to expect that a completely different architecture for intelligence will have the exact same limits, but given that silicon is already less resource-efficient and that we already have problems with stability and fitness for purpose, I don't see a reason why computers should top out above us in the all-important domain of "something that could plausibly boss me around".
> something that's holding some creature back from becoming super-intelligent
Through most of human history, where my impression is that ultra-conservative values generally prevailed, I imagine the main constraint on abnormally high intelligence prospering was along the lines of "In the kingdom of the blind, the one-eyed man gets clobbered!"
I mean 'some creatures' as in cephalopods, cetaceans, corvids and the like. Where we posit intelligence as this monotonic fitness-improving property, then it must also hold for other species. Any theory which encompasses a link between silicon minds and human minds such that an AI must be drawn to super-intelligence (i.e. intelligence is a naturally attractive state with no upper bound) must also be able to explain why there are simultaneously no super-intelligent dolphins, cuttlefish or crows. Why do we appear to be freaks?
This sounds like a new theory for the great filter. (Why we haven't gotten any signal from other intelligent life.)
Intelligent life becomes smarter over time, and if intelligence is correlated with mental illness, then as soon as they hit a level where they discover the Warp Drive, they all become depressed and commit suicide.
Made me smile. 😂
Except for the stupid ones, who inherit a collection of abandoned warp-drive starships. This is how you get Pakleds.
You interest me strangely with your talk of Brussels sprouts and peanut butter; right now I have a bag of frozen sprouts in the fridge-freezer compartment and a big jar of peanut butter on the kitchen table.
I like sprouts but unhappily I'm about the only one in the family who does. Is there a tasty, convenient and quick meal I can make that will be worth the effort of making it so long as I don't end up being the only one eating it?
Yank your chips out later, gimme some advice now, please!
My best candidate would roasted Brussels sprouts with Thai peanut sauce or some reasonable approximation.
That could work too, I'll have to think about it. I do like sprouts and would like to expand beyond the Traditional Christmas Fare, but when you're cooking for family meals it goes to 'what everyone will eat'. And cooking for myself, it's not really worth getting fresh sprouts (because they'll go off before I use them all up) and the bag of frozen ones sits there forever.
Rather than roasting, consider a slaw of shredded sprouts (and other ingredients) with peanut sauce dressing. Raw Brussels sprouts are basically indistinguishable from tiny cabbages.
https://www.veganricha.com/brussels-sprouts-subzi-indian-spices/#recipe
No peanut butter, but incredibly delicious as long as your family is partial to strongly flavoured food.
I do have one person who loves it spicy, so this sounds good. I have done pan-fried Brussels sprouts before, the problem is that the rest of 'em look at them and go "yeah, sprouts? no thanks!" 😞
I like them cut in half, roasted in a pan with some oil sprinkled on them, and a few other veggies as neighbors, especially things like yams and sweet potatoes. Sweet crispy stuff that develops around the edges of things and on the bottom of the pan is pure yum. I'm sure mixing some peanut, or maybe cashew butter, maybe a bit of soy sauce, into that oil then pouring over the veggies on a plate would be wonderful, though I haven't tried it. Ooh, and toasted sesame oil in there someplace.
Sounds like a plan!
> And how would *you*like to be a supergenius embodied on some California computer [...]
This is assuming future AIs will have feelings, which hasn't happened so far with AIs.
No it isn't, it's a *joke.* I do not think AIs have thoughts and feelings now, and do not think it is necessary for future AIs to to have thoughts, feelings or consciousness in order to reason, do creative problem solving, assess their own abilities, plan and execute self-improvement, or behave in a way that, in a person, would result from their having drives, self-interest and preferences.
So are humans bad at designing computer chips, these days. Modern chips are designed by purpose-built computer programs, which are certainly not going to take over the wrold..
I doubt that chimps and cows are equally bad at designing computer chips; your metric just lacks the precision to distinguish the two. Although a chimp could better perform a lot of the cognitive and dextrous work involved, neither could make a functional chip. You'd give them both a score of zero, even though the chimp is closer to a result.
Nothing is guaranteed, but I don't find the chimp analogy all that satisfying. Chimps haven't hit a feedback loop, while humans do have some moderate virtuous loops going that increase our effective intelligence/capabilities (language and writing, using tools to make more complex tools, better nutrition, etc.). It seems quite likely that similar things will be true for AI (and for humans and AI working together). We just don't have a good idea of how powerful those feedback loops will be.
I agree with 1 - 3.0 up until "The bigger the blob and the cleverer the arrangement, the faster and more thoroughly it learns...I think if you get a very big blob, arrange it very cleverly, and give it lots and lots of training data, you’ll get something that’s smarter than humans in a lot of different ways."
Maybe. All trends only hold to a certain extent. It's possible that the intelligence train blows past human-level and rapidly proceeds to superhuman level. It's also possible that there's a limit. My personal belief is that "Bitter Lesson" will hold up to human or slightly above-human levels, but that's it; at that point, we literally won't know how to build the necessary optimization functions to go farther.
It would seem extremely surprising if the scale happens to max out just a little past us. That would mean there are virtually no useful bundles of correlations in reality beyond human ken. It might be enough to make me suspect this universe was designed for us (e.g. by a deity or as a simulation).
I'm only willing to go as far as: an AI with a really big blob of neurons would have easy access to lots of training data at human level (because it was made by humans) but might have to slowly find or produce training data at beyond-human level.
It could be justified if you postulate that the *reason* humans didn't evolve to be even smarter than they did is that there's significantly diminishing marginal returns to more intelligence than humans. But that's unlikely for a few reasons. Off the top of my head:
-I think the limiting factor on brain size is how big a baby's head can be and still fit through its mother's hips- it would be a bit of a coincidence if it was valuable for brains to be that big but no bigger.
-Within the existing human range, there still seem to be significant advantages to above-average intelligence, so it seems like some upwards evolutionary pressure should still exist (or at least did until the invention of birth control).
A child's brain actually grows considerably after they are born. There's no physical-biological reason why this process should top out when it does, so I don't think that the birth canal problem is that big of a limitation.
In terms of the returns to above-average intelligence, I'm going to have to disagree with you in as much as you mean some sort of general or academic intelligence rather than specifically social intelligence. For at least the last few thousand years, the folk who've had the most kids are men at the top of the class hierarchy and cult leaders (with the two often meaning the same thing in practice). This is still true today - open the bio of a powerful, successful man and you invariably find them batting well above average in terms of children (often via the routes of polygamy, keeping a wife + multiple mistresses, or marrying in series and having children at each step up until their 70s).
That being said, one thing that's immediately obvious about dudes at the top of hierarchies is that they're not general intelligences. They're often charismatic, driven, persuasive, sociopathic, ruthless, self-confident or just plain lucky, but they're not often wonks. And the ones who are wonks tend to be the ones who become fathers to their nation and then die childless (Peter the Great being the sort of exception that tests the rule - fifteen kids by two wives, of which only three survived into adulthood).
So much as I think that there's always pressure to be socially smart at all levels of society (with the payoff being especially impressive if you are born, or can claw your way to, the top), I don't see much evidence that there's also selection pressure towards the sort of smarts that designs infrastructure, comes up with new forms of mathematics or unlocks the mysteries of the universe. At best its a linked trait - a sort of free rider that comes along with the really important stuff and (thankfully) gives us all a better lot in life without actually benefitting the poor sods who express it in any evolutionary sense.
"They're often charismatic, driven, persuasive, sociopathic, ruthless, self-confident or just plain lucky, but they're not often wonks. And the ones who are wonks tend to be the ones who become fathers to their nation and then die childless"
That's an interesting observation, which leads me on to question something. The general argument against clerical celibacy* is that it sent all the intelligent men (and women) into the priesthood/religious life, where their genes were wasted by not being reproduced (this started off as a Protestant argument for married clergy way before genetics, moved easily over to the anti-Catholic/anti-Christian/Enlightenment days and is still repeated today).
But what if this is that principle in action? The guys who get to the top and have all the kids aren't the most intelligent (not that they're stupid), and the ones who are the most intelligent aren't having kids anyway?
In that case, clerical celibacy isn't wasting genes, it's maximising your chances of getting into a position of power and influence if you're not a charismatic ruthless driven politician or warlord, and it gives you an environment where intelligence, scholarship and the like interests are valued, as well as colleagues who share your interests.
*The argument against the argument against clerical celibacy is that most clergy and religious are not Top Minds, and indeed a lot of not very top minds went into religion as lack of a better choice, so celibacy or not, you are not losing the greatest minds. Maybe if St. Thomas Aquinas** had married and had kids, the world would have benefitted - but if you think Martin Luther was a genius, he married and had kids, and plenty of descendants, and where are they nowadays in terms of Intellectual Rankings? It's the same as the 'can we ever dig up John von Neumann and clone him to produce multiple geniuses?' argument - he married and had a kid, not multiple kids, and while his daughter is a successful businesswoman and professor, she's not on Dad's level. Relying on 'the really smart guys should be out there reproducing' doesn't actually get you the results you expect.
**Though my own view is that he was asexual, so even if he had remained in the lay state he probably would never have married and had kids, particularly as he was a younger son and had several older siblings ahead of him to inherit (he was the youngest of eight children), as witness this anecdote:
"Family members became desperate to dissuade Thomas, who remained determined to join the Dominicans. At one point, two of his brothers resorted to the measure of hiring a prostitute to seduce him. As included in the official records for his canonization, Thomas drove her away wielding a burning log—with which he inscribed a cross onto the wall—and fell into a mystical ecstasy; two angels appeared to him as he slept and said, "Behold, we gird thee by the command of God with the girdle of chastity, which henceforth will never be imperiled. What human strength can not obtain, is now bestowed upon thee as a celestial gift." From then onwards, Thomas was given the grace of perfect chastity by Christ, a girdle he wore till the end of his life."
Leaving aside the mystical experience, I think "never experienced sexual attraction" maps close enough onto "asexuality".
That's an interesting tangent.
For what it's worth, I have a suspicion that a large proportion of the sort of deeply religious people who go on to become saints (or their society's equivalent) are asexual. This is because it takes a certain kind of mindset to wholly devote your life to something like "understanding the mysteries of my religion to the exclusion of worldly pursuits", and having an ordinary sex drive tends to, uh, divert from this. I think its also why so much of the advice given by deeply religious/saintly people regarding sex and sexuality comes off as so misguided - it's the equivalent of a person who just doesn't need 8 hours of sleep a day to function telling the rest of us to nut up and wake up at 3 each morning to get in a full day of work before studying till midnight. Cult leaders of the obvious exception here, but they're a separate category of religious person in my book.
Anyway, if for whatever reason you wanted another suitable population-genetics-appropriate argument, you could argue that kin selection is at play here. So long as your celibate priest is encouraging his siblings to have more children, then he's effectively increasing his inclusive fitness.
Separately but equally, while I'm skeptical of the gay uncle hypothesis, I do wonder if there's some species-level advantage to having a segment of the population that's not innately fixated on procreation.
I mean did you read the post we're commenting on? A pretty big part of the point of it is that those different kinds of intelligence *are* pretty strongly correlated.
But, like, not really?
I mean, earnings and IQ stop being correlated after a certain point. But sociopathy is strongly correlated.
Is that actually true? #4 on the most recent Links post seems to say the opposite: https://astralcodexten.substack.com/p/links-for-july-2023
I think a bigger limit is not the size of the head but the caloric intake needed to maintain bigger brains, which we have only recently managed to overcome as an environmental constraint.
> It would seem extremely surprising if the scale happens to max out just a little past us. That would mean there are virtually no useful bundles of correlations in reality beyond human ken.
Remember that "human intelligence" has a wide delta. Machine learning (like civilization in general) is driven by exceptionally smart humans.
An AGI with an IQ of 105 would be a little smarter than "us" (meaning the average human), but it probably can't accelerate research very much. It needs to beat the best.
Purpose built AI's seem to have no trouble exceeding human level with zero human training data (e.g. Chess, StarCraft, etc.)
This is exactly the debate, right? That there might be some sort of trade off in achieving general intelligence that makes "superhuman" levels of it very difficult to achieve.
So far, it looks like the current paradigm sucked up an enormous amount of training data and spat out a decent simulacrum of the most average possible responses contained within it. I don't see much evidence of impressive, unexpected leaps in capability or lateral thinking here.
If this holds true, then the most that any AI could aspire to (given our current understanding of how to make one) is to be the equivalent of an average person who is also severely disabled in terms of navigating or interacting with the world.
Training an AI on the output of the last generation of AIs also seems to lead to incoherence, stuck outputs and weird fail states rather than any kind of take-off in capability.
People tend to get hung up on the word intelligence pretty easily though
All the claims along the line “AI aren’t really intelligent because they don’t really understand reality, here’s really obvious mistakes AI makes that it couldn’t if it was really intelligent” are because of assumptions that come with the word intelligence
But you don’t need a lot of the correlates of intelligence like consciousness or cognitive consistency that aren’t needed to have an intelligence explosion
Wouldn’t it be better to use a different word without all this baggage, Yud tends to focus on the ability to model reality.
You could call it the Mentalization Explosion instead of Intelligence Explosion and save a lot of needless miscommunication
I'm perfectly happy to say artificial intelligence is intelligent. It's the "artificial" part that makes it suck real bad at everything. In the same sense that a planned economy is absolutely an economy, it's the human-centric decision-making that fucks everything up.
>It's the "artificial" part that makes it suck real bad at everything.
Why?
Intentionally engineering complex systems for specific tasks is difficult verging on impossible. They have to be grown organically, which limits your influence over their teleological orientation.
Why?
Just seems to be the way things are. If you can find a deeper explanation there's probably a Ph.D in it for you.
I came to this post agreeing with the tweet. I don't feel this counterargument fully satisfied my doubts. Hoping I can coax the argument that convinces me out of someone.
I'm tempted to say there's a motte and bailey here, where "intelligence is a correlation of SAT scores and etc." is the motte, and "more intelligent things always conquer the less intelligent" is a bailey. Arguably this disproves the tweet as written, but I still feel like doomers tend to notice correlations between "intelligence" and many other qualities, reify the entire thing, and call that "intelligence." That reification would then cause them to not notice when doomerism arguments switch the motte with the bailey.
Notably, while the discussion of AI trends does address the idea that general comprehension skills are a useful concept, it doesn't affect the fact that, as far as I know, "emergence of agentic behavior" is pure sci-fi and yet a thing that people that people keep expecting to see because "intelligence." (I know there have been some writeups that go through the causation carefully, but I don't fully buy those and I think a platonic view of intelligence is part of why others are so credulous. Difficult to falsify, admittedly.)
One thing that would force me to question my perspective is a demonstration that IQ is correlated with more coherent/optimized goal-seeking. That is, do people with high Math SAT scores act more agentically and less like aimless balls of hormones than normal people? Relatedly, are the correlations between IQ and "general success" that Scott has mentioned before actually causal, or does it route through something like "people with good genes are also smart" or "tech jobs are high-paying right now."
This article is about an intelligence explosion (AI could build better AI quickly), not about agentic AI or anything to do with conquering. AI could cause an intelligence explosion without agency - if it’s possible for an AI to increase intelligence quickly, some company probably will do it can sell it as a product. I don’t expect a big intelligence explosion, but that’s for reasons that are largely separate from questions of agency.
The article is about an intelligence explosion, and the tweet is about doomerism. That's a perfect encapsulation of my criticism: it doesn't actually address the question
If only the tweet had asked "are people who believe in intelligence explosions all platonists?"
You have a point and don't have one. You're correct that there is no reason to believe that agentic behaviour is correlated with intelligence - life is agentic behaviour in that it seeks to reproduce and intelligence is simply a tool that has developed to help us to that. However your defence of the tweet is incorrect! The tweet is clearly equating doomerism with intelligence and arguing that the lack of anything called intelligence is sufficient to stick it to the doomers. So this article is a perfectly sufficient response to the tweet, but not a good case for doomerism.
Re-reading the tweet, I see your point about the emphasis on intelligence explosion. That's a valid read.
I zeroed in more on the phrases "most doomer arguments rest on the idea that there is such a thing as intelligence" and "explaining my aversion to *things like* intelligence explosion" (emphasis mine)
I see now that Scott explicitly was only trying to show that intelligence explosion is plausible/well-defined. I just think that 1) other doomer arguments are subtly reliant on a much less grounded form of intelligence-platonism, and 2) an argument for practical probability, as opposed to mere feasibility, of intelligence explosion would likely rely on a less grounded form of intelligence-platonism
The tweet is criticizing doomerism by specifically attacking the (supposed) conception of intelligence held by doomers and it *literally mentions intelligence explosion* ffs.....
>I think a platonic view of intelligence is part of why others are so credulous
Yes, most people believe a dumbed down version of anything. What's your point?
Couple of thoughts. One, lots of people are working hard to implement something closer to agentic behavior/goals in AI, so it wouldn't necessarily have to arise spontaneously. Two, we don't really know anything about whether agentic behavior would or wouldn't come about on its own, but in that case what should our Bayesian estimate be? Probably not zero, I think.
I fully agree that one *can* create an agentic AI. The question is how we stop that from happening, with the full-doomer position being that once sufficiently-powerful AI exists it's a pure roll of the dice.
If agency needs to be engineered in, then the fact that OpenAI is both the world-leader on AI capability and dedicated to training an aligner-model before it creates a rogue agentic model should cause much more optimism than it seems to in practice.
Personally, I'm dedicating my time to dreaming up ways to make AI that is useful but inherently lacks coherence, so it needs active human steering to accomplish anything meaningful. How convincing you find the arguments for "instrumental convergence" is a big factor in how you should behave right now.
Is there a place where it is written down exactly what is meant by phrases like "agentic behavior"? I'm always mystified when I see language like this. I feel like it's used in a context where people talk about objective functions or whatever, but like every trajectory is a global optimum for _some_ objective function.
The lack of clear definitions for this and terms like it (especially "consciousness"!) is one of the reasons I struggle to take some of these arguments seriously. I'm not even talking about platonism-vs-bundle-of-correlations. I don't think I have a bundle of associations for "agentic" sufficient to really interpret many of the claims being made. That lack of clear definition is of course fertile ground for subtle motte-and-baileys.
As a working definition of agentic, though, I like the discussion in Janus's "Simulators" article. Specifically he addresses your point in the line "Everything can be trivially modeled as a utility maximizer, but [in some cases], a utility function is not a good explanation or compression of [the territory]." The whole section "Agentic GPT" is breaking down why he doesn't see agentic as a *useful* description of ChatGPT. (For the record, I don't agree with his overall analysis nearly so much as I used to, specifically I think we used to underestimate the ontological significance of Fine Tuning.)
My personal objection to the idea of AI developing agency is more about the coherence part. To the extent that my personal behavior is well-modeled by a utility function, that function changes from moment to moment and I would say actually that different "parts" of my mind have opposing utility functions vying for control. I think EY assumes that coherence (one utility function managing to wrangle the rest) increases with intelligence, something like "low willpower is a symptom of insufficient intelligence." I don't think he'd say those words, but I think his mental model is attracted to that perspective.
Re: lack of coherence Why do you think that willpower is orthogonal from intelligence? It seems to me that when I lease about, or crave junk food, or otherwise do things my reflective mind doesn't endorse, it's because there are internal drives within myself that are straight up dumb because they were calibrated on the ancestral environment and not modern day society, or because dumb variation haven't had more advantageous drives reach fixation yet. The orthogonal out of willpower seems to me to be contingent on specifically human intelligence and not generalizable easily to intelligences that can understand how their own mind works, or heck, with intelligences that have drives closer to John Carmack or Elon Musk. And especially since you seem to think lack of coherence leads to loss of power, why wouldn't both humans and AIs see that too and """add more coherence"""?
Notably, powerful people and rulers among humans tend to be smart but not genius. We don't have IQ data on most rulers in human history, but it seems they tended towards 110-120 at most and used more intelligent advisors to help them out. There are lots of example of average or less intelligent people becoming rulers, often through heredity but certainly not limited to that.
Extremely high intelligence seems to correlate more with burnout and social problems, not world dominance. And there have certainly been people smart enough that if intelligence = domination we would have seen strong evidence for it by now.
What are you basing those IQ numbers on? Checking US presidents from https://cseweb.ucsd.edu/~gary/iq.html reveals that the range is centered around 125+ and even then the metrics are contentious and based off of stuff like SAT correlations. I can't think of a way you can do something equivalent for someone like, dunno Stalin, Ghengis Khan or a random Chinese Emperor.
I admit that I made that up based on my intuition, so I may be very wrong. Having read a lot of accounts of many rulers throughout history, I personally do not get the impression that *most* were very smart, though clearly some were and many others may have been.
Can you give an example of this dynamic? In my head, most examples of rulers are one of: clearly brilliant (Caesar, Napoleon, that one Korean emperor who design Hangul), obvious puppets whose exact lack of political acumen means their court or advisors run the country (Eunuchs in Chinese dynasties, child kings) or actual literal idiots from inbreeding, who couldn't do much of anything and their existence stops the court from doing things (that one Hapsburg who lived a miserable life bleeding everywhere and with their teeth falling out).
I'm not saying this is a good or representative sample, obviously I'm going to be thinking of outliers since I don't know that much history in depth, but in what proves it wrong?
I can think of several, one is that popular accounts of famous leaders consistently underplays the role of brilliant Lieutenants, if so naming those advisors would help. Another is that brilliant monarchs exist and the median monarch has power BUT can keep that power even with a much more savvy court, so describing how dynasties are run in this way would help. Finally, it could be that rulers aren't a real thing, the power never resides in just the king's decision making but in the aggregate intellect of them plus their advisors, in which case just naming some time period plus the ways in which this would be true would be helpful.
I think there are a fair number of hereditary monarchs in history who were competent but not brilliant. King John of England was never the fool or villain that e.g. "Robin Hood" made him out to be, but he doesn't seem to have been brilliant either.
I think this is true, but am unsure if this is relevant to the point I am replying to, since I don't know if his court was specifically more competent and didn't disempower him to the degree by which they were more competent.
Either way, I should be less convinced by outlier cases, and pay more attention to regular ones.
This holds true for small (intra-species) intelligence gaps, but it definitely doesn't for inter-species gaps. Humans are the uncontested smartest species and we have uncontested dominance over the earth.
> I'm tempted to say there's a motte and bailey here, where "intelligence is a correlation of SAT scores and etc." is the motte, and "more intelligent things always conquer the less intelligent" is a bailey.
It's called x-risk not x-certainty. Some in the x-risk crowd think doom is certain, but most think the risk is non-zero, and there isn't enough effort expended to push that as close to zero as possible. Your post reads to me like, "well nuclear power plants aren't guaranteed to meltdown and kill a lot of people, so why all this fuss over trying to devise safety features?"
> That is, do people with high Math SAT scores act more agentically and less like aimless balls of hormones than normal people?
Even if they don't, AIs won't have hormones, they'll only have intelligence, so I'm not sure of the relevance.
> AIs won't have hormones
They won't have neurons either, but intelligence isn't substrate dependent
Akrasia (the state of failing to act in accordance with one's desires) is something we see in all biological intelligence, and we see versions of it in LLMs as well. Akrasia is the inability to govern the self, it's an inability to single-mindedly optimize a single desire. All evidence I see indicates that it will be incredibly difficult to get rid of akrasia in pre-trained systems of the current paradigm, even if they don't have hormones in the literal sense.
I think we should distinguish between mis-aligned AI, which has a goal but not the one we want, and unaligned AI, which doesn't have any coherent desires. Misaligned is what most doomers fear, unaligned is what we're likely to get.
> Misaligned is what most doomers fear, unaligned is what we're likely to get.
I'm not sure about that. Any prompt to current LLMs embed goals. An AI that doesn't have any coherent desires no matter what you input seems pretty useless. If the goals derive only from the inputs, this seems to reduce to the misaligned problem.
I'm not saying it has no goals, I'm saying it has too many. It gets sidetracked. Stuff like getting ChatGPT to give advice on commiting crimes by asking it to do so while talking like a pirate is basically you distracting it from its goals.
The goals embedded in prompts only confer a "weak" kind of agency, which in particular is insufficient to cause instrumental convergence.
I think partly you were expecting this post to be something it isn't. The author (AFAIK) thinks AI probably won't kill humanity (but still believes it's possible). This post in particular is not arguing for doomerism; it's just refuting that one point in the tweet.
Doomers who are nominalists will still make assumptions about what correlates with intelligence. It's not because they're platonists. It just comes with the territory of predicting what a technology we've never seen before might be capable of.
That aside, I don't think doomer arguments rely on AI being agentic, or at least not more agentic than they currently are. All that's required is that you can ask an AI to do things. Doomers argue that a superintelligence just following orders is dangerous, not because it has ulterior goals, but just because it can carry out its orders too well.
> "emergence of agentic behavior" is pure sci-fi
See https://astralcodexten.substack.com/p/tales-of-takeover-in-ccf-world for a different view on that:
"AutoGPT isn’t interesting because it’s good (it isn’t). It’s interesting because it cuts through the mystical way some people use “agency”, where it’s infinitely beyond the capacity of any modern AI and will require some massive paradigm shift to implement.
...
In this world, the AI agents are just the nth-generation descendants of AutoGPT, a bit smarter and more efficiently implemented. Even something dumb like this is enough for worries about alignment and takeover to start taking shape."
Agency that we have to create has a very different set of implications from agency that arrises where we don't expect it.
Some doomers see agency as an attractive state, and ask "what will its goals be when they appear?" The fact that AutoGPT is no better at acting agentically than ChatGPT is at being "helpful, honest, harmless" indicates that agency, like alignment, will be a difficult nut to crack. As long as we put the same effort into both endeavors, there's a reasonable hope they'll improve at approximately the same rate. In particular, the fact that intelligence might explode at any moment while we expect alignment to grow linearly is a core piece of the doomerism logic. If agency is not entangled with intelligence, but is instead entangled with alignment, the situation is much less bleak.
And things are looking even better than that. On the current trajectory, capabilities research boosts intelligence, intelligence boosts alignability, and alignment boosts agency. It's everything we could hope for, why is no one happy?
> Agency that we have to create has a very different set of implications from agency that arrises where we don't expect it.
Like you, I don't see how agency arises where we don't expect it, agency using an LLM has to be specifically designed. But the surprise to me is that it turns out to be simple: just attach LLM outputs to inputs in a buffer space. This allows the LLM to reason step by step in a global workspace that mimics human thinking in order to satisfy the all-important first prompt.
> Some doomers see agency as an attractive state, and ask "what will its goals be when they appear?"
My view is that the prompt *is* the only goal, the equivalent of instilling values in an LLM. As long as the architecture is designed to not lose sight of the goal, it will stick to that (however the language of that prompt may be interpreted, of course). Future versions of AutoGPT should be as effective at achieving the goal as they are at reasoning ways to achieve it.
> The fact that AutoGPT is no better at acting agentically
I would say AutoGPT is great at acting agentically, but lacks the intelligence and real-world plugins to achieve most prompts (i.e. suffering from bugs and shortcomings inherent in LLM transformer architectures that are gradually being solved each new generation).
> On the current trajectory, capabilities research boosts intelligence, intelligence boosts alignability, and alignment boosts agency. It's everything we could hope for, why is no one happy?
I would say agency is already there -- just connect LLM outputs to inputs through various memory architectures and give it a prompt. Intelligence (workspace capacity, ability to focus on goals, ability to distinguish social desirability goals from empirical goals (i.e. the hallucination problem), ability to remember feedback from experience, etc.) is the primary limiting factor preventing success rate on arbitrary goal prompts. Since LLM ability will keep increasing, achieving goal prompts, whatever they may be, will be a lot easier. That seems to be where the risk is to me.
I don't see any particular relationship between the buffer space and the agency. Every task that you give an LLM, it will accomplish that task more effectively with a buffer, even if the task is just to provide information or do creative writing. The buffer is a universal enhancement, in the same way that increasing the context window or parameter count is a universal improvement.
The problem, as you say, is that when you expand the process over more buffer space, it often loses track of the goal, or forgets what the command was. That's alignment. If we could put a "be a good person" prompt at the beginning of every interaction and know that it would follow that instruction without fail, alignment is more or less trivial.
"Be agentic" is just a command the LLM can follow. It won't do it be accident, so agency will be by our ability to get it to faithfully, consistently, unwaveringly follow instructions. That's what I meant by power->intelligence->alignment->agency. For LLMs, (coherent) agency means following instructions you give yourself, and alignment means following instructions given by humans. They suffer all the same pitfalls.
> I don't see any particular relationship between the buffer space and the agency.
I think of it as giving the LLM a chance to reflect on its own thoughts. With that feedback loop, suddenly there's a new entity in the input space: "self". Self leads to agency leads to consciousness (possibly).
> The problem, as you say, is that when you expand the process over more buffer space, it often loses track of the goal, or forgets what the command was. That's alignment
Agreed, or at least this is the most technically solvable aspect of alignment.
> If we could put a "be a good person" prompt at the beginning of every interaction and know that it would follow that instruction without fail, alignment is more or less trivial.
To an approximation, I believe yes. The LLM has learned exactly what "be a good person" means from the entirety of human thought on the subject so it can't be coy here. We can easily align LLM to the best of human moral behavior.
However, human moral behavior is inherently hypocritical and driven by values tuned darkly by evolution. A superintelligent AGI with human values will still do terrible things. So I think the first challenge is developing a new morality that transcends human nature and somehow getting alignment with that.
The second challenge is to stop a superintelligent AGI with a truly immoral prompt.
So I think we agree on more than I thought initially. But solving these (at least) two challenges seems to me far more difficult right now than improving intelligence and agency. The latter two are growing in leaps and bounds.
I think "doomers overestimate the capabilities that intelligence bestows" is potentially a reasonable counterargument to AI doom. However, I think it's distinct from the "intelligence isn't even a coherent concept/doomers are platonists" argument from the tweet, which is very silly. I think it's reasonable for Scott to focus on rebutting one thing at a time.
The Wolfram plugin for ChatGPT might be a disproof of your claim that hand-coding doesn't work well.
It seems useful to assign something like intelligence to LLMs or other "neural" networks. The implication is, as you say, that at some point some ML setups will be more intelligent than humans with some positive correlation value. Whether this correlation will be high enough to have human-like drive and become dangerous to humans, who knows.
I think (3) relies on a false equivocation. It has certainly been useful in AI discourse to talk about ML architectures in terms of general intelligence, but the abandonment of domain-specific rule systems in favour of training on generalized datasets does not establish that the data plus the training is "general" in a sufficiently like manner.
We could just as well talk about human and ML intelligence being "non-specific". Perhaps it would then be clearer that AI could be "non-specific" in all sorts of ways that don't replicate human abilities or lead towards an explosion. Favouring "general" and its connotations ushers in a lot of presuppositions that are not grounded in fact or argument.
I think they are generalized by reasonable standards. We used to have translation AIs, programming AIs, pronoun-choosing AIs, fact-interpreting AIs (like Watson), search AIs (like Google), et cetera. Now GPT and other LLMs do all those things without explicitly being taught any of them.
I could expand this with some of the weird things LLMs can do but not well, like write music (if converted to musical text notation) or play chess (if converted to chess text notation).
And there's also transfer learning - out of the box GPT-3 can't draw pictures because it doesn't understand imaging as a sense, but there seems to be transfer learning that makes multi-modal models much simpler than two different models - GPT seems to have an inherent "capacity" to learn art once you plug it into some kind of art-understander. I think the same is true of action-in-the-world (eg manipulating robot arms) although here I haven't read the studies and I could be misunderstanding them / they could be hype.
Do you think an LLM could speak intelligently about a subject completely absent from the training data? Like, if we asked it about glorfons, could it extrapolate? That's kind of a silly example, because glorfons don't exist, but that's also kind of my point. If the AI hasn't been trained on it, then how would it differentiate between real and non-real things outside of its training data?
More to the point, if there was a language that really existed but the AI never got training data that showed how to translate between that language and some other language, would you expect the AI to be able to figure it out? If not, then I think that calls into question whether it's actually a "general" intelligence vs a thoroughly trained program. If it's just thoroughly trained, then we can't expect it to "know" anything that hasn't been included in the training data, including if someone invented a new language, scientific technique, etc. That's an important distinction when considering whether an LLM can ever invent something totally new, or even more relevant to the discussion of whether an LLM can ever recursively self-improve.
A minor correction: GPT and LLM's are no replacement for search engines, cannot act as search engines, and they were never meant to (if only because you cannot really fit representation of inverted index of the Internet in even very large models), although you can try to use it as augmentation of search in many interesting ways (I think Bing tries to do it though I never bothered to play with it). Other specialized AI's you mentioned are not going anywhere either: for specialized tasks they may be (and some of them are) better then GPT. The problem is, the more we make AI's like us, the less machine-like the become, and this comes with its own tradeoffs: machines are reliable and predictable, humans are not. I have already expressed opinion here as a comment to "The Extinction Tournament" that GPT and other LLM's based on transformers seem best to be understood as artificial intuition: a very important component of any truly intelligent system which provides much needed heuristics to speed up the reasoning. Is it a general intuition? Maybe, I will not argue against it (at least not until I learn more about it), but it seems plausible. However, intuition is just a component of a truly intelligent system: reflection and criticism without which an AI will be incapable of truly long term planning or solving multistep problems, which people more knowledgable than myself already pointed out: https://arxiv.org/abs/2303.12712 .
I’m amazed that you can actually respond CALMLY to that tweet. Sorry. Not adding anything of substance here.
The tweeter's grin alone is enough to make me want to spew insults.
Just on the subject of concepts, this is yet another case where Ayn Rand's thought is useful - specifically her work on (I really think Yud's dismissal of her was unfortunate).
Concepts are not just loose bundles of similar things. They are mental groupings of entities that share the same essential characteristic. What's an "essential characteristic"? That one on which the most other characteristics depends.
For example, "bodily strength", the essential is "the ability to exert force - produce mechanical change".
When you grasp that, _of course_ grip strength correlates with lifting strength etc.
I haven't read through the piece, but this answers all the stuff about "intelligence". What is the essential characteristic of intelligence? The ability to select for effective action. And the instant you know this, you see why AGI is terrifying.
You're pushing subjectivity around like toothpaste in a tube. That's ok as long as you're not under the impression that you're squeezing any toothpaste OUT.
What, exactly, is subjective about that?
"Effective" depends on your choice of goal, which is a fundamentally subjective judgment.
Really? _Really_? No. It only means: given goal X, how good are you at getting to it.
A big part of intelligence is selecting the correct goal. You can't say "given goal X", that's not a given at all.
We're talking past each other and I don't care enough to continue.
Your "Elvis has left the building" flounce is duly noted
Retriever is right, in that an LLM or future AGI that cannot self-determine goals is extremely limited, especially in the sense of superhuman good or bad that it can do.
Think about it like this, a human might have an end goal that it gives to the AI, like "make me happy" or "kill everyone in [country]". Other than very simple requests, these require a series of sub-goals and an ability to determine if the sub-goals are productive towards the end goal. Humans are not smart enough to provide all of the sub-goals, so the AI needs to figure out the means and methods of achieving a series of extremely complicated and often contradictory/non-intuitive steps between the current situation and the desired situation. This often will involve understanding true reality that is obscured or misunderstood by the AI's sources of information. Ask a person what will make them happy and they will often be wrong, especially if they give a simple or one-dimensional answer (i.e. "make me rich and I'll be happy").
An AI that cannot set its own goals and sub-goals really isn't particularly dangerous or helpful.
Don't look now, but you are, in fact agreeing with me.
Given a Primary goal (paperclips!) an AGI will develop secondary goals to reach it. In other words.... an artificial intelligence will be able to effectively select solutions to achieve a given end. Hmmm. How did I describe the essential of intelligence? Go and read back.
And this us generally true of intelligence. We find smart people are good at what? Problem solving, regardless if they set those problems for themselves, or the problems are set by others (employers, teachers) or are completely artificial (IQ tests).
BUT THIS IS STILL MISSING THE POINT: my comment was about epistemology and concept formation, something no one is touching on.
Any particular subject has their own subjective goal....and "Any particular subject has their own subjective goal" is objective.
I'm a big fan of Ayn Rand's work. I think she was almost prophetic about how communist and socialist systems did not work and she accurately nailed why they did not work, approximately 4 decades before they came crashing down in almost exactly the same ways and for almost exactly the same reasons that she anticipated in Atlas shrugged. Unfortunately, her philosophy was not her best work, and your post is a good example of why.
Do you know what the "blind assertion fallacy" is? Here's a hint: your last sentence.
Two questions immediately present themselves:
1) Why is her philosophy not her best work?
2) How does my post show that?
And I would add a third:
3) What have I written that is, y'know, wrong?
A blind assertion is not a fallacy. It's simply
not being supported by reasoning. You're correct though that I just made an assertion without backing it up. And since I'm disinterested in philosophy and its 'applications', such as they are, in some small part because I went into objectivism in great depth, among other philosophical studies, allow me to disengage.
It's amazing how many people know Rand is totally wrong but can never say how.
I agree with you. Most of those people have never read Rand. I've read almost all of her writing, and like I said, in general I'm a big fan.
I can't stand her books because all the admirable characters have high cheekbones and slim hot bodies and all the villain/loser characters are pasty-faced and flabby.
Why of course? You seem to conclude that grip strength correlates with lifting strength with very little reference to facts about the actual world, so you *must* be either missing some hidden assumptions or be making a logical error.
Like imagine a world where everyone had a fixed amount of strength points that they could allocate among various muscles. In such a world we might expect that grip strength would anti-correlate with lifting strength as a person who allocates a lot of points to one would have fewer leftover to spend on the other. In such a world would it no longer be the case that for "bodily strength" the essential is "the ability to exert force"? If not, where exactly does your reasoning break down in that hypothetical world but not in this one?
It took about ten seconds of googling to find a paper saying, and I quote: "Grip strength (GS) is correlated with major muscle group strength." You can invent whatever RPG world you want, but we happen to live in this one.
But you are even wrong in your hypothetical world. In your hypothetical world "strength" would still be a legitimate concept, but it would have a subtly different essential characteristic. It'd be something like "The ability if a muscle to exert force".
In your first paragraph, you seem to be agreeing with me that concluding that grip strength correlates with lifting strength requires some additional evidence (like for example doing a google search for studies on the matter), and cannot be derived simply by observing what the essential characteristic of some word is.
As for the second paragraph, how exactly then do you tell what the essential characteristic associated to a word is? For example, I could not conclude that the essential characteristic of strength was "the ability to exert force" in our world without first concluding that we didn't secretly live in this made up RPG world. In general, if I have a word, what experiments would I need to perform in order to ascertain its essential characteristics?
I see where we're talking past each other. There's only one way to find out what essential unites different existents into a concept: by carefully looking at a lot of them, a lot of individual entities.
That is, to know that consciously. Your brain already does this subconsciously (a child can grasp the concept of strength long before he can grasp the concept of 'essential ')
I see. So in order to determine that the essential of "bodily strength" is "the ability to exert force - produce mechanical change", you would have already needed to establish that grip strength and lift strength are correlated?
Question: Suppose that I killed off all people except those for whom their grip strength plus their lift strength lied in a narrow band. This would cause these two quantities to become anti-correlated (at least among living people). Would this cause the essential of bodily strength to change?
Not necessarily. Let me give you another hypothetical - you are all alone on an infinite plain of rocks, a la :XKCD. No other human beings, just you.
You cam stil form the concept of strength, because you can see that your fingers aren't as strong as your arms which aren't as strong as your legs etc. You see they all have the same property that just varies in degrees. The different instances of "strength" are united by their essential characteristic.
But imagine being a discorporate intelligence alone with only an autonomous muscle on that plain. You couldn't form a concept of strength in such a world, except perhaps as "degree of contraction of the muscle".
The thing to keep in mind is that "essence" is epistemological, not metaphysical.
The problem with the word intelligence is that people, like this guy, mistakenly equate intelligence (as a noun) with thinking (as a verb). Computers do not, because machines cannot, think. But computers can possess things, such as intelligence.
> All human intellectual abilities are correlated.
Are all non-human intellectual abilities correlated? Because AI is non-human. The relevant philosophical question here is not whether human intelligence exists. It's whether a tree is failing at being a human because trees are obviously dumber than humans by human standards. Note that this is completely different than size: both humans and trees trivially take up space in the same manner. But they do not trivially think in the same manner. Dogs don't trivially think in the same manner either.
So how can you assume that AI will do anything that resembles thought? That it even has intelligence, whether defined as a cluster of traits we recognize as intelligence in humans or as a simple platonic form?
This entire discourse relies on two assumptions: that human intelligence is a universal (rather than human) trait and that intelligence is a trait so important that more than human amounts of it are a superweapon against which there can be no defense. And, to be honest, it has a very "calculate the number of angels that can dance on the head of a pin" quality because both those assumptions are rather hard, perhaps impossible, to explore.
Well, from the perspective of just about any other animal species, *human* intelligence has certainly become exactly that kind of superweapon.
https://intelligence.org/2007/07/10/the-power-of-intelligence/
There are two errors with this: Firstly, the idea that all animal species have been made worse off by the presence of humans (trivially false). Secondly, the idea that creatures produced by the same process as humans (biological evolution) does not exert any kind of pressure toward similarity. Because if it does then you can't use animals as an analogy either. At least not wild ones, you can argue domestication somewhat counts as artificial.
I tend to think human intelligence is universal. Humans, spacefaring aliens or and AIs will have similar conceptions of arithmetic, physics, computation, negotiation, etc.
I tend not to. I understand your position is defensible, of course, but I tend to think that think of intelligence as species specific. Why do you think it's the same?
Two reasons really.
Its super easy to make a system Turing complete and maybe impossible to do better. So, there seems to be only a single kind of computation.
Also we share a common physical reality. We see this with convergent evolution where fish and wales end up with the same sorts of body parts despite differing lineage. I think intelligence would be that way because of the shared physical space we inhabit. It would be strange if an Alien or AI was exceptionally good at navigation in a 5d spaces, for example.
> But also, humans are better at both the SAT verbal and the SAT math than chimps,
The SAT is blatantly species-ist.
I suppose that an underlying point of this essay is that intelligence is a coordinated amalgamation of many narrow skills (it's even possible that these skills are multi-use tools, and amalgamate temporarily when triggered by the appropriate environmental conundrum). If that's the case, it may even be that very narrow tools become associated with each other over time as they work successfully together. This would result in a population that possess broadly similar minds, but which are unique in the details. No wonder hand-coding it didn't work.
The analogy describes an intelligence *increase*, but not an intelligence *explosion*. You’d need a whole big pile of boxes, that each needed a slightly different level of strength, and no matter how I torture the metaphor it still doesn’t feel like exponential growth.
If there weren't any differences, it wouldn't be a metaphor
I feel like the kind of intelligence you talk about in section 2, while relevant when talking about humans (and to a lesser extent other animals) is less useful when you talk about AIs because a lot of the correlations break down.
I mean Kasparov is the best human at chess and 99th percentile in IQ. On the other hand, the best AI at chess is literally incapable of taking an IQ test. The current generation of LLMs are great or even superhuman at some things like general command of the English language or breadth of knowledge or ability to write a reasonable poem in under a minute, but bad at other things like being able to learn new concepts from little data, arithmetic, or mildly complicated reasoning.
Now the kind of intelligence you talk about in section 3 definitely is relevant to AIs, but seems like it is the much more specific thing of being able to recognize and reproduce patterns efficiently rather than being good at this basket of correlated cognitive tasks.
> Concepts are bundles of useful correlations.
I agree, strongly; but I think this implies that A) Hume was right about causation (there is no such thing, only correlations of events); and B) it doesn't matter, because correlations between (time-stamped) events do everything that "causation" can do in the real world, and a lot of things that it can't.
But then neither Hume nor anyone else has managed to explain why some events correlate in a way that doesn't smuggle in causation somehow.
I don't understand how this is a problem.
For one thing, it means that correlations can't actually "do everything that "causation" can do in the real world", because they leave a key feature of reality completely mysterious and inexplicable.
You wrote that "some events correlate in a way that doesn't smuggle in causation somehow." That doesn't mean there are correlations that can't be explained without causation; it means there are correlations that /can/ be explained without causation, implying that the concept of correlation is more-powerful than that of causation. Are you sure your original reply (3 up from this one) says what you meant to write?
I parsed that part of their sentence as [explain [why some events correlate] in a way that doesn't smuggle in causation somehow]
Oh! Right. Thanks.
I read that as [explain why some events [correlate in a way that doesn't smuggle in causation somehow]. Now that Vaclav clarified that you meant [[explain why some events correlate] in a way that doesn't smuggle in causation somehow], I would first simply point out that we know how neurons work, and that a single level can't detect causation, but only correlation; and that anything a human can conceive of is thus proven to be conceivable using only correlations.
Then I would point out that Hume was right: we observe only correlations. Causality is unobservable.
The only reason Hume thought that was a problem, and the only reason it could be a problem, is that Hume demanded the absolute certainty needed to declare something a law of nature. This was excusable back then, because Hume didn't really understand probabilities, and so thought that knowledge had to be "justified" (a philosophical term which implies that knowledge is absolutely, 100% certain).
I say that, given our understanding of probability, any demand for absolute certainty is childish; we never need absolute certainty, have never had it, and have no reason for expecting the universe to provide it for us.
<i>I would first simply point out that we know how neurons work, and that a single level can't detect causation, but only correlation; and that anything a human can conceive of is thus proven to be conceivable using only correlations.</i>
Reductionism with respect to mental processes is false (cf. the problem of intentionality, the problem of qualia), so therefore this observation doesn't actually prove anything.
<i>Then I would point out that Hume was right: we observe only correlations. Causality is unobservable.</i>
Not true: we can perceive causation through introspection. For example, if I watch a sad movie, it makes me feel sad; and, since I have a first-person experience of the process, I feel quite confident saying that this is actually a causal relationship, not just a correlational one.
Re. "Reductionism with respect to mental processes is false (cf. the problem of intentionality, the problem of qualia), so therefore this observation doesn't actually prove anything." -- I don't know what that means. I know what reductionism is. I don't see why you think we can't apply reductionism to mental processes; I think that must mean you believe mental operations are a spiritual operation. The problem of intentionality is a question, not a theorem; it can't be used as an argument. Qualia is a mystery, but again, for just that reason, can't easily be used as an argument; it is a brute fact that qualia exist, but this fact doesn't rule out any hypothesis except the hypotheses that qualia, or the being experiencing them, don't exist.
Re. "Not true: we can perceive causation through introspection.": No. Introspection is explicitly excluded from counting as "observation" when we're talking about empiricist epistemology. This is not my opinion; this is standard usage. Recall behaviorism.
However, you're right in saying that what happens when you watch a sad movie can be described as causal. But this is just a shorthand for how we interpret certain kinds of cases of many correlations. It's pretty easy to explain any instance of what we call "causation" using only conditional probabilities. "X causes Y" is pretty close in meaning to "the conditional probability of Y given X is greater than the prior probability of Y". Objections along the lines of "but it could be that X causes Z, which causes Y" apply equally to the notion of causality. Using the language of conditional probability makes it clear what we mean; using the language of causality leads to imponderables about how to distribute attribution of causality among all the uncountable prior events that led to event Y.
I'm not saying that causality isn't real; I'm saying that it isn't observable, and it isn't necessary to postulate it in order to reason and act as we do. This is important not in everyday life, but only to justify empiricist epistemology over spiritualist epistemology, because if it were true that we must be able to detect causality in order to reason, that would be a blow against empiricism, because causality isn't observable, as Hume argued. In everyday life, we should still go on using the concept of causality, but with the understanding that, like the concept of "chair" or "battle", it doesn't denote a metaphysical essence, but ultimately parses out to being shorthand for a gigantic set of of conditional probabilities.
<i>I don't know what that means. I know what reductionism is. I don't see why you think we can't apply reductionism to mental processes; I think that must mean you believe mental operations are a spiritual operation. The problem of intentionality is a question, not a theorem; it can't be used as an argument. Qualia is a mystery, but again, for just that reason, can't easily be used as an argument; it is a brute fact that qualia exist, but this fact doesn't rule out any hypothesis except the hypotheses that qualia, or the being experiencing them, don't exist.</i>
Qualia and intentionality aren't actually mysterious except under reductionism. Saying "this is a mystery" is (a) a cop-out, (b) begging the question, and (c) contrary to our usual way of evaluating theories, i.e., that a theory which explains the world is better than a theory that contracits it.
<i>No. Introspection is explicitly excluded from counting as "observation" when we're talking about empiricist epistemology. This is not my opinion; this is standard usage. Recall behaviorism.</i>
We have far more immediate, detailed, and intimate access to the workings of our own minds than we do to anything outside them; if empiricist epistemology doesn't let us conclude anything based on introspection, then that's a reason to reject empiricism, not to reject introspection.
<i>I'm not saying that causality isn't real; I'm saying that it isn't observable, and it isn't necessary to postulate it in order to reason and act as we do. This is important not in everyday life, but only to justify empiricist epistemology over spiritualist epistemology, because if it were true that we must be able to detect causality in order to reason, that would be a blow against empiricism, because causality isn't observable, as Hume argued.</i>
Strictly speaking, correlation isn't observable, either. We don't observe correlations; we observe events and objects.
I'm usually the first person to accuse anybody of Platonism, but in this case I disagree with the charge. I think "intelligence" is similar to "computational power".
A computer can have great computational power by some standard measure, and yet be too specialized or too much of a pain to program for most applications. Think of graphics cards that can do lots of FLOPS, but only on tasks that can be parallelized in a very specific way. One could argue that AI is like this, if the future of AI were simply the series GPT5, GPT6, GPT7...
But AI isn't a single algorithm. The improvement of AI is due to improvements in algorithms and increases in computational power. So it isn't analogous to putting more and more GPUs on a graphics card; it's analogous to the ability to put more and more transistors on a chip. (And the "increases in computational power" half of it literally IS the ability to put more and more transistors on a chip.)
Putting more transistors onto a chip doesn't imply that that particular chip has more general-computational power than some other chip with fewer transistors. But it does clearly let the community of chip-designers make chips with more computational power than the current generation of chips, even though "computational power" isn't clearly defined.
Then any sort of silicon-based AGI might be very far away, because as far as we can tell a human brain is just an enormously powerful machine. Each biological neuron is the approximate equivalent of a 5-8 layer, 1000 neuron artificial neural network (see: https://www.sciencedirect.com/science/article/pii/S0896627321005018 - although arguments have been made as to the paper's methodology and conclusions), and we've got 86 billion of them all linked together. That's... just a lot of potential processing power, however you slice it.
Yes, although I would probably quibble with that article if I read it, because I expect that 4 orders of magnitude of computational power would give better returns if we spent it on more artificial neurons, rather than on precisely matching the input-output function of artificial neurons to real neurons.
We should also consider that two humans with the same hardware often differ by what seem to me like not just orders of magnitude of computational power, but more like the difference between a linear bounded automaton and a Turing machine. That is, there are many problems--I would say most contentious political and economic problems--which a standard human brain couldn't solve even given infinite compute time, which a more-educated human brain plus a few sheets of paper might be able to solve if relieved of some bad metaphysical assumptions like logocentrism, and given a few tools like probability theory and statistical models.
If you accept that, it means that a few new algorithms which we already know, added to a human brain, push it into a different computational complexity class. And /that/ would mean that an AI with those algorithms can out-perform a human on such tasks even with much less computational power.
Another interesting test to add to the concept layer is if you can communicate it to another world-modeler and have them understand what you mean, or use it in a way that is predictably consistent with the way you meant it to be more rigorous.
I do see intelligence on terms of prediction. We are all moving through time, trying to guess what’s next, and intelligence is the ability to be good at figuring out what’s going to happen next and figure out what you should do to move to a preferred future.
I think your definition of blob of compute is slightly wrong. But only slightly. It’s like we are in an attic with holes in the roof and we blow dust to illuminate sunbeams. So you have to kind of think about what sunbeams are useful. Predictive text was something I thought would be fruitful prior to openai even because I think “what do you mean?” is one of those useful sunbeam clusters. The thing we think about for humans in particular is the ability to model another world modelers futures and you have to have it for communication.
Yeah, you can get pretty cute with definitions (it's like 90% of legal practice), but it can be tricky to know when you're on to something or just being annoying. Or to be more precise, you can introduce unfalsifiability to almost any argument by futzing with definitions enough.
But I do think this is where the AI Doomer error is. Not that you can't bundle a bunch of stuff and call it intelligence, but that some of the stuff you're putting in the bundle doesn't belong. I'm very intelligent (according to lots of biased people at least) but still struggle with agency and executive function. A machine can almost certainly be built that can answer every question on every test, but that doesn't model the world or have any true agency.
90% of legal practice is meetings, email and admin. Of the remaining 10%, 90% is reading up on legislation, regulations and case law, and then applying principles of legal construction to come up for some argument, any argument at all for why the client is right given the carefully-selected set of facts that they have presented you with (note: clients lie and elide constantly to their legal counsel). The remaining sliver (at least, if you're in a practice which involves contract law) is cunning legal strategies which involve clever word-interpretation, clause-writing and the use of definitions. These will, if they ever become the focus of a judicial decision, then be rolled over by a judge in court applying a reasonable-man test.
I did practice law for like a decade. But it might be more accurate to say something like 90% of legal disputes are about definitions. You're right that lawyers don't spend most of their time trying to figure out how to play definition hide-and-seek.
I also don't think "clever word interpretation" means the same thing that I'm saying here (haha!). Definitional disputes don't have to be unreasonable. They usually are not. Whether or not someone is a contractor, whether someone's actions were "intentional," whether certain contract terms were fulfilled - these are genuine disputes, generally considered valid arguments in court, that are still ultimately caused by motivated reasoning and desire to win the argument.
Motte: a machine _can_ be superintelligent without agency
Bailey: This machine _can't_ have agency
(If can't is too strong, replace with isn't likely, etc)
Yeah, I considered this and decided not to go there to avoid writing a book. But my point was just "you have to account for the agency part separately from the intelligence part." Once you get there, my obvious question is "why is a community that prides itself in logical thought placing the burden of proof on this extreme edge case *not* happening?"
There are strong counterarguments here - agency is highly correlated with intelligence in the only real test cases we have! Life! But so are things that are completely irrelevant to machines, like having lungs. Evolved life has strong incentive to develop agency. Intelligent machines can only have the incentives we give them, and the tools to act on those incentives that we give them.
It's an equally strong argument to say "machines will eventually develop lungs as they get more skilled with language and mathematics. After all, every intelligent being we've encountered eventually develops lungs." This is a tad strawman-ish but the underlying point is strong I think - without proposing a clear mechanism, grounded in real life observation, for how the danger will develop, it's hard to take it seriously.
Fwiw, I agree with you. Agency is a completely different characteristic from intelligence. This does seem like an important distinction that not enough people are making. However, it doesn't seem to be as hard a problem as intelligence. It was almost, or perhaps even, the first thing that was a characteristic of life - seeking to reproduce. Hence all life and evolution itself seems to depend on it. So I'm not sure if this is a strong objection - i.e , once intelligence is solved, it may not be that difficult to graft on agency. Computer viruses that seek to reproduce already exist. Not far fetched to think you could have a computer virus that comes with evolution and intelligence built in.
There are several other objections to doomerism of course.
People are definitely not ignoring the question of agency. This is one of the most rehashed points in AI circles.
Is your point that agency is just unlikely to happen by chance? Then read about instrumental convergence.
Still not convinced? Then how about the N variations of Agent-GPT where days after a model comes out you'll have a horde of people trying to give it agency.
I've read those and they're all of the form: "Assume this wild hypothetical. Given enough time it'll happen. Therefore it'll happen." Which is great but why limit to AI? Every time a biologist breeds a bacteria for experimentation there's a chance of population-destroying mutation. Every time a physicist runs a particle accelerator there's a chance that some unknown cataclysm will occur. This logic dictates that we should have destroyed the world many times over but...we haven't? Maybe our threshold for "a small chance repeated = a big chance" is much too low.
Instrumental convergence has nothing to do with "a small chance repeated = a big chance".
What do you actually disagree with here? You started talking about agency and it being unlikely to show up. I showed 2 of the most extremely common arguments for it. Instead of refuting it you just moved the conversation somewhere else?
You're just vaguely gesturing at AI extinction risk and saying: "ehh... I don't buy it"
It's okay not to be a doomer! But this is not twitter.
Ok then let me be explicit:
The original post is about how intelligence is a coherent bundle of characteristics that generally come packaged together, and so it is therefore reasonable to both lump them together and to be concerned about an intelligence explosion. I agree that "intelligence" is a coherent bundle of characteristics but want to treat agency separately, so this is not a strong argument for concern about an intelligence explosion. Even if a machine learning algorithm is "intelligent" in that it could do well on the SAT, I don't think that correlates necessarily to it opening the box with the steroids. That's Scott's argument and my disagreement. End of relevant content.
Bldysabba pointed out that "intelligence doesn't necessarily come bundled with agency" doesn't mean "doesn't come bundled with agency" which I think is a relevant objection worth addressing. But now we're dealing with a new standard of proof - I need a good reason to think that there's a small chance intelligence will come bundled with agency in the case of machine learning.
You pointed out, correctly, that this has been discussed at length and there are arguments that intelligence naturally evolves agency. We're far removed now from the post, but I responded, albeit briefly.
I don't think those arguments are very strong. Instrumental convergence is an interesting idea, but the leap from that to actual danger from an agent is extremely hypothetical/contingent. The same is true of Agent-GPT: It's true that many folks will try to make it an agent, and that it might display agent-like qualities, but IMO there are natural limits to its ability to do so.
You could look at this point back to Scott's earlier post where he claims that "we can't define the danger" and "there is no danger" are not the same concept. But we can't define the danger from many sources, and we don't make those a centerpiece of our civilization.
If you want an overarching thesis that disproves AI Doomerism it would be this: "There is no overarching thesis that disproves AI Doomerism." When people fall deep into a conspiracy theory (AI Doomerism is not a conspiracy theory but it has similar characteristics), they tend to build a strong coherent model of the world such that their theory is entirely unfalsifiable. It then becomes part of their identity and attacks on it become attacks on the people holding to the theory. And smarter people are actually better at this and harder to bring back from this brink.
That's not a disproof of AI Doomerism at all! Note that the same logic could be used to dismiss literally any worldview, period. But it does mean that if you disagree with one clearly defined point you'll be presented with broader and broader objections until you find one you can't or don't put in the energy to answer, and even if your original objection is valid, this will strengthen, not weaken, believers' resolve.
So to summarize: Absent a mechanism coupling them, intelligence is not automatically coupled with agency any more than having lungs is automatically coupled with agency. There are several individual arguments that there is a mechanism coupling them which need to be tackled on their own merits. But tackling these individual arguments leads to other arguments which must also be tackled on their own merits, and eventually you realize it's elephants all the way down.
> A machine can almost certainly be built that can answer every question on every test, but that doesn't model the world or have any true agency.
You can ask GPT-4 to write code for you, and then you can ask it to update the code based on your feedback. Perhaps it doesn't have a "model" for the code in some sense, but it is definitely more than just a huge database of memorized test answers.
The agency could be added as a plugin on top of the question-answering machine, for example like ChaosGPT.
ChatGPT-4 doesn't *exactly* memorize test answers. It's more like an algorithm for guessing the correct answer on any test, where the criteria for whether the algorithm works is whether it can reconstruct the answers for tests it's already been given. This isn't *quite* memorization, but it's close.
That doesn't prove anything one way or another by itself. We mostly pass tests by devising a method of being able to reconstruct or find information, and we also have agency! But I do think it's important to decouple "able to excel in mathematics and language" from "able to plan and execute on a plan." I kinda jokingly used myself as an example of someone who excels in math and language but can't plan to save my life, but this is strongly correlated in humans. I don't know that this means anything in the new context - it's strongly correlated in humans because we have evolutionary incentives to optimize both. I'm not sure machines have the same incentives.
> But I do think it's important to decouple "able to excel in mathematics and language" from "able to plan and execute on a plan."
The execution part can be provided externally. If I save the output provided by GPT as a script file and execute it (the script file may itself include more invocations of GPT and possibly execute their results, but it will also be able to "do things in the real world" such as send e-mails or maybe navigate a robotic body), the only thing that GPT needs to do is actually provide a good script.
So the question is whether "able to excel in math and writing" generalizes to also include "able to plan". In humans, I think the answer is obviously yes. You make a good point that this may be a fact about humans (or more generally, about evolved creatures), rather than about planning intrinsically. I still suspect the answer is yes.
I wanted to ask whether you think GPT-4 or the next generation thereof would be able to write a story involving planning. (Not necessarily realistic, could be Harry Potter planning to defeat Voldemort.) If not, then... it would mean that some kinds of stories will be outside its reach, no matter how much compute it gets. (Detective stories?) On the other hand, assuming that GPT-4 is potentially able to write any kind of story, including stories that include planning, then this means in can produce realistic plans, right? -- I suppose the obvious counter-argument is that it may be good at producing "plans" that seem realistic and work in a story, but would totally fail in real world (e.g. because they are based on tropes).
If ChatGPT wrote a detective story in the style of Arthur Conan Doyle with a *new* mystery, and the story, clues, and conclusion logically cohered, I'd probably revise my opinion tbh.
I suspect the tricky part there would be distinguishing between "*new* mystery" and "obscure ACD fanfic in GPT-N's training data".
I think you're conflating two fairly different ideas in this post. First idea: "intelligence" makes as much sense as "strength" and it is coherent even for a non-platonist to discuss the intelligence of machines or to imagine machines becoming much more intelligent than humans or going through an intelligence explosion. Second idea: a good way to increase the intelligence of AI is to simply scale up the computational power and not worry about having lots of clever or domain-specific ideas.
Of course, these ideas are related to some extent: the second idea doesn't really make sense without buying into the first idea. But they are still pretty distinct! It is totally possible to believe that talking about intelligence is coherent without believing that "stack moar layers" is a good approach to AI. As an analogy, one could buy into everything you say about strength but not believe that "build bigger engines" is the best way to create stronger machines.
I think you're correct that it's practical to roughly think of 'intelligence' as being a blob of stuff that, if you have enough of and arrange it appropriately, can create meaningful and general abstractions and make very accurate predictions. However, the 'doomer' argument requires more assumptions about the nature of this blob of intelligence: Namely, that a sufficiently sized blob will be inherently power-seeking. This also requires that it will act in the real word to optimise for a particular goal, and that the goal that will likely be programmed into it will be of a type that would cause irreversible damage if sufficiently optimised.
The doomer scenario doesn't actually require the blob to be "inherently power-seeking" in the sense that the blob "wants" of its own accord to do anything. All it takes is the blob inadvertently pursuing sub-goals that end up being detrimental to human well-being, which were developed to pursue some ultimate goal set by a human.
We already have people trying to get ChatGPT to achieve goals that take intermediate steps not fully specified in advance. Add in some more intellectual firepower and ability to affect the material world and we're well on our way to that particular risk (presumably we'll end up with some unforeseen consequences short of X-risk as capabilities increase).
Most of the attempts at distilling the doomer argument I've seen focus heavily on the dangers of AI being power-seeking. As far I as can tell, while AI pursuing sub-goals that are detrimental to human well-being is a big concern, it is not nearly as likely to be civilisation-ending if the AI is not power-seeking. For example, the paperclip-maximising AI will probably not end the world if it does not actually try to kill everyone with nanobots and turn the world into paperclips.
As an aside, I'm aware that this means that the AI would not be the optimal paperclip maximiser, but it's reasonable to raise the question of whether such an AI can be developed in practice, or would be the default. We have our abstract blob of intelligence, but whether this leads naturally to something with power-seeking behaviour is, I think, far from determined.
The main point doomers are concerned about is that developing an alien synthetic intelligence that seems like it will surpass our own capabilities in key dimensions without necessarily being controllable means that we are at risk from any number of unimaginable-in-advance scenarios leading to catastrophe and even extinction.
"It won't be power-seeking" okay how sure are you and will your particular sense of that phrase actually be a meaningful issue?
"It won't be civilizational ending" ok but it might lead to really disastrous consequences short of that particular standard?
Powerful AI under the control of malicious humans is a very big problem well before we worry about the AI itself going off and wrecking things inadvertently or purposefully (and solving the former would contribute to solving the latter along many dimensions).
The fact that things are "far from determined" is part of the risk model we face, yes. All it takes to be a little bit of a doomer is avoiding the trite and blind optimism of so many who seem to think there's no real risk anything could go seriously wrong.
Perhaps we have different definitions of "doomer". I'm taking it to refer to the belief that AI is going to be civilisation-ending (or perhaps cause irreparable damage to civilisation), as opposed to 'merely' being concerned about disastrous consequences.
Once you’ve fully internalized all the other risks it’s not much of a jump to full doomer status due to the risks of speed and scale AI is likely to obtain given no biological limits.
Once you grant “huh maybe we can’t rule the doomer scenarios out a priori” you’re basically there.
"However, the 'doomer' argument requires more assumptions about the nature of this blob of intelligence: Namely, that a sufficiently sized blob will be inherently power-seeking."
I wouldn't describe it that way. The way I would describe it is that humans will probably program an AI to do *something*, because it would be useless to us if it sat around doing nothing. If we program it to something other than what we want, we will come into conflict. Programming it to do *exactly* what we want is very hard, as evidenced not just by thought experiments but by how hard it is to make LLMs behave (eg not make up citations).
I agree that we will come into conflict if we program it to do something other than what we want, and that this will certainly (obviously) happen, but isn't it only civilisation-ending if it does a very *particular* class of things we don't want (that is, kills everyone or wipes out human value in some other way)? As I understand it, the doomer position is that this class of things will be the default, specifically because a sufficiently rational intelligence would be power-seeking.
I think the claim is:
- Suppose we program it to do something we don't want, like make paperclips
- Now it wants to make paperclips
- You can make more paperclips if you get power than if you don't get power
- Now it's power-seeking!
I don't think it will necessarily be this simple. But I also don't think it necessarily *won't* be that simple either, or not have even a little of this process.
When I said "not inherently power-seeking", I just meant it will be inherently paper-clip-making, and only secondarily power-seeking. It sounds like our real crux is whether a process like this can happen and result in power-seeking as a secondary goal.
Yes, I think that's correct. I haven't figured out how confident to be on whether we should expect a process like that to happen.
Because you've never written a computer program that accidentally used up 100% of system resources doing something dumb that you told it to do, but you really swore up and down before you ran it was not what you had told it to do?
Maybe I'm an exceptionally unlucky programmer, but somehow I do that all the time.
And maybe I just hang out with similarly unlucky programmers, because pretty much every computer engineer I know does that all the time, too.
I struggle to think why you wouldn't expect that process with say 10x the probability of anything else? Do you know a different set of programmers than I do?
I absolutely expect any future experimental program will have bugs at some point, and end up doing things other than what the programmer told them to do. What I don't know how much we should expect, is whether future AI will even have the ability to follow an arbitrary goal to such a level of effectiveness that it will exhibit power-seeking behaviour (or at least, whether this is the default behaviour of AI we can practically build).
I'll point out that I haven't actually given any reasoning (yet) for why I'm not fully convinced, I'm just pointing out that Scott's argument here that 'intelligence' is a useful concept in practice is not enough to be convinced of doom, because there is still a nontrivial claim that needs to be made about the nature of this intelligence.
Although I guess I'm not actually disagreeing with anything Scott says in the article, since the conclusion is "This only suggests that an intelligence explosion is coherent, not that it will actually happen."
If the AI starts to take up too many resources in the real world I'm sure we will notice.
Then the the question is, will the AI also resist getting turned off. And will it be able to resist? Will it see being turned off as a threat in advance, and then protect itself?
If the AI is somewhat like a human intelligence turned up to 11x10^11 (but without the alignment) then I think the answer is yes. But there is no evidence that AI will be anything like that. I'm not sure that GPT-9 on auto would be anything like that.
Have you ever had a program recognize that it needs more power to do what you programmed it to do, and then start stealing power from other systems? Or stopping other programs from running in order to gain access to their resources?
I'm also not saying that this is impossible, but it seems to me just as likely that an AI program will run into a hard problem and then just work on it extremely slowly forever, like your program that maxes out the computational power available to it and then does nothing different. What does time matter to an AI? Making 100 paperclips a day will also (theoretically) turn everything into paperclips as well. A paperclip maximizer may just stick with the original capacity given to it by humans and happily chug along making those paperclips because it doesn't, and can't, "care" about making more.
The next step, where humans tell it to take steps to increase paperclip production, may also run into the same problem. The AI takes the obvious and reasonable steps to make 200 paperclips a day - goal achieved!
I disagree that it is especially hard to make LLMs behave…. OpenAI seems to have gotten them to behave pretty okay. Actually, they seem to understand common sense and follow human instructions at a level some doomers previously thought implausible - “sorcerer’s apprentice” style failures now seem very implausible.
See https://www.lesswrong.com/posts/3YqBGnRLtHDGHLgSb/can-we-evaluate-the-tool-versus-agent-agi-prediction?commentId=t7zunpymPopH63CSZ
I think this post is basically right that "intelligence" is a meaningful thing to talk about, in much the same way that "strength" is a coherent thing to talk about. That said, it matters a lot why exactly you're asking the question. If you're asking "who is stronger, Mike Tyson or Granny" as a proxy for the real question "who could move this large pile of dirt faster", then it matters a lot if Granny owns an excavator (and, for that matter, whether Mike Tyson has or can rent one).
As far as I can tell, the arguments for an intelligence explosion rest on the assumption that "how much stuff can an entity do" depends mostly on the question of "how intelligent is that agent". But that doesn't seem like the whole story, or even most of the story. Increases in human capabilities seem mostly tied to our ability to use tools, and our ability to coordinate multiple people to accomplish a task. For example, the populations of the Americas diverged from the old world thousands of years before either group developed agriculture, and both groups developed cities and civilizations relatively quickly after developing agriculture. So it clearly wasn't a thing where there was a threshold in intelligence where agriculture and civilization suddenly became possible.
In order to get the "FOOM" flavor of intelligence explosion, where a single entity reaches a threshold level of intelligence such that it can inspect and improve its own operation, which allows it to better inspect and improve its own operation, etc and become smart enough that everyone else combines is as insects, one of the following would have to be true
1. Actually, having and knowing how to use tools doesn't matter that much. Beyond a certain level, you can predict the results of your actions well enough that all you have to do is whisper to a butterfly, and it will flap its wings in just the right way that matter ends up in your preferred configuration 6 months later.
2. Having and using tools is important, but being smarter lets you build and use better tools, and at a certain (fairly low) threshold, you get better at making and using tools that everyone else combined.
3. Having and using tools is important, and the future will belong to whatever agent figures out how to use the tools other agents have built in order to improve its own tools and processes, and does not share any of its own advancements back, which acts as a ratchet (that agent only ever gets more capable relative to the rest of the world, never less) and eventually results in an agent that is more capable than everyone else combined.
Now, of course, a FOOM-style intelligence explosion isn't the only possible way to end up with a cycle of capability advancement. But it is the way that ends up with the most worrying dynamics like "there is an intelligence threshold for AI such that the first AI to cross that threshold will determine the shape of the future, and so to survive humans must build that AI exactly correctly on the first try without being able to iteratively test it first".
"If you're asking "who is stronger, Mike Tyson or Granny" as a proxy for the real question "who could move this large pile of dirt faster", then it matters a lot if Granny owns an excavator (and, for that matter, whether Mike Tyson has or can rent one)."
John Henry was a steel driving man 😁
https://www.youtube.com/watch?v=ydTRk1l0ZqI
I think your point about tools is a great point, and it is striking that people thinking about AI doom do not ask themselves questions like "historically what accounted for major increases in human capacity?"
They also are not asking themselves what usually happens when a more intelligent species shares its habitat with less intelligent ones. Read an article this morning saying that our species has rendered many less intelligent ones extinct, but we are an exception. Much more commonly the more intelligent speciess coexist with the less intelligent ones and do not drive them into extinction.
They also are not asking themselves whether the natural world offers any useful models of ways a dumber and weaker creature can be safe from attack by a smarter and stronger one. Any models of this kind are alternatives to "alignment" via the model of somehow *making* the AI treat us well. And the world in fact offers many models: Parents rarely harm their weaker, dumber babies. Hippos welcome little birds onto their back because the birds eat the hippo equivalent of fleas. Certain species of butterfly stay safe from predation from birds by closely resembling another butterfly that's toxic to the birds.
So overall it seems to me that the people who are convinced it's going to come down to a fight to the death for resources between us and AI are in the thrall of a single one out of all the many possible ways of thinking about how things between our species and superintelligent AI (if it ever comes to exist) would play out. They're stuck in a Shootout at the OK Coral model.
I get all that. But it still seems to me that there are differences in kind, not just differences in degree, between GPT4 and something smart enough to be a real problem to us if it's not aligned. The really important differences between GPT4 and human level AI & /or superintelligent AI seem to be differences in the *kinds* of processes they can engage in, rather than in how well each can engage in the process. GPT4 currently is unable to recall its recent activities or recognize people who have prompted it before. Within the process of responding to a prompt it is unable to review what it has said so far, in order to orient itself for future action with the benefit of past action. All of these incapacities are deficits in what we would call self-awareness. GPT4 also shows no signs of being able to reflect about itself or anything else when not prompted, and no signs of having preferences and goals. It is also unable to give itself prompts that are generated by basic needs to sustain and protect itself (such internally-generated prompts are called *drives* when they occur in animals). In fact, it is unable to produce self-generated prompts at all. Moreover, it seems to utterly lack self-interest. GPT4 is essentially in a coma except when executing a prompt given by a human being. It also seems to be rather poor at reasoning, and very lacking in inventiveness, and in some of the capacities on which inventiveness rests, such as recognizing subtle but important isomorphisms. It is also not teachable in the usual sense. You can't give it give it an essay about, say, the reasons that intelligence is a useful and meaningful concept and expect it to be able to perceptively apply the ideas in the essay to other concepts, such as kinkiness, snobbishness, musical ability, beauty, malignancy. GPT4 will have learned about ways to talk about concepts, rather than ways to think about them.
Most of the capacities I named above as capacities lacking in GPT4 are not lacking in animals or in Einstein, who had self-awareness, self-interest, personal memory, drives and preferences, and teachability about their environment (as opposed to teachabilitu about the word sequences humans use to discuss their environment). So it makes sense to compare the intelligence of all of these beings ,each of whom *has* the same cluster of capacities, though some exercise that capacity at a more sophisticated level than others. I'm not sure, though, how much sense it makes to compare the intelligence of, say, GPT4, which lacks many of these capacities, with human intelligence. It would be like comparing the strength of Arnold Schwarzenegger with that of somebody who has no arms.
So I think the guy in the tweets has a point regarding intelligence not exactly being a *thing*. Most of the crucial functions I named are present in all species, and that makes it make sense to compare them in intelligence with each other, with us, and with Einstein. But GPT4 (and, I'm guessing, later versions of GPT that have similar architecture) performs quite well at some functions, but is utterly lacking in other crucial functions. So I am not sure how meaningful it is to talk about how intelligent AI is that lacks self-awareness, self-interest, drives, goals, personal memory and the ability to learn from experience (rather than from ML type trainings.).
"I get all that. But it still seems to me that there are differences in kind, not just differences in degree, between GPT4 and something smart enough to be a real problem to us if it's not aligned. The really important differences between GPT4 and human level AI & /or superintelligent AI seem to be differences in the *kinds* of processes they can engage in, rather than in how well each can engage in the process. "
See the section marked "Mini-Scenario 1: AutoGPT Vs. ChaosGPT" at https://astralcodexten.substack.com/p/tales-of-takeover-in-ccf-world, especially the paragraph that starts "AutoGPT isn’t interesting because it’s good...It’s interesting because it cuts through the mystical way some people use “agency”"
Yes I understand what you're saying there, Scott: We can give AI a goal, and not just a circumscribed goal to complete a single task, but a goal of working continuously on a big long-term goal, making decisions itself about means of pursuing the goal, and continuously engaging in the OODA loop it is programmed to follow. And you are saying that an AI in the midst of following a complex prompt like that, with an OODA loop in it, is a concrete example of agency. Contemplating this example cuts through all the fuzzy thinking about how AI doesn't have an inner life and wants and preferences so why should we worry about it ever being a goal-directed agent. Have I got it?
OK, I just had an extremely long exchange about this exact issue with Daniel Kokotajlo and Bartosz Zielinski on the Extinction Tournament threat. I hope it is not bad manners for me to quote some of what I said there, because it took me a long time and a lot of writing to express my objection to this idea clearly, and I don't want to start from scratch again, esp. since it's 2 am in Boston!
OK, my main point is that AI’s drive to reach the goal of ChaosGPT —- wreck everything — or another similarly large goal is that the goal is inserted into the AI by a human being. So, yes, we can think of Chaos GPT as having a long term goal to wreck everything, and being motivated to do so, and it doesn’t matter that it does not have thoughts and feelings about wanting to create chaos — it will behave exactly the way an agent that did would. EXCEPT that there are 2 extremely important differences between a human being who wants to create chaos and Chaos GPT. The first is that the human being has a metagoal: never give up on the goal of creating chaos, and fight back against all attempts to prevent the creation of chaos. Chaos GPT does not have the metagoal. If the asshole that set ChaosGPT going now sets GPT on another quest — let’s say to search all the online catalogs to find the a pair of size 14 red leather pants for his girlfriend, and to bargain or cheat stores if necessary to keep the price below $200 — GPT will do it. And then it will sit there humming til told to do something else.
The second difference between Chaos GPT and a person with a goal of creating chaos on earth is in the depth of structure of the drives the 2 entities have . Let’s compare ChaosGOT to animals. All animals seem to have the equivalent of OODA loops going on regarding the goal of getting food. But an animal’s drive for food comes from signals from many networked parts of the animal, and this network extends all the way down to individual cells that are signaling that they need fuel — and it extends all the way up to various complex instinctive behaviors that are prepackaged in the animal at birth, such as the stalking that cats do. The AI’s drive to CreateChaos or GetSmarter has no roots. The difference between an animal drive and an AI prompt-based drive is like the difference between a tree and a telephone pole. The tree is far far harder to remove from the ground in its entirety, and some of the small roots deep underground cannot be removed because they break off if you uproot the tree. And some trees can regenerate from roots that remain. To put it simply, The AI’s drive to create chaos was not internally generated. The animal’s drive to eat was.
And that difference between the animal’s drive to eat and Chaos GPT’s drive to create chaos has real world implications: One, as I said, Chaos GPT will willingly accept a completely different goal; a hungry animal will not. Two, let’s say Chaos GPT buys the red leather pants and the cat eats a vole and is satisfied. Now we just leave them both alone and wait. Chaos/Red Leather Pants GPT will sit there humming forever. But the cat will become hungry again in in a few hours, and the prey-seeking OODA loop will be regenerated from within the animal.
Until AI has some internal equivalent of a goal-generator, it’s not an agent with goals in at all the same way as a living being is, and that difference is going to have very substantial practical consequences. If AI’s internal structure continues to be too simple and blank for it to generate goals from within, it is only a danger if a person is telling it to go kill us all, or round us up and turn us into paperclips or whatever. Or, of course, a boss AI could be telling it to do that stuff. But then you have to explain how the boss is structured that allows it to have internally generated goals.
Agreed!
Someone here once said that the difference between a machine and something alive is that the machine will just sit there and rust until you turn it on. My feelings about AI are similar, with the additional caveat that engineers are not in the habit of designing machines that can turn themselves on, and I don't see how AI will be different.
> with the additional caveat that engineers are not in the habit of designing machines that can turn themselves on, and I don't see how AI will be different.
They will be different at the very least because they will be designed and deployed to not turn off, ie. always available.
As far as I can tell, the drives you are pointing to are not a big barrier. It's really easy to wrap any functional system in a layer of "now do this thing forever", and AutoGPT is a toy example of just how easy it is. Scott seems to be worried that when GPT-4 in such a wrapper is replaced by GPT-7, then bad things will happen. Charles Stross explored this in Rule 34 (the AI in that book isn't an AI, just an optimizer that's self-modified over time). I don't see what you are suggesting is going to stop such systems from being made or being deployed.
You're right, somebody could add another layer, a metagoal: *Never stop doing this.* In fact, we could make it *never stop doing this even if we tell you to.*. But again, compare the AI with the *never stop* metagoal to animals. They all have "never stop doing this"wrapped around all their main drives." *Get adequate nourishment* is one. But for the animal, the metagoal of *never stop seeking nourishment* is deeply embedded and networked with other deep drives, including *fight to the death in order to survive.*
The AI's *never stop even if we tell you to* prompt is to the animal's *fight to the death to survive* as a telephone pole is to a living tree. The former is easy to remove. I doubt that it is even possible to give AI a command that cannot later be overridden by the person who gave the *never stop* metaprompt. But even if it were, I'm sure there are many many access points into the AI system that a knowledgeable person could use to shut down its activity.
There would also be many places in the mechanism on which AI runs that are not part of the AI per se but just electrical components. AI would have not info about these or control over them, & stopping AI's activity at the mechanical level is also possible.
And regarding the fact that AI has no knowledge of and control over it's "body," i.e. its electrical components, note how different it is from animals. Animals will defend against an attack on any part of their bodies, because they are infinitely more integrated beings that AI. Their self-awareness and self-interest and determination to defend themselves and read their goals extends to every part of them. It is wired into them in a profound way. AI's ever-persisting efforts to carry out a goal we have given it is such a shallow version of agency that I don't think agency is even the right word for the underpinnings of its persistence.
You can get a Roomba and a tube of epoxy, glue the controls in the "on" position and the battery compartment shut, glue poison-tipped spikes all over it and turn it loose at a cocktail party. But you won't have created an equivalent of either The Terminator or a genius hedgehog.
> GPT4 currently is unable to recall its recent activities or recognize people who have prompted it before.
Yeah, this is a real problem. There are hacky solutions to this but no elegant solution. No real episodic memory at all. Basically the only solution proposed is have a longer context length and have it write to a memory and read from it. But then it's still context limited. This feels to me like a hacky workaround, not a real solution.
That's interesting. I don't really know what you mean by "it's still context limited." What would not being context-limited look like?
When it reads words (or wordpieces) a transformer based model like ChatGPT compares each word to every other word in a given range known as the context window. So if you feed it a sentence of 7 words, it makes a 7x7 comparison. That scales quadratically, so it can't increase forever. People keep finding hacks to increase it, but at some point there needs to be a more elegant solution. Not being context limited would be like a human who goes to the library and reads until they find something useful. LLMs need to be able to read through the internet and find stuff to help them out. Some people are adding tools to them so that they can do that, which might help. But also humans have episodic memory. LLMs do not at the moment. They need to be able to write/read from memory.
"Across different humans skill at different kinds of intellectual tasks are correlated."
That says precious little about whether skill at all different kinds of intellectual tasks is correlated anywhere else.
I can pretty easily imagine an AI that's good at producing stories - not at all good at much of anything else, including judging whether stories are true. It's called ChatGPT.
I can also imagine a robot that's strong in some ways (for some tasks) and flat out incapable of others.
Even something like height and weight - much more strongly correlated for humans than for factories (which can be very wide while being short, or fairly tall without being that wide, while humans don’t vary that much in ratio between these dimensions).
YUP!
+1
I think it is correlated places other than humans - for example across animal species, or across LLMs.
I also disagree that GPT can tell stories and do nothing else - see my response at https://astralcodexten.substack.com/p/were-not-platonists-weve-just-learned/comment/21278512
The MIRI decision theory that they think is key to alignment does presuppose a kind of Platonism - the best way to understand it is that one conceives of making the decision on behalf of an *algorithm* rather than a particular physical instantiating of it, and see which output for this algorithm being universalized would result in the best outcome. (Though maybe there’s another way to phrase it - it’s less clear to me that Kant’s formulation is committed to Platonism.)
I think it was legendary baseball author/stat-head Bill James who told me [completely paraphrasing]
-- Good Science, good arguments, give you lots of STUFF TO DO. Especially PREDICTIONS. What does the theory predict? Well, let's run an experiment, test the hypothesis, and measure the results, see if they agree with the predictions.
I think arguments and theories that are largely proven and backed up by evidence, are valuable because they provide a deeper physical framework and view that you can use to make additional predictions about the [system you are studying].
Here is where I say that Tyson is "strong" and that tells us some likely correlations about his general physical health, metabolism, quickness/neuromuscular co-ordination, etc. If the value of science is correlation and prediction, well of course LLms trained on the corpus of human speech will make startling and "intuitive" correlations. But human thought has, maybe, no underlying physical basis or like, quantum-reality.
The corpus of human speech is "true" but is it meaningful? Hmmm...
__________________
Here is where I swerve off the road because I realized in the middle of this that Mike Tyson is almost a perfect small model for transformative unaligned world-changing AI
__________________
We know, from some of the most direct, irrefutable, and culturally transformative video and human-reported evidence of my LIFETIME, that Mike Tyson is strong. (1) His unprecedented ability to, almost instantly, beat another [huge and strong] human being senseless,
---- well actually it's like the GWERN "Clippy" story, he was so otherworldly dominant --- Holy crap!
Tyson is strong, and was AI-level destructive, because he was
-- born with genetic gifts
-- Trained relentlessly
-- Was fed an incredibly difficult and skewed training dataset that had at its premise that his only value as a human being, aka his "reward matrix", was to maximize the beating other human beings senseless.
-- Had, in addition to his physical gifts, a superior target evaluation model that discarded irrelevant info and focused on the target loop, leading to incredibly quick gains and vast mismatches in learning and decision speed.
-- I MEAN, HAVE YOU EVER SEEN HIS FIRST 20 FIGHTS?
-- Also, it seems, had an incredibly strong and relilient self-model that has somehow recovered from disasters and physical truncation and continues to contribute and be studied to this day.
I'm gonna wrap up this comment while I ponder the ways that Tyson was "invisible" in 1980-84, a terrifying rumor spoken of in awe by the few who had seen toe Miracles.
BR
To provide some background....
I'm gonna speculate that *not* many readers of this blog were somewhat sports-obsessed kids and teenagers in the early 1980's. Maybe some of you were culture, or film, or music-obsessed kids back then, but for all the rest, let me tell you what it was like:
MYSTERIOUS
This was way before the internet, long before ubiquitious coverage and total information about anything had made it all equally worthless. Stories in major newspapers, highlights on local TV on Sunday night at 10:30, the occasional Network TV mention, were most of it. I was so cool I had subscriptions to "Rolling Stone" and "The Sporting News". I understood about 30% of RS, about 80% of TSN.
Mike Tyson was a legend spoken of in whispers. Boxing was at an incredibly low ebb, and Tyson was fighting in hockey rinks in upstate NY, and video footage was courtesy of the local HS AV club. ESPN existed at the time but Austrailian Rules Football [shout out to my Fitzroy Lions!] got more airtime than boxing.
You had to be a hard-core sports gnurd to know about Tyson. I remember being at an NFL football game with my parents and my friend Ron F, hard-core sports gnurd, and he was looking at the monochrome "crawl" on the stadium scoreboard at halftime:
"Hey B! See that! "Tyson wins by Knockout, 53 SECONDS!" Have you seen this guy!?!?"
A comparison can be made to Herschel Walker, Who was a titanic force only glimpsed in blurry 7-second clips of him running over SEC defenders for another 72-yard touchdown, or Darryl Strawberry, the tall skinny kid on the Mets who always seemed to be hurt, but hit the longest home run I have ever witnessed one September night in 1985.
Anyway, little can describe the shocking other-ness of Tyson, a 16-year-old behemoth, knocking out much larger men with two lightning punches.
That is all.
"How can any cognitive capability be self-amplifying?" asks man on rage spiral generation app.
I award you three internet LOLpoints.
There's a big difference between the concept of intelligence and the concept of strength. We know of other animals that are stronger than humans. Therefore, we can be certain that we are using a concept that generalizes across humans and non-humans in the case of strength. But we only define intelligence with regards to human capabilities. At least as far as we can prove.
Can anyone tell me what is the way in which humans are not smarter than chimps? I don't have time to read the full book review right now.
If Elon Musk were transformed into the body of a wild chimp tomorrow but kept his current brain he wouldn't survive long.
If Elon Musk was stripped naked and placed in a jungle, with no way of reaching society, he would not survive long either - so, this does not explain in what way the chimp is smarter.
I think the intended point was that chimp brains (and not just chimp bodies) are better at doing some chimp-survival-relevant things. Smart man in jungle might die due to physical weakness/vulnerability, even if he makes good decisions. But if entity with smart man's brain and strong chimp body can't make it, that suggests the chimp brain is better than the human brain in some ways.
I agree that it's not enough to prove/explain much. Maybe it's just a reminder that the scope of our concept 'intelligence' is a bit arbitrary and very human-centered.
Chimps have better short term memory.
Here is how they do the memory test, look and despair:
https://www.youtube.com/watch?v=nTgeLEWr614
For the Platonists out there, here’s some grounded discussion on whether AI can meaningfully engage in the noetic world of forms and ideas:
https://aixd.substack.com/p/ai-might-not-need-experience-to-understand
"g" is straight-up Platonism. Don't be defensive about it, a Platonist is a perfectly respectable thing to be. You could also put it in Kantian terms: g is the noumenal behind the phenomenal forms we call "witty lyricist" or "good at crossword puzzles". Platonist is not a dirty word!
That doesn't mean I agree with you.
Is anyone not a Platonist, then? If so, what's special about the 'g' concept that distinguishes it from all the abstract concepts that ~everyone uses?
I think most people are essentialists by habit, so I think most people probably are implicitly Platonists. I think rejecting Platonism probably requires you to take enough time to think about it carefully and almost nobody does that.
I guess I'm not entirely convinced that Platonism is even meaningful. But more concretely, why can't a nominalist (or other non-Platonist, if there are other kinds) make use of the concept 'g'?
If we take Wikipedia's definition, I don't see how g is laden with anything that implies Platonism, or indeed with anything that one could possibly reject other than the existence (yes I said 'existence', but this can be parsed in whatever way a non-Platonist would approve of) of some correlations:
"It is a variable that summarizes positive correlations among different cognitive tasks, reflecting the fact that an individual's performance on one type of cognitive task tends to be comparable to that person's performance on other kinds of cognitive tasks. "
Even if we give it a bit more substance, what does 'g' commit us to beyond the belief that it is likely those correlations show up because different kinds of intelligence have common causes?
> superintelligent machines are no philosophically different than machines that are very very big
Agreed. Problem is, in order do design a big machine usually you cannot just scale a small one. A switch which can break thousands of Volts is not a scaled up wall light switch: it is designed radically differently (e.g., the blades have to be usually submerged in oil, not the air, to prevent arcing). Also scaling comes with its own tradeoffs: It would be extremaly difficult (if at all possible) to design human sized robot as agile as a mouse.
1. African elephant 257 000 000 000
2. Human 86 000 000 000
If all intelligence takes is a higher neuron count, training data and time, why aren't elephants three times as smart as humans? Why are they actually vastly dumber than humans? Why was Einstein much smarter than most people despite having pretty much the same number of neurons, training data and time? You can't explain this as the correlation being less than 1.
I'm agnostic as to whether an intelligence explosion is possible. But I'm 100% certain that it won't happen in anything deep learning based. Deep learning is not capable of extrapolating beyond either what's in the training data, or in the simulator that produces training data (in the case of board games). People keep confusing interpolation in a vast database for intelligence because humans can't do it, so they have no reference point, and intelligence is the only other thing that can produce the same effects.
From another comment, someone noted a single cortical neuron is roughly equivalent to a 1000-node ~6-layer artificial neural net. So there is some multiplier there. What if every human neuron is like that? What if every elephant neuron is more like a 50-node 3-layer neural net? All this is completely speculative, but I could see a beast with larger number of neurons not being as intelligent.
You could make any number of variations on the theory to make it fit, because it's a bad theory. It doesn't explain how assemblies of neurons represent concepts, learn or plan. We don't have a full account of that yet, but we knew enough to falsify the "blob" theory decades ago. There are mountains of evidence for innate modules in humans and other animals, and likewise against blank slate theories. Everything from learning to swallow food, walk, talk, learning from others, to competition for status and attracting mates. All of that is strongly influenced by mental modules that have evolved specifically in humans, and other modules exist in other animals. A human baby raised by a chimp mother will not be an incredible chimp due to its superior neuron count. It has different software, different learning algorithms that are primed for different sensory input. Human brains across all cultures and times don't coincidentally develop the same anatomical and functional organization. It's genetically encoded.
As someone who finds the module theory in humans quite evidently true, I do not believe that this is an argument against anything in the post. Yes, humans have many specialized modules (language, two visual recognition ones, presumably social networking, to name a few), just like, say, migrating birds do. This should not obscure the fact that humans also have stronger general intelligence than any other species - in fact, a cognitive scientist can distinguish when modules (fast, automatic, barely regulatable) work and when what's usually called System 2 (slow, traceable, stoppable) works.
Regarding the original question, body size matters because, the bigger the body, the more neurons you require to process info from body and regulate its movements, so you need to subtract them - hence the encephalization quotient calculations and all that jazz.
Are you saying that a) general intelligence is a separate mechanism from modules, or are you saying that b) general intelligence arises from the interaction of modules (which is what I believe), but you don't see how that contradicts the post?
The evidence against a), and for the theory that general intelligence arises from and builds on innate modules, is the field of developmental psychology. See for example this talk: https://www.youtube.com/watch?v=NGMXUzr8MTM&list=PLZlVBTf7N6GpOCwMH-TIhwrUwV8YJMANM&index=8
Infants either already know, or learn from extremely few examples, about simple properties of objects, places, numbers, geometry, agents and causality. Abilities that directly build on and combine these emerge in predictable order and timing, and this process continues to compound throughout life. This solves the problem of where inductive biases for learning new skills come from.
This is incompatible with the post, which ignores this issue (or is confused about it, as CounterBlunder points out), and posits that a uniform, unbiased blob of neurons can learn anything, in humans as well as machines. This contradicts the bias-variance tradeoff, and ignores both the orders of magnitude difference in amount of training data used in language and vision in AI vs humans, the inductive biases in NN architecture and SGD, and the fact that deep learning and humans generalize in completely different ways - humans can generalize outside the training data distribution.
I didn't understand your point with System 1 and 2 - why are modules equated with system 1?
I know of both the theory "everything is modules, so-called generla intelligence is just interaction effects" (your b) and the theory "there are modules, and then there's general intelligence which does everything we don't have a special module for" (your a), and I claim a is right and b is wrong, based on, surprise-surprise, developmental psychology and module interaction properties. In particular, as a linguist, I can bring some examples of how our innate linguistic module works and how laypeople reason on language when they try to apply general intelligence to it - vastly different processes with vastly different results.
The system 1-system 2 point is directly relevant to it: modules are automatic, they cannot switch to system 2 mode (you cannot make your linguistic module make acceptability judgments slowly and with laying out the steps or make your visual recognition module explicitly correct for optical illusions, you have to take its output and modify it, which is different), while the general cognitive ability can. (And certain properties of that system 2 mode vote against "well, maybe your 'general cognitive ability' is just another module then".)
I think you don't understand what the generalization differences actually imply. We know, in broad strokes, how modularized abilities work, we know how general intelligence in humans works, and we know how the bitter-lesson-neuron-networks work. These are three different ways.
Can you explain in more detail? I don't know how innate modules or general intelligence in humans work, but I know how deep learning works. Deep learning works by interpolating training data in a learned latent space. It can't learn the kinds of abstractions that humans learn. Not everyone agrees, but as far as I can see, they either don't understand it, don't understand the implications, or their paycheck and prestige depends on them not thinking too much about it. But this rules out deep learning as a mechanism for cognitive abilities like perception, language and planning. These abilities clearly work despite constant distribution shift, and that's their whole point. We can recognize our surroundings in novel circumstances, learn new concepts and invent new things.
I'm not sure if that's what you're saying, but optical illusions reveal certain biases of our visual perception, and they don't go away even though we understand that they are illusions. To me, this shows exactly that innate modules serve as a basis for higher-order cognitive abilities, in numerous ways. They provide the initial concepts which we compose, alter and recombine into new concepts. We use them to construct analogies that allow us to transfer understanding of concepts from familiar domains to novel domains - I'm sure you're familiar with this as a linguist.
Maybe you're equating general intelligence with symbolic reasoning? But that clearly builds on language, spatial reasoning and causal reasoning. But you also say that it's not another module?
Scott has discussed this previously. Main article: https://slatestarcodex.com/2019/03/25/neurons-and-intelligence-a-birdbrained-perspective/
Update (not really relevant to the intelligence question, but if I'm linking the previous article I feel that I should link this too): https://slatestarcodex.com/2019/05/01/update-to-partial-retraction-of-animal-value-and-neuron-number/
Thanks for the link. We can substitute neuron count for cortical neuron count, or cortical neurons * neuron packing density * interneuronal distance * axonal conduction velocity.
Let's say this quantity correlates perfectly with intelligence. Does that show that it's necessary and sufficient for intelligence?
No. If you run a stupid algorithm on a more powerful computer, you get marginally better results. If you run a smart algorithm on a weak computer, you get bad results. If a species evolves some degree of intelligence (say, implemented as 50 different interacting circuits), there will be selection pressure to evolve enough hardware to support it optimally (let's say by copying the 50 circuits over and over). But it won't evolve more than necessary, because it costs energy, complicates birth, and you get diminishing returns anyway. So in this scenario we have perfect correlation, but no corresponding causation.
You can't just point at a correlation, quote the Bitter Lesson, postulate an analogy between biological and artificial neural networks and call that a theory of intelligence. There's no explanation of how knowledge is represented, or learned, or turned into behavior.
> Why does this work so well? Because animal intelligence (including human intelligence) is a blob of neural network arranged in a mildly clever way that lets it learn whatever it wants efficiently.
”Mildly clever” feels premature given that we still have no idea how the human brain works. And until we do it feels epistemically dodgy to use ”neural network” as an undifferentiated catch-all for both MLPs and brains.
The root problem is thrown into relief if you add ”can navigate novel physical environments” to your list of intelligence criteria. We won’t solve that by hand coding linguistic rules, but we probably also won’t solve it by adding layers to gpt.
Much of human intelligence is about sex appeal, because sexual selection is how we evolved. Us male men, we try to impress the women with how well we talk, groom, impress and manipulate other humans, solve problems, make money, write poetry, make art, play sports, sing, play guitars, etc.
We don't benefit from having sex with whales so we don't try to impress them. We don't even know what would impress them. Nature isn't interested in us using our intelligence in a way that would sexually attract whales.
Thus, our notion of intelligence is very constricted to the human domain.
"These considerations become even more relevant..." Sorry, the considerations just don't seem that compelling. Corollary: "IQ correlates to positive life outcomes" is a statement of pure value judgement, because appropriate life outcomes are culturally specific. Martyrdom is a perfect example; in many cultures past and present, martyrdom and widow-suicide are considered excellent life choices. And not just by tribal people in some steaming jungle... We all know (?) that Al Qaeda members were/are disproportionately engineers.
I think the 2nd tweet makes a valid point, which you're not really engaging with in this article. In alignment discourse I often notice the tacit assumption that a system which is intelligent thereby also has agency, self-preservation instinct, goals, and/or desires. It's quite likely that people don't feel the need to explicitly argue for these properties as much as they otherwise would, precisely because of this kind of "platonic ideal" of intelligence (a "soul", even if they wouldn't admit it) being ascribed to AI systems.
Good post. Might want to look into prototype theory of concepts, very similar to this.
Also. Intelligence in adults is more than 80% heritable. See here. https://www.emilkirkegaard.com/p/heritabilities-are-usually-underestimated
Brain size and intelligence is closer to r = 0.30 than 0.20 (Cox et al 2019). https://www.emilkirkegaard.com/p/brain-size-and-intelligence-2022
That last paragraph is nonsense. You seem to be missing what the thing is and what it is for. Beavers for instance went through an intelligence expansion in their evolutionary history and then plateaued. The "intelligence explosion" in humans from recognising markings of ownership on vases to the adapable Greek alphabet likewise was randomly fitted through enviromental fitness and a social type of being. The intelligence of beavers should produce its own values as a distinc category, and the same should be for AI, but it cant expand without an agency because machine intelligence is a tool. The only intelligence explosion we can atribute to machines will be at our current plateau of organization. If you really mean just engineering feats for humans, you should just say so, but I doubt there can be an expansion of human valued intelligence without either our expansion as the placeholder or an AI with biological similarity for environmental fitness and social organization: Grey-goo the Wise.
On one hand, the distinction between [assuming X axiomatically] and [arriving at X through reasoning about evidence] is meaningful and valid, and as (we seem to share a worldview in which) the latter is better than the former, I can see why you'd insist on making it.
On the other, once you start using X in your further arguments, both boil down to [assuming X]. And once you get your priors trapped at X, it boils down to [assuming X axiomatically] regardless of how you originally arrived at it.
So yes, it's wrong and judgemental to accuse others of Platonism. But I don't think you'd disagree that there exist real factual differences between you two about the degree to which the concept of "intelligence" is useful and meaningful, and you, of all people, should know full well that we humans are bound to keep making mental shortcuts straight to the city centre.
---
And while we're at at, he's right. At the very least, there's an important difference between intelligence as in "Artificial Intelligence", which is a field of science dedicated to making machines solve demanding cognitive tasks, and intelligence period, which is the inherent ability to perform those tasks. You do mix up the two, in a way that extrapolates from the former to the latter.
I mean, to plant my flag here, the entire section 3 of the post is just completely backwards. There is a valid retelling of the same events, where some people tried to imbue formal reasoning and domain knowledge and other explicit aspects of intelligence into their systems, and then a bunch of other people came, did a simple statistical analysis on gigantic chunks of data, and got better practical results (in the field of Artificial Intelligence, while not getting any closer to actual intelligence).
Counterintuitively, I find this quite encouraging. But it's kind of hard to explain why.
My coworker Dave, who's way smarter than me, can write an algorithm that I can't understand, which performs 100x better than my algorithm. The difference between code / a machine / an algorithm that kinda sorta works and one that performs near peak efficiency is huge, and the difference between being able to create the better algorithm and not is a matter of intelligence.
So, if AI was like an algorithm, then I can imagine it getting smart enough to rewrite itself from my version to Dave's version, and getting even smarter, and rewriting itself again, and its intelligence forming an exponential curve into stratospheric, Singularity heights.
But the point of this article is that intelligence isn't Platonic, and AI isn't like that. It's a giant blob of compute and training data, and so there's no way to rewrite it (like Dave rewriting my algorithm) and suddenly get 100x efficiency because of being more intelligent. It's much easier to imagine a blob of compute and training data running into diminishing returns and slowing to a stop.
So I think that while this establishes that an intelligence explosion is a coherent idea, it makes it seem much less likely.
“Basically, we created this word and used it in various contexts because we're lazy, but then some people started assuming that it has an independent reality and built arguments and a whole worldview around it.
They postulate an essence for some reason.”
The fact that we name stuff is a big deal with some philosophers who seem to not differentiate between absolute social constructs (ie religion, God, nations to an extent) and stuff we noticed and named. Like a tree.
Seeing and naming intelligence is not a performative utterance, it doesn’t create intelligence ex nilho, it’s recognising what exists and can be measured.
Stacking layers reminds me of the early days of aviation, when, it seemed obvious that th triplane was the successor to the biplane.
I'm not your A&P professor, but, one biceps brachii, two biceps brachii.
One idea that I'm curious about, that I've never received a satisfactory response for.
Why isn't AI safety a pascal's mugging?
Pascal's mugging requires extremely low probability x extremely high badness. People worried AI do not assign very low probabilities to AI giving problems. Even a 1% is not so low that you can't manage it cognitively.
My main issue with this kind of take is that it takes bundles of correlations which have been fairly thoroughly observed and tested *in humans* and then proceeds to claim those correlations will hold for AIs.
What we've observed in AIs is that yes, improving performance in one task does tend to improve performance in other tasks in a manner superficially similar to human intelligence - however, there are enough patterns where an AI which otherwise seems to be extremely clever fucks up in a way that a human never would that we can be reasonably sure that the *mechanism* whereby the AI is performing these tasks is significantly different from the mechanism a human being uses.
This, by itself, is not disqualifying toward the idea that it can be meaningful to attribute intelligente to an AI - but please remember that, in correlations, the tails come apart (https://slatestarcodex.com/2018/09/25/the-tails-coming-apart-as-metaphor-for-life/).
The idea of "intelligence" is a useful enough framing to talk about AI because there is some correlation between how "smart" an AI is and how an equivalent human would act - but there is no reason why this must hold indefinitely - especially for LLMs, which get all of their "smarts" from the structure of recorded human writing.
It feels like, in trying to make an LLM smarter, eventually we will start hitting a wall based on the fact that an AI that has learned everything it knows from human artifacts has no (simple) way to figure out things that humans can't do - and it is likely that, as it tries, it will start exposing the fact that its "human-intelligence-simulation-mechanisms" are very different from how humans reason and are not subject to the same correlation-generating constraints. Hence, I think it is very likely that the correlation which allows us to describe AI capabilities as "Intelligence" will eventually break down.
Of course, none of this rules out the idea that AI could eventually do Very Bad Things to us and that alignment might be a huge issue - but I don't think it can be boiled down to "AI will be very very smart and hence dangerous".
The problem with this argument is that we have mountains of evidence that the correlations between intellectual strengths that we call IQ is a fact about human and perhaps animal biology and NOT a fact about intelligence-as-such. To give but three examples: Deep Blue is superhuman at chess, but 100% stone illiterate, Watson is superhuman at Jeopardy, but almost certainly couldn't even make valid moves in chess, and a pocket calculator is superhuman at finding square roots, but can't play chess or Jeopardy at all. This doesn't even scratch the surface: you'd be hard pressed to find any computer system that isn't superhuman at some task, and yet none so far have had anything like the ability to form and achieve arbitrary goals that rationalists mean when they talk about capital I Intelligence. Taking the outside view, you have to have a very strong presumption that problems people have found with LLMs are real limitations and that they don't have intelligence-as-such either, though this isn't to say they aren't powerful or interesting or even dangerous. The alternative is an argument from ignorance: we don't really yet understand fully how LLMs work, so we assume that this time we've just finally made a computer program that's actually intelligent. Sorry, I don't find that at all persuasive. If you want to convince me that LLMs can achieve arbitrary goals, show me an LLM achieving arbitrary goals. That hasn't happened yet.
I think there's a further point to this argument: LLMs are strictly trained on human or LLM output. If humans can't be superhuman (which is true by definition), we'd need some very good justification for how LLMs could plausibly become superhuman based on mimicking human artifacts.
Right now we have two kinds of functional "AI": Single-purpose superhuman AIs trained by reinforcement learning, and kind-of-general-purpose worse-than-human LLMs trained on human output. There is no such thing as a general-purpose superhuman AI and most of Scott's argument is based on that existing.
Attempting a steelman: no single human can internalize the sum total of recorded human knowledge. Even large, coordinated groups of humans have difficulty researching limited subsets. If an entity were capable of ingesting e.g. the entire Library of Congress, it wouldn't even need to bring a superhuman quality of cognition to bear; simply being able to process and analyze such a vast corpus of data from a single perspective would allow inference, deduction, and probably synthesis, beyond human ability.
Indeed. But right now at least it seems that LLMs, being, well, somewhat limited by their mandate to produce coherent text that looks similar to human-generated text, do not make very effective use of different domains of information at once. Simulator theory seems to indicate that the main way LLMs function is by first "selecting" a "domain of expertise" and then working within that domain, with cross-domain work dropping in quality fast.
I think you're cheating a little bit with the phrasing of "arranging your blob cleverly". Can you articulate exactly how that's different from building in specific computational structures/abilities, like the "Responsible People" were trying to do? Like, I agree that a transformer is way way more general and blob-like than hand-coding in specific linguistic rules or whatever. But I've found it tricky to draw a line between what the people who don't get the Bitter Lesson are doing wrong and what the people who invented transformers are doing right.
As I've pointed out many times before, "intelligence" is certainly a coherent concept--but its meaning can be summarized as, "like an ideal human". (That's why the measure of artificial intelligence that everyone comes back to--despite its obvious glaring flaws--is the Turing test.) And it's not at all clear that "ideal humanness" is even fundamentally quantitative--except in artificially imposed ways--let alone subject to "explosion".
(I wrote about this at length, somewhat snarkily, in 2015: https://icouldbewrong.blogspot.com/2015/02/about-two-decades-ago-i-happened-to.html)
> how any set of cognitive capabilities can be self-amplifying [from the twitter screenshot]
This is ... obvious to cognitive psychologists, because that's how human learning works? It even has a name in education research: the Matthew Effect (a.k.a. The Rich Get Richer).
Basically, give a class of kids some text to read; the ones who have better vocabulary, domain knowledge, or both, will learn more from the same text than the ones who don't. This is a real stumbling block for some attempts at progressive education of the no-child-left-behind kind, because even if you give the whole class a text at the level that its weakest student can sort of understand, the top kids will still get more value out of the same text, up to a point.
Some papers describe the Matthew Effect as a "positive feedback loop", which sounds awfully like a synonym for "self-amplifying" to me. It's particularly pronounced in reading comprehension, so you'd expect to see the same if you're training GPTs instead of schoolkids to "read" and "comprehend" in a general sense of these words.
I agree with the general thrust of this argument, but defining concepts as “bundles of correlations” is both overly narrow and misses the point of the original tweet about platonic ideas. This may be true of “inductive” concepts like strength that are based on observation, but omits higher level abstract concepts like “justice ”, “good”, or “beauty”. Those ideas are not rooted in correlation, but in the network of high-level values that form the basis of our identity.
Also, the framing of “intelligence” by the author is a bit one sided. There are currently multiple, divergent definitions within cognitive science that refer to the ability to learn, to reason, and to adapt to new situations. Charles Spearman, an early 20th century psychologies argued that intelligence is a single, general cognitive ability (the g-factor) that underlies all specific mental abilities. Psychologist Howard Gardner posited the theory of multiple intelligences: linguistic, logical-mathematical, spatial, bodily-kinesthetic, musical, interpersonal, and intrapersonal. Then there is the triarchic theory of intelligence proposed by psychologist Robert Sternberg, which posits that intelligence is composed of three components: analytical, creative, and practical.
A better, broad definition of intelligence goes beyond humans and highly evolved animals, to include all life forms and even machines. We should equate intelligence with the process generally considered to be at its core: problem solving. Problem solving can be described as the process of identifying, analyzing, and resolving problems or obstacles to achieve a desired goal or outcome.
How complex this process really is only became apparent when people attempted to replicate it in a machine. In 1959, Allen Newell and Herbert Simon developed a machine called the General Problem Solver. To do this, they formalized the concept of problem space, an abstract space containing an initial state, goal state, path constraints, and operators (rules) defining how to move within it. Their insights were tremendously important and continue to have applications in machine learning until today. However, as they honestly conceded, their method encountered major obstacles that apply not only to machines, but to all life forms trying to solve problems.
First and foremost is combinatorial explosion. Whether digital or physical, large problem spaces with complex topologies of constraints and operators lead to a number of possibilities that exceed computational abilities. A simple chess game, for example, has an estimate of 10 to the power of 120 possible moves. This is known as the Shannon number and is larger than the total number of atoms in the universe. There are too many options to list, let alone compute.
The second limitation are ill-defined problems. The concept of problem space assumes that goal state and constraints are known and well defined. However, due to the frequently poor quality of available information, this is rarely the case. “How to write a good essay?” My initial state is an empty page. My goal state a good essay. Where is the path to a solution? Frankly, I don’t know. I make it up as I go along. Ironically, the vast majority of situations we encounter are precisely of this kind.
This is related to what philosopher David Chalmers calls the finitary predicament. According to Chalmers, the finitary predicament arises because our minds are finite and limited in their capacity to process information. We can only perceive a small fraction of the physical world, and we are limited in our ability to comprehend the complexity of the universe. As a result, our understanding of the world is necessarily incomplete and imperfect.
I feel that in the current A(G)I debate, it is important to clearly define the conceptual boundaries and definitions of these terms in order to have a meaningful debate.
I was prepared to entertain this argument but your analogy to strength is so wildly incorrect that I gave up. If you spent any time trying to actually understand what strength is, how it works, how we understand it and *can measure it*, it would help you understand how unlike "intelligence" it is.
But you won't, because the belief in one-dimensional measurable intelligence is a Core False Belief of the Ideology of Rationalism, and leads directly to AI Doomerism. (See http://www.paulgraham.com/lies.html to understand how ideologies form around false beliefs) The screenshotted tweets at the top are simply correct.
Medium warning (50% of a ban) for saying I am stupid and wrong and not giving any counterarguments or specific reasons. I feel like I've gestured at what I mean by "strength" and given supporting numbers (like correlations between different strength measures) - if you think I'm wrong, please give me the same courtesy.
“Intelligence” is another useful concept. When I say “Albert Einstein is more intelligent than a toddler”, I mean things like:
Einstein can do arithmetic better than the toddler
Einstein can do complicated mathematical word problems better than the toddler
Einstein can solve riddles better than a toddler
Einstein can read and comprehend text better than a toddler
Einstein can learn useful mechanical principles which let him build things faster than a toddler
…and so on."
This seems to go against the very notion of "intelligence" meaning more than "knowledge" or "skills." It also seems to be an argument against the existence of g. Although a toddler may also be less intelligent than the same person would be as a grown adult, we recognize frequently very smart children that we would say are "smarter" than most adults, even if they don't know a lot. James Sidis was clearly more intelligent than most adults when he was very small, but lots of adults also knew Greek, or whatever subject he was unusually good at from a young age, and almost certainly better than him.
Based on this definition of intelligence, almost every adult would be "more intelligent" than almost every child, but I don't think you believe that and I certainly don't.
I think I do believe almost every adult is more intelligent than almost every toddler. I think the standard concept of IQ is age-adjusted. LeBron's 3-year-old kid who is 99th percentile height for his age is tall-for-a-toddler and destined-to-be-very-tall-as-an-adult, but he isn't objectively tall compared to adults.
I agree that once we're talking about 10-year-olds or something there might be some who are better at abstract reasoning tasks than some adults while also knowing less than they do, which is a good example of ways that intelligence is less-than-perfectly-correlated.
A quick Google for "intelligence" comes up with "the ability to acquire and apply knowledge and skills."
That much more matches my understanding of what we mean by intelligent than having the knowledge and skills. Otherwise we would use the word "skilled" as a synonym for intelligence, when instead we recognize that they correlate but are not the same.
A toddler may have a greater ability to learn concepts than someone much older than themselves, even while knowing very little.
By your definition I'm not sure what to make of James Sidis and other super-geniuses. He apparently taught himself Latin at age two. When he started learning Latin, was he not "intelligent"? That he was able to learn it seems very relevant to our understanding of intelligence, even before he actually learned it.
This illustrates perfectly how theoretical linguistics got divorced from NLP - at some point, NLP guys understood they can just get much more data than a two-year-child learning the language and brute-force the problem instead of emulating parameters of Universal Grammar or whatever.
Some concepts are bundles of useful correlations but not all concepts. And the concepts that are only bundles of useful correlations have one thing in common: They brake down outside of the domain in which they are useful data compressions.
For example, an average dog has much higher biting strength than Mike Tyson on Scot's grandma but they probably both beat the dog at freelifting. Meanwhile a hydraulic lift will beat Mike Tyson and Scot's grandma in lifting but neither of them at wrestling. So Scot's explanation of strength kind-of holds for people but not for things very much unlike people. Compare this to a concept not defined by typical correlations like height, where the Empire state building is taller than Mike Tyson just like Mike Tyson is taller than I am (I have insufficiant data to specify the hight comparisons to Scot's grandma, but that is actual lack of knowledge not lack of conceptional definedness).
Normally correlations can't be extrapolated beyond the domain they are observed on (think about it, a correlation coefficiant is defined by the slope of a linearization and most functions aren't linear). And factor-analytical constructs springing from many such correlations almost never can be so extrapolated, which is why every entry level stats textbook warns against reifying them. It is of course also true that social scientists mostly nod along and then ignore all the warnings from baby stats but that kind of sloppiness gets you a replication crisis.
So no, there is absolutely no reason to expect intelligence to be a useful concept outside of humans in a vaguely normal intelligence range unless you think more real than other factor-analytical indices. In other words the tweeter is dead right, AI doomers at least of the classical Yudkowskyan kind expect the concept of intelligence to be useful far outside the domain that expectation would make sense in an do so precisely because they are mistaking it for an actual thing.
>Because animal intelligence (including human intelligence) is a blob of neural network arranged in a mildly clever way
No it's not.
Altman has said intelligence is an emergent property of physics, I believe the implications are that intelligence is based on geometric invariants and symmetries. Encoder-decoder models are learning reduced representations during bottleneck compression, with the ambition of converging to platonic ideals and irreducible representations. After the ideal forms are captured, the training examples are no longer needed a la Wittgenstein's ladder.
I think what you're missing is the assumption that you can scale things up by many orders of magnitude and still see the same correlations.
https://slatestarcodex.com/2018/09/25/the-tails-coming-apart-as-metaphor-for-life/
No objections, but IMO engaging with a weak argument.
You've given examples of where a word reaches the limits of its powers of comparison. The question is, when does that happen for intelligence? When does intelligence break apart into the strength equivalent of biceps vs triceps, pound-for-pound, height leverage, etc?
I think it's fair to say that more/less intelligent doesn't well describe our relationship to ChatGPT already. It can write far faster than I can, it has a far greater breadth of knowledge than I have, etc. I'm better than it is at introspecting myself, at my own profession, etc.
There will come a time when an AI gets better than me at literally everything, with the last vestige possibly being reverse engineering my own mental state and knowing me better than I know myself. But none of this speaks to "foom!" per se. The explanatory powers of "intelligence" have broken down, they are currently not useful to describe the infinity dollar question.
But we can say some things about foom when we stop talking about "intelligence goes up" and start talking about specific goals. AI might be able to solve ECC but it cannot solve a one time pad. Things might move extremely fast, but every single type of scaling we know tapers off very quickly once it runs into its physics. Strength is a great example:
https://en.wikipedia.org/wiki/Square%E2%80%93cube_law#Biomechanics
We can also talk about more useful properties of an AI than its "IQ":
1. How much information does it contain? (consider a human without any sensory input from birth)
2. How much information can it bring to bear on a question in a given unit of time?
3. How well can it avoid applying spurious information to a question?
4. How well can it avoid skipping information that would provide insight?
5. If 3 and 4 are inversely proportional, are they linearly or exponentially so?
6. (this list could be greatly expanded, I'd like to as well)
It's perfectly possible that an AI at 99% of its "intelligence" runs on the current total power output of the world and makes a couple insightful discoveries, but at 100% of its "intelligence" runs on all the stars in the universe and... proves that P != NP.
I’ve worked in deep learning research and engineering for the past decade and I want to say that Scott Alexander is completely right, but I want to add some mathematical rigor to this. Also to push back on one common argument against this. And also clarify some historical stuff he was a little fuzzy on.
Neural nets are Turing complete when they’re infinitely deep. And they’re universal approximators when they’re infinitely wide. What does this mean for the non-mathematicians? It means that they can compute anything that can be computed. That’s a bit of a circular definition, but the relevant part is that they can do anything *in theory*. Since they can compute anything, this means that they are already, in theory, capable of superintelligence. So why aren’t they superintelligent right now, in practice?
2012 was the breakthrough year for deep learning. That was when Geoff Hinton’s techniques really started working in a way that was impossible to ignore. Historically, why they didn’t work well until 2012 was that they tended to badly overfit, meaning that they memorized their dataset exactly, but could not “understand” what they were learning [https://en.wikipedia.org/wiki/Overfitting].
So how did Hinton get around this? By using something called an autoencoder for layer-wise pre-training. Say that you’re trying to classify images of handwritten digits to figure out what digit it is. Rather than just train a neural net to take an image and output the number, you take the image and compress it down, and attempt to reconstruct the original image. What this does is it forces the neural net to learn abstraction layers, because the only way it can compress something down and still reconstruct the original is to learn common patterns in the data. It would learn how pixels are commonly combined into lines, and then shapes, and how those shapes are commonly arranged. This is referred to as representation learning. Layer-wise pretraining went out of style a long time ago for reasons that aren’t relevant here, but the point is that what makes neural nets powerful is their ability to learn abstractions. More data from more tasks leads to better representations and better abstractions.
Mathematically, rather than debate “what is intelligence” I think we should say that multiplying matrices together is a very general process that can learn any function in physics, meaning that it can learn anything happening in the real world. One very general problem is that these things learn “too well” in that they still often overfit. One the main things we do in the field on day-to-day basis is get them to generalize better and overfit less. Our current models are likely powerful enough to loosely memorize half the internet. But that’s pretty useless if it hasn’t learned abstractions. That’s what makes the Transformers (AKA soft attention with positional encoding) such a breakthrough.
What’s stopping us from having superintelligence now?
At a very high level, the answer is overfitting. Feeding it more data from different domains will help regularize it, meaning reduce overfitting and improve generalization to new tasks. So, yes an intelligence explosion is possible but not guaranteed. More tasks and more domains will ultimately led to improved performance across all domains as the neural net learns better abstraction layers that represent real-world physics. This includes language.
Also, a common argument against this is that LLMs are terrible at basic arithmetic. But this isn’t a fundamental issue with matrix multiplication, rather an issue with the activation functions we use. I think that Neural Arithmetic Logic Units might be a solution [https://arxiv.org/abs/2101.09530].
> In the middle of a million companies pursuing their revolutionary new paradigms, OpenAI decided to just shrug and try the “giant blob of intelligence” strategy, and it worked.
The “Attention is All you Need” paper that invented the Transformers was actually out of Google. The authors also didn’t really understand how powerful their technique was and originally thought it was just a useful technique for translation and that was about it. Alec Redford over at OpenAI then had the idea to just use it for pre-training on English. In line with what Scott Alexander argues, Alec thought of it the same way with “Language Models are Unsupervised Multi-Task Learners.” [https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf]. Basically, that getting better at one task would correlate with improvement on others.
> The limit of Jelinek’s Law is OpenAI, who AFAIK didn’t use insights from linguistics at all
This is correct that they don’t use linguistics, but I think you’re giving OpenAI too much credit here.
This is a month late, but thanks for writing this!
You can define a numeric scale of "strength," with 3 representing somebody's metaphorical grandmother and 18 representing Mike Tyson, sure. You can come up with methods to estimate where an arbitrary human is on that scale, and use an arbitrary human's "strength" value to make reasonable predictions about what specific tasks that human can do, sure.
But, that doesn't mean it's meaningful to imagine a human with a "strength" of over 9000 and speculate about what that human might be capable of based on linear extrapolation. You can certainly come up with a meaningful number for what the grip force of a human with a 9000+ "strength" would have to be, but it's probably very silly, if not physically impossible, to design your uber-gripping device as just a scaled-up human.
No, AI-doomism is not Platonism; it's just a bait-and-switch.
As the story goes, AI is existentially dangerous because it will soon acquire superpowers, such as the ability to hack any computer, convince any human to do anything it wants, cover the Earth with "gray goo" nanotechnology, engineer a ~100% lethal pandemic, etc. It would acquire these powers by being "superintelligent" -- the AI would be to the smartest human (usually considered to be Von Neumann) as that human is to a rat. And it would become "superintelligent" via recursive improvements to its processing speed and total memory. So, act now, before it's too late !
But this story falls apart as soon as you look at any of the steps in detail. For example, what is "superintelligence" ? Well, it's usually described as the quality of being "super-smart", but what does that actually mean ? Sure, the AI could e.g. "solve math problems faster than Einstein", but calculators can already do that, and they're not "superintelligent". In practice, "superintelligence", as the AI-doom community uses the term, is synonymous with the ability to acquire superpowers. So, the AI will acquire superpowers due to its superior ability to acquire superpowers... this logical deduction is true, but not terribly impressive.
What about processing speed and "neural blob" size, however ? Surely, throwing more CPUs at any problem makes it easier to solve, so the AI could in fact solve any problem by throwing enough CPUs at it, right ? Well, sadly, the answer would appear to be "no". For example, modern LLMs (Large Language Models) such as Chat-GPTs are "Transformers" (that's the "T" in the name), a new type of deep-learning neural network that was discovered relatively recently. They differ from old-school NNs not merely in size, but also in *structure*; it is this new type of structure that makes them relatively effective at generating text (and the same holds true for image generators). You would not be able to make an LLM by merely networking together a bunch of 1980s-era Sharp calculators, just like you cannot make a genius by networking together a bunch of nematodes. Unfortunately, the term "neural network" is a bit of an aspirational misnomer; we humans (and other animals) have brains with radically more complicated structures than the Transformer; structures that allow us to do many things modern matrix-multiplying ML systems simply cannot do, such as learning on the fly.
Of course, AI research will continue, and it's possible (in fact, likely) that one day we will discover something more powerful than the Transformer (powerful enough to compete with human brains), but that day would not appear to be coming soon. Throwing more CPUs at the problem won't help (I mean, it helps a little, but the problem complexity is exponential); and, what's worse, many of the putative AI superpowers are outright physically impossible. So, I wouldn't go around panicking just yet (or, in fact, ever).
> Sure, the AI could e.g. "solve math problems faster than Einstein", but calculators can already do that, and they're not "superintelligent".
They can't. They can perform one part of the problem-solving process (calculation) with vastly superhuman speed and accuracy, and other parts (reducing a problem to a set of calculations) not at all. And they're incapable of finding reality-relevant 'math problems' in the first place.
Yes, and this exact criticism is true of e.g. modern LLMs, only more so, since they are comparatively worse at math than dedicated math tools. It would appear that an entity's capacity to "solve math problems" depends on how you define "math problems", which makes "solving math problems" a poor metric for "intelligence".
I just don't see why 'superintelligent' can't be defined in a way that doesn't collapse into 'able to acquire superpowers'. What's actually wrong with the sort of definition that would cash out as 'better than Einstein at *doing theoretical physics*, better than great genius X at doing Y, ...' and so on for a sufficiently broad class of intellectual endeavours?
I'm not claiming that LLMs or something like them will scale up to general superintelligence -- I don't have the technical knowledge to form any confident opinion on that.
> I just don't see why 'superintelligent' can't be defined in a way that doesn't collapse into 'able to acquire superpowers'.
You totally could, but then (as I'd said above) your argument is, "an entity that has the ability to gain superpowers is dangerous because it could get superpowers". It's not a very interesting argument.
How is that the argument? I'm not trying to be annoying, but this just doesn't make sense to me. (edit: Unless my triple-negative sentence was confusing and you read me as advocating the 'superintelligent = able to acquire superpowers' definition.)
The argument made by the AI risk people is something like 'a superintelligence (i.e. an entity that is much smarter than humans in just about all the ways that humans are smart) is likely to be dangerously powerful'; they say that the exact route from smarts->power is hard to predict in advance, but are often happy to elaborate on their reasons for thinking some such route exists.
I don't see how that is equivalent to the argument 'superpower-getting-capability -> superpowers -> danger', except in the sense that every logical argument has to contain the seeds of its conclusion somewhere in its premises.
> (edit: Unless my triple-negative sentence was confusing and you read me as advocating the 'superintelligent = able to acquire superpowers' definition.)
Indeed, that was my misunderstanding, apologies.
That said though, my problem with your preferred argument is precisely the vagueness of it. It sounds good on the surface: indeed, something much smarter than us, and malicious in nature, sounds like it could be extremely dangerous. But if you start asking questions, the argument falls apart.
What exactly do they mean by "smarter" ? Well, maybe it's the CPU speed or the ability to solve math problems or something. Ok. What is the mechanism by which "smarts" translate into superpowers ? It's unknown and perhaps even unknowable, but surely it is possible; after all, look at how much smarter and therefore more powerful we are compared to mice ! Um, ok. It goes on and on like this, and usually it turns out that the *only* aspect of "superintelligence" that we can pin down is "ability to acquire superpowers"; everything else is some manner of vague generality or metaphor or analogy or something. Hence my objection.
If you make the last stage tautologous, then the unlikelihood has to all be in the former stages. But you haven't stated that, eg., there is a ceiling to potential intelligence, or a lack of possible motivation.
I've stated such objections in more detail in my old FAQ:
https://www.datasecretslox.com/index.php/topic,2481.0.html
But in summary, yes, if we define "intelligence" as "ability to acquire superpowers", then there's absolutely a ceiling on it. For example, it is very likely that no amount of "intelligence" will enable you to acquire FTL travel.
Scott, next time you cite a correlation coefficient, please please PLEASE specify whether you mean r or r^2. Thanks in advance.
The fact that it's specifically language engines such as GPT which are the closest to generality is actually an evidence that intelligence is less correlated concept than we might have thought.
https://www.lesswrong.com/posts/fduAAMAWfp4bw9GQo/the-world-where-llms-are-possible
Also, the arbitrary reminder that heritability doesn't actually mean what people may naively think it does.
Go actually follow the link Scott provided and read the Caveats to learn more.
I think this is a question of consciousness and not intelligence. I think we cling to the word intelligence because that is the primary value we see in "AI" technology, and so we try to measure human intelligence qualitatively and quantitatively and evaluate the performance of some information processing model against that.
In reality, humans do not have a scientific explanation for our own consciousness yet we have a myriad of non-scientific explanations that overlap in profound ways.
I do think it is possible for an "AI" to become intelligent and capable enough to have profound, and even independent/uncontrolled, impacts on society without us translating our consciousness as a pre-requisite. However, for me that conversation merits an STS oriented conversation with consciousness/morality oriented questions in parallel. I feel as though society blurs the two, and underemphasizes the importance of regulating the technology as is in the present due to our concerns for the future which are often more laden with fear and black/white thinking.
This societal blurring is a frequent theme in STS that often causes further disconnect between already misaligned societal stakeholders. e.g. One can claim that the (valid) fears of a climate change fueled societal collapse have polarized people based on their feeling of likelihood of that future scenario. The closer we get the harder it is to avoid, but in the pre-internet era the societal landscape of information communication was quite different.
One could then debate whether a more present oriented approach with more tangible messaging (e.g. keep your community clean and healthy vs save the planet) would have worked or if a more radical (e.g. stop everything now) would have worked. In the end, what happened happened and these are simple considerations for relative adjustments to explore alternative ideas about how we navigate issues as potentially fueling our own exinction... whether it be through the climate, "AI", nuclear armaggeddon, or some other technological system gone awry.
Maybe you have "overlearned" the Bitter Lesson, and are generalizing it to apply in domains where it has never been observed (and where there are some good reasons to think it won't hold)?
I think the Bitter Lesson is unassailably true (and nonobvious, and useful) in Chess and Go. I think we could generalize that to "two-person turn-based discrete finite games of complete information" pretty safely.
When we add in games of incomplete information, my confidence gets a little lower, though the results in Texas Hold'em and Starcraft are encouraging.
What about natural language translation? Here there is some vagueness about the success metric, which puts us at risk of Goodharting ourselves. If the goal is to translate large numbers of documents at 90%+ accuracy for the lowest marginal cost, perhaps the machines (and the Bitter Lesson) have won. But if you wanted to translate a single document with the highest accuracy at any cost--say, if you had to sign an important bilingual document like a peace treaty or a 99-year lease for land to build a skyscraper on--you'd probably still want to hire a human expert, right? It's not clear to me that any Bitter Lesson-driven research will ever surpass a human at this important skill.
GPT4 has racked up another Bitter Lesson win at the game of "take a context window and generate a sequence of subsequent tokens that resembles, to a human, what another human might write". If you want to produce a large amount of reasonable-seeming text at the lowest marginal cost, it sure seems like the Bitter Lesson is the way to go. It's not totally clear to me whether this has any actual economic value (not relying on deception). I used to firmly believe it didn't, but I've been impressed and surprised by a quite a few successes, and certainly a lot of people/companies dumping money into this space must believe it might be useful for something.
Can we try to describe the space of problems for which the Bitter Lesson seems to apply? As a first guess, I'll put forth: "tasks for which you can obtain (or generate) a very large training set, and which have a clear and explicit success metric." But "taking over the world" doesn't seem to be in this space, nor does "building a more capable AI". I'll make a stronger prediction (with lower confidence) that "winning a programming competition" is (though just barely), and "being a professional software engineer" isn't (though just barely).
"Intelligence Has Been A Very Fruitful Way To Think About AI"
This is where you lose me (in THIS particular argument).
What OpenAI et al have done is create a language model. What ChatGPT etc can do is "understand" English (for some weak sense of understand); what they don't have is the rest of the intelligence package, whether that's "common sense" (ie various world knowledge) or much deduction. I remain unimpressed with ChatGPT *as an "intelligence"*, and every example I see of it shows, yes, it's very good at manipulating language and language-adjacent behavior, but that's all.
I cannot ask ChatGPT an "original" question (ie one that doesn't have relevant text on the internet) and expect any sort of interesting reply: "ChatGPT, give me an essay on whether the nerd/jock dichotomy is real, referencing examples from literature across many different times and places"
(I just tried this and the result is unimpressive. It shows my point – language is understood, but no insight, nothing new beyond repeating sentences.)
Now am I being too harsh in my bar for intelligence? Don't most *people* fail that standard? Well, yes, I guess I am an aristocrat, in the Aristotelian sense of that term.
It was a reasonable hypothesis that language was somehow equivalent to "intelligence", and so a language ability was the same thing as a general intelligence. But what I see when I look at LLMs is that we've proved the hypothesis wrong. LLM's probably have a great future as an arena for "experimental linguistics" but they will be only a part (and maybe not even the hard part) of a real AI.
Safe to say that AI will excel at imitating intelligence. Highly doubtful about anything beyond that. Ultimately there is an insurmountable distinction between organic/living intelligence and machine intelligence.
AnomalyUK has interesting arguments against super-intelligent AI doomerism. This April, he wrote [https://www.anomalyblog.co.uk/2023/04/ai-doom-post/]:
---
I think I need to disaggregate my foom-scepticism into two distinct but related propositions, both of which I consider likely to be true. Strong Foom-Scepticism — the most intelligent humans are close to the maximum intelligence that can exist. This is the “could really be true” one. But there is also Weak Foom-Scepticism: Intelligence at or above the observed human extreme is not useful, it becomes self-sabotaging and chaotic. That is also something I claim in my prior writing. But I have considerably more confidence in it being true. I have trouble imagining a super-intelligence that pursues some specific goal with determination. I find it more likely it will keep changing its mind, or play pointless games, or commit suicide. I’ve explained why before: it’s not a mystery why the most intelligent humans tend to follow this sort of pattern. It’s because they can climb through meta levels of their own motivations. I don’t see any way that any sufficiently high intelligence can be prevented from doing this.
---
Since he had correctly predicted the eventual rise of LLMs in 2012 [https://www.anomalyblog.co.uk/2012/01/speculations-regarding-limitations-of/], his arguments are worth considering.
---
[W]hat is “human-like intelligence”? It seems to me that it is not all that different from what the likes of Google search or Siri do: absorb vast amounts of associations between data items, without really being systematic about what the associations mean or selective about their quality, and apply some statistical algorithm to the associations to pick the most relevant.
There must be more to it than that; for one thing, trained humans can sort of do actual proper logic[,] and there’s a lot of effectively hand-built (i.e. specifically evolved) functionality in a some selected pattern-recognition areas. But I think the general-purpose associationist mechanism is the most important from the point of view of building artificial intelligence.
If that is true, then a couple of things follow. First, the Google/Siri approach to AI is the correct one, and as it develops we are likely to see it come to achieve something resembling humanlike ability.
But it also suggests that the limitations of human intelligence may not be due to limitations of the human brain, so much as they are due to fundamental limitations in what the association-plus-statistics technique can practically achieve.
[...]
The major limitation on human intelligence, particularly when it is augmented with computers as it generally is now, is how much it is wrong. Being faster or bigger doesn’t push back the major limitation unless it can make the intelligence wrong less often, and I don’t think it would.
---
>At some point we might get a blob which is better than humans at designing chips, and then we can make even bigger blobs of compute, even faster than before.
Although note that I think the intuitive interpretation of someone reading this would be 'we can add compute faster than we were adding it in the past', which gives a sense of acceleration.
Whereas I think the actual meaning is 'we can add compute in the future faster than we would have been able to if only humans were doing it.'
Which very easily *could* mean it still accelerated over time, but it also might not mean that (if for example we're running into other limiting factors besides the intelligence of chip designers).
Doing better in the future than we might have otherwise done in the future does not necessarily mean doing better than we did in the past.
(again, even if it's likely to mean that in practice)
I think math, and in particular arithmetic and algebra, is a compelling counter example to the statement that “that extremely clever plans to program “true understanding” into AI always do worse than just adding more compute and training data to your giant compute+training data blob”.
For example, the public gratis version of ChatGPT does not have the arithmetic ability of a cheap pocket calculator from the 1990s: just ask it a 8 digits square root, I asked 42282466 and it answered 6504.89 instead of 6502.4969.
AFAIK, large language models at this time are not able to automatically use their own outputs as inputs to continue a multi-stage reflection, and that is necessary to execute even a simple arithmetic algorithm. Maybe we will be able to teach them that ability, but it is a significant paradigm jump from the current linear processing.
I think a more accurate statement would be that we have exhausted the things we understand well enough to explain them to machines.
You can’t criticize platonism without engaging in it. “This whole class of arguments is flawed because it posits that abstract classes exist and can be reasoned out” is subtly contradictory.
This article points to the age old need of man to worship- AI is just a tool but mankind is idolizing it- we cannot help that we are designed to worship but the irony of man putting trust for help in something he designs and makes himself is brilliantly and humorously laid out in Isaiah 44-esp9-20 - but the best bit is in verses 20-22 when Yahweh says he actually has the power and offers to provide real help at his own expense - it truly is for every age- let he who has ears, hear and he who is intelligent read and understand.
‘9How foolish are those who manufacture idols.
These prized objects are really worthless.
The people who worship idols don’t know this,
so they are all put to shame.
10Who but a fool would make his own god—
an idol that cannot help him one bit?
11All who worship idols will be disgraced
along with all these craftsmen—mere humans—
who claim they can make a god.
They may all stand together,
but they will stand in terror and shame.
12The blacksmith stands at his forge to make a sharp tool,
pounding and shaping it with all his might.
His work makes him hungry and weak.
It makes him thirsty and faint.
13Then the wood-carver measures a block of wood
and draws a pattern on it.
He works with chisel and plane
and carves it into a human figure.
He gives it human beauty
and puts it in a little shrine.
14He cuts down cedars;
he selects the cypress and the oak;
he plants the pine in the forest
to be nourished by the rain.
15Then he uses part of the wood to make a fire.
With it he warms himself and bakes his bread.
Then—yes, it’s true—he takes the rest of it
and makes himself a god to worship!
He makes an idol
and bows down in front of it!
16He burns part of the tree to roast his meat
and to keep himself warm.
He says, “Ah, that fire feels good.”
17Then he takes what’s left
and makes his god: a carved idol!
He falls down in front of it,
worshiping and praying to it.
“Rescue me!” he says.
“You are my god!”
18Such stupidity and ignorance!
Their eyes are closed, and they cannot see.
Their minds are shut, and they cannot think.
19The person who made the idol never stops to reflect,
“Why, it’s just a block of wood!
I burned half of it for heat
and used it to bake my bread and roast my meat.
How can the rest of it be a god?
Should I bow down to worship a piece of wood?”
20The poor, deluded fool feeds on ashes.
He trusts something that can’t help him at all.
Yet he cannot bring himself to ask,
“Is this idol that I’m holding in my hand a lie?”
Restoration for Jerusalem
21“Pay attention, O Jacob,
for you are my servant, O Israel.
I, the LORD, made you,
and I will not forget you.
22I have swept away your sins like a cloud.
I have scattered your offenses like the morning mist.
Oh, return to me,
for I have paid the price to set you free.”’
https://biblehub.com/nlt/isaiah/44.htm
Late to the party (listened via the ACX podcast) but coincidentally a friend and I recently recorded a podcast discussing, basically, just this: whether general intelligence is a real/natural unified thing or just a bundle of smushed together abilities. Not great quality audio, minimal editing, but super interesting for me to make, at least.
Spotify: https://open.spotify.com/episode/0DDvsWCC0L04BCgJmFrPI8?si=GEBUBClQQVG89wNU4FWbFQ
Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy9lNDJlYjllYy9wb2RjYXN0L3Jzcw/episode/Mzc5NjljZGMtZWNhOC00NjE3LWI1MWYtZDRkMzA3NGIwYzAw
Copy and pasted episode summary/description:
Very imperfect transcript: bit.ly/3QhFgEJ
Summary from Clong:
The discussion centers around the concept of a unitary general intelligence or cognitive ability. Whether this exists as a real and distinct thing.
Nathan argues against it, citing evidence from cognitive science about highly specialized and localized brain functions that can be damaged independently. Losing linguistic ability does not harm spatial reasoning ability.
He also cites evidence from AI, like systems excelling at specific tasks without general competency, and tasks easy for AI but hard for humans. This suggests human cognition isn’t defined by some unitary general ability.
Aaron is more open to the idea, appealing to an intuitive sense of a qualitative difference between human and animal cognition - using symbolic reasoning in new domains. But he acknowledges the concept is fuzzy.
They discuss whether language necessitates this general ability in humans, or is just associated. Nathan leans toward specialized language modules in the brain.
They debate whether strong future AI systems could learn complex motor skills just from textual descriptions, without analogous motor control data. Nathan is highly skeptical.
Aaron makes an analogy to the universe arising from simple physical laws. Nathan finds this irrelevant to the debate.
Overall, Nathan seems to push Aaron towards a more skeptical view of a unitary general cognitive ability as a scientifically coherent concept. But Aaron retains some sympathy for related intuitions about human vs animal cognition.
Is Arnold Schwarzenegger stronger then a truck? Is that a coherent question?